Class DataSets

java.lang.Object

deepnetts.data.DataSets

public class DataSets
extends Object

Data set utility methods ans constants.

Author:

Zoran Sevarac

Field Summary

Fields

Modifier and Type	Field	Description
static final String	DELIMITER_COMMA
static final String	DELIMITER_SEMICOLON
static final String	DELIMITER_SPACE
static final String	DELIMITER_TAB

Constructor Summary

Constructors

Constructor	Description
DataSets()

Method Summary

Modifier and Type	Method	Description
static TabularDataSet	createBatchedDataset(TabularDataSet<?> dataSet, int batchSize)
static javax.visrec.ml.data.DataSet	createBatchedDatasetGPU(ImageSet imageSet, int batchSize)
static TabularDataSet	createBatchedDatasetGPU(TabularDataSet<?> dataSet, int batchSize)
static CsvFormat	detectCsvFormat(String fileName)
static float[]	oneHotEncode(String hotLabel, String[] allLabels)	Returns one hot encoded vector for the given label.
static TabularDataSet	readCsv(File csvFile, int numInputs, int numOutputs, boolean hasColumnNames, String delimiter)	Creates and returns data set from specified CSV file.
static javax.visrec.ml.data.DataSet	readCsv(String fileName, int numInputs, int numOutputs)	Create data set from CSV file, using coma (,) as default delimiter and no header (column names) in first row.
static TabularDataSet	readCsv(String fileName, int numInputs, int numOutputs, boolean hasColumnNames)
static TabularDataSet	readCsv(String fileName, int numInputs, int numOutputs, boolean hasColumnNames, String delimiter)
static TabularDataSet	readCsv(String fileName, int numInputs, int numOutputs, String delimiter)
static MaxScaler	scaleToMax(javax.visrec.ml.data.DataSet dataSet)
static MinMaxScaler	scaleToMinMax(javax.visrec.ml.data.DataSet dataSet)
static TrainTestSplit	trainTestSplit(javax.visrec.ml.data.DataSet<?> dataSet, double split)

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DELIMITER_SPACE

public static final String DELIMITER_SPACE

See Also:

• Constant Field Values

DELIMITER_COMMA

public static final String DELIMITER_COMMA

See Also:

• Constant Field Values

DELIMITER_SEMICOLON

public static final String DELIMITER_SEMICOLON

See Also:

• Constant Field Values

DELIMITER_TAB

public static final String DELIMITER_TAB

See Also:

• Constant Field Values

Constructor Details

DataSets

public DataSets()

Method Details

readCsv

public static TabularDataSet readCsv(File csvFile,
 int numInputs,
 int numOutputs,
 boolean hasColumnNames,
 String delimiter)
                              throws FileNotFoundException,
IOException

Creates and returns data set from specified CSV file. Empty lines are skipped

Parameters:

csvFile - CSV file

numInputs - number of input values in a row

numOutputs - number of output values in a row

hasColumnNames - true if first row contains column names

delimiter - delimiter character used to separate values in a row

Returns:

instance of data set with values loaded from file

Throws:

FileNotFoundException - if file was not found

IOException - if there was an error reading file TODO: Detect if there are labels in the first line, if there are no labels, set class1, class2, class3 in classifier evaluation! and detect type of attributes Move this method to some factory class or something? or as a default method in data set? TODO: Autodetetect delimiter; column type

readCsv

public static TabularDataSet readCsv(String fileName,
 int numInputs,
 int numOutputs,
 boolean hasColumnNames,
 String delimiter)
                              throws IOException

Throws:

IOException

readCsv

public static TabularDataSet readCsv(String fileName,
 int numInputs,
 int numOutputs,
 boolean hasColumnNames)
                              throws IOException

Throws:

IOException

readCsv

public static TabularDataSet readCsv(String fileName,
 int numInputs,
 int numOutputs,
 String delimiter)
                              throws IOException

Throws:

IOException

readCsv

public static javax.visrec.ml.data.DataSet readCsv(String fileName,
 int numInputs,
 int numOutputs)
                                            throws IOException

Create data set from CSV file, using coma (,) as default delimiter and no header (column names) in first row.

Parameters:

fileName - Name of the CSV file

numInputs - Number of input columns

numOutputs - Number of output columns

Returns:

Throws:

IOException

detectCsvFormat

public static CsvFormat detectCsvFormat(String fileName)
                                 throws FileNotFoundException,
IOException

Throws:

FileNotFoundException

IOException

scaleToMax

public static MaxScaler scaleToMax(javax.visrec.ml.data.DataSet dataSet)

scaleToMinMax

public static MinMaxScaler scaleToMinMax(javax.visrec.ml.data.DataSet dataSet)

oneHotEncode

public static float[] oneHotEncode(String hotLabel,
 String[] allLabels)

Returns one hot encoded vector for the given label. One-Hot encoded vector is a binary array in which each position corresponds to one label, and all elements are zero, except the one which corresponds to hotLabel which has value of one. Index of hotLabel in allLabels array, determines which position in vector should be one. Vector size equals to the number of labels in allLabels array.

Parameters:

hotLabel - one label to encode

allLabels - all labels (used to determine size and hot position of encoded vector)

Returns:

one hot encoded vector for given label

trainTestSplit

public static TrainTestSplit trainTestSplit(javax.visrec.ml.data.DataSet<?> dataSet,
 double split)

createBatchedDataset

public static TabularDataSet createBatchedDataset(TabularDataSet<?> dataSet,
 int batchSize)

createBatchedDatasetGPU

public static TabularDataSet createBatchedDatasetGPU(TabularDataSet<?> dataSet,
 int batchSize)

createBatchedDatasetGPU

public static javax.visrec.ml.data.DataSet createBatchedDatasetGPU(ImageSet imageSet,
 int batchSize)