Training NN with csv file

[saviord7]

Question

Goof afternoon!

I saw a lot of image training in you example, but I am confused how I can make csv reader for fill training model next.\

I also saw this CSVDataset and I have some question:

1. There are 2 columns: first - data, secon - label
   If I have csv file with 10 columns (9 - data, 1 - label ) how I can rightly read all data:

 NDArray datum = manager.create(encode(record.get("url")));
        NDArray label = manager.create(Float.parseFloat(record.get("isMalicious")));

I need to read for one column like:

 NDArray datum = manager.create(record.get("1 col"))
NDArray datum = manager.create(record.get("2 col"))
...
NDArray datum = manager.create(record.get("9 col"))

Or I can do it with some another way?

2. How I can fit training module with this dataset?
   For example, I will have next:

  CSVDataset build() throws IOException {
            String csvFilePath = "examplecsv";
            try (Reader reader = Files.newBufferedReader(Paths.get(csvFilePath));
                 CSVParser csvParser =
                         new CSVParser(
                                 reader,
                                 CSVFormat.DEFAULT
                                         .withHeader("1 col", "2col",...'9 col' "labelColumn")
                                         .withFirstRecordAsHeader()
                                         .withIgnoreHeaderCase()
                                         .withTrim())) {
                csvRecords = csvParser.getRecords();
            }
            return new CSVDataset(this);

Next I can do it like that?

 RandomAccessDataset trainingSet = CSVDataset()

[stu1130]

1. Yes, once you get the CSVRecord, you can get data either by index (record.get(int i)) or name (record.get(String name)). Both of them return a string, then you need to convert it to number as currently DJL doesn't support creating a NDArray from string.
2. We use builder pattern to create the dataset.

RandomAccessDataset trainingDataset = CSVDataset.builder().build().
...
EasyTrain.fit(..., ..., trainingDataset, ...);

[zachgk]

With CSV and tabular data, usually data is put into the model as a 1D "feature vector". So, you want to convert all of your data columns into a numerical representation and then concatenate those numbers to form an array. You can do this in standard Java.

Once you have your feature array, you can do NDArray datum = manager.create(featureArray) to convert the whole data array at once.

[saviord7]

1. Yes, once you get the CSVRecord, you can get data either by index (record.get(int i)) or name (record.get(String name)). Both of them return a string, then you need to convert it to number as currently DJL doesn't support creating a NDArray from string.
2. We use builder pattern to create the dataset.

RandomAccessDataset trainingDataset = CSVDataset.builder().build().
...
EasyTrain.fit(..., ..., trainingDataset, ...);

Did I undesrtand right that I need to read all columns by this method:

 NDArray datum = manager.create(record.get("1 col"))
NDArray datum = manager.create(record.get("2 col"))
...
NDArray datum = manager.create(record.get("9 col"))

And after these steps my 'datum' will have all my data comparabled to my labels ?

[saviord7]

With CSV and tabular data, usually data is put into the model as a 1D "feature vector". So, you want to convert all of your data columns into a numerical representation and then concatenate those numbers to form an array. You can do this in standard Java.

Once you have your feature array, you can do NDArray datum = manager.create(featureArray) to convert the whole data array at once.

For example my data except column's names are in number fromat (it sensor data ) , so I dont need to convert ?

[stu1130]

1. Yes, once you get the CSVRecord, you can get data either by index (record.get(int i)) or name (record.get(String name)). Both of them return a string, then you need to convert it to number as currently DJL doesn't support creating a NDArray from string.
2. We use builder pattern to create the dataset.

RandomAccessDataset trainingDataset = CSVDataset.builder().build().
...
EasyTrain.fit(..., ..., trainingDataset, ...);

Did I undesrtand right that I need to read all columns by this method:

 NDArray datum = manager.create(record.get("1 col"))
NDArray datum = manager.create(record.get("2 col"))
...
NDArray datum = manager.create(record.get("9 col"))

And after these steps my 'datum' will have all my data comparabled to my labels ?

Yes, the code snippet looks like this

int size = record.size();
float[] data = new float[size];
for (int i = 0; i < size; ++i) {
    data[i] = Float.parseFloat(record.get(i));
}
// here you create a row of csv data
NDArray array = manager.create(data);

[frankfliu]

@saviord7
I create a PR to added a concrete CsvDataset that you can use directly: #208

I will add a document a bit later.