Is it possible to automatically match the header when reading a locally stored csv?

[sc303030]

Can't I automatically match the header when I read the locally stored csv? column1, column2 keep generating the key like this. I think it's okay to read it from the s3 bucket, is there no way to do it from the local file?

current

{"message":"name,tel,stuName,stuBirth,stuSex","column1":"name","column2":"tel","column3":"stuName","column4":"stuBirth","column5":"stuSex"}
{"message":"kimkim33,010-6666-1001,kimhehehe,19970917,female","column1":"kimkim33","column2":"010-6666-1001","column3":"kimhehehe","column4":"19970917","column5":"female"}

expexted

{"message":"name,tel,stuName,stuBirth,stuSex","column1":"name","column2":"tel","column3":"stuName","column4":"stuBirth","column5":"stuSex"}
{"message":"kimkim33,010-6666-1001,kimhehehe,19970917,female","name":"kimkim33","tel":"010-6666-1001","stuName":"kimhehehe","stuBirth":"19970917","stuSex":"female"}

yaml

version: 2
sample-pipeline:
  workers: 4
  source:
    file:
      path: "/mnt/storage1/student.csv"
      record_type: "event"
  buffer:
    bounded_blocking:
      buffer_size: 1024
      batch_size: 256
  processor:
    - csv:
        column_names_source_key: "1"
  sink:
    - stdout:

[dlvenable]

@sc303030 , I do not believe this is possible in the current version of Data Prepper (2.6.2). Our upcoming release of 2.7.0 will support codecs in the file source. So you will be able to use either the csv codec or use the newline codec with the header_destination property. This will read very much as the S3 source does.

You can also build from source to try this out now. See the following documentation:

https://github.com/opensearch-project/data-prepper/blob/main/docs/developer_guide.md

[sc303030]

Thanks to this, we are able to specify a header. Thank you very much.