Parquet Sink Codec

[umayr-codes]

Description

• ParquetSinkCodec implementation added
• Supports both Logical and Primitive Data Types

Issues Resolved

Resolves #2403

Check List

• New functionality includes testing.
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[asuresh8]

Can you make integration tests like the ones in https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/s3-source/src/integrationTest/java/org/opensearch/dataprepper/plugins/source?

[asuresh8]

You haven't addressed my previous comment. I want to see integration tests for all codecs where files are actually written to an S3 bucket and then read and asserted that they are correct.

Again see https://github.com/opensearch-project/data-prepper/tree/18de3fff6b06be80738d522a74f5144e5042da7a/data-prepper-plugins/s3-source/src/integrationTest/java/org/opensearch/dataprepper/plugins/source

[omkarmmore95]

Team is working on integration test.

[omkarmmore95]

You haven't addressed my previous comment. I want to see integration tests for all codecs where files are actually written to an S3 bucket and then read and asserted that they are correct.

Again see https://github.com/opensearch-project/data-prepper/tree/18de3fff6b06be80738d522a74f5144e5042da7a/data-prepper-plugins/s3-source/src/integrationTest/java/org/opensearch/dataprepper/plugins/source

As suggested by @dlvenable we have written integration test which integrates input and output codec and assert that the input data to input codec is equal to output data from outputcodec. Same we would be writing for Parquet as well.

[asuresh8]

Where are the integration tests for all the codecs?

I would like to see some instructions like the bottom of https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/s3-source/README.md to run the integration tests and I want to see the command + output that runs the integration test. Similar to the the integration tests for the source:

./gradlew :data-prepper-plugins:s3-source:integrationTest -Dtests.s3source.region=<your-aws-region> -Dtests.s3source.bucket=<your-bucket> -Dtests.s3source.queue.url=<your-queue-url>

[umayr-codes]

Where are the integration tests for all the codecs?

I would like to see some instructions like the bottom of https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/s3-source/README.md to run the integration tests and I want to see the command + output that runs the integration test. Similar to the the integration tests for the source:

./gradlew :data-prepper-plugins:s3-source:integrationTest -Dtests.s3source.region=<your-aws-region> -Dtests.s3source.bucket=<your-bucket> -Dtests.s3source.queue.url=<your-queue-url>

The integration of all codecs are in the respective PRs. ITs for CSV, JSON and Avro codecs are in PR #2986 (in S3SinkServiceIT class).

[umayr-codes]

Where are the integration tests for all the codecs?

I would like to see some instructions like the bottom of https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/s3-source/README.md to run the integration tests and I want to see the command + output that runs the integration test. Similar to the the integration tests for the source:

./gradlew :data-prepper-plugins:s3-source:integrationTest -Dtests.s3source.region=<your-aws-region> -Dtests.s3source.bucket=<your-bucket> -Dtests.s3source.queue.url=<your-queue-url>

Please see the readme.md file of parquet-codecs project. The gradle command to run the Integration tests is already in place.

[kkondaka]

Since you are anyways creating a new parquetRecord, I think it makes sense to add the "tagsTargetKey" also here without creating a 'modifiedEvent', right? I think you should be write like

      if (tagsTargetKey != null) {
            parquetRecord.put(tagsTargetKey, event.getMetadata().getTags());
       }. 
       writer.write(parquestRecord)

[umayr-codes]

Hi Krishna, the logic behing modifying the event itself is because we are generating schema from event in case user doesn't supply any schema.If event won't be modified to contain tags target key and tags then parquet record won't be able to store it either.