Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PLUGIN-1778] Add schema management to remote task execution flow in Wrangler service #709

Merged
merged 1 commit into from
May 1, 2024

Conversation

vanathi-g
Copy link
Contributor

@vanathi-g vanathi-g commented Apr 29, 2024

Changes

  • Add new inputSchema field to RemoteDirectiveRequest
  • Set the inputSchema obtained from request in TransientStore created in RemoteExecutionTask
  • Add a new RemoteDirectiveResponse class that wraps the output rows and output schema together for returning from RemoteExecutionTask

Copy link
Contributor

@tivv tivv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a specific case handling. Any other TransientStore usage won't work. Should we just serialize and send the whole store over the wire both ways?

@vanathi-g
Copy link
Contributor Author

vanathi-g commented Apr 30, 2024

Thanks for pointing that out @tivv. I did initially implement that approach with the type handler for serializing the transient store: eb981ab

But then later I thought that sending the entire transient store might add unnecessary overhead and if there are serialization/deserialization issues with some other variable in the store, schema handling would break.

We can add it back in as a follow up item with more rigorous testing maybe? Right now this issue is a release blocker so I tried to fix it with minimal changes

@vanathi-g
Copy link
Contributor Author

vanathi-g commented Apr 30, 2024

@samdgupi seeing that you have added a kryo serializer for rows in a previous PR under a feature flag, you have to add support for schema as well now

Copy link
Contributor

@tivv tivv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a comment to TransientStoreKeys to update this code on any additonal values

@samdgupi samdgupi force-pushed the schema-remote-execution-fix branch from fc79157 to c1f636c Compare April 30, 2024 18:11
@samdgupi samdgupi added the build Triggers unit test build label Apr 30, 2024
@samdgupi samdgupi force-pushed the schema-remote-execution-fix branch from c1f636c to 3898779 Compare April 30, 2024 18:42
@samdgupi
Copy link
Contributor

@samdgupi seeing that you have added a kryo serializer for rows in a previous PR under a feature flag, you have to add support for schema as well now

@vanathi-g Updated KryoSeializer to support RemoteDirectiveResponse. Please check the PR and give LGTM

Copy link
Contributor Author

@vanathi-g vanathi-g left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, looks like there are build issues due to checkstyle

@samdgupi samdgupi force-pushed the schema-remote-execution-fix branch from 3898779 to 53a9825 Compare May 1, 2024 06:19
@samdgupi samdgupi merged commit 564f734 into develop May 1, 2024
4 of 5 checks passed
@samdgupi samdgupi deleted the schema-remote-execution-fix branch May 1, 2024 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Triggers unit test build
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants