-
Notifications
You must be signed in to change notification settings - Fork 56
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[PLUGIN-1778] Add schema management to remote task execution flow in Wrangler service #709
Conversation
eb981ab
to
34eb0c7
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks like a specific case handling. Any other TransientStore usage won't work. Should we just serialize and send the whole store over the wire both ways?
Thanks for pointing that out @tivv. I did initially implement that approach with the type handler for serializing the transient store: eb981ab But then later I thought that sending the entire transient store might add unnecessary overhead and if there are serialization/deserialization issues with some other variable in the store, schema handling would break. We can add it back in as a follow up item with more rigorous testing maybe? Right now this issue is a release blocker so I tried to fix it with minimal changes |
34eb0c7
to
fc79157
Compare
@samdgupi seeing that you have added a kryo serializer for rows in a previous PR under a feature flag, you have to add support for schema as well now |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add a comment to TransientStoreKeys to update this code on any additonal values
fc79157
to
c1f636c
Compare
c1f636c
to
3898779
Compare
@vanathi-g Updated KryoSeializer to support RemoteDirectiveResponse. Please check the PR and give LGTM |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, looks like there are build issues due to checkstyle
3898779
to
53a9825
Compare
Changes
RemoteDirectiveRequest
TransientStore
created inRemoteExecutionTask
RemoteDirectiveResponse
class that wraps the output rows and output schema together for returning from RemoteExecutionTask