Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Substrait conversion layer #48

Open
jurplel opened this issue Feb 3, 2024 · 2 comments
Open

Substrait conversion layer #48

jurplel opened this issue Feb 3, 2024 · 2 comments
Labels
enhancement New feature or request

Comments

@jurplel
Copy link
Member

jurplel commented Feb 3, 2024

  • Put it in a separate crate, parallel to [optd-datafusion-bridge](https://github.com/cmu-db/optd/tree/main/optd-datafusion-bridge)
  • There exists a crate under the datafusion tree, datafusion-substrait, that claims to convert from datafusion to substrait representations. However, on the physical side (which is what we need to convert), it seems like it has the same problem that would be encountered if we were to send impl ExecutionPlan trait objects around, in that it doesn't seem to know the type or any information about each node. It seems like we wouldn't want to use it, in this case, and instead convert directly from optd nodes. I may be wrong on this---check for yourself here.
  • Bonus points: make sure serialization is working (can be a separate/later PR)
@jurplel jurplel self-assigned this Feb 4, 2024
@jurplel
Copy link
Member Author

jurplel commented Feb 4, 2024

I'll give this a go later this week if that's good w/ everyone

@amol-
Copy link

amol- commented Feb 5, 2024

@jurplel If you find anything that Substrait plans might be lacking on regarding a proper representation of Physical Plans, I would be interested in hearing them. We are definitely looking for opportunities to improve Substrait as we can learn a lot from actual usage of it.

@jurplel jurplel added the enhancement New feature or request label Feb 8, 2024
@jurplel jurplel removed their assignment Jul 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants