Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor!: isolate arrow code #266

Open
wants to merge 39 commits into
base: main
Choose a base branch
from

Conversation

varshith257
Copy link
Contributor

@varshith257 varshith257 commented Oct 12, 2024

Please be sure to look over the pull request guidelines here: https://github.com/spaceandtimelabs/sxt-proof-of-sql/blob/main/CONTRIBUTING.md#submit-pr.

Please go through the following checklist

Rationale for this change

Currently, arrow is one of the elements that must be disabled to enable no_std support. There are many instances of #[cfg(feature = "arrow")] throughout the codebase, which leaves the code cluttered. To make code readable and manageable , arrow module is introduced.

Closes #237
/claim #237

What changes are included in this PR?

  • Moved all [cfg(feature = "arrow")] related code to separate modules and cracked cyclic dependencies along the way without any massive disruption.
  • Bumped Arrow from 51 -> 53

Are these changes tested?

Yes

@varshith257 varshith257 marked this pull request as draft October 12, 2024 15:55
@varshith257 varshith257 marked this pull request as ready for review October 12, 2024 18:56
@varshith257
Copy link
Contributor Author

@JayWhite2357 Have a review in your free time.

@algora-pbc algora-pbc bot mentioned this pull request Oct 12, 2024
3 tasks
@iajoiner iajoiner changed the title refactor: arrow code refactor!: arrow code Oct 15, 2024
@iajoiner iajoiner changed the title refactor!: arrow code refactor!: isolate arrow code Oct 15, 2024
@iajoiner
Copy link
Contributor

@varshith257 Really thanks for your PR! Actually it would be better if you bump Arrow to 53.1.0 in a separate PR. Or I can do it.

@varshith257
Copy link
Contributor Author

@iajoiner I will bump to its latest version in follow-up PR if needee

Copy link
Contributor

@JayWhite2357 JayWhite2357 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good so far. The main concern I have is around cyclic dependencies.

The resulting code must not introduce cyclic dependencies and should resolve any existing ones with arrow related code. This is one of the primary reasons for this cleanup: in order to more easily identify and resolve these.

In particular, nothing in base should depend on anything outside of base.

I've added comments to this effect. If you resolve those along with @iajoiner's concerns, we should be good on this PR.

@varshith257
Copy link
Contributor Author

@JayWhite2357 Will bump with the fixes

@varshith257
Copy link
Contributor Author

@melsonic
Copy link

@JayWhite2357 @iajoiner Can have a look at it here? https://github.com/spaceandtimelabs/sxt-proof-of-sql/actions/runs/11417714964/job/31770330872#step:4:766

I am not sure what it making to fail

I believe It's because of being unable to find the implementations in crates/proof-of-sql/src/base/arrow/arrow_query_result.rs.

@JayWhite2357
Copy link
Contributor

I am not sure what it making to fail

@varshith257

replace

RecordBatch::try_from(query_result).unwrap()

with

RecordBatch::try_from(query_result.table).unwrap()

This is caused because I asked you to remove this code:

impl<S: Scalar> TryFrom<QueryData<S>> for RecordBatch {
    type Error = ArrowError;

    fn try_from(value: QueryData<S>) -> Result<Self, Self::Error> {
        Self::try_from(value.table)
    }
}

varshith257 and others added 2 commits October 20, 2024 10:09
This is a final fix and all CI will pass though
iajoiner
iajoiner previously approved these changes Oct 21, 2024
Copy link
Contributor

@iajoiner iajoiner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@varshith257 Really thanks for your great work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Isolate arrow code
4 participants