Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PERF] native streaming parquet #1193

Merged
merged 6 commits into from
Jul 29, 2023
Merged

[PERF] native streaming parquet #1193

merged 6 commits into from
Jul 29, 2023

Conversation

samster25
Copy link
Member

@samster25 samster25 commented Jul 27, 2023 •
edited
Loading

  • Improves perf of parquet reader by performing pipelining between compute and IO
  • Uses rayon and oneshot channels to fan out expensive work to blocking threads

@samster25 samster25 changed the title [Perf] native streaming parquet [PERF] native streaming parquet Jul 27, 2023
@samster25
Copy link
Member Author

samster25 commented Jul 27, 2023 •
edited by eventual-github-bot bot
Loading

/benchmark

✅ Daft Benchmark [#1] @ 11ba2c7

Download artifacts


---
History
Most recent attempts appear on the left.

✅

@codecov
Copy link

codecov bot commented Jul 27, 2023 •
edited
Loading

Codecov Report

Merging #1193 (0d421cc) into main (440b8b3) will not change coverage.
Report is 6 commits behind head on main.
The diff coverage is n/a.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #1193   +/-   ##
=======================================
  Coverage   88.37%   88.37%           
=======================================
  Files          54       54           
  Lines        5576     5576           
=======================================
  Hits         4928     4928           
  Misses        648      648

@samster25 samster25 merged commit bacd70e into main Jul 29, 2023
18 checks passed
@samster25 samster25 deleted the sammy/native-streaming-parquet branch July 29, 2023 01:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@samster25