Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEAT] Enable feature-flagged native downloader in daft.read_parquet #1190

Merged
merged 13 commits into from
Jul 31, 2023
Merged

[FEAT] Enable feature-flagged native downloader in daft.read_parquet #1190

merged 13 commits into from
Jul 31, 2023

Commits on Jul 27, 2023

  1. [FEAT] Enable stub code for native downloader for top-level daft.read… 

    …_parquet
    Jay Chia committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    ef15788 View commit details
    Browse the repository at this point in the history

  2. Add filesystem metadata API

    Jay Chia committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    503e79f View commit details
    Browse the repository at this point in the history

  3. Share get_parquet_metadata functionality across schema inference and … 

    …file metadata retrieval
    Jay Chia committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    70dcb12 View commit details
    Browse the repository at this point in the history

  4. Use injected info in source_info in glob_path_with_stats

    Jay Chia committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    858e5bc View commit details
    Browse the repository at this point in the history

  5. Ensure IOConfig imports are guarded by TYPE_CHECKING to avoid circula… 

    …r imports
    Jay Chia committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    1e5309e View commit details
    Browse the repository at this point in the history

  6. Integrate new parquet APIs

    Jay Chia committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    0b76eee View commit details
    Browse the repository at this point in the history

  7. Test native downloader for Parquet reads in test_creation.py

    Jay Chia committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    429b05c View commit details
    Browse the repository at this point in the history

  8. Add missing flag propogation in daft/execution/execution_step.py

    Jay Chia committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    b3a0836 View commit details
    Browse the repository at this point in the history

  9. Add fixes to make PyIOConfig pickleable

    Jay Chia committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    0f1b076 View commit details
    Browse the repository at this point in the history

  10. Add a table cast for native read_parquet

    Jay Chia committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    37b0976 View commit details
    Browse the repository at this point in the history

  11. Use JSON serialization and __reduce__ instead of get/setstate for ser… 

    …ialization
    Jay Chia committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    dba0cec View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2023

  1. Merge branch 'main' into jay/parquet-pyarrow-switch-to-table

    @jaychia
    jaychia authored Jul 31, 2023
    Configuration menu
    Copy the full SHA
    8311610 View commit details
    Browse the repository at this point in the history

  2. Fix Cargo.lock

    Jay Chia committed Jul 31, 2023
    Configuration menu
    Copy the full SHA
    34423d5 View commit details
    Browse the repository at this point in the history