Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PARQUET-2261: add statistics for better estimating unencoded/uncompressed sizes and finer grained filtering #197

Merged
merged 34 commits into from
Nov 14, 2023
Merged

PARQUET-2261: add statistics for better estimating unencoded/uncompressed sizes and finer grained filtering #197

merged 34 commits into from
Nov 14, 2023

Commits on Aug 22, 2023

  1. Draft: Initial proposal

    @emkornfield
    emkornfield committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    e311a8a View commit details
    Browse the repository at this point in the history

  2. Update src/main/thrift/parquet.thrift 

    Co-authored-by: Gang Wu <[email protected]>
    @emkornfield @wgtmac
    emkornfield and wgtmac committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    c60568b View commit details
    Browse the repository at this point in the history

  3. Separate out structure

    @emkornfield
    emkornfield committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    d341446 View commit details
    Browse the repository at this point in the history

  4. revert cumulative, change decimal to minimum precisions

    @emkornfield
    emkornfield committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    e49b908 View commit details
    Browse the repository at this point in the history

  5. Change field to variable width bytes

    @emkornfield
    emkornfield committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    f9fba05 View commit details
    Browse the repository at this point in the history

  6. update based on feedback

    @emkornfield
    emkornfield committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    97d22a5 View commit details
    Browse the repository at this point in the history

  7. more language smithing

    @emkornfield
    emkornfield committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    c6e244b View commit details
    Browse the repository at this point in the history

  8. change name to unencoded

    @emkornfield
    emkornfield committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    24fafcc View commit details
    Browse the repository at this point in the history

  9. Update src/main/thrift/parquet.thrift 

    Co-authored-by: Gang Wu <[email protected]>
    @emkornfield @wgtmac
    emkornfield and wgtmac committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    0556bfb View commit details
    Browse the repository at this point in the history

  10. Update src/main/thrift/parquet.thrift 

    Co-authored-by: Gang Wu <[email protected]>
    @emkornfield @wgtmac
    emkornfield and wgtmac committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    6fe15bb View commit details
    Browse the repository at this point in the history

  11. address some comment

    @emkornfield
    emkornfield committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    c5ca0e1 View commit details
    Browse the repository at this point in the history

  12. separate histogram and add it to index

    @emkornfield
    emkornfield committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    766e62d View commit details
    Browse the repository at this point in the history

  13. Update src/main/thrift/parquet.thrift 

    Co-authored-by: Gabor Szadovszky <[email protected]>
    @emkornfield @gszadovszky
    emkornfield and gszadovszky committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    a0d0d43 View commit details
    Browse the repository at this point in the history

  14. Update src/main/thrift/parquet.thrift 

    Co-authored-by: Gabor Szadovszky <[email protected]>
    @emkornfield @gszadovszky
    emkornfield and gszadovszky committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    5edbf34 View commit details
    Browse the repository at this point in the history

  15. move stats

    @emkornfield
    emkornfield committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    0dfc307 View commit details
    Browse the repository at this point in the history

  16. Update src/main/thrift/parquet.thrift 

    Co-authored-by: Gang Wu <[email protected]>
    @emkornfield @wgtmac
    emkornfield and wgtmac committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    f00f611 View commit details
    Browse the repository at this point in the history

  17. fix some formatting

    @emkornfield
    emkornfield committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    cb68370 View commit details
    Browse the repository at this point in the history

  18. Apply suggestions from code review 

    apply wording recommendations.

Co-authored-by: Antoine Pitrou <[email protected]>
    @emkornfield @pitrou
    emkornfield and pitrou committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    b7ebf1c View commit details
    Browse the repository at this point in the history

  19. Update src/main/thrift/parquet.thrift 

    Co-authored-by: mwish <[email protected]>
    @emkornfield @mapleFU
    emkornfield and mapleFU committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    407443f View commit details
    Browse the repository at this point in the history

  20. Update src/main/thrift/parquet.thrift 

    Co-authored-by: mwish <[email protected]>
    @emkornfield @mapleFU
    emkornfield and mapleFU committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    ab3ef49 View commit details
    Browse the repository at this point in the history

  21. reflow comments, remove trailing whitespace

    @emkornfield
    emkornfield committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    a9ec32f View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2023

  1. update based on feedback on PR

    @emkornfield
    emkornfield committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    d0b051c View commit details
    Browse the repository at this point in the history

  2. rename size_estimate_statistics

    @emkornfield
    emkornfield committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    8591f23 View commit details
    Browse the repository at this point in the history

  3. clarify histogram comment

    @emkornfield
    emkornfield committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    98d1881 View commit details
    Browse the repository at this point in the history

  4. rename field

    @emkornfield
    emkornfield committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    0181c5e View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2023

  1. update language on omitted fields

    @emkornfield
    emkornfield committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    9fa9f9c View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2023

  1. Update src/main/thrift/parquet.thrift 

    Co-authored-by: JFinis <[email protected]>
    @emkornfield @JFinis
    emkornfield and JFinis authored Sep 6, 2023
    Configuration menu
    Copy the full SHA
    4f8dcf0 View commit details
    Browse the repository at this point in the history

  2. Update src/main/thrift/parquet.thrift 

    Co-authored-by: JFinis <[email protected]>
    @emkornfield @JFinis
    emkornfield and JFinis authored Sep 6, 2023
    Configuration menu
    Copy the full SHA
    ce7904d View commit details
    Browse the repository at this point in the history

  3. Update src/main/thrift/parquet.thrift 

    Co-authored-by: JFinis <[email protected]>
    @emkornfield @JFinis
    emkornfield and JFinis authored Sep 6, 2023
    Configuration menu
    Copy the full SHA
    1752f2d View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2023

  1. flatten structures

    @emkornfield
    emkornfield committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    5ee0864 View commit details
    Browse the repository at this point in the history

  2. update indices

    @emkornfield
    emkornfield committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    b4a703d View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2023

  1. Update src/main/thrift/parquet.thrift 

    Co-authored-by: Gang Wu <[email protected]>
    @emkornfield @wgtmac
    emkornfield and wgtmac authored Sep 8, 2023
    Configuration menu
    Copy the full SHA
    7081735 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2023

  1. remove redundant phrasing

    @emkornfield
    emkornfield committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    aee8c0e View commit details
    Browse the repository at this point in the history

  2. Update src/main/thrift/parquet.thrift 

    Co-authored-by: JFinis <[email protected]>
    @emkornfield @JFinis
    emkornfield and JFinis authored Sep 12, 2023
    Configuration menu
    Copy the full SHA
    d7856b1 View commit details
    Browse the repository at this point in the history