Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] add support for spilling to on-heap memory before spilling to file #356

Open
wants to merge 59 commits into
base: main
Choose a base branch
from

Commits on Jul 3, 2023

  1. Folder: scripts, third_party, website, .github

    relative pr:
    Update build dependencies oap-project#185
    zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    1184743 View commit details
    Browse the repository at this point in the history
  2. Folder: common, connector

    relative pr:
    
    add decimal column reader support oap-project#254
    Add utility method MemoryUsageTracker::highUsage() oap-project#227
    Support parquet read case sensitive option oap-project#126
    Make varchar and varbinary compatible oap-project#115
    Create folder if not exits on HDFS write oap-project#267
    zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    db2b662 View commit details
    Browse the repository at this point in the history
  3. Folder: core

    relative pr:
    
    Add expand op in velox oap-project#199
    Add ValueStreamNode operator oap-project#204
    Allow decimal in casting string to int oap-project#215
    zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    34ce137 View commit details
    Browse the repository at this point in the history
  4. Folder: dwio

    relative pr:
    
    add support for reading ORC oap-project#229
    Parquet: Optimize parquet write perf oap-project#238
    Expand timestamps in page reader oap-project#260
    Add processedStrides and processedSplits metrics oap-project#264
    zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    871ae83 View commit details
    Browse the repository at this point in the history
  5. Folder: exec

    relative pr:
    
    Fix hashjoin runtime issue oap-project#106
    INVALID_STATE on HashJoin when spill is turned on oap-project#154
    SIGABRT on DecimalAvgAggregate<UnscaleLongDecimal, UnscaleShortDecimal> when spilling is engaged oap-project#236
    Support kPreceeding & kFollowing for window range frame type oap-project#287
    zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    de7528f View commit details
    Browse the repository at this point in the history
  6. Folder: expression

    relative pr:
    
    Allow decimal in casting string to int oap-project#215
    Add mapping from named_struct to row_constructor oap-project#214
    Fix semantic issues in cast function oap-project#280
    zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    e7134e3 View commit details
    Browse the repository at this point in the history
  7. Folder: function

    relative pr:
    
    Fix replace SparkSQL function oap-project#277
    Support kPreceeding & kFollowing for window range frame type oap-project#287
    support timestamp hash oap-project#269
    Spark sum can overflow oap-project#101
    Support float & double types in pmod function oap-project#157
    Implement datetime functions in velox/sparksql. oap-project#81
    Fix type check in MapFunction oap-project#273
    Let function validation fail for lookaround pattern in RE2-based implementation oap-project#124
    Register lpad/rpad functions for Spark SQL. oap-project#63
    Support substring_index sql function oap-project#189
    Fix First/Last aggregate functions intermediate type and support decimal oap-project#245
    Support date_add spark sql function oap-project#144
    zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    44b2919 View commit details
    Browse the repository at this point in the history
  8. Folder: row, serializers

    relative pr:
    
    Serialize and deserialize RowVector oap-project#250
    zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    38cb28a View commit details
    Browse the repository at this point in the history
  9. Folder: substrait

    relative pr:
    
    Check a fallback case in validation: using literal partition key in window function oap-project#148
    Fix might_contain validate fallback and support struct literal oap-project#137
    Implement datetime functions in velox/sparksql. oap-project#81
    Parse options in SingularOrList correctly oap-project#48
    Add SingularOrList support oap-project#45
    Support if then in filter oap-project#74
    Fix semi join output type and support existence join oap-project#67
    Support decimal as partition column oap-project#167
    Add the window support oap-project#61
    Add expand operator oap-project#65
    Support more cases of filter and its pushdown oap-project#14
    zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    e94db80 View commit details
    Browse the repository at this point in the history
  10. Folder: type, vector

    relative pr:
    
    Support more data types for read filter oap-project#139
    Fix cast double to decimal oap-project#179
    Fix casting from string to decimal oap-project#281
    Support cast decimal to int oap-project#177
    Fix null on overflow and multiply as spark precision and support cast varchar to decimal oap-project#169
    Disable tokenizing the path by dot oap-project#109
    Serialize and deserialize RowVector oap-project#250
    Support datetime pattern in spark oap-project#94
    zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    d9654f1 View commit details
    Browse the repository at this point in the history
  11. Folder: website

    zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    ca1be07 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    166496d View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    4f8394c View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    d9c76a0 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    29c0acc View commit details
    Browse the repository at this point in the history
  16. Fix use pre-build arrow (oap-project#289)

    In Ubuntu, thrift will be installed manually in setup scripts, but Arrow still compile thrift, then Velox use system thrift and pre-build arrow/parquet.
    In Centos, Velox could not found system thrift, so it will compile Arrow and thrift twice.
    Since Arrow will compile thrift in all environments, lets use these pre-build shared libs to save time and keep toolchain consistency.
    
    Support native dependency could be overrided by env variables.
    Yohahaha authored and zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    63976a9 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    3e386b9 View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    8ae1d76 View commit details
    Browse the repository at this point in the history
  19. [DWIO] refactor the reader of dwrf/orc (oap-project#261)

    Co-authored-by: zuochunwei <[email protected]>
    2 people authored and zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    9c331a0 View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    7e9b9b7 View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    e8084ce View commit details
    Browse the repository at this point in the history
  22. update dnf cache on centos (oap-project#302)

    Signed-off-by: Yuan Zhou <[email protected]>
    zhouyuan authored and zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    9347af7 View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    d42da52 View commit details
    Browse the repository at this point in the history
  24. 2023.5.31 rebase problem fix

    rui-mo authored and zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    2faf17d View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    bcb35ad View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    cbaa882 View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    9104343 View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    37584f0 View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    9dff543 View commit details
    Browse the repository at this point in the history
  30. ld (oap-project#317)

    Co-authored-by: zuochunwei <[email protected]>
    2 people authored and zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    b001065 View commit details
    Browse the repository at this point in the history
  31. Configuration menu
    Copy the full SHA
    249e152 View commit details
    Browse the repository at this point in the history
  32. Configuration menu
    Copy the full SHA
    0d8c2c6 View commit details
    Browse the repository at this point in the history
  33. Add flag for sort in window (oap-project#321)

    * add flag for sort in window op
    
    * minor change
    zhli1142015 authored and zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    e6f99f0 View commit details
    Browse the repository at this point in the history
  34. Configuration menu
    Copy the full SHA
    0903e4b View commit details
    Browse the repository at this point in the history
  35. Configuration menu
    Copy the full SHA
    87b46c9 View commit details
    Browse the repository at this point in the history
  36. Fix too much small row group issue in Velox parquet writer (oap-proje…

    …ct#326)
    
    Co-authored-by: youxiduo <[email protected]>
    2 people authored and zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    0589260 View commit details
    Browse the repository at this point in the history
  37. rebase to velox 4.27 300370f

    ‘zhaozhenhui’ authored and zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    5e7b4b0 View commit details
    Browse the repository at this point in the history
  38. Configuration menu
    Copy the full SHA
    cbd91d3 View commit details
    Browse the repository at this point in the history
  39. Configuration menu
    Copy the full SHA
    feafeb3 View commit details
    Browse the repository at this point in the history
  40. Configuration menu
    Copy the full SHA
    2bf58aa View commit details
    Browse the repository at this point in the history
  41. Fix velox_parquet_e2e_filter_test CI

    Co-authored-by: youxiduo <[email protected]>
    2 people authored and zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    c3a8948 View commit details
    Browse the repository at this point in the history
  42. Fix format-check (oap-project#334)

    * format
    zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    66dc508 View commit details
    Browse the repository at this point in the history
  43. Configuration menu
    Copy the full SHA
    89b7174 View commit details
    Browse the repository at this point in the history
  44. Configuration menu
    Copy the full SHA
    975aec0 View commit details
    Browse the repository at this point in the history
  45. Configuration menu
    Copy the full SHA
    96b8b85 View commit details
    Browse the repository at this point in the history
  46. Quick fix velox UT and format (oap-project#340)

    * quick fix velox UT
    zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    b7f0375 View commit details
    Browse the repository at this point in the history
  47. Configuration menu
    Copy the full SHA
    7520160 View commit details
    Browse the repository at this point in the history
  48. Fix decimal add (oap-project#344)

    * fix decimal add
    * remove unused code
    zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    b8d2333 View commit details
    Browse the repository at this point in the history
  49. fix rebase compile

    zhejiangxiaomai committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    d151629 View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2023

  1. Revert "ld (oap-project#317)" (oap-project#338)

    This reverts commit bb6a570.
    jinchengchenghh authored and zhejiangxiaomai committed Jul 4, 2023
    Configuration menu
    Copy the full SHA
    875d0b9 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    482e4b3 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    43697df View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    78e96ff View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2023

  1. Configuration menu
    Copy the full SHA
    386f25e View commit details
    Browse the repository at this point in the history
  2. Fix permission problem (oap-project#352)

    Co-authored-by: ‘zhaozhenhui’ <‘[email protected]’>
    zhejiangxiaomai and ‘zhaozhenhui’ authored Jul 5, 2023
    Configuration menu
    Copy the full SHA
    4e95d70 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2023

  1. Configuration menu
    Copy the full SHA
    da13c97 View commit details
    Browse the repository at this point in the history
  2. Use presto abs (oap-project#345)

    * Use presto abs
    jinchengchenghh authored Jul 6, 2023
    Configuration menu
    Copy the full SHA
    a62ddb3 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    68be4e0 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2023

  1. spillheap

    zuochunwei committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    8aad40e View commit details
    Browse the repository at this point in the history