[feature](datalake) Add BucketShuffleJoin support for bucketed hive tables #27784

Nitin-Kashyap · 2023-11-29T17:10:19Z

Add BucketShuffleJoin support for bucketed hive tables generated by Spark. (27783)

Proposed changes

Issue Number: close #27783

1. Original planner updated to consider BucketShuffle for bucketed hive table
2. Neerids planner updated for bucketShuffle join on hive tables.
3. Added spark style hash calculation in BE for shuffle on one side.

###Sample Output:s

github-actions

clang-tidy made some suggestions

be/src/vec/columns/column_decimal.cpp

be/src/vec/columns/column_map.cpp

be/src/vec/columns/column_string.cpp

github-actions · 2023-11-30T03:36:28Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions

clang-tidy made some suggestions

be/src/vec/columns/column_vector.cpp

github-actions · 2023-11-30T05:55:31Z

clang-tidy review says "All clean, LGTM! 👍"

morningman · 2023-12-01T01:41:01Z

Hi @Nitin-Kashyap , thanks for your contribution.
Could you please provide some create table stmt of hive table on spark side,
so that we can test this case?

morningman · 2023-12-01T01:42:24Z

BTW, is it only suitable for "spark created" hive bucket table?
What if the hive table is created by other system with different hash function?

Nitin-Kashyap · 2023-12-01T04:27:03Z

Hi @Nitin-Kashyap , thanks for your contribution. Could you please provide some create table stmt of hive table on spark side, so that we can test this case?

@morningman Please find the sample test I used for this case: -

CREATE TABLE parquet_test (
     user_id INT,
     key       VARCHAR(20),
     part      VARCAHAR(10)
)
USING parquet
PARTITIONED BY (part)
CLUSTERED BY (user_id) INTO 3 BUCKETS;

INSERT INTO parquet_test2 VALUES (31, 'U31', 'IN'),  (11,'U11','IN'), (21, 'U21', 'IN');

Nitin-Kashyap · 2023-12-01T04:54:38Z

BTW, is it only suitable for "spark created" hive bucket table? What if the hive table is created by other system with different hash function?

@morningman Yes, for current scope it will understand only Spark created bucketed table, it identifies this by Properties defined by spark for bucket specification.

I plan to take up supporting for Hive, Hudi as well in some time (hopefully in next PR); for this I have left a place holder THashType [HIVE_MOD: Hive and Hudi use the same hash method] however for hudi some more changes on FE side need to do for identifing type bucket id from file path.

github-actions · 2023-12-02T12:28:11Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-12-02T12:28:42Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-12-04T05:14:30Z

clang-tidy review says "All clean, LGTM! 👍"

fe/fe-core/src/main/java/org/apache/doris/catalog/external/HMSExternalTable.java

fe/fe-core/src/main/java/org/apache/doris/planner/external/HiveScanNode.java

fe/fe-core/src/main/java/org/apache/doris/planner/DataPartition.java

fe/fe-core/src/main/java/org/apache/doris/planner/DistributedPlanner.java

fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java

fe/fe-core/src/main/java/org/apache/doris/planner/external/FileQueryScanNode.java

be/src/vec/columns/column.h

be/src/vec/utils/util.hpp

gensrc/thrift/Partitions.thrift

github-actions · 2023-12-04T18:20:47Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-02-23T05:40:04Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-02-23T05:53:23Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-02-23T06:38:35Z

clang-tidy review says "All clean, LGTM! 👍"

Nitin-Kashyap · 2024-02-23T06:42:12Z

run buildall

doris-robot · 2024-02-23T08:13:06Z

TeamCity be ut coverage result:
Function Coverage: 35.68% (8549/23961)
Line Coverage: 27.51% (69374/252223)
Region Coverage: 26.65% (35975/135012)
Branch Coverage: 23.46% (18393/78404)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a13919ca473b2abfe5bf7b177061c4da8416f4d9_a13919ca473b2abfe5bf7b177061c4da8416f4d9/report/index.html

github-actions · 2024-02-26T05:25:22Z

clang-tidy review says "All clean, LGTM! 👍"

Nitin-Kashyap · 2024-03-01T05:44:41Z

run buildall

github-actions · 2024-03-01T05:52:29Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-03-01T07:17:10Z

TeamCity be ut coverage result:
Function Coverage: 35.60% (8550/24014)
Line Coverage: 27.42% (69395/253124)
Region Coverage: 26.57% (36001/135482)
Branch Coverage: 23.39% (18394/78630)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a5ce2395a2ea0752c987d764edc0b39f902f82ce_a5ce2395a2ea0752c987d764edc0b39f902f82ce/report/index.html

morningman · 2024-03-01T09:30:14Z

Hi @Nitin-Kashyap , I submitted a PR to your branch
Nitin-Kashyap#1
Please review

github-actions · 2024-03-01T09:43:34Z

clang-tidy review says "All clean, LGTM! 👍"

morrySnow

LGTM

Nitin-Kashyap · 2024-03-04T14:29:00Z

run buildall

github-actions · 2024-03-28T07:59:31Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In build-support/shell-check.sh line 30:
    source "${DORIS_HOME}/env.sh" &>/dev/null
           ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In build.sh line 34:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In build.sh line 108:
    "${MVN_CMD}" clean
     ^--------^ SC2154 (warning): MVN_CMD is referenced but not assigned.


In build.sh line 280:
if [[ ! -f "${DORIS_THIRDPARTY}/installed/lib/libbacktrace.a" ]]; then
            ^-----------------^ SC2154 (warning): DORIS_THIRDPARTY is referenced but not assigned.


In build.sh line 474:
    ENABLE_PCH                  -- ${ENABLE_PCH}
                                   ^-----------^ SC2154 (warning): ENABLE_PCH is referenced but not assigned.


In build.sh line 538:
    MAKE_PROGRAM="$(command -v "${BUILD_SYSTEM}")"
                                ^-------------^ SC2154 (warning): BUILD_SYSTEM is referenced but not assigned.


In build.sh line 545:
    echo "-- Use ccache: ${CMAKE_USE_CCACHE}"
                         ^-----------------^ SC2154 (warning): CMAKE_USE_CCACHE is referenced but not assigned.


In build.sh line 551:
    "${CMAKE_CMD}" -G "${GENERATOR}" \
     ^----------^ SC2154 (warning): CMAKE_CMD is referenced but not assigned.
                       ^----------^ SC2154 (warning): GENERATOR is referenced but not assigned.


In build_plugin.sh line 25:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In build_plugin.sh line 105:
        "${MVN_CMD}" clean
         ^--------^ SC2154 (warning): MVN_CMD is referenced but not assigned.


In contrib/udf/build_udf.sh line 39:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In contrib/udf/build_udf.sh line 123:
    "${CMAKE_CMD}" -G "${GENERATOR}" -DCMAKE_BUILD_TYPE="${CMAKE_BUILD_TYPE}" ../
     ^----------^ SC2154 (warning): CMAKE_CMD is referenced but not assigned.
                       ^----------^ SC2154 (warning): GENERATOR is referenced but not assigned.


In contrib/udf/build_udf.sh line 124:
    "${BUILD_SYSTEM}" -j "${PARALLEL}"
     ^-------------^ SC2154 (warning): BUILD_SYSTEM is referenced but not assigned.


In fe_plugins/auditloader/build.sh line 25:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In fe_plugins/auditloader/build.sh line 29:
"${MVN_CMD}" clean package -DskipTests
 ^--------^ SC2154 (warning): MVN_CMD is referenced but not assigned.


In fs_brokers/apache_hdfs_broker/build.sh line 25:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In fs_brokers/apache_hdfs_broker/build.sh line 35:
"${MVN_CMD}" package -DskipTests
 ^--------^ SC2154 (warning): MVN_CMD is referenced but not assigned.


In generated-source.sh line 29:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In run-be-ut.sh line 42:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In run-be-ut.sh line 132:
    ENABLE_PCH          -- ${ENABLE_PCH}
                           ^-----------^ SC2154 (warning): ENABLE_PCH is referenced but not assigned.


In run-be-ut.sh line 218:
MAKE_PROGRAM="$(command -v "${BUILD_SYSTEM}")"
                            ^-------------^ SC2154 (warning): BUILD_SYSTEM is referenced but not assigned.


In run-be-ut.sh line 220:
echo "-- Use ccache: ${CMAKE_USE_CCACHE}"
                     ^-----------------^ SC2154 (warning): CMAKE_USE_CCACHE is referenced but not assigned.


In run-be-ut.sh line 224:
"${CMAKE_CMD}" -G "${GENERATOR}" \
 ^----------^ SC2154 (warning): CMAKE_CMD is referenced but not assigned.
                   ^----------^ SC2154 (warning): GENERATOR is referenced but not assigned.


In run-be-ut.sh line 241:
    -DDORIS_JAVA_HOME="${JAVA_HOME}" \
                       ^----------^ SC2154 (warning): JAVA_HOME is referenced but not assigned.


In run-be-ut.sh line 303:
if [[ -d "${DORIS_THIRDPARTY}/installed/lib/hadoop_hdfs/" ]]; then
          ^-----------------^ SC2154 (warning): DORIS_THIRDPARTY is referenced but not assigned.


In run-be-ut.sh line 424:
        cmd1="${LLVM_PROFDATA} merge -o ${profdata} ${profraw}"
              ^--------------^ SC2154 (warning): LLVM_PROFDATA is referenced but not assigned.


In run-be-ut.sh line 427:
        cmd2="${LLVM_COV} show -output-dir=${DORIS_TEST_BINARY_DIR}/report -format=html \
              ^---------^ SC2154 (warning): LLVM_COV is referenced but not assigned.


In run-cloud-ut.sh line 140:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In run-cloud-ut.sh line 160:
MAKE_PROGRAM="$(command -v "${BUILD_SYSTEM}")"
                            ^-------------^ SC2154 (warning): BUILD_SYSTEM is referenced but not assigned.


In run-cloud-ut.sh line 166:
"${CMAKE_CMD}" -G "${GENERATOR}" \
 ^----------^ SC2154 (warning): CMAKE_CMD is referenced but not assigned.
                   ^----------^ SC2154 (warning): GENERATOR is referenced but not assigned.


In run-cloud-ut.sh line 177:
    "${CMAKE_USE_CCACHE}" \
     ^-----------------^ SC2154 (warning): CMAKE_USE_CCACHE is referenced but not assigned.


In run-fe-ut.sh line 25:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In run-fe-ut.sh line 117:
        "${MVN_CMD}" test jacoco:report -DfailIfNoTests=false -Dtest="$1"
         ^--------^ SC2154 (warning): MVN_CMD is referenced but not assigned.


In thirdparty/build-thirdparty.sh line 42:
    . "${DORIS_HOME}/env.sh"
      ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In thirdparty/build-thirdparty.sh line 155:
if [[ "${CC}" == *gcc ]]; then
       ^---^ SC2154 (warning): CC is referenced but not assigned.


In thirdparty/build-thirdparty.sh line 208:
check_prerequest "${CMAKE_CMD} --version" "cmake"
                  ^----------^ SC2154 (warning): CMAKE_CMD is referenced but not assigned.


In thirdparty/build-thirdparty.sh line 340:
        "${CMAKE_CMD}" -G "${GENERATOR}" -DCMAKE_INSTALL_PREFIX="${TP_INSTALL_DIR}" -DEVENT__DISABLE_TESTS=ON \
                           ^----------^ SC2154 (warning): GENERATOR is referenced but not assigned.


In thirdparty/build-thirdparty.sh line 343:
    "${BUILD_SYSTEM}" -j "${PARALLEL}"
     ^-------------^ SC2154 (warning): BUILD_SYSTEM is referenced but not assigned.


In thirdparty/build-thirdparty.sh line 1123:
    local ld="${DORIS_BIN_UTILS}/ld"
              ^----------------^ SC2154 (warning): DORIS_BIN_UTILS is referenced but not assigned.

For more information:
  https://www.shellcheck.net/wiki/SC2154 -- BUILD_SYSTEM is referenced but no...
  https://www.shellcheck.net/wiki/SC1091 -- Not following: ./env.sh: openBina...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions · 2024-03-29T09:30:23Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In build-support/shell-check.sh line 30:
    source "${DORIS_HOME}/env.sh" &>/dev/null
           ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In build.sh line 34:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In build.sh line 108:
    "${MVN_CMD}" clean
     ^--------^ SC2154 (warning): MVN_CMD is referenced but not assigned.


In build.sh line 280:
if [[ ! -f "${DORIS_THIRDPARTY}/installed/lib/libbacktrace.a" ]]; then
            ^-----------------^ SC2154 (warning): DORIS_THIRDPARTY is referenced but not assigned.


In build.sh line 474:
    ENABLE_PCH                  -- ${ENABLE_PCH}
                                   ^-----------^ SC2154 (warning): ENABLE_PCH is referenced but not assigned.


In build.sh line 538:
    MAKE_PROGRAM="$(command -v "${BUILD_SYSTEM}")"
                                ^-------------^ SC2154 (warning): BUILD_SYSTEM is referenced but not assigned.


In build.sh line 545:
    echo "-- Use ccache: ${CMAKE_USE_CCACHE}"
                         ^-----------------^ SC2154 (warning): CMAKE_USE_CCACHE is referenced but not assigned.


In build.sh line 551:
    "${CMAKE_CMD}" -G "${GENERATOR}" \
     ^----------^ SC2154 (warning): CMAKE_CMD is referenced but not assigned.
                       ^----------^ SC2154 (warning): GENERATOR is referenced but not assigned.


In build_plugin.sh line 25:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In build_plugin.sh line 105:
        "${MVN_CMD}" clean
         ^--------^ SC2154 (warning): MVN_CMD is referenced but not assigned.


In contrib/udf/build_udf.sh line 39:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In contrib/udf/build_udf.sh line 123:
    "${CMAKE_CMD}" -G "${GENERATOR}" -DCMAKE_BUILD_TYPE="${CMAKE_BUILD_TYPE}" ../
     ^----------^ SC2154 (warning): CMAKE_CMD is referenced but not assigned.
                       ^----------^ SC2154 (warning): GENERATOR is referenced but not assigned.


In contrib/udf/build_udf.sh line 124:
    "${BUILD_SYSTEM}" -j "${PARALLEL}"
     ^-------------^ SC2154 (warning): BUILD_SYSTEM is referenced but not assigned.


In fe_plugins/auditloader/build.sh line 25:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In fe_plugins/auditloader/build.sh line 29:
"${MVN_CMD}" clean package -DskipTests
 ^--------^ SC2154 (warning): MVN_CMD is referenced but not assigned.


In fs_brokers/apache_hdfs_broker/build.sh line 25:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In fs_brokers/apache_hdfs_broker/build.sh line 35:
"${MVN_CMD}" package -DskipTests
 ^--------^ SC2154 (warning): MVN_CMD is referenced but not assigned.


In generated-source.sh line 29:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In run-be-ut.sh line 42:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In run-be-ut.sh line 132:
    ENABLE_PCH          -- ${ENABLE_PCH}
                           ^-----------^ SC2154 (warning): ENABLE_PCH is referenced but not assigned.


In run-be-ut.sh line 218:
MAKE_PROGRAM="$(command -v "${BUILD_SYSTEM}")"
                            ^-------------^ SC2154 (warning): BUILD_SYSTEM is referenced but not assigned.


In run-be-ut.sh line 220:
echo "-- Use ccache: ${CMAKE_USE_CCACHE}"
                     ^-----------------^ SC2154 (warning): CMAKE_USE_CCACHE is referenced but not assigned.


In run-be-ut.sh line 224:
"${CMAKE_CMD}" -G "${GENERATOR}" \
 ^----------^ SC2154 (warning): CMAKE_CMD is referenced but not assigned.
                   ^----------^ SC2154 (warning): GENERATOR is referenced but not assigned.


In run-be-ut.sh line 241:
    -DDORIS_JAVA_HOME="${JAVA_HOME}" \
                       ^----------^ SC2154 (warning): JAVA_HOME is referenced but not assigned.


In run-be-ut.sh line 303:
if [[ -d "${DORIS_THIRDPARTY}/installed/lib/hadoop_hdfs/" ]]; then
          ^-----------------^ SC2154 (warning): DORIS_THIRDPARTY is referenced but not assigned.


In run-be-ut.sh line 424:
        cmd1="${LLVM_PROFDATA} merge -o ${profdata} ${profraw}"
              ^--------------^ SC2154 (warning): LLVM_PROFDATA is referenced but not assigned.


In run-be-ut.sh line 427:
        cmd2="${LLVM_COV} show -output-dir=${DORIS_TEST_BINARY_DIR}/report -format=html \
              ^---------^ SC2154 (warning): LLVM_COV is referenced but not assigned.


In run-cloud-ut.sh line 140:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In run-cloud-ut.sh line 160:
MAKE_PROGRAM="$(command -v "${BUILD_SYSTEM}")"
                            ^-------------^ SC2154 (warning): BUILD_SYSTEM is referenced but not assigned.


In run-cloud-ut.sh line 166:
"${CMAKE_CMD}" -G "${GENERATOR}" \
 ^----------^ SC2154 (warning): CMAKE_CMD is referenced but not assigned.
                   ^----------^ SC2154 (warning): GENERATOR is referenced but not assigned.


In run-cloud-ut.sh line 177:
    "${CMAKE_USE_CCACHE}" \
     ^-----------------^ SC2154 (warning): CMAKE_USE_CCACHE is referenced but not assigned.


In run-fe-ut.sh line 25:
. "${DORIS_HOME}/env.sh"
  ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In run-fe-ut.sh line 117:
        "${MVN_CMD}" test jacoco:report -DfailIfNoTests=false -Dtest="$1"
         ^--------^ SC2154 (warning): MVN_CMD is referenced but not assigned.


In thirdparty/build-thirdparty.sh line 42:
    . "${DORIS_HOME}/env.sh"
      ^--------------------^ SC1091 (info): Not following: ./env.sh: openBinaryFile: does not exist (No such file or directory)


In thirdparty/build-thirdparty.sh line 155:
if [[ "${CC}" == *gcc ]]; then
       ^---^ SC2154 (warning): CC is referenced but not assigned.


In thirdparty/build-thirdparty.sh line 208:
check_prerequest "${CMAKE_CMD} --version" "cmake"
                  ^----------^ SC2154 (warning): CMAKE_CMD is referenced but not assigned.


In thirdparty/build-thirdparty.sh line 340:
        "${CMAKE_CMD}" -G "${GENERATOR}" -DCMAKE_INSTALL_PREFIX="${TP_INSTALL_DIR}" -DEVENT__DISABLE_TESTS=ON \
                           ^----------^ SC2154 (warning): GENERATOR is referenced but not assigned.


In thirdparty/build-thirdparty.sh line 343:
    "${BUILD_SYSTEM}" -j "${PARALLEL}"
     ^-------------^ SC2154 (warning): BUILD_SYSTEM is referenced but not assigned.


In thirdparty/build-thirdparty.sh line 1123:
    local ld="${DORIS_BIN_UTILS}/ld"
              ^----------------^ SC2154 (warning): DORIS_BIN_UTILS is referenced but not assigned.

For more information:
  https://www.shellcheck.net/wiki/SC2154 -- BUILD_SYSTEM is referenced but no...
  https://www.shellcheck.net/wiki/SC1091 -- Not following: ./env.sh: openBina...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

… generated by Spark. (27783) 1. Original planner updated to consider BucketShuffle for bucketed hive table 2. Neerids planner updated for bucketShuffle join on hive tables. 3. Added spark style hash calculation in BE for shuffle on one side. 4. Added shuffle hash selection based on left(non-shuffling) side.

github-actions · 2024-07-05T06:04:26Z

clang-tidy review says "All clean, LGTM! 👍"

Nitin-Kashyap force-pushed the feature-hiveBucketShuffle branch from 0488e3b to c5e23b2 Compare November 29, 2023 17:10

github-actions bot added the meta-change label Nov 29, 2023

github-actions bot reviewed Nov 29, 2023

View reviewed changes

be/src/vec/columns/column_decimal.cpp Outdated Show resolved Hide resolved

be/src/vec/columns/column_map.cpp Show resolved Hide resolved

be/src/vec/columns/column_string.cpp Outdated Show resolved Hide resolved

Nitin-Kashyap force-pushed the feature-hiveBucketShuffle branch from b4464d4 to f9e42ab Compare November 30, 2023 04:48

github-actions bot reviewed Nov 30, 2023

View reviewed changes

be/src/vec/columns/column_vector.cpp Outdated Show resolved Hide resolved

Nitin-Kashyap force-pushed the feature-hiveBucketShuffle branch 2 times, most recently from ed212e1 to eaf29b0 Compare November 30, 2023 05:47

morningman self-assigned this Nov 30, 2023

Nitin-Kashyap force-pushed the feature-hiveBucketShuffle branch from eaf29b0 to 34c701c Compare December 2, 2023 12:19

wm1581066 requested review from morningman and morrySnow December 3, 2023 10:51

Nitin-Kashyap force-pushed the feature-hiveBucketShuffle branch from 34c701c to d25350a Compare December 4, 2023 05:05

morningman reviewed Dec 4, 2023

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/planner/external/FileQueryScanNode.java Outdated Show resolved Hide resolved

HappenLee reviewed Dec 4, 2023

View reviewed changes

be/src/vec/columns/column.h Outdated Show resolved Hide resolved

HappenLee reviewed Dec 4, 2023

View reviewed changes

be/src/vec/utils/util.hpp Outdated Show resolved Hide resolved

HappenLee reviewed Dec 4, 2023

View reviewed changes

gensrc/thrift/Partitions.thrift Outdated Show resolved Hide resolved

Nitin-Kashyap force-pushed the feature-hiveBucketShuffle branch from d25350a to 28039b8 Compare December 4, 2023 18:12

Nitin-Kashyap requested review from morningman and HappenLee December 4, 2023 18:12

Nitin-Kashyap force-pushed the feature-hiveBucketShuffle branch from 6f15fc1 to f84c004 Compare February 23, 2024 05:42

Nitin-Kashyap force-pushed the feature-hiveBucketShuffle branch from f84c004 to a13919c Compare February 23, 2024 06:27

Nitin-Kashyap requested a review from morrySnow February 23, 2024 06:42

morningman force-pushed the feature-hiveBucketShuffle branch from a13919c to 92adb1e Compare February 26, 2024 05:14

Nitin-Kashyap force-pushed the feature-hiveBucketShuffle branch from 92adb1e to a5ce239 Compare March 1, 2024 05:41

morrySnow reviewed Mar 1, 2024

View reviewed changes

Nitin-Kashyap force-pushed the feature-hiveBucketShuffle branch from 83b98e5 to 8daa14a Compare March 28, 2024 07:58

Nitin-Kashyap force-pushed the feature-hiveBucketShuffle branch from 8daa14a to bd53e2c Compare March 29, 2024 09:29

Nitin-Kashyap and others added 6 commits July 5, 2024 11:08

[opt] refine some method name

10c4f56

fixed murmur-hash calculation

0fa8549

fixed murmur-hash calculation-2

d5966b7

fixed murmur-hash calculation-3

f7d0d40

fix local_exchange

d6e888e

Nitin-Kashyap force-pushed the feature-hiveBucketShuffle branch from bd53e2c to d6e888e Compare July 5, 2024 05:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature](datalake) Add BucketShuffleJoin support for bucketed hive tables #27784

[feature](datalake) Add BucketShuffleJoin support for bucketed hive tables #27784

Nitin-Kashyap commented Nov 29, 2023 •

edited

Loading

github-actions bot left a comment

github-actions bot commented Nov 30, 2023

github-actions bot left a comment

github-actions bot commented Nov 30, 2023

morningman commented Dec 1, 2023

morningman commented Dec 1, 2023

Nitin-Kashyap commented Dec 1, 2023 •

edited

Loading

Nitin-Kashyap commented Dec 1, 2023 •

edited

Loading

github-actions bot commented Dec 2, 2023

github-actions bot commented Dec 2, 2023

github-actions bot commented Dec 4, 2023

github-actions bot commented Dec 4, 2023

github-actions bot commented Feb 23, 2024

github-actions bot commented Feb 23, 2024

github-actions bot commented Feb 23, 2024

Nitin-Kashyap commented Feb 23, 2024

doris-robot commented Feb 23, 2024

github-actions bot commented Feb 26, 2024

Nitin-Kashyap commented Mar 1, 2024

github-actions bot commented Mar 1, 2024

doris-robot commented Mar 1, 2024

morningman commented Mar 1, 2024

github-actions bot commented Mar 1, 2024

morrySnow left a comment

Nitin-Kashyap commented Mar 4, 2024

github-actions bot commented Mar 28, 2024

github-actions bot commented Mar 29, 2024

github-actions bot commented Jul 5, 2024

[feature](datalake) Add BucketShuffleJoin support for bucketed hive tables #27784

Are you sure you want to change the base?

[feature](datalake) Add BucketShuffleJoin support for bucketed hive tables #27784

Conversation

Nitin-Kashyap commented Nov 29, 2023 • edited Loading

Proposed changes

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 30, 2023

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 30, 2023

morningman commented Dec 1, 2023

morningman commented Dec 1, 2023

Nitin-Kashyap commented Dec 1, 2023 • edited Loading

Nitin-Kashyap commented Dec 1, 2023 • edited Loading

github-actions bot commented Dec 2, 2023

github-actions bot commented Dec 2, 2023

github-actions bot commented Dec 4, 2023

github-actions bot commented Dec 4, 2023

github-actions bot commented Feb 23, 2024

github-actions bot commented Feb 23, 2024

github-actions bot commented Feb 23, 2024

Nitin-Kashyap commented Feb 23, 2024

doris-robot commented Feb 23, 2024

github-actions bot commented Feb 26, 2024

Nitin-Kashyap commented Mar 1, 2024

github-actions bot commented Mar 1, 2024

doris-robot commented Mar 1, 2024

morningman commented Mar 1, 2024

github-actions bot commented Mar 1, 2024

morrySnow left a comment

Choose a reason for hiding this comment

Nitin-Kashyap commented Mar 4, 2024

github-actions bot commented Mar 28, 2024

sh-checker report

github-actions bot commented Mar 29, 2024

sh-checker report

github-actions bot commented Jul 5, 2024

Nitin-Kashyap commented Nov 29, 2023 •

edited

Loading

Nitin-Kashyap commented Dec 1, 2023 •

edited

Loading

Nitin-Kashyap commented Dec 1, 2023 •

edited

Loading

`sh-checker report`

`sh-checker report`