Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Refactor](inverted index) refactor inverted index compound predicates evaluate logic #38908

Merged
merged 36 commits into from
Sep 2, 2024

Conversation

airborne12
Copy link
Member

@airborne12 airborne12 commented Aug 6, 2024

Proposed changes

This PR addresses several key issues related to the compound condition support in the inverted index, and optimization for index skipping without returning to the table:

  1. Unified Handling of expr and column predicate:

    • Combined the processing of inverted index-related column predicate and expr.
    • Ensured that compound conditions involving both column predicate and expr are processed uniformly to reduce complexity and improve robustness.
  2. Optimized the Execution of Compound Conditions:

    • Removed the logic in scan_operator that normalized compound predicates by pushing down logic to _common_expr_ctxs_push_down where expr contexts are managed.
    • Added evaluate_inverted_index support to the vexpr and function layers, such as function comparison and function collection_in.
      • Introduced new data structures in VExprContext to store results from evaluate_inverted_index, thus facilitating quick lookup and application of these results during execution.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added the doing label Aug 6, 2024
@airborne12 airborne12 marked this pull request as draft August 6, 2024 01:45
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@@ -123,6 +204,18 @@ class VCompoundPred : public VectorizedFnCall {
}

Status execute(VExprContext* context, Block* block, int* result_column_id) override {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'execute' has cognitive complexity of 113 (threshold 50) [readability-function-cognitive-complexity]

    Status execute(VExprContext* context, Block* block, int* result_column_id) override {
           ^
Additional context

be/src/vec/exprs/vcompound_pred.h:206: +1, including nesting penalty of 0, nesting level increased to 1

        if (context->get_inverted_index_result_column().contains(this)) {
        ^

be/src/vec/exprs/vcompound_pred.h:210: +2, including nesting penalty of 1, nesting level increased to 2

            for (int i = 0; i < result_column->size(); i++) {
            ^

be/src/vec/exprs/vcompound_pred.h:218: +1, including nesting penalty of 0, nesting level increased to 1

        if (children().size() == 1 || !_all_child_is_compound_and_not_const()) {
        ^

be/src/vec/exprs/vcompound_pred.h:224: +1, including nesting penalty of 0, nesting level increased to 1

        RETURN_IF_ERROR(_children[0]->execute(context, block, &lhs_id));
        ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/vec/exprs/vcompound_pred.h:224: +2, including nesting penalty of 1, nesting level increased to 2

        RETURN_IF_ERROR(_children[0]->execute(context, block, &lhs_id));
        ^

be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/vec/exprs/vcompound_pred.h:236: +1, including nesting penalty of 0, nesting level increased to 1

        if (lhs_is_nullable) {
        ^

be/src/vec/exprs/vcompound_pred.h:250: nesting level increased to 1

        auto get_rhs_colum = [&]() {
                             ^

be/src/vec/exprs/vcompound_pred.h:251: +2, including nesting penalty of 1, nesting level increased to 2

            if (rhs_id == -1) {
            ^

be/src/vec/exprs/vcompound_pred.h:252: +3, including nesting penalty of 2, nesting level increased to 3

                RETURN_IF_ERROR(_children[1]->execute(context, block, &rhs_id));
                ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/vec/exprs/vcompound_pred.h:252: +4, including nesting penalty of 3, nesting level increased to 4

                RETURN_IF_ERROR(_children[1]->execute(context, block, &rhs_id));
                ^

be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/vec/exprs/vcompound_pred.h:262: +3, including nesting penalty of 2, nesting level increased to 3

                if (rhs_is_nullable) {
                ^

be/src/vec/exprs/vcompound_pred.h:270: nesting level increased to 1

        auto return_result_column_id = [&](ColumnPtr res_column, int res_id) -> int {
                                       ^

be/src/vec/exprs/vcompound_pred.h:271: +2, including nesting penalty of 1, nesting level increased to 2

            if (result_is_nullable && !res_column->is_nullable()) {
            ^

be/src/vec/exprs/vcompound_pred.h:271: +1

            if (result_is_nullable && !res_column->is_nullable()) {
                                   ^

be/src/vec/exprs/vcompound_pred.h:280: nesting level increased to 1

        auto create_null_map_column = [&](ColumnPtr& null_map_column,
                                      ^

be/src/vec/exprs/vcompound_pred.h:282: +2, including nesting penalty of 1, nesting level increased to 2

            if (null_map_data == nullptr) {
            ^

be/src/vec/exprs/vcompound_pred.h:291: nesting level increased to 1

        auto vector_vector_null = [&]<bool is_and_op>() {
                                  ^

be/src/vec/exprs/vcompound_pred.h:301: +2, including nesting penalty of 1, nesting level increased to 2

            if constexpr (is_and_op) {
            ^

be/src/vec/exprs/vcompound_pred.h:302: +3, including nesting penalty of 2, nesting level increased to 3

                for (size_t i = 0; i < size; ++i) {
                ^

be/src/vec/exprs/vcompound_pred.h:307: +1, nesting level increased to 2

            } else {
              ^

be/src/vec/exprs/vcompound_pred.h:308: +3, including nesting penalty of 2, nesting level increased to 3

                for (size_t i = 0; i < size; ++i) {
                ^

be/src/vec/exprs/vcompound_pred.h:321: +1, including nesting penalty of 0, nesting level increased to 1

        if (_op == TExprOpcode::COMPOUND_AND) {
        ^

be/src/vec/exprs/vcompound_pred.h:324: +2, including nesting penalty of 1, nesting level increased to 2

            if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
            ^

be/src/vec/exprs/vcompound_pred.h:324: +1

            if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                                    ^

be/src/vec/exprs/vcompound_pred.h:324: +1

            if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                               ^

be/src/vec/exprs/vcompound_pred.h:324: +1

            if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                                                      ^

be/src/vec/exprs/vcompound_pred.h:327: +1, nesting level increased to 2

            } else {
              ^

be/src/vec/exprs/vcompound_pred.h:328: +3, including nesting penalty of 2, nesting level increased to 3

                RETURN_IF_ERROR(get_rhs_colum());
                ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/vec/exprs/vcompound_pred.h:328: +4, including nesting penalty of 3, nesting level increased to 4

                RETURN_IF_ERROR(get_rhs_colum());
                ^

be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/vec/exprs/vcompound_pred.h:330: +3, including nesting penalty of 2, nesting level increased to 3

                if ((lhs_all_true && !lhs_is_nullable) ||    //not null column
                ^

be/src/vec/exprs/vcompound_pred.h:330: +1

                if ((lhs_all_true && !lhs_is_nullable) ||    //not null column
                                                       ^

be/src/vec/exprs/vcompound_pred.h:330: +1

                if ((lhs_all_true && !lhs_is_nullable) ||    //not null column
                                  ^

be/src/vec/exprs/vcompound_pred.h:331: +1

                    (lhs_all_true && lhs_all_is_not_null)) { //nullable column
                                  ^

be/src/vec/exprs/vcompound_pred.h:334: +1, nesting level increased to 3

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                       ^

be/src/vec/exprs/vcompound_pred.h:334: +1

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                                                               ^

be/src/vec/exprs/vcompound_pred.h:334: +1

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                                          ^

be/src/vec/exprs/vcompound_pred.h:335: +1

                           (rhs_all_false && rhs_all_is_not_null)) {
                                          ^

be/src/vec/exprs/vcompound_pred.h:338: +1, nesting level increased to 3

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                       ^

be/src/vec/exprs/vcompound_pred.h:338: +1

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                                                              ^

be/src/vec/exprs/vcompound_pred.h:338: +1

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                                         ^

be/src/vec/exprs/vcompound_pred.h:339: +1

                           (rhs_all_true && rhs_all_is_not_null)) {
                                         ^

be/src/vec/exprs/vcompound_pred.h:342: +1, nesting level increased to 3

                } else {
                  ^

be/src/vec/exprs/vcompound_pred.h:343: +4, including nesting penalty of 3, nesting level increased to 4

                    if (!result_is_nullable) {
                    ^

be/src/vec/exprs/vcompound_pred.h:345: +5, including nesting penalty of 4, nesting level increased to 5

                        for (size_t i = 0; i < size; i++) {
                        ^

be/src/vec/exprs/vcompound_pred.h:348: +1, nesting level increased to 4

                    } else {
                      ^

be/src/vec/exprs/vcompound_pred.h:353: +1, nesting level increased to 1

        } else if (_op == TExprOpcode::COMPOUND_OR) {
               ^

be/src/vec/exprs/vcompound_pred.h:356: +2, including nesting penalty of 1, nesting level increased to 2

            if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) {
            ^

be/src/vec/exprs/vcompound_pred.h:356: +1

            if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) {
                                                   ^

be/src/vec/exprs/vcompound_pred.h:356: +1

            if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) {
                              ^

be/src/vec/exprs/vcompound_pred.h:356: +1

            if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) {
                                                                    ^

be/src/vec/exprs/vcompound_pred.h:359: +1, nesting level increased to 2

            } else {
              ^

be/src/vec/exprs/vcompound_pred.h:360: +3, including nesting penalty of 2, nesting level increased to 3

                RETURN_IF_ERROR(get_rhs_colum());
                ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/vec/exprs/vcompound_pred.h:360: +4, including nesting penalty of 3, nesting level increased to 4

                RETURN_IF_ERROR(get_rhs_colum());
                ^

be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/vec/exprs/vcompound_pred.h:361: +3, including nesting penalty of 2, nesting level increased to 3

                if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                ^

be/src/vec/exprs/vcompound_pred.h:361: +1

                if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                                        ^

be/src/vec/exprs/vcompound_pred.h:361: +1

                if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                   ^

be/src/vec/exprs/vcompound_pred.h:361: +1

                if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                                                          ^

be/src/vec/exprs/vcompound_pred.h:364: +1, nesting level increased to 3

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                       ^

be/src/vec/exprs/vcompound_pred.h:364: +1

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                                                              ^

be/src/vec/exprs/vcompound_pred.h:364: +1

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                                         ^

be/src/vec/exprs/vcompound_pred.h:365: +1

                           (rhs_all_true && rhs_all_is_not_null)) {
                                         ^

be/src/vec/exprs/vcompound_pred.h:368: +1, nesting level increased to 3

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                       ^

be/src/vec/exprs/vcompound_pred.h:368: +1

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                                                               ^

be/src/vec/exprs/vcompound_pred.h:368: +1

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                                          ^

be/src/vec/exprs/vcompound_pred.h:369: +1

                           (rhs_all_false && rhs_all_is_not_null)) {
                                          ^

be/src/vec/exprs/vcompound_pred.h:372: +1, nesting level increased to 3

                } else {
                  ^

be/src/vec/exprs/vcompound_pred.h:373: +4, including nesting penalty of 3, nesting level increased to 4

                    if (!result_is_nullable) {
                    ^

be/src/vec/exprs/vcompound_pred.h:375: +5, including nesting penalty of 4, nesting level increased to 5

                        for (size_t i = 0; i < size; i++) {
                        ^

be/src/vec/exprs/vcompound_pred.h:378: +1, nesting level increased to 4

                    } else {
                      ^

be/src/vec/exprs/vcompound_pred.h:383: +1, nesting level increased to 1

        } else {
          ^

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@@ -123,6 +191,18 @@ class VCompoundPred : public VectorizedFnCall {
}

Status execute(VExprContext* context, Block* block, int* result_column_id) override {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'execute' has cognitive complexity of 113 (threshold 50) [readability-function-cognitive-complexity]

    Status execute(VExprContext* context, Block* block, int* result_column_id) override {
           ^
Additional context

be/src/vec/exprs/vcompound_pred.h:193: +1, including nesting penalty of 0, nesting level increased to 1

        if (context->get_inverted_index_result_column().contains(this)) {
        ^

be/src/vec/exprs/vcompound_pred.h:197: +2, including nesting penalty of 1, nesting level increased to 2

            for (int i = 0; i < result_column->size(); i++) {
            ^

be/src/vec/exprs/vcompound_pred.h:205: +1, including nesting penalty of 0, nesting level increased to 1

        if (children().size() == 1 || !_all_child_is_compound_and_not_const()) {
        ^

be/src/vec/exprs/vcompound_pred.h:211: +1, including nesting penalty of 0, nesting level increased to 1

        RETURN_IF_ERROR(_children[0]->execute(context, block, &lhs_id));
        ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/vec/exprs/vcompound_pred.h:211: +2, including nesting penalty of 1, nesting level increased to 2

        RETURN_IF_ERROR(_children[0]->execute(context, block, &lhs_id));
        ^

be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/vec/exprs/vcompound_pred.h:223: +1, including nesting penalty of 0, nesting level increased to 1

        if (lhs_is_nullable) {
        ^

be/src/vec/exprs/vcompound_pred.h:237: nesting level increased to 1

        auto get_rhs_colum = [&]() {
                             ^

be/src/vec/exprs/vcompound_pred.h:238: +2, including nesting penalty of 1, nesting level increased to 2

            if (rhs_id == -1) {
            ^

be/src/vec/exprs/vcompound_pred.h:239: +3, including nesting penalty of 2, nesting level increased to 3

                RETURN_IF_ERROR(_children[1]->execute(context, block, &rhs_id));
                ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/vec/exprs/vcompound_pred.h:239: +4, including nesting penalty of 3, nesting level increased to 4

                RETURN_IF_ERROR(_children[1]->execute(context, block, &rhs_id));
                ^

be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/vec/exprs/vcompound_pred.h:249: +3, including nesting penalty of 2, nesting level increased to 3

                if (rhs_is_nullable) {
                ^

be/src/vec/exprs/vcompound_pred.h:257: nesting level increased to 1

        auto return_result_column_id = [&](ColumnPtr res_column, int res_id) -> int {
                                       ^

be/src/vec/exprs/vcompound_pred.h:258: +2, including nesting penalty of 1, nesting level increased to 2

            if (result_is_nullable && !res_column->is_nullable()) {
            ^

be/src/vec/exprs/vcompound_pred.h:258: +1

            if (result_is_nullable && !res_column->is_nullable()) {
                                   ^

be/src/vec/exprs/vcompound_pred.h:267: nesting level increased to 1

        auto create_null_map_column = [&](ColumnPtr& null_map_column,
                                      ^

be/src/vec/exprs/vcompound_pred.h:269: +2, including nesting penalty of 1, nesting level increased to 2

            if (null_map_data == nullptr) {
            ^

be/src/vec/exprs/vcompound_pred.h:278: nesting level increased to 1

        auto vector_vector_null = [&]<bool is_and_op>() {
                                  ^

be/src/vec/exprs/vcompound_pred.h:288: +2, including nesting penalty of 1, nesting level increased to 2

            if constexpr (is_and_op) {
            ^

be/src/vec/exprs/vcompound_pred.h:289: +3, including nesting penalty of 2, nesting level increased to 3

                for (size_t i = 0; i < size; ++i) {
                ^

be/src/vec/exprs/vcompound_pred.h:294: +1, nesting level increased to 2

            } else {
              ^

be/src/vec/exprs/vcompound_pred.h:295: +3, including nesting penalty of 2, nesting level increased to 3

                for (size_t i = 0; i < size; ++i) {
                ^

be/src/vec/exprs/vcompound_pred.h:308: +1, including nesting penalty of 0, nesting level increased to 1

        if (_op == TExprOpcode::COMPOUND_AND) {
        ^

be/src/vec/exprs/vcompound_pred.h:311: +2, including nesting penalty of 1, nesting level increased to 2

            if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
            ^

be/src/vec/exprs/vcompound_pred.h:311: +1

            if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                                    ^

be/src/vec/exprs/vcompound_pred.h:311: +1

            if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                               ^

be/src/vec/exprs/vcompound_pred.h:311: +1

            if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                                                      ^

be/src/vec/exprs/vcompound_pred.h:314: +1, nesting level increased to 2

            } else {
              ^

be/src/vec/exprs/vcompound_pred.h:315: +3, including nesting penalty of 2, nesting level increased to 3

                RETURN_IF_ERROR(get_rhs_colum());
                ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/vec/exprs/vcompound_pred.h:315: +4, including nesting penalty of 3, nesting level increased to 4

                RETURN_IF_ERROR(get_rhs_colum());
                ^

be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/vec/exprs/vcompound_pred.h:317: +3, including nesting penalty of 2, nesting level increased to 3

                if ((lhs_all_true && !lhs_is_nullable) ||    //not null column
                ^

be/src/vec/exprs/vcompound_pred.h:317: +1

                if ((lhs_all_true && !lhs_is_nullable) ||    //not null column
                                                       ^

be/src/vec/exprs/vcompound_pred.h:317: +1

                if ((lhs_all_true && !lhs_is_nullable) ||    //not null column
                                  ^

be/src/vec/exprs/vcompound_pred.h:318: +1

                    (lhs_all_true && lhs_all_is_not_null)) { //nullable column
                                  ^

be/src/vec/exprs/vcompound_pred.h:321: +1, nesting level increased to 3

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                       ^

be/src/vec/exprs/vcompound_pred.h:321: +1

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                                                               ^

be/src/vec/exprs/vcompound_pred.h:321: +1

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                                          ^

be/src/vec/exprs/vcompound_pred.h:322: +1

                           (rhs_all_false && rhs_all_is_not_null)) {
                                          ^

be/src/vec/exprs/vcompound_pred.h:325: +1, nesting level increased to 3

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                       ^

be/src/vec/exprs/vcompound_pred.h:325: +1

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                                                              ^

be/src/vec/exprs/vcompound_pred.h:325: +1

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                                         ^

be/src/vec/exprs/vcompound_pred.h:326: +1

                           (rhs_all_true && rhs_all_is_not_null)) {
                                         ^

be/src/vec/exprs/vcompound_pred.h:329: +1, nesting level increased to 3

                } else {
                  ^

be/src/vec/exprs/vcompound_pred.h:330: +4, including nesting penalty of 3, nesting level increased to 4

                    if (!result_is_nullable) {
                    ^

be/src/vec/exprs/vcompound_pred.h:332: +5, including nesting penalty of 4, nesting level increased to 5

                        for (size_t i = 0; i < size; i++) {
                        ^

be/src/vec/exprs/vcompound_pred.h:335: +1, nesting level increased to 4

                    } else {
                      ^

be/src/vec/exprs/vcompound_pred.h:340: +1, nesting level increased to 1

        } else if (_op == TExprOpcode::COMPOUND_OR) {
               ^

be/src/vec/exprs/vcompound_pred.h:343: +2, including nesting penalty of 1, nesting level increased to 2

            if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) {
            ^

be/src/vec/exprs/vcompound_pred.h:343: +1

            if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) {
                                                   ^

be/src/vec/exprs/vcompound_pred.h:343: +1

            if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) {
                              ^

be/src/vec/exprs/vcompound_pred.h:343: +1

            if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) {
                                                                    ^

be/src/vec/exprs/vcompound_pred.h:346: +1, nesting level increased to 2

            } else {
              ^

be/src/vec/exprs/vcompound_pred.h:347: +3, including nesting penalty of 2, nesting level increased to 3

                RETURN_IF_ERROR(get_rhs_colum());
                ^

be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/vec/exprs/vcompound_pred.h:347: +4, including nesting penalty of 3, nesting level increased to 4

                RETURN_IF_ERROR(get_rhs_colum());
                ^

be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/vec/exprs/vcompound_pred.h:348: +3, including nesting penalty of 2, nesting level increased to 3

                if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                ^

be/src/vec/exprs/vcompound_pred.h:348: +1

                if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                                        ^

be/src/vec/exprs/vcompound_pred.h:348: +1

                if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                   ^

be/src/vec/exprs/vcompound_pred.h:348: +1

                if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                                                          ^

be/src/vec/exprs/vcompound_pred.h:351: +1, nesting level increased to 3

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                       ^

be/src/vec/exprs/vcompound_pred.h:351: +1

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                                                              ^

be/src/vec/exprs/vcompound_pred.h:351: +1

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                                         ^

be/src/vec/exprs/vcompound_pred.h:352: +1

                           (rhs_all_true && rhs_all_is_not_null)) {
                                         ^

be/src/vec/exprs/vcompound_pred.h:355: +1, nesting level increased to 3

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                       ^

be/src/vec/exprs/vcompound_pred.h:355: +1

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                                                               ^

be/src/vec/exprs/vcompound_pred.h:355: +1

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                                          ^

be/src/vec/exprs/vcompound_pred.h:356: +1

                           (rhs_all_false && rhs_all_is_not_null)) {
                                          ^

be/src/vec/exprs/vcompound_pred.h:359: +1, nesting level increased to 3

                } else {
                  ^

be/src/vec/exprs/vcompound_pred.h:360: +4, including nesting penalty of 3, nesting level increased to 4

                    if (!result_is_nullable) {
                    ^

be/src/vec/exprs/vcompound_pred.h:362: +5, including nesting penalty of 4, nesting level increased to 5

                        for (size_t i = 0; i < size; i++) {
                        ^

be/src/vec/exprs/vcompound_pred.h:365: +1, nesting level increased to 4

                    } else {
                      ^

be/src/vec/exprs/vcompound_pred.h:370: +1, nesting level increased to 1

        } else {
          ^

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/segment_v2/inverted_index_reader.h Outdated Show resolved Hide resolved
be/src/vec/exprs/vcompound_pred.h Show resolved Hide resolved
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/segment_v2/inverted_index_reader.h Outdated Show resolved Hide resolved
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/segment_v2/inverted_index_reader.h Outdated Show resolved Hide resolved
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/exprs/vcompound_pred.h Outdated Show resolved Hide resolved
be/src/vec/exprs/vcompound_pred.h Show resolved Hide resolved
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/exprs/vmulti_match_predicate.cpp Outdated Show resolved Hide resolved
be/src/vec/exprs/vmulti_match_predicate.h Outdated Show resolved Hide resolved
be/src/vec/exprs/vmulti_match_predicate.h Outdated Show resolved Hide resolved
be/src/olap/rowset/segment_v2/segment_iterator.cpp Outdated Show resolved Hide resolved
be/src/olap/rowset/segment_v2/segment_iterator.cpp Outdated Show resolved Hide resolved
be/src/vec/exprs/vexpr.cpp Outdated Show resolved Hide resolved
be/src/vec/exprs/vectorized_fn_call.cpp Outdated Show resolved Hide resolved
be/src/vec/exprs/vectorized_fn_call.cpp Outdated Show resolved Hide resolved
be/src/vec/exprs/vectorized_fn_call.cpp Outdated Show resolved Hide resolved
be/src/vec/exprs/vectorized_fn_call.cpp Outdated Show resolved Hide resolved
be/src/vec/exprs/vcompound_pred.h Outdated Show resolved Hide resolved
be/src/vec/exprs/vcompound_pred.h Show resolved Hide resolved
@airborne12 airborne12 marked this pull request as ready for review August 13, 2024 01:34
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/exprs/vcompound_pred.h Outdated Show resolved Hide resolved
@@ -123,6 +210,16 @@
}

Status execute(VExprContext* context, Block* block, int* result_column_id) override {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'execute' has cognitive complexity of 111 (threshold 50) [readability-function-cognitive-complexity]

    Status execute(VExprContext* context, Block* block, int* result_column_id) override {
           ^
Additional context

be/src/vec/exprs/vcompound_pred.h:212: +1, including nesting penalty of 0, nesting level increased to 1

        if (context->get_inverted_index_result_column().contains(this)) {
        ^

be/src/vec/exprs/vcompound_pred.h:222: +1, including nesting penalty of 0, nesting level increased to 1

        if (children().size() == 1 || !_all_child_is_compound_and_not_const()) {
        ^

be/src/vec/exprs/vcompound_pred.h:228: +1, including nesting penalty of 0, nesting level increased to 1

        RETURN_IF_ERROR(_children[0]->execute(context, block, &lhs_id));
        ^

be/src/common/status.h:626: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/vec/exprs/vcompound_pred.h:228: +2, including nesting penalty of 1, nesting level increased to 2

        RETURN_IF_ERROR(_children[0]->execute(context, block, &lhs_id));
        ^

be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/vec/exprs/vcompound_pred.h:240: +1, including nesting penalty of 0, nesting level increased to 1

        if (lhs_is_nullable) {
        ^

be/src/vec/exprs/vcompound_pred.h:254: nesting level increased to 1

        auto get_rhs_colum = [&]() {
                             ^

be/src/vec/exprs/vcompound_pred.h:255: +2, including nesting penalty of 1, nesting level increased to 2

            if (rhs_id == -1) {
            ^

be/src/vec/exprs/vcompound_pred.h:256: +3, including nesting penalty of 2, nesting level increased to 3

                RETURN_IF_ERROR(_children[1]->execute(context, block, &rhs_id));
                ^

be/src/common/status.h:626: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/vec/exprs/vcompound_pred.h:256: +4, including nesting penalty of 3, nesting level increased to 4

                RETURN_IF_ERROR(_children[1]->execute(context, block, &rhs_id));
                ^

be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/vec/exprs/vcompound_pred.h:266: +3, including nesting penalty of 2, nesting level increased to 3

                if (rhs_is_nullable) {
                ^

be/src/vec/exprs/vcompound_pred.h:274: nesting level increased to 1

        auto return_result_column_id = [&](ColumnPtr res_column, int res_id) -> int {
                                       ^

be/src/vec/exprs/vcompound_pred.h:275: +2, including nesting penalty of 1, nesting level increased to 2

            if (result_is_nullable && !res_column->is_nullable()) {
            ^

be/src/vec/exprs/vcompound_pred.h:275: +1

            if (result_is_nullable && !res_column->is_nullable()) {
                                   ^

be/src/vec/exprs/vcompound_pred.h:284: nesting level increased to 1

        auto create_null_map_column = [&](ColumnPtr& null_map_column,
                                      ^

be/src/vec/exprs/vcompound_pred.h:286: +2, including nesting penalty of 1, nesting level increased to 2

            if (null_map_data == nullptr) {
            ^

be/src/vec/exprs/vcompound_pred.h:295: nesting level increased to 1

        auto vector_vector_null = [&]<bool is_and_op>() {
                                  ^

be/src/vec/exprs/vcompound_pred.h:305: +2, including nesting penalty of 1, nesting level increased to 2

            if constexpr (is_and_op) {
            ^

be/src/vec/exprs/vcompound_pred.h:306: +3, including nesting penalty of 2, nesting level increased to 3

                for (size_t i = 0; i < size; ++i) {
                ^

be/src/vec/exprs/vcompound_pred.h:311: +1, nesting level increased to 2

            } else {
              ^

be/src/vec/exprs/vcompound_pred.h:312: +3, including nesting penalty of 2, nesting level increased to 3

                for (size_t i = 0; i < size; ++i) {
                ^

be/src/vec/exprs/vcompound_pred.h:325: +1, including nesting penalty of 0, nesting level increased to 1

        if (_op == TExprOpcode::COMPOUND_AND) {
        ^

be/src/vec/exprs/vcompound_pred.h:328: +2, including nesting penalty of 1, nesting level increased to 2

            if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
            ^

be/src/vec/exprs/vcompound_pred.h:328: +1

            if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                                    ^

be/src/vec/exprs/vcompound_pred.h:328: +1

            if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                               ^

be/src/vec/exprs/vcompound_pred.h:328: +1

            if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                                                      ^

be/src/vec/exprs/vcompound_pred.h:331: +1, nesting level increased to 2

            } else {
              ^

be/src/vec/exprs/vcompound_pred.h:332: +3, including nesting penalty of 2, nesting level increased to 3

                RETURN_IF_ERROR(get_rhs_colum());
                ^

be/src/common/status.h:626: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/vec/exprs/vcompound_pred.h:332: +4, including nesting penalty of 3, nesting level increased to 4

                RETURN_IF_ERROR(get_rhs_colum());
                ^

be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/vec/exprs/vcompound_pred.h:334: +3, including nesting penalty of 2, nesting level increased to 3

                if ((lhs_all_true && !lhs_is_nullable) ||    //not null column
                ^

be/src/vec/exprs/vcompound_pred.h:334: +1

                if ((lhs_all_true && !lhs_is_nullable) ||    //not null column
                                                       ^

be/src/vec/exprs/vcompound_pred.h:334: +1

                if ((lhs_all_true && !lhs_is_nullable) ||    //not null column
                                  ^

be/src/vec/exprs/vcompound_pred.h:335: +1

                    (lhs_all_true && lhs_all_is_not_null)) { //nullable column
                                  ^

be/src/vec/exprs/vcompound_pred.h:338: +1, nesting level increased to 3

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                       ^

be/src/vec/exprs/vcompound_pred.h:338: +1

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                                                               ^

be/src/vec/exprs/vcompound_pred.h:338: +1

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                                          ^

be/src/vec/exprs/vcompound_pred.h:339: +1

                           (rhs_all_false && rhs_all_is_not_null)) {
                                          ^

be/src/vec/exprs/vcompound_pred.h:342: +1, nesting level increased to 3

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                       ^

be/src/vec/exprs/vcompound_pred.h:342: +1

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                                                              ^

be/src/vec/exprs/vcompound_pred.h:342: +1

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                                         ^

be/src/vec/exprs/vcompound_pred.h:343: +1

                           (rhs_all_true && rhs_all_is_not_null)) {
                                         ^

be/src/vec/exprs/vcompound_pred.h:346: +1, nesting level increased to 3

                } else {
                  ^

be/src/vec/exprs/vcompound_pred.h:347: +4, including nesting penalty of 3, nesting level increased to 4

                    if (!result_is_nullable) {
                    ^

be/src/vec/exprs/vcompound_pred.h:349: +5, including nesting penalty of 4, nesting level increased to 5

                        for (size_t i = 0; i < size; i++) {
                        ^

be/src/vec/exprs/vcompound_pred.h:352: +1, nesting level increased to 4

                    } else {
                      ^

be/src/vec/exprs/vcompound_pred.h:357: +1, nesting level increased to 1

        } else if (_op == TExprOpcode::COMPOUND_OR) {
               ^

be/src/vec/exprs/vcompound_pred.h:360: +2, including nesting penalty of 1, nesting level increased to 2

            if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) {
            ^

be/src/vec/exprs/vcompound_pred.h:360: +1

            if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) {
                                                   ^

be/src/vec/exprs/vcompound_pred.h:360: +1

            if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) {
                              ^

be/src/vec/exprs/vcompound_pred.h:360: +1

            if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) {
                                                                    ^

be/src/vec/exprs/vcompound_pred.h:363: +1, nesting level increased to 2

            } else {
              ^

be/src/vec/exprs/vcompound_pred.h:364: +3, including nesting penalty of 2, nesting level increased to 3

                RETURN_IF_ERROR(get_rhs_colum());
                ^

be/src/common/status.h:626: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/vec/exprs/vcompound_pred.h:364: +4, including nesting penalty of 3, nesting level increased to 4

                RETURN_IF_ERROR(get_rhs_colum());
                ^

be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/vec/exprs/vcompound_pred.h:365: +3, including nesting penalty of 2, nesting level increased to 3

                if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                ^

be/src/vec/exprs/vcompound_pred.h:365: +1

                if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                                        ^

be/src/vec/exprs/vcompound_pred.h:365: +1

                if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                   ^

be/src/vec/exprs/vcompound_pred.h:365: +1

                if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                                                          ^

be/src/vec/exprs/vcompound_pred.h:368: +1, nesting level increased to 3

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                       ^

be/src/vec/exprs/vcompound_pred.h:368: +1

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                                                              ^

be/src/vec/exprs/vcompound_pred.h:368: +1

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                                         ^

be/src/vec/exprs/vcompound_pred.h:369: +1

                           (rhs_all_true && rhs_all_is_not_null)) {
                                         ^

be/src/vec/exprs/vcompound_pred.h:372: +1, nesting level increased to 3

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                       ^

be/src/vec/exprs/vcompound_pred.h:372: +1

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                                                               ^

be/src/vec/exprs/vcompound_pred.h:372: +1

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                                          ^

be/src/vec/exprs/vcompound_pred.h:373: +1

                           (rhs_all_false && rhs_all_is_not_null)) {
                                          ^

be/src/vec/exprs/vcompound_pred.h:376: +1, nesting level increased to 3

                } else {
                  ^

be/src/vec/exprs/vcompound_pred.h:377: +4, including nesting penalty of 3, nesting level increased to 4

                    if (!result_is_nullable) {
                    ^

be/src/vec/exprs/vcompound_pred.h:379: +5, including nesting penalty of 4, nesting level increased to 5

                        for (size_t i = 0; i < size; i++) {
                        ^

be/src/vec/exprs/vcompound_pred.h:382: +1, nesting level increased to 4

                    } else {
                      ^

be/src/vec/exprs/vcompound_pred.h:387: +1, nesting level increased to 1

        } else {
          ^

@airborne12
Copy link
Member Author

run buildall

1 similar comment
@airborne12
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37873 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ff8049b19384fd7dbacfc906f7f381dd2684c895, data reload: false

------ Round 1 ----------------------------------
q1	18169	4493	4358	4358
q2	2548	220	178	178
q3	11525	1189	1110	1110
q4	10242	748	813	748
q5	8177	2837	2764	2764
q6	226	148	136	136
q7	983	614	601	601
q8	9540	2019	2050	2019
q9	7447	6528	6526	6526
q10	7053	2201	2169	2169
q11	445	238	239	238
q12	390	224	224	224
q13	18890	3033	3041	3033
q14	278	235	242	235
q15	516	478	489	478
q16	513	390	380	380
q17	992	646	680	646
q18	7523	6819	6825	6819
q19	5523	1039	1172	1039
q20	697	312	343	312
q21	3868	2941	2877	2877
q22	1102	983	997	983
Total cold run time: 116647 ms
Total hot run time: 37873 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4550	4265	4270	4265
q2	380	272	276	272
q3	2871	2625	2547	2547
q4	1900	1622	1578	1578
q5	5369	5393	5415	5393
q6	219	132	131	131
q7	2052	1680	1687	1680
q8	3175	3346	3325	3325
q9	8429	8316	8397	8316
q10	3338	3107	3096	3096
q11	614	483	499	483
q12	767	590	596	590
q13	16268	3042	3047	3042
q14	304	288	270	270
q15	528	461	478	461
q16	457	420	415	415
q17	1797	1488	1513	1488
q18	7531	7493	7533	7493
q19	1671	1450	1512	1450
q20	2010	1847	1806	1806
q21	9100	4987	5141	4987
q22	1110	1001	1013	1001
Total cold run time: 74440 ms
Total hot run time: 54089 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183921 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ff8049b19384fd7dbacfc906f7f381dd2684c895, data reload: false

query1	907	381	369	369
query2	6451	1883	1937	1883
query3	6651	213	221	213
query4	31032	23397	23283	23283
query5	4218	486	487	486
query6	285	188	163	163
query7	4594	289	294	289
query8	260	208	195	195
query9	8563	2465	2435	2435
query10	435	252	259	252
query11	17209	15080	14853	14853
query12	152	96	101	96
query13	1629	360	366	360
query14	9861	6476	6967	6476
query15	235	168	179	168
query16	7710	458	489	458
query17	1503	550	541	541
query18	1811	283	282	282
query19	200	152	149	149
query20	112	110	110	110
query21	212	101	100	100
query22	4228	3983	3865	3865
query23	33817	33276	33179	33179
query24	12095	2814	2838	2814
query25	698	399	399	399
query26	1781	153	156	153
query27	2871	264	270	264
query28	7629	2009	2007	2007
query29	1121	412	412	412
query30	307	150	160	150
query31	958	753	756	753
query32	99	56	57	56
query33	755	288	283	283
query34	967	449	459	449
query35	831	729	727	727
query36	1078	927	912	912
query37	188	86	85	85
query38	3982	3903	3790	3790
query39	1432	1393	1382	1382
query40	275	114	115	114
query41	49	46	46	46
query42	114	96	103	96
query43	530	471	482	471
query44	1252	744	730	730
query45	193	167	169	167
query46	1131	738	696	696
query47	1857	1756	1765	1756
query48	369	286	289	286
query49	1217	444	434	434
query50	801	407	410	407
query51	6795	6742	6700	6700
query52	102	90	89	89
query53	255	185	186	185
query54	961	451	442	442
query55	82	76	79	76
query56	270	257	264	257
query57	1140	1090	1101	1090
query58	254	221	231	221
query59	2958	2739	2795	2739
query60	297	270	267	267
query61	121	113	224	113
query62	815	646	663	646
query63	226	189	179	179
query64	6300	2241	1709	1709
query65	3203	3123	3177	3123
query66	1377	332	348	332
query67	15685	14706	14767	14706
query68	6425	547	540	540
query69	687	392	283	283
query70	1099	1051	1064	1051
query71	541	291	266	266
query72	7546	2294	2024	2024
query73	815	313	342	313
query74	8895	8809	8798	8798
query75	4954	2679	2610	2610
query76	4617	975	924	924
query77	789	313	302	302
query78	9734	9098	9015	9015
query79	8664	525	522	522
query80	1020	475	478	475
query81	582	222	224	222
query82	804	133	130	130
query83	319	145	145	145
query84	265	77	72	72
query85	1353	273	259	259
query86	438	293	289	289
query87	4509	4127	4254	4127
query88	4922	2300	2228	2228
query89	517	290	282	282
query90	2007	186	184	184
query91	123	94	123	94
query92	65	47	50	47
query93	7080	541	540	540
query94	946	278	276	276
query95	346	259	259	259
query96	619	260	260	260
query97	3194	3070	3034	3034
query98	220	197	199	197
query99	1519	1235	1257	1235
Total cold run time: 313110 ms
Total hot run time: 183921 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.23 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ff8049b19384fd7dbacfc906f7f381dd2684c895, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.07
query5	0.49	0.48	0.50
query6	1.13	0.73	0.73
query7	0.03	0.01	0.01
query8	0.06	0.05	0.05
query9	0.54	0.49	0.46
query10	0.55	0.55	0.53
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.60	0.58
query14	0.78	0.76	0.77
query15	0.88	0.82	0.83
query16	0.38	0.37	0.37
query17	0.96	0.99	1.01
query18	0.23	0.22	0.21
query19	1.92	1.68	1.67
query20	0.01	0.01	0.01
query21	15.39	0.73	0.64
query22	5.12	5.59	2.53
query23	18.30	1.49	1.30
query24	2.11	0.21	0.23
query25	0.16	0.08	0.07
query26	0.29	0.21	0.21
query27	0.45	0.24	0.22
query28	13.31	1.01	1.01
query29	12.57	3.28	3.28
query30	0.24	0.05	0.05
query31	2.88	0.39	0.38
query32	3.28	0.49	0.47
query33	2.94	2.94	2.97
query34	17.22	4.35	4.41
query35	4.41	4.44	4.43
query36	0.66	0.46	0.49
query37	0.19	0.15	0.14
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.12	0.12
query41	0.09	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.96 s
Total hot run time: 31.23 s

@airborne12
Copy link
Member Author

run buildall

@airborne12
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39704 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e0e384b6d69e7782d5276be9e0c0032761da6517, data reload: false

------ Round 1 ----------------------------------
q1	17622	4826	4744	4744
q2	2015	184	178	178
q3	11660	1126	1170	1126
q4	10508	889	754	754
q5	7787	2972	2988	2972
q6	250	144	140	140
q7	1051	614	621	614
q8	9531	2134	2177	2134
q9	9282	6940	6923	6923
q10	7067	2249	2261	2249
q11	463	249	248	248
q12	445	241	234	234
q13	18703	3079	3064	3064
q14	300	244	242	242
q15	534	498	492	492
q16	524	409	394	394
q17	1018	688	686	686
q18	7485	6890	6958	6890
q19	6390	1215	1137	1137
q20	714	351	331	331
q21	4197	3428	3147	3147
q22	1154	1050	1005	1005
Total cold run time: 118700 ms
Total hot run time: 39704 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4995	4790	4752	4752
q2	403	278	281	278
q3	3184	2910	2955	2910
q4	2155	1894	1851	1851
q5	5840	5738	5823	5738
q6	255	143	149	143
q7	2293	1775	1763	1763
q8	3555	3748	3701	3701
q9	8845	9007	8876	8876
q10	3731	3407	3486	3407
q11	646	521	513	513
q12	860	654	644	644
q13	17241	3273	3283	3273
q14	327	292	305	292
q15	578	517	504	504
q16	508	455	453	453
q17	1983	1591	1581	1581
q18	8235	8021	7978	7978
q19	1890	1655	1796	1655
q20	2255	1890	1919	1890
q21	5895	5427	5631	5427
q22	1176	1057	1038	1038
Total cold run time: 76850 ms
Total hot run time: 58667 ms

@airborne12
Copy link
Member Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/exprs/vcompound_pred.h Outdated Show resolved Hide resolved
}

if (all_pass && !res.is_empty()) {
context->set_inverted_index_result_for_expr(this, res);
}
return Status::OK();
}

Status execute(VExprContext* context, Block* block, int* result_column_id) override {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'execute' has cognitive complexity of 111 (threshold 50) [readability-function-cognitive-complexity]

    Status execute(VExprContext* context, Block* block, int* result_column_id) override {
           ^
Additional context

be/src/vec/exprs/vcompound_pred.h:141: +1, including nesting penalty of 0, nesting level increased to 1

        if (fast_execute(context, block, result_column_id)) {
        ^

be/src/vec/exprs/vcompound_pred.h:144: +1, including nesting penalty of 0, nesting level increased to 1

        if (children().size() == 1 || !_all_child_is_compound_and_not_const()) {
        ^

be/src/vec/exprs/vcompound_pred.h:150: +1, including nesting penalty of 0, nesting level increased to 1

        RETURN_IF_ERROR(_children[0]->execute(context, block, &lhs_id));
        ^

be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/vec/exprs/vcompound_pred.h:150: +2, including nesting penalty of 1, nesting level increased to 2

        RETURN_IF_ERROR(_children[0]->execute(context, block, &lhs_id));
        ^

be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/vec/exprs/vcompound_pred.h:162: +1, including nesting penalty of 0, nesting level increased to 1

        if (lhs_is_nullable) {
        ^

be/src/vec/exprs/vcompound_pred.h:176: nesting level increased to 1

        auto get_rhs_colum = [&]() {
                             ^

be/src/vec/exprs/vcompound_pred.h:177: +2, including nesting penalty of 1, nesting level increased to 2

            if (rhs_id == -1) {
            ^

be/src/vec/exprs/vcompound_pred.h:178: +3, including nesting penalty of 2, nesting level increased to 3

                RETURN_IF_ERROR(_children[1]->execute(context, block, &rhs_id));
                ^

be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/vec/exprs/vcompound_pred.h:178: +4, including nesting penalty of 3, nesting level increased to 4

                RETURN_IF_ERROR(_children[1]->execute(context, block, &rhs_id));
                ^

be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/vec/exprs/vcompound_pred.h:188: +3, including nesting penalty of 2, nesting level increased to 3

                if (rhs_is_nullable) {
                ^

be/src/vec/exprs/vcompound_pred.h:196: nesting level increased to 1

        auto return_result_column_id = [&](ColumnPtr res_column, int res_id) -> int {
                                       ^

be/src/vec/exprs/vcompound_pred.h:197: +2, including nesting penalty of 1, nesting level increased to 2

            if (result_is_nullable && !res_column->is_nullable()) {
            ^

be/src/vec/exprs/vcompound_pred.h:197: +1

            if (result_is_nullable && !res_column->is_nullable()) {
                                   ^

be/src/vec/exprs/vcompound_pred.h:206: nesting level increased to 1

        auto create_null_map_column = [&](ColumnPtr& null_map_column,
                                      ^

be/src/vec/exprs/vcompound_pred.h:208: +2, including nesting penalty of 1, nesting level increased to 2

            if (null_map_data == nullptr) {
            ^

be/src/vec/exprs/vcompound_pred.h:217: nesting level increased to 1

        auto vector_vector_null = [&]<bool is_and_op>() {
                                  ^

be/src/vec/exprs/vcompound_pred.h:227: +2, including nesting penalty of 1, nesting level increased to 2

            if constexpr (is_and_op) {
            ^

be/src/vec/exprs/vcompound_pred.h:228: +3, including nesting penalty of 2, nesting level increased to 3

                for (size_t i = 0; i < size; ++i) {
                ^

be/src/vec/exprs/vcompound_pred.h:233: +1, nesting level increased to 2

            } else {
              ^

be/src/vec/exprs/vcompound_pred.h:234: +3, including nesting penalty of 2, nesting level increased to 3

                for (size_t i = 0; i < size; ++i) {
                ^

be/src/vec/exprs/vcompound_pred.h:247: +1, including nesting penalty of 0, nesting level increased to 1

        if (_op == TExprOpcode::COMPOUND_AND) {
        ^

be/src/vec/exprs/vcompound_pred.h:250: +2, including nesting penalty of 1, nesting level increased to 2

            if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
            ^

be/src/vec/exprs/vcompound_pred.h:250: +1

            if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                                    ^

be/src/vec/exprs/vcompound_pred.h:250: +1

            if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                               ^

be/src/vec/exprs/vcompound_pred.h:250: +1

            if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                                                      ^

be/src/vec/exprs/vcompound_pred.h:253: +1, nesting level increased to 2

            } else {
              ^

be/src/vec/exprs/vcompound_pred.h:254: +3, including nesting penalty of 2, nesting level increased to 3

                RETURN_IF_ERROR(get_rhs_colum());
                ^

be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/vec/exprs/vcompound_pred.h:254: +4, including nesting penalty of 3, nesting level increased to 4

                RETURN_IF_ERROR(get_rhs_colum());
                ^

be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/vec/exprs/vcompound_pred.h:256: +3, including nesting penalty of 2, nesting level increased to 3

                if ((lhs_all_true && !lhs_is_nullable) ||    //not null column
                ^

be/src/vec/exprs/vcompound_pred.h:256: +1

                if ((lhs_all_true && !lhs_is_nullable) ||    //not null column
                                                       ^

be/src/vec/exprs/vcompound_pred.h:256: +1

                if ((lhs_all_true && !lhs_is_nullable) ||    //not null column
                                  ^

be/src/vec/exprs/vcompound_pred.h:257: +1

                    (lhs_all_true && lhs_all_is_not_null)) { //nullable column
                                  ^

be/src/vec/exprs/vcompound_pred.h:260: +1, nesting level increased to 3

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                       ^

be/src/vec/exprs/vcompound_pred.h:260: +1

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                                                               ^

be/src/vec/exprs/vcompound_pred.h:260: +1

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                                          ^

be/src/vec/exprs/vcompound_pred.h:261: +1

                           (rhs_all_false && rhs_all_is_not_null)) {
                                          ^

be/src/vec/exprs/vcompound_pred.h:264: +1, nesting level increased to 3

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                       ^

be/src/vec/exprs/vcompound_pred.h:264: +1

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                                                              ^

be/src/vec/exprs/vcompound_pred.h:264: +1

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                                         ^

be/src/vec/exprs/vcompound_pred.h:265: +1

                           (rhs_all_true && rhs_all_is_not_null)) {
                                         ^

be/src/vec/exprs/vcompound_pred.h:268: +1, nesting level increased to 3

                } else {
                  ^

be/src/vec/exprs/vcompound_pred.h:269: +4, including nesting penalty of 3, nesting level increased to 4

                    if (!result_is_nullable) {
                    ^

be/src/vec/exprs/vcompound_pred.h:271: +5, including nesting penalty of 4, nesting level increased to 5

                        for (size_t i = 0; i < size; i++) {
                        ^

be/src/vec/exprs/vcompound_pred.h:274: +1, nesting level increased to 4

                    } else {
                      ^

be/src/vec/exprs/vcompound_pred.h:279: +1, nesting level increased to 1

        } else if (_op == TExprOpcode::COMPOUND_OR) {
               ^

be/src/vec/exprs/vcompound_pred.h:282: +2, including nesting penalty of 1, nesting level increased to 2

            if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) {
            ^

be/src/vec/exprs/vcompound_pred.h:282: +1

            if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) {
                                                   ^

be/src/vec/exprs/vcompound_pred.h:282: +1

            if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) {
                              ^

be/src/vec/exprs/vcompound_pred.h:282: +1

            if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) {
                                                                    ^

be/src/vec/exprs/vcompound_pred.h:285: +1, nesting level increased to 2

            } else {
              ^

be/src/vec/exprs/vcompound_pred.h:286: +3, including nesting penalty of 2, nesting level increased to 3

                RETURN_IF_ERROR(get_rhs_colum());
                ^

be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/vec/exprs/vcompound_pred.h:286: +4, including nesting penalty of 3, nesting level increased to 4

                RETURN_IF_ERROR(get_rhs_colum());
                ^

be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/vec/exprs/vcompound_pred.h:287: +3, including nesting penalty of 2, nesting level increased to 3

                if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                ^

be/src/vec/exprs/vcompound_pred.h:287: +1

                if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                                        ^

be/src/vec/exprs/vcompound_pred.h:287: +1

                if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                   ^

be/src/vec/exprs/vcompound_pred.h:287: +1

                if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) {
                                                                          ^

be/src/vec/exprs/vcompound_pred.h:290: +1, nesting level increased to 3

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                       ^

be/src/vec/exprs/vcompound_pred.h:290: +1

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                                                              ^

be/src/vec/exprs/vcompound_pred.h:290: +1

                } else if ((rhs_all_true && !rhs_is_nullable) ||
                                         ^

be/src/vec/exprs/vcompound_pred.h:291: +1

                           (rhs_all_true && rhs_all_is_not_null)) {
                                         ^

be/src/vec/exprs/vcompound_pred.h:294: +1, nesting level increased to 3

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                       ^

be/src/vec/exprs/vcompound_pred.h:294: +1

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                                                               ^

be/src/vec/exprs/vcompound_pred.h:294: +1

                } else if ((rhs_all_false && !rhs_is_nullable) ||
                                          ^

be/src/vec/exprs/vcompound_pred.h:295: +1

                           (rhs_all_false && rhs_all_is_not_null)) {
                                          ^

be/src/vec/exprs/vcompound_pred.h:298: +1, nesting level increased to 3

                } else {
                  ^

be/src/vec/exprs/vcompound_pred.h:299: +4, including nesting penalty of 3, nesting level increased to 4

                    if (!result_is_nullable) {
                    ^

be/src/vec/exprs/vcompound_pred.h:301: +5, including nesting penalty of 4, nesting level increased to 5

                        for (size_t i = 0; i < size; i++) {
                        ^

be/src/vec/exprs/vcompound_pred.h:304: +1, nesting level increased to 4

                    } else {
                      ^

be/src/vec/exprs/vcompound_pred.h:309: +1, nesting level increased to 1

        } else {
          ^

}
VLOG_DEBUG << "begin to execute match directly, column_name=" << column_name
<< ", match_query_str=" << match_query_str;
InvertedIndexCtx* inverted_index_ctx = reinterpret_cast<InvertedIndexCtx*>(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: use auto when initializing with a cast to avoid duplicating the type name [modernize-use-auto]

Suggested change
InvertedIndexCtx* inverted_index_ctx = reinterpret_cast<InvertedIndexCtx*>(
auto* inverted_index_ctx = reinterpret_cast<InvertedIndexCtx*>(

cloud/src/recycler/hdfs_accessor.h Outdated Show resolved Hide resolved
@doris-robot
Copy link

TPC-H: Total hot run time: 38097 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2ad037ca7405542efb69fe491ffeab6b11b11305, data reload: false

------ Round 1 ----------------------------------
q1	17616	4919	4292	4292
q2	2027	180	190	180
q3	10464	1156	1155	1155
q4	10145	735	782	735
q5	7756	2834	2825	2825
q6	228	146	150	146
q7	987	600	602	600
q8	9359	2038	2063	2038
q9	7279	6578	6565	6565
q10	7071	2246	2262	2246
q11	468	252	243	243
q12	508	228	232	228
q13	18315	3044	3071	3044
q14	273	233	245	233
q15	530	477	481	477
q16	543	402	389	389
q17	985	636	698	636
q18	7551	6754	6891	6754
q19	1385	1044	1118	1044
q20	666	338	342	338
q21	3882	2918	2975	2918
q22	1123	1022	1011	1011
Total cold run time: 109161 ms
Total hot run time: 38097 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4308	4241	4266	4241
q2	379	273	277	273
q3	2912	2696	2661	2661
q4	1916	1669	1585	1585
q5	5391	5362	5389	5362
q6	224	133	141	133
q7	2089	1683	1667	1667
q8	3175	3358	3378	3358
q9	8500	8359	8406	8359
q10	3402	3183	3164	3164
q11	598	499	484	484
q12	791	615	622	615
q13	16947	3104	3067	3067
q14	305	275	282	275
q15	520	469	483	469
q16	480	427	420	420
q17	1792	1516	1501	1501
q18	7724	7591	7454	7454
q19	1670	1627	1643	1627
q20	2004	1813	1785	1785
q21	5414	5076	5179	5076
q22	1114	1000	1033	1000
Total cold run time: 71655 ms
Total hot run time: 54576 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184220 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2ad037ca7405542efb69fe491ffeab6b11b11305, data reload: false

query1	908	383	397	383
query2	6464	1892	1821	1821
query3	6654	221	230	221
query4	34822	23371	23338	23338
query5	4186	485	516	485
query6	286	174	166	166
query7	4594	294	284	284
query8	249	202	208	202
query9	8595	2455	2428	2428
query10	433	272	259	259
query11	16692	15117	14971	14971
query12	147	101	103	101
query13	1643	367	368	367
query14	9725	6454	6576	6454
query15	217	165	164	164
query16	7581	503	465	465
query17	1154	542	522	522
query18	1952	289	271	271
query19	196	138	137	137
query20	115	105	106	105
query21	204	103	100	100
query22	4333	3996	3890	3890
query23	34124	33229	33360	33229
query24	11952	2868	2882	2868
query25	634	369	358	358
query26	1226	148	150	148
query27	2482	269	268	268
query28	7368	2010	2000	2000
query29	910	398	402	398
query30	296	148	153	148
query31	974	720	751	720
query32	95	51	53	51
query33	746	301	279	279
query34	945	450	465	450
query35	855	738	729	729
query36	1098	933	914	914
query37	146	79	80	79
query38	4008	3748	3870	3748
query39	1431	1375	1369	1369
query40	201	111	114	111
query41	45	43	44	43
query42	117	98	100	98
query43	484	466	455	455
query44	1285	734	737	734
query45	195	167	165	165
query46	1119	733	739	733
query47	1865	1781	1767	1767
query48	351	286	285	285
query49	1071	414	413	413
query50	799	404	409	404
query51	6760	6712	6738	6712
query52	105	89	93	89
query53	253	182	181	181
query54	918	448	436	436
query55	79	76	82	76
query56	289	245	245	245
query57	1131	1040	1080	1040
query58	228	225	233	225
query59	2745	2677	2739	2677
query60	299	266	266	266
query61	98	95	96	95
query62	830	641	630	630
query63	215	182	188	182
query64	5265	2284	1729	1729
query65	3202	3129	3111	3111
query66	833	340	330	330
query67	15499	14903	14819	14819
query68	4550	537	545	537
query69	411	266	265	265
query70	1154	1135	1095	1095
query71	379	277	268	268
query72	6525	2256	1901	1901
query73	760	313	315	313
query74	9286	8747	8786	8747
query75	3465	2668	2717	2668
query76	2276	989	1008	989
query77	515	320	314	314
query78	10481	9184	10809	9184
query79	2101	532	533	532
query80	2067	494	497	494
query81	580	225	224	224
query82	305	136	136	136
query83	274	150	146	146
query84	269	74	74	74
query85	1474	268	266	266
query86	496	312	294	294
query87	4418	4320	4208	4208
query88	2867	2259	2262	2259
query89	374	286	281	281
query90	1782	188	184	184
query91	120	93	92	92
query92	64	50	48	48
query93	1048	534	532	532
query94	916	264	294	264
query95	356	253	253	253
query96	584	266	261	261
query97	3248	3031	3017	3017
query98	218	204	201	201
query99	1559	1270	1232	1232
Total cold run time: 291643 ms
Total hot run time: 184220 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.64 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2ad037ca7405542efb69fe491ffeab6b11b11305, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.04
query4	1.68	0.08	0.07
query5	0.48	0.49	0.49
query6	1.15	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.56	0.48	0.49
query10	0.55	0.55	0.54
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.58
query14	0.76	0.78	0.77
query15	0.83	0.81	0.81
query16	0.37	0.37	0.36
query17	1.04	1.05	0.96
query18	0.23	0.22	0.23
query19	1.87	1.76	1.71
query20	0.01	0.01	0.01
query21	15.38	0.75	0.66
query22	4.41	7.38	1.80
query23	18.31	1.43	1.23
query24	2.09	0.21	0.22
query25	0.16	0.08	0.09
query26	0.32	0.21	0.21
query27	0.46	0.23	0.23
query28	13.31	1.02	1.00
query29	12.62	3.40	3.38
query30	0.23	0.05	0.04
query31	2.89	0.40	0.38
query32	3.28	0.48	0.48
query33	2.95	3.00	2.96
query34	17.13	4.39	4.33
query35	4.39	4.46	4.53
query36	0.66	0.48	0.46
query37	0.19	0.16	0.16
query38	0.15	0.15	0.15
query39	0.04	0.03	0.03
query40	0.17	0.12	0.13
query41	0.09	0.05	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.16 s
Total hot run time: 30.64 s

Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@airborne12 airborne12 deleted the refactor-compound branch September 2, 2024 02:16
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

airborne12 added a commit that referenced this pull request Sep 3, 2024
…factor (#40324)

## Proposed changes
Make profile metrics right after #38908
airborne12 added a commit that referenced this pull request Sep 4, 2024
…nd (#40338)

## Proposed changes
Untokenized strings exceed ignore_above, they are written as null,
causing range query errors.
Introduced by #38908
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Sep 10, 2024
…s evaluate logic (apache#38908)

This PR addresses several key issues related to the compound condition
support in the inverted index, and optimization for index skipping
without returning to the table:

1. **Unified Handling of `expr` and `column predicate`**:
- Combined the processing of inverted index-related `column predicate`
and `expr`.
- Ensured that compound conditions involving both `column predicate` and
`expr` are processed uniformly to reduce complexity and improve
robustness.

2. **Optimized the Execution of Compound Conditions**:
- Removed the logic in `scan_operator` that normalized compound
predicates by pushing down logic to `_common_expr_ctxs_push_down` where
`expr` contexts are managed.
- Added `evaluate_inverted_index` support to the `vexpr` and function
layers, such as `function comparison` and `function collection_in`.
- Introduced new data structures in `VExprContext` to store results from
`evaluate_inverted_index`, thus facilitating quick lookup and
application of these results during execution.
airborne12 added a commit that referenced this pull request Sep 10, 2024
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Sep 10, 2024
…factor (apache#40324)

## Proposed changes
Make profile metrics right after apache#38908
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Sep 10, 2024
…nd (apache#40338)

Untokenized strings exceed ignore_above, they are written as null,
causing range query errors.
Introduced by apache#38908
xiaokang pushed a commit that referenced this pull request Sep 12, 2024
…_expr_pushdown is disabled (#40689)

If common_expr_pushdown is disabled, we cannot guarantee that all
conditions are processed by the inverted index after refactor of #38908
Consider a scenario where there is a column predicate and an expression
involving the same column in the SQL query,
such as 'a < 0' and 'abs(a) > 1'. 
This could potentially lead to errors.
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Sep 12, 2024
…_expr_pushdown is disabled (apache#40689)

If common_expr_pushdown is disabled, we cannot guarantee that all
conditions are processed by the inverted index after refactor of apache#38908
Consider a scenario where there is a column predicate and an expression
involving the same column in the SQL query,
such as 'a < 0' and 'abs(a) > 1'. 
This could potentially lead to errors.
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Sep 23, 2024
…_expr_pushdown is disabled (apache#40689)

If common_expr_pushdown is disabled, we cannot guarantee that all
conditions are processed by the inverted index after refactor of apache#38908
Consider a scenario where there is a column predicate and an expression
involving the same column in the SQL query,
such as 'a < 0' and 'abs(a) > 1'.
This could potentially lead to errors.
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Sep 27, 2024
…s evaluate logic (apache#38908)

This PR addresses several key issues related to the compound condition
support in the inverted index, and optimization for index skipping
without returning to the table:

1. **Unified Handling of `expr` and `column predicate`**:
- Combined the processing of inverted index-related `column predicate`
and `expr`.
- Ensured that compound conditions involving both `column predicate` and
`expr` are processed uniformly to reduce complexity and improve
robustness.

2. **Optimized the Execution of Compound Conditions**:
- Removed the logic in `scan_operator` that normalized compound
predicates by pushing down logic to `_common_expr_ctxs_push_down` where
`expr` contexts are managed.
- Added `evaluate_inverted_index` support to the `vexpr` and function
layers, such as `function comparison` and `function collection_in`.
- Introduced new data structures in `VExprContext` to store results from
`evaluate_inverted_index`, thus facilitating quick lookup and
application of these results during execution.
airborne12 added a commit that referenced this pull request Sep 29, 2024
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Sep 29, 2024
…factor (apache#40324)

## Proposed changes
Make profile metrics right after apache#38908
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Sep 29, 2024
…nd (apache#40338)

Untokenized strings exceed ignore_above, they are written as null,
causing range query errors.
Introduced by apache#38908
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Sep 29, 2024
…nd (apache#40338)

Untokenized strings exceed ignore_above, they are written as null,
causing range query errors.
Introduced by apache#38908
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Oct 8, 2024
…_expr_pushdown is disabled (apache#40689)

If common_expr_pushdown is disabled, we cannot guarantee that all
conditions are processed by the inverted index after refactor of apache#38908
Consider a scenario where there is a column predicate and an expression
involving the same column in the SQL query,
such as 'a < 0' and 'abs(a) > 1'.
This could potentially lead to errors.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.2-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants