-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feature](nereids)support correlated scalar subquery without scalar agg #39471
Conversation
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
run buildall |
TPC-H: Total hot run time: 38175 ms
|
ae4c653
to
5805433
Compare
run buildall |
TPC-H: Total hot run time: 37897 ms
|
TPC-DS: Total hot run time: 195603 ms
|
ClickBench: Total hot run time: 31.51 s
|
run buildall |
TPC-H: Total hot run time: 38022 ms
|
TPC-DS: Total hot run time: 196557 ms
|
ClickBench: Total hot run time: 31.69 s
|
if (child instanceof LogicalProject) { | ||
// keep NoneMovableFunction for later use | ||
for (NamedExpression output : ((LogicalProject<?>) child).getOutputs()) { | ||
if (output.containsType(NoneMovableFunction.class)) { | ||
childRequiredSlotBuilder.add(output.toSlot()); | ||
} | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think u should change this interface org.apache.doris.nereids.trees.plans.logical.LogicalProject#pruneOutputs
// unnest correlated scalar subquery may add count(*) and any_value() to project list | ||
// then there may be more than one expr, so we add all project exprs here |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we do more check here?
public ScalarSubquery(LogicalPlan subquery) { | ||
super(Objects.requireNonNull(subquery, "subquery can not be null")); | ||
this(Objects.requireNonNull(subquery, "subquery can not be null"), ImmutableList.of()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this(Objects.requireNonNull(subquery, "subquery can not be null"), ImmutableList.of()); | |
this(subquery, ImmutableList.of()); |
} else { | ||
return false; | ||
} | ||
} else if (plan instanceof LogicalJoin) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why only join return false directly? could u add some comment to explain it? what about other operator, such as sort, limit, window or generate?
// check if the query has top level scalar agg | ||
// if the correlated subquery doesn't have top level scalar agg | ||
// we need create one in subquery unnesting step | ||
private static boolean findTopLevelScalarAgg(Plan plan) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add fe ut for this static function
Map<SubqueryExpr, Optional<MarkJoinSlotReference>> subqueryToMarkJoinSlot, | ||
CascadesContext ctx, Optional<Expression> conjunct, | ||
boolean isProject, boolean singleSubquery, boolean isMarkJoinSlotNotNull) { | ||
private Pair<LogicalPlan, Optional<Expression>> addApply(SubqueryExpr subquery, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add fe ut for this function
// left child | ||
projects.addAll(childPlan.getOutput()); | ||
// markJoinSlotReference | ||
projects.addAll(markJoinSlot.isPresent() ? ImmutableList.of(markJoinSlot.get()) : ImmutableList.of()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
projects.addAll(markJoinSlot.isPresent() ? ImmutableList.of(markJoinSlot.get()) : ImmutableList.of()); | |
markJoinSlot.map(projects::add); |
.build(); | ||
|
||
return new LogicalProject(projects, newApply); | ||
ImmutableList.Builder<NamedExpression> projects = ImmutableList.builder(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ImmutableList.Builder<NamedExpression> projects = ImmutableList.builder(); | |
ImmutableList.Builder<NamedExpression> projects = ImmutableList.builderWithExpectedSize(childPlan.getOutput().size() + 3); |
boolean needAddScalarSubqueryOutputToProjects = isConjunctContainsScalarSubqueryOutput( | ||
subquery, conjunct, isProject, singleSubquery); | ||
boolean useNewSubquery = false; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
boolean useNewSubquery = false; | |
boolean needRuntimeAssertCount = false; |
LogicalAggregate<Plan> aggregate = new LogicalAggregate<>(ImmutableList.of(), | ||
ImmutableList.of(countAlias, anyValueAlias), subquery.getQueryPlan()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
group by will be add later?
5805433
to
8aa3e0c
Compare
run buildall |
TPC-H: Total hot run time: 37919 ms
|
TPC-DS: Total hot run time: 192664 ms
|
ClickBench: Total hot run time: 32.23 s
|
3ceda99
to
fdc19c1
Compare
run buildall |
TPC-H: Total hot run time: 38117 ms
|
TPC-DS: Total hot run time: 187253 ms
|
ClickBench: Total hot run time: 31.8 s
|
run buildall |
TPC-H: Total hot run time: 37897 ms
|
TPC-DS: Total hot run time: 192558 ms
|
ClickBench: Total hot run time: 32.06 s
|
6c77532
to
21ea260
Compare
ClickBench: Total hot run time: 31.89 s
|
ccdf83c
to
acac3c5
Compare
run buildall |
TPC-H: Total hot run time: 42691 ms
|
TPC-DS: Total hot run time: 199000 ms
|
ClickBench: Total hot run time: 30.87 s
|
acac3c5
to
a7dbcbc
Compare
run buildall |
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
TPC-H: Total hot run time: 41344 ms
|
TPC-DS: Total hot run time: 195561 ms
|
ClickBench: Total hot run time: 32.15 s
|
support correlated scalar subquery without scalar agg like:
select t1.c1 from t1 where t1.c2 > (select t2.c2 from t2 where t1.c1 = t2.c1);
after this pr, nereids would produce a correct plan for above sql.