Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SL-1627] Add full support for predicate pushdown, including partition pruning #1010

Closed
3 of 5 tasks
tlento opened this issue Jan 29, 2024 · 0 comments
Closed
3 of 5 tasks
Assignees
Labels
Medium priority Created by Linear-GitHub Sync Metricflow Created by Linear-GitHub Sync

Comments

@tlento
Copy link
Contributor

tlento commented Jan 29, 2024

This is a large-scoped issue for all filter pushdown optimization requirements.

At a high level, we want to get a place where end users can specify filters in a way that is both natural and allows for easy hinting that filter predicates should be pushed down as far as they can go. This requires three things on the input interface side:

  1. Allow users to split up predicates by semantic model in some natural way. This is done, actually, but we don't make use of it today. Users can, if they wish, provide a list of predicates which we will process as an intersection of filters (i.e., we will link them all with AND). For example, ["metric_time = '2021-01-01'", "user__country = 'CA' OR user__region = 'EMEA'"] is the same as metric_time = '2021-01-01' AND (user__country = 'CA' OR user__region = 'EMEA', but the former can have each statement pushed down to the constituent semantic models while the latter cannot (why not is a separate discussion, short version is we don't currently have the ability to do that level of semantic analysis).
  2. Allow users to construct filters allowing us to properly apply range adjustments to time filters on metrics requiring expanded windows. For example, a cumulative metric with a 7-day aggregation window must include the week preceding the first date in the filter to return correct results for every requested output row.
  3. Provide a way for users to ensure filters on partition columns are pushed down in such a manner that the underlying engine will trigger pruning.

Between our existing input interfaces and the addition of more robust partition specification options in the semantic manifest we should be able to provide all of these things. Indeed, we can deliver partial support for all of these scenarios without added updates to our input specifications. The sub-tasks listed here cover the major component pieces, and will be roughly addressed in the order in which they are created.

SL-1627

Tasks

  1. Metricflow
  2. Medium priority Metricflow
    tlento
  3. Low priority Metricflow
  4. Medium priority Metricflow
    tlento
  5. High priority Metricflow bug
    courtneyholcomb
@tlento tlento changed the title Add full support for predicate pushdown, including partition pruning [SL-1627] Add full support for predicate pushdown, including partition pruning Jan 29, 2024
@tlento tlento self-assigned this Jan 29, 2024
@tlento tlento added Metricflow Created by Linear-GitHub Sync Medium priority Created by Linear-GitHub Sync Metricflow Gap Created by Linear-GitHub Sync labels Jan 30, 2024
@tlento tlento added Metricflow Created by Linear-GitHub Sync and removed Metricflow Created by Linear-GitHub Sync Metricflow Gap Created by Linear-GitHub Sync labels Feb 8, 2024
@tlento tlento closed this as not planned Won't fix, can't repro, duplicate, stale Aug 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Medium priority Created by Linear-GitHub Sync Metricflow Created by Linear-GitHub Sync
Projects
None yet
Development

No branches or pull requests

1 participant