RFC: NGql Filter pushed down to the storaged #4787
Replies: 3 comments 4 replies
-
Thanks for posting this RFC. The general idea looks good to me. Link a detailed plan here: https://github.com/zhaojunnana/nebula/blob/master/docs/ngql_push_filter_down.md |
Beta Was this translation helpful? Give feedback.
-
Regarding of further optimizing the processing of super nodes, I'm wondering whether filtering more properties direcly in the storaged, as proposed in this issue, would be sufficient for your applications in your business. Although pushing down filters alleviate network transfers and the pressure on serialization and deserialization, the overhead of scanning the neighbourhood of super nodes is still there. A candidate for the worst case could be multiple concurrent queries scanning the same set of big super nodes. Similar to what have been applied in RDBMS, we could try implementing shared node/edge scans in the storaged that serve the lookup requests for multiple concurrent queries to amortize the cost. |
Beta Was this translation helpful? Give feedback.
-
完全依靠属性过滤解决超级节点确实是不可行的,业务使用中建模方面是有节点打散处理的,超级节点可以理解为可以通过属性区分的部分大节点,下推过滤后我们的业务目前可以支撑起来。 |
Beta Was this translation helpful? Give feedback.
-
Background:
Some super nodes in business use need to be filtered through attributes. Because the filtering does not push down the large storage layer, the query times out. Forcibly truncating the storage layer through limit [] will result in data loss. In some aspects, whether and how much filtering can be pushed down directly affects the performance of the entire database.
In the nebula source code, the push-down logic is in the optimizer. When the filter encounters LeftJoin and InnerJoin, it can't push down, but normal multi-hop query statements will trigger these two behaviors.
The main reason why InnerJoin cannot be pushed down is that the storage layer does not support variable filtering, and the LeftJoin may have DestExpression. If these two problems are solved, you can basically push them down.
General idea:
Reference PR:#4796
Beta Was this translation helpful? Give feedback.
All reactions