Multiple Update Graphs

[rcaudy]

I would like to consider introducing support for multiple update graphs to the query engine.

My rough idea is to combine an UpdateGraphProcessor instance and a LogicalClock instance into an UpdateContext, with a default instance thereof handling the common use cases. All BaseTable instances will need to record the UpdateContext they were created under, and all multi-table operations will need to ensure that their refreshing inputs are all from the same UpdateContext.

This would give us more freedom for a few use cases:

1. Unit test isolation: with independent UGPs and clocks, we can allow more intra-process parallelism if appropriate, and we can permanently eliminate the "cascading failures problem" caused by improper UGP state.
2. Parallel DAGs for health monitoring or other "non core" use cases.
3. Faster input table updates for snapshot-only use cases.

I'm interested in input regarding usability and other use cases.

[devinrsmith]

This would be a prerequisite for single-process multi-tenancy IMO. I can theorize situations where this could useful. (The user abstractions in this theoretical multitenant process could be very different than what we present today. I think this should be discussed separately if interested.)

[nbauernfeind]

Multiple UpdateGraphProcessors could easily "throw" data from one UGP to another using BarrageMessageProducer's creating BarrageMessages and importing into BarrageTables. This would happen at the chunk layer and would not require serializing to input streams.

I suspect there are scenarios where you would want two UGPs working well together -- one that has a long target cycle time and one that aims toward a much lower latency. The slowly iterating UGP would be computational heavy and the quickly iterating UGP would be reactionary to events based on the last major computation.

There might be other ways to cross these boundaries, but I can't think of a clean solution that is generic to the number of UGPs.

[chipkent]

I think the concept is good. We need to make sure that the abstractions the user sees make their life easy for constructing the UGPs, communicating between UGPs, etc.