[Serverless] Ensure that only one subscription instance is active

[alexeyzimarev]

When a subscription is deployed as a serverless function or container, it's hard to control how many instances are currently active. It will inevitably lead to multiple instances running simultaneously, processing the same events (which could be a problem) and overriding the checkpoint (another potential problem).

#161 suggests adding an HTTP API to subscriptions to report their positions. Another API could also expose the subscription presence with some unique id. The id could be generated on startup and remains static for a given running instance.

It would allow creating a self-managed distributed control plane for subscription services. It could work like this:

• A newly instantiated service makes several calls to its own HTTP endpoint, assuming the endpoint is load balanced with a round-robin strategy
• It will collect all the responses from running instances and can understand how many are there
• If there's only one instance, the subscription will mark itself as active. This status is included in the API response.
• If there are multiple instances, and one of them is active, the subscription doesn't start
• If there are multiple instances, and none of them is active, the subscription decides to become active after gossiping with other instances
• When a subscription receives a gossip call from some instance that wants to activate, it will ensure that it is not active. Otherwise, the request should be denied.

This process needs to run continuously, so when one active subscription gets shut down, some other instance will become active.

I foresee some race conditions during the elections, but I am not sure yet if it's relevant.

[alexeyzimarev]

It might be a lot easier to implement with some external expiring lease documents. Redis, for example, could work. The question is if it's possible to avoid using any external infrastructure.

[fbjerggaard]

I don't think depending on external infrastructure is such a big deal - There is already a dependency on a EventStore of some sort.

It also isn't only an issue in a serverless deployment - if I understood it right it is generally a problem with a service that is scaled out.

[alexeyzimarev]

I am inclined to that too.

I had a relatively long and deep experiment with Cloud run and I just can't make a single subscription instance to work. It appeared to be much harder and more complex than I thought, if things are kept only on the inside. Will continue with different options, preparing for my Event Sourcing Live talk.

[fbjerggaard]

A thought I just had was that it could (should?) be configurable during startup whether Eventuous is "cluster-aware" or not - kinda like Quartz.NET does it. Then it would be possible to run Eventuous in a single instance (or manually ensuring that only 1 instance of Eventuous runs) or let Eventuous handle it automatically via the registered ClusterHandler (Working name) - and for a start it would be pretty simple to only create a Redis ClusterHandler for a start - and then it could be expanded with other options down the road.