feat!: add app.state

[fubuloubu]

What I did

This PR adds worker shared state (accessible via app.state) and adds 2 new system tasks to support loading and storing a snapshot of that managed state. Currently this is just last_block_processed and last_block_seen.

This paves the way for the Parameter feature, but does not include that just yet.

BREAKING CHANGE: state snapshot features migrates from runner to worker and must be triggered by runner

How I did it

How to verify it

Checklist

• Passes all linting checks (pre-commit and CI jobs)
  - [ ] New test cases have been added and are passing
• Documentation has been updated
• PR title follows Conventional Commit standard (will be automatically included in the changelog)

[mikeshultz]

Took a quick initial pass. Kind of wish "state" wasn't used for so many different things, it's getting confusing.

[fubuloubu]

So given that distributed TaskIQ sequencing currently operates as multi-threading (and not multi-processing), this approach of worker-managed application state is okay as long as there isn't multiple workers trying to write the same state at the same time. We may need to integrate a lock to the AppState datatype in order to ensure this, but in terms of the mechanism it will actually function as long as there is only one container with one process operating the multi-threaded workers.

[fubuloubu]

TODO:

• write documentation about app.state and how it gets used

[mikeshultz]

I still don't really understand why the datastore is on the worker side. I'm going to have to basically re-invent that whole thing for the platform side, more or less reverting or duplicating some of this design. But I think a bunch of this was already touched in previous discussions so probably pointless to mention.

Will try and figure out how to make this work on the cluster side when I get a chance.

[fubuloubu]

I still don't really understand why the datastore is on the worker side.

Originally I don't think I actually understood what you were saying, but yes it is very clear to me now that needs to be inverted. The application state being worker side is what I wanted here, but the state snapshot loading and backup should be entirely on the runner client to perform. My mistake.

I hope you agree the solution should be to update this system function and add startup_state as a parameter it receives from the runner to update the local state on startup, then update this system function to just return the state snapshot back to the runner to write to disk; lastly, can move the AppDatastore back to the SDK's BaseRunner (and maybe make that a configurable ABC if you need to use it downstream, up to you).

[fubuloubu]

I still don't really understand why the datastore is on the worker side. I'm going to have to basically re-invent that whole thing for the platform side, more or less reverting or duplicating some of this design. But I think a bunch of this was already touched in previous discussions so probably pointless to mention.

Will try and figure out how to make this work on the cluster side when I get a chance.

@mikeshultz resolved in 07dd9f9

[mikeshultz]

One question on design inline. I'm good with it as is though.

[mikeshultz]

question: Do we have future plans for this handler? This more or less is just the runner sending these two values to the worker to send back to the runner.

Or maybe we intend to make this available to user code since it's called every checkpoint?

[fubuloubu]

State snapshots in general will be created using this mechanism (parameters will make use of this), but I added system: to the front of these specific internal state variables because then they cannot conflict with any attribute added to state directly by users (e.g. app.state.something = ... by definition cannot start with system:)

One of my ideas when adding parameters is that those are only allowed to be specific types, and only parameters will end up in state snapshots that get communicated back to the cluster services. All other non-parameter state will not be backed up in snapshots.

There may exist other features that make use of this, like nonce tracking, although they may need better concurrency guarantees (like using a lock)