[POC/RFC] Operations queue

[dantleech]

This is an experiment for an operations queue which unities the seperate scheduledInserts, scheduledUpdates, scheduledReorders, etc. into a single queue.

This enables persistance operations to be committed out in the order in which they were programmed.

This makes sense for move, reorder and delete operations which can have multiple entries in the stack for a given document. It makes less sense for "update" because the changeset is not recorded for each operation ("persist") but is calculated once before comitting.

Lots of test fails here and this is not necessarily the solution we need, just an experiment.

[dbu]

wow, quite a rewrite. imo it makes sense to attempt this, its the natural way. you can also look at the code in jackalope to get inspired, we have a quite similar approach in the jackalope unit of work (called ObjectManager there).

note that add and remove can happen through modified child fields and children collections, and move through assignment. all of those will only be detected at flush time. but i would say that is live and if you do stuff where the order matters, with these changes you at least stand a chance to do things.

benjamin told me that for the ORM, they have something that calculates the transitions needed to get from the persisted state to the current memory state (in terms of what needs to be done first and after that and so on). this would work even for the implicit add/remove/move operations and be exactly in the declarative spirit of doctrine. but i imagine it could be quite hard to calculate.

[dantleech]

In regard to implicitly calculating the changes, the use case that breaks the current system is:

move: /foobar > /barfoo
create: /foobar

Create is done before move, so it doesn't work. We could add some sepcial code to detect this. The question I am asking myself is if there are other scenarios where the order of operations might matter. - and to what extent recording the order of events is important.

The operations queue is a much more intuitive and natural way to process the commit, and so it should be easier to maintain and have less bugs. Performance I guess depends on the size of the queue and on any compression we can apply to the queue before flushing. I don't know if it would be inherently slower than the implicit method, but after optimization it should be negligible one way or the other.

Just musing.

[dbu]

i think the idea of leveraging ocramius/changeset is interesting here. we would record changes as we notice them, but for the implicit changes (adding a child, renaming something and so on) calculated in the uow its gonna be more or less undetermined, but for all explicit calls at least the order would be preserved. this has potential for quite some BC breaks - we should make sure to calculate in the order that we previously had, to minimize them.

[dantleech]

I understand it as we only need an operation queue for tree operations (move, delete, create, reorder, etc). The documents and their data would be handled by the changeset. So first you would run-off the operation queue, then you would apply the changeset -- having mandatory UUIDs will be of great use here.

Shall we schedule this for 2.0? Without mandatory UUIDs this would be more tricky I think, and as you mention there will be BC breaks.

[dbu]

this sounds a lot like 2.0 to me, yes.

you are right about field changes. but there are quite some tricky things to keep in mind (field changes on moved documents and detect that we don't need the changes on a deleted document, for example)

[dantleech]

Really need to have another crack at this. Maybe there is away that doesn't involve rewriting the UOW.