Optimistic concurrency support

[curquiza]

Because of the GitHub bug that refuses to transfer discussions, just following this feature request here: meilisearch/meilisearch#1886

[dikatok]

@curquiza thanks for the help, so to reiterate the problem I am facing.

One thing that is preventing me from moving from Elasticsearch to Meilisearch is the lack of optimistic concurrency. This is helpful when the document updates are done asynchronously, not directly as per user command, especially when it is driven by change data capture or projection of an event-sourced system.

It does not have to be fancy, for instance, having a numeric version field that can be used to check whether the new document has a value greater than the current value or not, if it is the case replace it, otherwise fail the update. For reference, Elasticsearch has this capability under external versioning here.

Checking the current version of the document on the application level does not cut it, since there might be concurrent replacement requests from another process.

I am not sure how the async update is implemented in Meilisearch, but if a queue-like structure is used, maybe just drop updates with a version less than the current document version?

Thanks

[gmourier]

Hi @dikatok! Thanks for this feedback 👋

To summarize, you would like to be able to put version numbers on each document so that MeiliSearch does not take versions inferior or equal to the current version of a document during updates?

This could be an interesting feature to implement. How do you determine the version number of a document on your side before?

Thanks!

[dikatok]

@gmourier sorry for the delayed reply

Exactly.

In my case, I was using a transactional database as my single source of truth, which enables me to safely increment the version number of a document each updates.

Then, some async processes will be triggered to update the projection in other data stores, such as elasticsearch.

[dikatok]

I am curious, is this kind of feature request open for contribution?

[gmourier]

It can be!

In summary, the process we currently use is this :

In case we detect that adding this feature to MeiliSearch can have a positive impact and can help a majority of users, we prioritize the feature in the roadmap because we already have a lot of features requested that are useful. We think that it allows us to better communicate the direction that MeiliSearch has chosen to take.

Next to that, a specification is written to help the implementation and sharing of information among everyone before starting it.

Now, I understand that this is a process that from the outside may seem quite tedious and I would like external contributors to continue to propose features that have value so that we can work together to steer them in the right direction. 🤓

[dikatok]

I see, that is understandable. About the writing of the specification, do external contributors have a hand in the process? Or they are in complete control of the internal team?

[gmourier]

Specifications are open for comment during the review phase. It allows to bring up potential unresolved pain points and to validate the API or the behaviors.

To answer you, yes, we would like to have feedback helping us when we started to shape a solution to specification for the different needs we have collected in the discussions.

What we are doing now is making a product ideation phase when we decide to solve a particular need. This ideation can be followed in the product repository issues section. Here is an example. This allows people to participate and criticize our proto-solution. When the proto-solution seems okay, we start the process of specification. The specification is then open for review to get feedback on a nearly concrete final state.