Snapshot creation on demand

[Magorx15]

It would be really nice to have the same feature as dump has, create dump with a simple /dump request but with snapshots.
In my head it would work just the same, send request to /snapshot, a task gets added to the queue and makes the snapshot just like the schedule one. The response would have the same structure, having the id (name) of the snapshot etc.

For our use case it is simple:
We want to have the snapshot as soon as the indexing is done, so we can use that snapshot for later (use it on different servers that might not have the specs to index (low memory, disk space etc))
Now, we achieve this with a very high MEILI_SCHEDULE_SNAPSHOT value (3600) so we are certain the indexing has been finished and then the script just waits until the snapshot file has been created. This is very time consuming.
Our script (getting data, indexing etc) takes 55 minutes to finish. Waiting for the snapshot scheduler to start and then to create snapshot takes around 110 minutes, so this process takes around 3 hours to finish.

[oluademola]

Hello @Magorx15. Thank you for the feedback. Indeed you can't create snapshots on demand, and you can only schedule them for now as seen in the dedicated documentation. Since this is a feature request, I'd move this discussion to the product repo to be handled by the product team. Perhaps they may have some suggestions or workaround for you.

[gmourier]

After a discussion with one of our users, making snapshots on demand is convenient for various use cases.

For that particular user, the idea is to index a bunch of documents belonging to an entity (an index), make a snapshot of the instance, and expose the search on demand via the Cloud Run service. (It is also a need for scale to zero in that context).

The initial idea was to index documents when a user buys that entity to be searched later. Still, as the task queue is shared per index, it's better to isolate the indexing process per purchased resources, so indexing speed is always guaranteed when there's a lot of traffic.

[gmourier]

Hello everyone 👋,

Here is the current prototype you can use to create snapshots on-demand.

How to run the prototype?

Use the following Docker image:

docker run -it --rm -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:prototype-snapshot-on-demand-0

How to use it?

Create a snapshot on-demand.

Send a POST request to the newly added endpoint /snapshots

Meilisearch returns a 202 Accepted HTTP response containing the task id if you wish to track the snapshot creation progress via GET /tasks/:task_uid.

API key and permission

If you're using an API key that doesn't have * for its actions property, you must create a new API key that specifies snapshots.create within the actions property.

By default, both the master key and the default admin API key can process a snapshot creation.

Snapshots location

Snapshot files are created in the path specified via --snapshot-dir or MEILI_SNAPSHOT_DIR.

Note about snapshot scheduling

You can create snapshots on demand even if the snapshot scheduling feature is enabled.

Both snapshot creations will be processed in the order of arrival in the task queue, and --snapshot-dir will contain the last processed one.

---

⚠️ We do NOT recommend using this prototype in production. This is only for test purposes.

Everyone is more than welcome to give feedback and to report any issue or bug you might encounter when using this prototype. Thanks in advance for your involvement. It means a lot to us ❤️

[gmourier]

Hey everyone 👋

With v1.5, it's now possible to create snapshots on demand.

Closing this discussion for now, please open a new one if needed!

Thanks!