Experimental feature: Limit the number of batched tasks

[Kerollmops]

Feature name	How to enable	Description	Missing for stabilization
limit the number of batched tasks	You can export the env variable, the command line argument, or the config file to limit Meilisearch to processing only a given number of tasks simultaneously. # You can either use this environment variable MEILI_EXPERIMENTAL_MAX_NUMBER_OF_BATCHED_TASKS=100 # directly use this CLI argument --experimental-max-number-of-batched-tasks 100 # Or via the config file experimental_max_number_of_batched_tasks = 100	Reduces the size of the batched tasks to reduce the RAM usage of the engine	We could remove this flag and make the engine sufficiently good to do choose the right number of tasks to process simultaneously

How to use the feature?

1. Run Meilisearch with the environment variable or the command line argument flag described above
2. Profit.

Why is this feature not stable yet?

We are confident that this parameter will become useless in a short time. We are exposing this option to help the engine while we improve it. Once we make sure the engine is good enough to compute the limit of tasks it can auto-batch by itself, we will remove this parameter, and we don't want this to be a breaking change.

🗣️ You are welcome to give feedback about the score details or ask any question on its usage; we are eager to collect feedback on the feature

[gh2k]

Another issue beyond RAM usage here is the no space left on device problem that you can run into when ingesting large amounts of data in a short space of time: meilisearch/meilisearch#2844 / meilisearch/meilisearch#2765

(This is particularly an issue in Docker/k8s, where pointing TMPDIR at a mounted volume slows things down a lot but root disk size can be limited.)

[mj520]

This is very useful and necessary、Looking forward to adding and default to the official version

[dureuill]

Thanks for your feedback. We'd really prefer if Meilisearch could determine the batch size itself dynamically, rather than the user having to specify it manually. We're still working on ways to improve the performance of indexing operations.