Experimental feature: Reduce Indexing Memory Usage

[Kerollmops]

Hey everyone 👋

For the v1.2 of Meilisearch (which will be released on 5th June), we will introduce an experimental feature that should help reduce the memory usage of the engine as an experimental feature.

Feature Abstract

This flag enables the MDB_WRITEMAP option of LMDB, making the internal key-value store use much less RAM than usual. Unfortunately, it can reduce the write speed of it and therefore slow down the engine. You can read more about this LMDB option in the lmdb.h header file.

This flag cannot be enabled on Windows. This is due to a different way of handling memory maps on this system than on others i.e. Unix-like.

How to use the feature?

• Launch meilisearch using the --experimental-reduce-indexing-memory-usage CLI flag or the MEILI_EXPERIMENTAL_REDUCE_INDEXING_MEMORY_USAGE environment variable. This option does not accept any values.
• Index documents into memory and notice less RAM used by Meilisearch than without the flag.

---

What is an experimental feature

• The API and the behavior of an experimental feature can break between two minor versions of Meilisearch.
• The whole experimental feature process can break between 2 minor versions of Meilisearch.
• The newly introduced experimental features or improvements can be removed in the future. Providing a feature as “experimental” does not guarantee it will be stable one day.
• The experimental features are supposed to be unstable regarding the API usage but not bugged.

⚠️ Experimental features are not recommended to be used in production

Why is this feature not stable yet?

We have yet to determine if a lot impacts performance and whether the RAM usage reduction is significant enough. Could you give us your feedback on whether we want to ship it?

When will the feature potentially be stable?

We have yet to determine when this feature could be stable since we need to iterate on it. As explained previously, we do not guarantee it will be stable someday.

Depending on the feedback we get, we plan to iterate on the feature every 1 or 2 releases (Currently, our release cycle is two months long).

🗣️ You are welcome to give feedback about the feature or ask any questions; we can guide you through this new feature usage!

[Zabrane]

Hi @Kerollmops
What's the status of this feature (experimental_reduce_indexing_memory_usage)?
Can we safely turn it on forever?
We're always using the latest Meilisearch. Now on 1.10.0.

[macraig]

Hi @Zabrane , the feature is still experimental, meaning we don't recommend using it in production environments. At the moment, we don't have a date for stabilization, but we’d appreciate any feedback you might have. How’s it been working for you? Any insights you can share about its impact on performance and whether the RAM usage reduction is significant enough would be really helpful.

[Zabrane]

Hi @Zabrane , the feature is still experimental, meaning we don't recommend using it in production environments. At the moment, we don't have a date for stabilization, but we’d appreciate any feedback you might have. How’s it been working for you? Any insights you can share about its impact on performance and whether the RAM usage reduction is significant enough would be really helpful.

@macraig Ah I see. We will turn this experimental feature ON in a small cluster and I will report here my findings.

[macraig]

That would be great @Zabrane ! Looking forward to your findings!