Greek language support

[nantoniadis]

In Greece, we use some symbols to pronounce our words. For example μπάλα. But while searching, people often skip these symbols. So, instead of μπάλα, they type μπαλα. Meilisearch treats them as separate words, while in fact, they are not.

Is there any way to treat some characters the same? The list of characters are:
α = ά
ο = ό
ι = ί
η = ή
ε = έ
ω = ώ
υ = ύ
ς = σ

[curquiza]

Hello @nantoniadis
Thanks for your feature request!
Pinging @ManyTheFish here, since he's the main maintainer of the language support 😇

[ManyTheFish]

Hello @nantoniadis, thank you for creating this discussion.

I see 2 possible improvements to your issue.

Remove nonspacing-marks

α = ά
ο = ό
ι = ί
η = ή
ε = έ
ω = ώ
υ = ύ

In Meilisearch v1.0.0, we already decompose characters from their nonspacing-marks, for example, ώ (U+03CE) is decomposed in ω (U+03C9) + ◌́ (U+0301), moreover a nonspacing-marks remover exists in Meilisearch but isn't activated for Greek in this version.

Possible enhancement

Make a PR activating nonspacing-marks normalizer for Greek script:
https://github.com/meilisearch/charabia/blob/38ba8570c3b7531fd44bb8773328681fcd21a7f9/charabia/src/normalizer/nonspacing_mark.rs#L28-L31

I've created a good first issue on the dedicated repository:
meilisearch/charabia#178

Normalize Sigma

ς = σ

Meilisearch v1.0.0 doesn't normalize final-sigma, and so, ς is considered different than σ.

Possible enhancement

Create a new normalizer converting ς into σ.

I've created a good first issue on the dedicated repository:
meilisearch/charabia#179

[koufopoulosf]

Hello @ManyTheFish,

May I please ask your valuable opinion about the support of Greek language when using Meilisearch?

Would it be considered stable to be used in a production environment?

Thank you very much for your time.

Best Regards,

Filippos

[koufopoulosf]

Hello @ManyTheFish ,

Thank you very much for your valuable response!

I will definitely try it out! May I please ask you 4 more questions?

1. Is there any specific setup we need to follow to make meilisearch secure for production use and fault tolerant? Or if it crashes, to work again in the next few seconds? Because compared to Typesense, it doesn't support cluster mode yet.

2. How does Meilisearch compare to other search engines like lnx-search, typesense, sphinx, etc. regarding the accuracy of the results it returns for a given query? Ofcourse that would heavily depend on the dataset, but would meilisearch provide more accurate results on average compared to other search engines? To be more specific, on the meilisearch.com, if you type "aitress", instead of returning the movie "Waitress", it returns "Actress". It has a tedency to replace internal letters than to check for additional ones missing left and right.

3. What are the hardware requirements for 500mb of data in total? Maybe 2 columns, for both the title and the description. Like what CPU, RAM, Disk space etc would be needed for that amount of data (500mb) to work effectively with high traffic?

4. Is there any admin panel for meilisearch with analytics where we can see what are the most frequent requests/searches?

Thank you once again for your valuable time 🙏 and I can't wait to try out meilisearch!

[ManyTheFish]

1 Is there any specific setup we need to follow to make meilisearch secure for production use and fault tolerant? Or if it crashes, to work again in the next few seconds? Because compared to Typesense, it doesn't support cluster mode yet.

There is no high availability feature so far in Meilisearch. I'm probably not the best person to answer, but, setting an auto-restart using systemctl should be a good start, Meilisearch start is fast and the unavailability should be short. Making my own stress tests, I found out that the minimal requirement in terms of RAM for a Meilisearch instance is 2GB if you want to avoid most of the crashes.
If High availability is an important subject for you and you want to follow it, the team is currently implementing a prototype of a clusterized version of Meilisearch, I put below the dedicated discussion on discord:
https://discord.com/channels/1006923006964154428/1075374699774742618/1088560171162271744

2.1 How does Meilisearch compare to other search engines like lnx-search, typesense, sphinx, etc. regarding the accuracy of the results it returns for a given query? Ofcourse that would heavily depend on the dataset, but would meilisearch provide more accurate results on average compared to other search engines?

I am currently working on evaluating the relevancy and the search performances of Meilisearch using TREC datasets. These datasets are used by researchers to compare different search algorithms and provide queries with the expected relevant documents for each of them. However, we will use them mainly to compare Meilisearch versions and monitor the progress more than comparing Meilisearch with their challengers. Moreover, the first part of these benchmarks focuses on English datasets so the results couldn't be considered for other Languages like Greek. The only thing we have so far to monitor relevancy is the user's feedback.

2.2 To be more specific, on the meilisearch.com, if you type "aitress", instead of returning the movie "Waitress", it returns "Actress". It has a tedency to replace internal letters than to check for additional ones missing left and right.

Technically, if waitress isn't found with aitress, it's because a typo on the first letter is counted as 2 typos and Meilisearch accepts 2 typos only on words containing at least 8 characters. I join the dedicated documentation here.

Why a typo on the first letter is counted as 2 typos?

1. It's for performance reasons, computing a typo on the first letter is way slower than computing any other typos, so we chose to add a penalty on the first letter.
2. People rarely perform typos on the first letter and on the last letter, which are the 2 most important letters in a word for its readability making these typos easy to catch and are instantly corrected by the writer.
   I don't manage to find the psychological study but:

If you are a natvie egnilsh spaeekr, you wuold not hvae any dificulites to udnesrtnad tihs setnesne depsite the fact that the itneranl letetrs are shuflfed!

3 What are the hardware requirements for 500mb of data in total? Maybe 2 columns, for both the title and the description. Like what CPU, RAM, Disk space etc would be needed for that amount of data (500mb) to work effectively with high traffic?

The minimal requirement of Meilisearch is 2GB of RAM and 10 times your data in terms of disk space. However, if you want to index documents at the same time as a search request spike, I'd personally recommend 4GB of RAM. CPUs are mainly used to parallelize the search load and don't impact the instance stability.

4 Is there any admin panel for meilisearch with analytics where we can see what are the most frequent requests/searches?

If you speak about the content of the searches (what the end user typed), no we don't save this. And well... it's my personal point of view but I think that's better in terms of privacy. 😬

[koufopoulosf]

Thank you very much for your detailed reply once again! I highly appreciate it! @ManyTheFish

Regarding High Availability, combined with the last part about hardware requirements, would an 8GB RAM and 4 AMD EPIC VCPUs be enough to handle, let's say, 500 concurrent requests without an issue using nginx? What happens above that number? Would a 16GB RAM with 4VCPUs be an overkill and 8GB Ram enough? or maybe deploy another one server as fallback behind a load balancer in case one fails due to high traffic or for any other reason? These are some of my final questions. Ofcourse, I guess disabling the search as you type option would make it more efficient and thus that's what I plan to use (click a button to send the query and not by search as you type). I mean, if someone uses the search as a critical feature of his web application, that everyone who joins the website uses, he would expect this to not fail during high traffic times. If there are any measures that you would suggest about avoiding such mission-critical feature of web app (meilisearch to stop working), that would be really awesome!

I totally agree about your point on the typos on first and last letter. Regarding the content of searches, I really agree with your point regarding privacy! However I was referring to a "Google trends" like tool, where you do not see personal queries, but most frequent/trending searches overall.

Thank you once again for your awesome support! I guess meilisearch is the go-to option for everyone seeking a great (and most importantly, simple to use) search functionality!

[koufopoulosf]

@ManyTheFish I'd appreciate your response, especially on hardware requirements for concurrent users. Thanks🙏

[ManyTheFish]

Hello @koufopoulosf,
Sorry for the delay, to answer your question, it's hard to say how Meilisearch would fit your expectations.
However, for really big datasets 8GB of RAM is a good start, and if you have any issue with it, don't hesitate to take 16GB.
About the search load, you can expect to have around 15 requests per second and per available CPU.

I mean, if someone uses the search as a critical feature of his web application, that everyone who joins the website uses, he would expect this to not fail during high traffic times. If there are any measures that you would suggest about avoiding such mission-critical feature of web app (meilisearch to stop working), that would be really awesome!

We are working on it, but we can't provide this kind of feature for the moment, the only thing I could say is to synchronize manually several Meilisearch instances and set a load balancer dispatching the requests between instances.

Regarding the content of searches, I really agree with your point regarding privacy! However I was referring to a "Google trends" like tool, where you do not see personal queries, but most frequent/trending searches overall.

I understand, so far we don't store searches in Meilisearch so we can't provide aggregated search metrics

Again, sorry for the delay,

HAve a good week!

[gmourier]

Hey folks!

v1.1 has been released and now supports normalizing diacritics and final sigma ✨