Arabic language support

[Hadryan]

All varieties of Arabic combined are spoken by perhaps as many as 422 million speakers (native and non-native) in the Arab world, making it the fifth most spoken language in the world. (Wikipedia)

Related: https://github.com/meilisearch/MeiliSearch/issues/553

[Kerollmops]

Hey,

As I answered to your previous language support request, I would like some feedback on the current state of the engine. We currently split by whitespace and when it comes to CJK character we split them one by one and consider each of those a different word.

Thank you!

[mzeidhassan]

Hi @Kerollmops , I think if you use what Apache Solr uses, it would be great.

Character normalization factories are very important.

solr.ArabicNormalizationFilterFactory
solr.ArabicStemFilterFactory
solr.ICUFoldingFilterFactory

So, to have a good Arabic search engine, you need to be able to search by stem. Also, there should be some kind of character normalization, while there is an option for users to get exact searches if they wish, i.e.: ignoring normalization. Something like quoted search.

For example, in Arabic, we have alef hamza above, alef hamza below, and alef hamza. Here how hey look like:

أ
إ
ا

Character normalization should be able to treat all variations of hamza as the last one. So, hamza above and hamza below can be ignored. Think of it as in German:
Typing "uber", should be able to get 'Über', and 'ueber'.

Hope this helps.

[curquiza]

I add here the comment of @mohamed-foly that could be definitely useful
meilisearch/meilisearch#2248 (comment)

[curquiza]

Another interesting related issue: meilisearch/charabia#36

[curquiza]

Also, we've just released a new version of our tokenizer, making it easier to contribute. Meaning anyone can add an Arabic normalizer/segmenter and then, make Meilisearch finally compliant with the Arabic language as the Arabic community expects Meilisearch to work 😄

Feel free to open any PR or even ask any question on the tokenizer repo, named charabia 🎉

[LoaiDev]

any updates on this?

[gmourier]

Hi @LoaiDev, as we don't have any native Arabic-speaking technical person in the core team at the moment, we hope to have an external contribution on the tokenizer repo to move this topic forward.

We have made contributions easier to provide on the tokenizer lately to favor that language-specific addition. Feel free to contribute; it will be a massive help for Meilisearch. 😊

[ManyTheFish]

Hello all!
I come back to you to make some updates.
For Hacktoberfest we've created several issues to enhance the Arabic support of Meilisearch:

• Implement an efficient Nonspacing Mark Normalizer charabia#137:

A normalization allowing Meilisearch to retrieve words containing nonspacing marks like diacritics or hamza.

• Arabic script: Implement specialized Segmenter charabia#133:

Enhance the Arabic Language support by segmenting agglutinated words like الشجرة

All these issues are open to external contributions during the whole month, so don't hesitate to contribute! 🧑‍💻
I'll be glad to help you if you have any questions.

This is another step in enhancing Arabic Language support, depending on future feedback, we will be able to go further.

Thanks for all your feedback! ✍️

[ManyTheFish]

Hello all!

I have a simple question about Arabic ligatures, is it only useful to format texts or does it have a particular sense compared to the decomposed form? For instance ﰲ (U+FC32) is composed of ف (U+0641) + ي (U+064A), is it sensible to decompose them?

note that if I write both composing letters together, GitHub seems to automatically add the ligature, making ﰲ (U+FC32) and في (U+0641) + (U+064A) similar for a non-Arabic writter. 🤔

Thanks a lot for your help!

[ManyTheFish]

Hey @mohamed-foly, thanks for your answer!
I don't suggest splitting these two characters into 2 words, my question is: does ﰲ means the same as في?

To give a bit more context, today, Meilisearch consider them as 2 completely different things.

[LoaiDev]

they are the same word. I never knew this even existed

[LoaiDev]

if that's the case for all arabic words that means that now meilisearch can only match full words because most letters will change to a different unicode if a letter is written after it

[mzeidhassan]

hi @ManyTheFish
Arabic ligatures are rarely used out of professional liturgical typing. That said, they are indeed 2 different things. In most texts, the composed form is the widely used. As @LoaiDev mentioned, I didn't know these even existed, but apparently there are a bunch of them.

I haven't seen any Arabic keyboard that supports this form though.

Maybe, it would be better if you give option to normalize such ligatures, i.e. replacing ﰲ with في

By the way, whenever I copy that ﰲ in Notepad, or any text editor, it shows up as a box (gets corrupted).

[ManyTheFish]

@LoaiDev and @mzeidhassan, thank you for your help.
So the issue about Compatibility Decomposition (charabia#139) should enhance Meilisearch Arabic language support by unifying both Unicode versions of Arabic ligatures.

[DrAliRagab]

I hope if anyone can help with this: meilisearch/charabia#204

[hamdallah90]

any update on this?

[curquiza]

Hello @hamdallah90
The improvement described by @DrAliRagab in this comment is already available in a release candidate (v1.2.0-rc.2) and will be release officially in v1.2.0 on 5th of June 😊

[DrAliRagab]

Yes, both Arabic normalizer and segmenter are improved.

[macraig]

v1.2.0 is now officially out and includes normalizer and segmentation improvements.

[LoaiDev]

Hello, commenting on this again in case someone can work on it. In arabic there is "ال" which is the same as "the" but it is at the start of the word instead of separate. so "باب" would mean "door" but "الباب" would mean "the door" I am not sure how meilisearch handles "the" in english but I think this also should be handled someway in arabic

[ManyTheFish]

Hello @LoaiDev,
Did you try the last version of Meilisearch? @DrAliRagab worked on this specific case and ensured that "ال" is split from the linked word, if you encounter an issue on this it's probably a bug, do you manage to find a document containing "الباب" by searching "باب"?

Have a good day!

[lordjoo]

Actulay we also are facing a similar case where we have "للباب" and try searching with "باب" and no documents found

[ManyTheFish]

Interesting, could you send a text that should be found with the query "باب" but isn't retrieved by Meilisearch, please?

[usamamhmod]

للباب/ بالباب/ لباب/ الباب
بابه / بابها/ بابك / ...