Persian language support

[Ja7ad]

Hello,

In the new version (1.10), index settings support the localizedAttributes feature based on this issue.

I checked Notion, and I couldn't find support for the Persian language, only Arabic is supported.

Language settings in v1.10

How can I add the Persian language?

[ManyTheFish]

Hello @Ja7ad,
Meilisearch relies on Charabia to tokenize the documents and the queries,
If you make a PR on this repository, it will impact future Meilisearch versions.

First, we need to define the Language specificities we want to support; I don't know anything about Persian, so we'll have to help each other.

There are 2 main steps during the tokenizing process that we could specialize for Persian:

1. the segmentation process defines how the words are split. Is it only based on the separators, or is there any additional rule for splitting words?
2. the normalization process defines rules to unify tokens that are considered equivalent for a human but are technically different for a machine. To give some examples of already supported normalization:
   a) case: Hello <=> hello
   b) kashida: الحمد <=> الحمــــــد
   c) spoofing: Ð (Latin Capital Letter Eth) <=> Đ (Latin Capital Letter D with Stroke)

Then, when we define the changes to make, you'll be able to create a PR on Charabia; the repository is made easy to contribute to, even for people unfamiliar with Rust. The CONTRIBUTING.md file is an excellent starting point for implementing a normalizer or a segmenter.

Thank you for creating this discussion. 🪴
Let me know if you have any questions!

See you!

[Ja7ad]

Thank you, I write issue:

meilisearch/charabia#304