German Language support

[sa-]

Hey there, first of all thanks for building this!

Wanted to ask if there were any plans to include better tokenization support for German rather than just whitespace. German is a language that can stick multiple "tokens" together without a space.

e.g. when I type in "kochplatte", I would expect "induktionskochplatte" to be just as relevant as other results that only say "kochplatte"

[ErikBrendel]

Agreed! It is a pity that searching for parts of compound words does not work. When searching for "Abo", documents containing "Jahresabo" should also be found.

[ManyTheFish]

Hello @sa- @ErikBrendel,
So far, I didn't investigate so much time in how to better segment the German Language,
However, if you have any suggestions of a words dictionary or even a Rust library that you trust to handle the segmentation of the German Language, it would be helpful.
Moreover, if you know a bit about Rust development or are interested to try it, our tokenizer repository, Charabia, is a good way to contribute easily to a Rust project, on top of it, if you need help, I can guide you during your implementation.

Thank you for your interest in the project!

[sa-]

Hello!

I have not worked tried any of these tools myself but I can point you in the direction of this collection of resources: https://github.com/adbar/German-NLP#linguistic-processing

[breunigs]

Tantivy appears to have utilities to deal with compound words, although it relies on a (stemmed) dictionary: https://docs.rs/tantivy/latest/tantivy/tokenizer/struct.SplitCompoundWords.html

[ManyTheFish]

Hello @breunigs,
Thank you for the suggestion; yes, we should rely on a dictionary to split German words properly, but the engine team can't allocate that much time to implement it. However, the tokenizer is an external library and is easy to contribute, so don't hesitate to come:
https://github.com/meilisearch/charabia

An alternate solution could be to use the dictionary setting to force Meilisearch to split German words:
https://www.meilisearch.com/docs/reference/api/settings#dictionary

Thanks again for your suggestion,

See you

[luflow]

@ManyTheFish I would like to contribute to charabia but am a bit unsure about the licensing of the dictionary - I would like to use an existing dictionary because creating one on my own sounds too erroneous and time consuming. Which licenses are ok for the project to include into charabia?

[luflow]

First version ready4review: meilisearch/charabia#303

[ManyTheFish]

Hello @luflow, thank you a lot for your contribution,
I will check your PR and merge it in order to release it with Meilisearch v1.11.0! 🙏

[luflow]

Awesome @ManyTheFish if you need anything else in this manner, just write me or your feedback into the PR, would love to see it in v1.11.0!