Need for maintained fastText project

[juhoinkinen]

I noticed the official fastText repository was archived Mar 19, 2024 and is thus no more maintained at all.

Annif already installs fastText from the fasttext-wheel PyPI project, which builds code from https://github.com/messense/fasttext-wheel (Annif initially switched to that because of this installation issue).

At some point installation from the fasttext-wheel project probably won't work. In the fastText mailing list someone already reported a compatibility issue with NumPy 2.0. Actually when I now try to install the fasttext backend dependencies of Annif on Python 3.12 on my laptop, Poetry complains Unable to find installation candidates for fasttext-wheel (0.9.2) (two weeks ago, but after NumPy 2.0 release, CICD successfully installed fastText).

If we want retain the fasttext backend in Annif for a long term, some alternative provider for fastText should be found. There are a number of more or less active fastText forks.

[juhoinkinen]

An interesting candidate is Floret, available in PyPI.

It could be more memory-efficient than fastText:

In order to store word and subword vectors in a more compact format, we turn to an algorithm that's been used by spaCy all along: Bloom embeddings. Bloom embeddings (also called the "hashing trick", or known as HashEmbed within spaCy's ML library thinc) can be used to store distinct representations in a compact table by hashing each entry into multiple rows in the table. By representing each entry as the sum of multiple rows, where it's unlikely that two entries will collide on multiple hashes, most entries will end up with a distinct representation.

It should provide the same functionality as fastText:

floret supports all existing fasttext commands and does not modify any fasttext defaults.

Edit: Just switching to use the floret provided fastText algorithm needs very minimal changes: c6d23d8