FTS Xapian plugin for Dovecot

What is this?

This project intends to provide a straightforward, simple and maintenance free, way to configure FTS plugin for Dovecot, leveraging the efforts by the Xapian.org team.

This effort came after Dovecot team decided to deprecate "fts_squat" included in the dovecot core, and due to the complexity of the Solr plugin capabilitles, unneeded for most users.

If you feel donating, kindly use Paypal : [email protected]

Debugging/Support

Please submit requests/bugs via the GitHub issue tracker. A Matrix Room exists also at : #xapian-dovecot:matrix.grosjo.net

Availability in major distributions

THis plugin is readly available in major distributions under the name "dovecot-fts-xapian"

• Archlinux : https://archlinux.org/packages/?q=dovecot-fts-xapian
• Debian : https://packages.debian.org/bookworm/dovecot-fts-xapian
• Fedora : https://src.fedoraproject.org/rpms/dovecot-fts-xapian

Configuration - dovecot.conf file

Update your dovecot.conf file with something similar to:

(Example in conf.d/90-fts.conf )

mail_plugins = (...) fts fts_xapian

(...)

plugin {
    fts = xapian
    fts_xapian = partial=3 full=20

    fts_autoindex = yes
    fts_enforced = yes

    fts_autoindex_exclude = \Trash

    # Index attachements
    fts_decoder = decode2text
}

service indexer-worker {
    # Increase vsz_limit to 2GB or above.
    # Or 0 if you have rather large memory usable on your server, which is preferred for performance)
    vsz_limit = 2G
    # This one must be 0
    process_limit = 0
}

service decode2text {
    executable = script /usr/libexec/dovecot/decode2text.sh
    user = dovecot
    unix_listener decode2text {
        mode = 0666
    }
}

Make sure also that dovecot is started with enough files opening capacity (ideally set 'LimitNOFILE=65535' in the systemd start file).

Configuration - Indexing options

Option	Optional	Description	Possible values	Default value
partial & full	no	NGram values for header fields	between 3 and 20 characters	3 & 20
verbose	yes	Logs verbosity	0 (silent), 1 (verbose) or 2 (debug)	0
lowmemory	yes	Memory limit before disk commit	0 (default, meaning 250MB), or set value (in MB)	0
detach	yes	Allow Xapian closing to be detached from main process	0 (no), 1 (yes)	0

Set detach=1 only if your mail storage in on a partition on which you can force the uid/gid. For some reasons, Devoct write as root instead so the filesystem must correct until dovecot team fixes the bug. detach=1 speeds up very much the process.

Configuration - NGrams details

The partial & full parameters are the NGram values for header fields, which means the keywords created for fields (To, Cc, ...) are between 3 and 20 chars long. Full words are also added by default (if not longer than 245 chars, which is the limit of Xapian capability).

Example: "john@doe" will create joh, ohn, hn@, ..., john@d, ohn@do, ..., and finally john@doe as searchable keywords.

Configuration - Index updating

Please make sure that "ulimit" is high enough. Typically, set "DefaultLimitNOFILE=16384:524288" in /etc/systemd/system.conf

Just restart Dovecot:

sudo service restart dovecot

If this is not a fresh install of dovecot, you need to re-index your mailboxes:

doveadm index -A -q \*

• With argument -A, it will re-index all mailboxes, therefore may take a while.
• With argument -q, doveadm queues the indexing to be run by indexer process. Remove -q if you want to index immediately.

You shall put in a cron the following command (daily for instance) to cleanup indexes :

doveadm fts optimize -A

Building yourself - Prerequisites

You are going to need the following things to get this going:

* Dovecot 2.2.x (or above)
* Xapian 1.2.x (or above)
* ICU 50.x (or above)
* SQlite 3.x

You will need to configure properly Users Home Directories in dovecot configuration

Building yourself - Installing the Dovecot plugin

First install the following packages, or equivalent for your operating system.

Ubuntu:
apt-get build-dep dovecot-core 
apt-get install dovecot-dev git libxapian-dev libicu-dev libsqlite3-dev autoconf automake libtool pkg-config

Archlinux:
pacman -S dovecot
pacman -S xapian-core icu git sqlite

FreeBSD:
pkg install xapian-core
pkg install xapian-bindings
pkg install icu
pkg install git

Fedora:
dnf install sqlite-devel libicu-devel xapian-core-devel
dnf install dovecot-devel git 

Clone this project:

git clone https://github.com/grosjo/fts-xapian
cd fts-xapian

Compile and install the plugin.

autoupdate
autoreconf -vi
./configure --with-dovecot=/path/to/dovecot
make
sudo make install

Note: if your system is quite old, you may change gnu++20 by gnu++11 in src/Makefile.in

Replace /path/to/dovecot by the actual path to 'dovecot-config'. Type 'locate dovecot-config' in a shell to figure this out. On ArchLinux , it is /usr/lib/dovecot.

For specific configuration, you may have to 'export PKG_CONFIG_PATH=...'. To check that, type 'pkg-config --cflags-only-I icu-uc icu-io icu-i18n', it shall return no error.

The module will be placed into the module directory of your dovecot configuration

---

Thanks to Aki Tuomi [email protected], Stephan Bosch [email protected], Paul Hecker [email protected]