NAME

squatter - Cyrus IMAP documentation

Create SQUAT and Xapian indexes for mailboxes

SYNOPSIS

general:
squatter [ -C config-file ] [mode] [options] [source]
i.e.:
squatter [ -C config-file ] [ -v ] [ -a ] [ -S seconds ] [ -Z ]
squatter [ -C config-file ] [ -v ] [ -a ] [ -i ] [ -N name ] [ -S seconds ] [ -r ] [ -Z ] mailbox...
squatter [ -C config-file ] [ -v ] [ -a ] [ -i ] [ -N name ] [ -S seconds ] [ -r ] [ -Z ] -u user...
squatter [ -C config-file ] [ -v ] [ -a ] -R [ -n channel ] [ -d ] [ -S seconds ] [ -Z ]
squatter [ -C config-file ] [ -v ] [ -a ] -f synclogfile [ -S seconds ] [ -Z ]
squatter [ -C config-file ] [ -v ] -t srctier(s)... -z desttier [ -B ] [ -F ] [ -U ] [ -T reindextiers ] [ -X ] [ -o ] [ -S seconds ] [ -u user... ]

DESCRIPTION

NOTE:

squatter creates a new text index for one or more IMAP mailboxes. The index is a unified index of all of the header and body text of each message in a given mailbox. This index is used to significantly reduce IMAP SEARCH times on a mailbox.

mode is one of indexer, search, rolling, synclog, compact or audit.

By default, squatter creates an index of ALL messages in the mailbox, not just those since the last time that it was run. The -i option is used to select incremental updates. Any messages appended to the mailbox after squatter is run, will NOT be included in the index. To include new messages in the index, squatter must be run again, or on a regular basis via crontab, an entry in the EVENTS section of cyrus.conf(5) or use rolling mode (-R).

In the first synopsis, squatter indexes all mailboxes.

In the second synopsis, squatter indexes the specified mailbox(es). The mailboxes are space-separated.

In the third synopsis, squatter indexes the specified user(s) mailbox(es).

For the latter two index modes (mailbox, user) one may optionally specify -r to recurse from the specified start, or -a to limit action only to mailboxes which have the shared /vendor/cmu/cyrus-imapd/squat annotation set to “true”.

In the fourth synopsis, squatter runs in rolling mode. In this mode squatter backgrounds itself and runs as a daemon (unless -d is set), listening to a sync log channel chosen using the -n option, and set up using the sync_log_channels setting in imapd.conf(5). Very soon after messages are delivered or uploaded to mailboxes squatter will incrementally index the affected mailbox (see notes, below).

In the fifth synopsis, squatter reads a single sync log file and performs incremental indexing on the mailbox(es) listed therein. This is sometimes useful for cleaning up after problems with rolling mode.

In the sixth synopsis, squatter will compact indices from srctier(s) to desttier, optionally reindexing (-X) or filtering expunged records (-F) in the process. The optional -T flag may be used to specify members of srctiers which must be reindexed. These files are eventually copied with rsync -a and then removed by rm. rsync can increase the load average of the system, especially when the temporary directory is on tmpfs. To throttle rsync it is possible to modify the call in imap/search_xapian.c and pass -\-bwlimit=<number> as further parameter. The -o flag may be used to direct that a single index be copied, rather than compacted, from srctier to desttier. The -u flag may be used to restrict operation to the specified user(s).

For all modes, the -S option may be specified, causing squatter to pause seconds seconds after each mailbox, to smooth loads.

When using the Xapian engine the -Z option may be specified, for the indexing modes. This tells squatter to consult the Xapian internally indexed GUIDs, rather than relying on what’s stored in cyrus.indexed.db, allowing for recovery from broken cyrus.indexed.db at the sacrifice of efficiency.

NOTE:

NOTE:

squatter reads its configuration options out of the imapd.conf(5) file unless specified otherwise by -C.

OPTIONS

EXAMPLES

squatter is typically deployed via entries in cyrus.conf(5), in either the DAEMON or EVENTS sections.

For the older SQUAT search engine, which offers poor performance in rolling mode (-R) we recommend triggering periodic runs via entries in the EVENTS section, as follows:

Sample entries from the EVENTS section of cyrus.conf(5) for periodic squatter runs:

EVENTS {


    # reindex changed mailboxes (fulltext) approximately every three hours


    squatter1   cmd="/usr/bin/ionice -c idle /usr/lib/cyrus/bin/squatter -i" period=180


    # reindex all mailboxes (fulltext) daily


    squattera   cmd="/usr/lib/cyrus/bin/squatter" at=0117
}

For the newer Xapian search engine, and with sufficiently fast storage, the rolling mode (-R) offers advantages. Use of rolling mode requires that squatter be invoked in the DAEMON section.

Sample entries for the DAEMON section of cyrus.conf(5) for rolling squatter operation:

DAEMON {


  # run a rolling squatter using the default sync_log channel "squatter"


  squatter cmd="squatter -R"


  # run a rolling squatter using a specific sync_log channel


  squatter cmd="squatter -R -n indexer"
}

NOTE:

NOTE:

NOTE:

[NB: More examples needed]

HISTORY

Support for additional search engines was added in version 3.0.

The following command-line switches were added in version 3.0:

-F -R -X -d -f -o -u

The following command-line settings were added in version 3.0:

-S <seconds>, -T <directory>, -f <synclogfile>, -n <channel>, -t srctier..., -z desttier

FILES

/etc/imapd.conf, /etc/cyrus.conf

SEE ALSO

imapd.conf(5), cyrus.conf(5)

AUTHOR

The Cyrus Team, Nic Bernstein (Onlight)

COPYRIGHT

1993–2023, The Cyrus Team