InterIMAP - Fast bidirectional synchronization for QRESYNC-capable
IMAP servers
interimap [OPTION ...] [COMMAND] [MAILBOX
...]
interimap performs stateful synchronization between two
IMAP4rev1 servers. Such synchronization is made possible by the
QRESYNC IMAP extension; for convenience reasons servers must also
support the LIST-EXTENDED, LIST-STATUS (or NOTIFY) and
UIDPLUS IMAP extensions. See also the supported extensions
section below.
Stateful synchronization is only possible for mailboxes supporting
persistent message Unique Identifiers (UID) and persistent storage of
mod-sequences (MODSEQ); any non-compliant mailbox will cause interimap to
abort. Furthermore, because UIDs are allocated not by the client but
by the server, interimap needs to keep track of associations
between local and remote UIDs for each mailbox. The synchronization state of
a mailbox consists of its UIDNEXT and HIGHESTMODSEQ values on
each server; it is then assumed that each message with UID smaller than
UIDNEXT have been replicated to the other server, and that the
metadata (such as flags) of each message with MODSEQ at most
HIGHESTMODSEQ have been synchronized. Conceptually, the synchronization
algorithm is derived from RFC 4549 with the RFC 7162 (sec. 6)
amendments, and works as follows:
- 1.
- SELECT (on both servers) a mailbox the current UIDNEXT or
HIGHESTMODSEQ values of which differ from the values found in
the database (for either server). Use the QRESYNC SELECT
parameter from RFC 7162 to list changes (vanished messages and flag
updates) since HIGHESTMODSEQ to messages with UID smaller than
UIDNEXT.
- 2.
- Propagate these changes onto the other server: get the corresponding UIDs
from the database, then:
- a.
- issue an UID STORE command, followed by UID EXPUNGE,
to remove messages that have not already been deleted on both servers;
and
- b.
- issue some UID STORE commands to propagate flag updates
(send a single command for each flag list in order the reduce the
number of round trips).
(Conflicts may occur if the metadata of a message has been updated
on both servers with different flag lists; in that case, interimap
issues a warning and updates the message on each server with the
union of both flag lists.) Repeat this step if the server sent some
updates in the meantime. Otherwise, update the
HIGHESTMODSEQ value in the database.
- 3.
- Process new messages (if the current UIDNEXT value of the
mailbox differs from the one found in the database) by issuing an
UID FETCH command; process each received message on-the-fly
by issuing an APPEND command with the message's RFC822 body,
FLAGS and INTERNALDATE. Repeat this step if the server received new
messages in the meantime. Otherwise, update the UIDNEXT value in the
database. Go back to step 2 if the server sent some metadata (such
as flag) updates in the meantime.
- 4.
- Go back to step 1 to proceed with the next unsynchronized mailbox.
By default, interimap synchronizes each mailbox listed by
the LIST "" "*" IMAP command;
the list-mailbox, list-select-opts and ignore-mailbox
options from the configuration file can be used to shrink that list and save
bandwidth. However if some extra argument are provided on the command line,
interimap ignores said options and synchronizes the given
MAILBOXes instead. Note that each
MAILBOX is taken “as is”; in particular,
it must be UTF-7 encoded, unquoted, and the list wildcards `*' and
`%' are not expanded.
If the synchronization was interrupted during a previous run while
some messages were being replicated (but before the UIDNEXT or
HIGHESTMODSEQ values have been updated), interimap performs
a “full synchronization” on theses messages:
downloading the whole UID and flag lists on each servers allows
interimap to detect messages that have been removed or for which
their flags have changed in the meantime. Finally, after propagating the
offline changes for these messages, interimap resumes the synchronization
for the rest of the mailbox.
Specifying one of the commands below makes interimap perform
an action other than the default QRESYNC-based
synchronization.
- --repair [MAILBOX ...]
- List the database anomalies and try to repair them. (Consider only the
given MAILBOXes if non-optional arguments are provided.) This is
done by performing a so-called “full synchronization”,
namely: 1/ download all UIDs along with their flag list both from the
local and remote servers; 2/ ensure that each entry in the database
corresponds to an existing UID; and 3/ ensure that both flag lists match.
Any message found on a server but not in the database is replicated on the
other server (which in the worst case, might lead to a message duplicate).
Flag conflicts are solved by updating each message to the union of both
lists.
- --delete MAILBOX [MAILBOX
...]
- Delete the given MAILBOXes on each target (by default each server
plus the database, unless --target specifies otherwise) where it
exists. Note that per the IMAP4rev1 standard deletion is not
recursive. Thus MAILBOX's children are not
deleted.
- --rename SOURCE DEST
- Rename the mailbox SOURCE to DEST on each target (by default
each server plus the database, unless --target specifies
otherwise) where it exists. interimap aborts if DEST
already exists on either target. Note that per the IMAP4rev1 standard
renaming is recursive. Thus SOURCE's children are moved to become
DEST's children instead.
- --config=FILE
- Specify an alternate configuration file. Relative paths start from
$XDG_CONFIG_HOME/interimap, or ~/.config/interimap if the
XDG_CONFIG_HOME environment variable is unset.
- --target={local,remote,database}
- Limit the scope of a --delete or --rename command to the given
target. Can be repeated to act on multiple targets. By default all three
targets are considered.
- --watch[=seconds]
- Don't exit after a successful synchronization. Instead, keep synchronizing
forever. Sleep for the given number of seconds (by default 1 minute
if --notify is unset, and 15 minutes if --notify is set) between
two synchronizations. Setting this options enables SO_KEEPALIVE on the
socket for types other than tunnel.
- --notify
- Whether to use the IMAP NOTIFY extension to instruct the server
to automatically send updates to the client. (Both local and
remote servers must support RFC 5465 for this to work.) This
greatly reduces IMAP traffic since interimap can rely on server
notifications instead of manually polling for updates. If the connection
remains idle for 15 minutes (configurable with --watch), then
interimap sends a NOOP command to avoid being logged out for
inactivity.
- -q, --quiet
- Try to be quiet.
- --debug
- Turn on debug mode. Debug messages are written to the given
logfile. Note that this include all IMAP traffic (except literals).
Depending on the chosen authentication mechanism, this might include
authentication credentials.
- -h, --help
- Output a brief help and exit.
- --version
- Show the version number and exit.
Unless told otherwise by the --config=FILE command-line
option, interimap reads its configuration from
$XDG_CONFIG_HOME/interimap/config (or
~/.config/interimap/config if the XDG_CONFIG_HOME
environment variable is unset) as an INI file. The syntax of the
configuration file is a series of OPTION=VALUE lines organized
under some [SECTION]; lines starting with a `#' or `;' character
are ignored as comments. The [local] and [remote] sections
define the two IMAP servers to synchronize. Valid options
are:
- database
- SQLite version 3 database file to use to keep track of associations
between local and remote UIDs, as well as the UIDVALIDITY,
UIDNEXT and HIGHESTMODSEQ of each known mailbox on both
servers. Relative paths start from
$XDG_DATA_HOME/interimap, or
~/.local/share/interimap if the XDG_DATA_HOME
environment variable is unset. This option is only available in
the default section. (Default: HOST.db, where
HOST is taken from the [remote] or
[local] sections, in that order.)
- list-mailbox
- A space separated list of mailbox patterns to use when issuing the initial
LIST command (overridden by the MAILBOXes given as
command-line arguments). Note that each pattern containing special
characters such as spaces or brackets (see RFC 3501 for the exact
syntax) must be quoted. Furthermore, non-ASCII names must be UTF-7
encoded. Two wildcards are available: a `*' character matches zero
or more characters, while a `%' character matches zero or more
characters up to the mailbox's hierarchy delimiter. This
option is only available in the default section. (The default
pattern, *, matches all visible mailboxes on the
server.)
- list-select-opts
- An optional space separated list of selectors for the initial LIST
command. (Requires a server supporting the LIST-EXTENDED IMAP
extension.) Useful values are SUBSCRIBED (to list only
subscribed mailboxes), REMOTE (to also list remote mailboxes on a
server supporting mailbox referrals), and RECURSIVEMATCH (to
list parent mailboxes with children matching one of the above
list-mailbox patterns). This option is only
available in the default section.
- ignore-mailbox
- An optional Perl Compatible Regular Expressions (PCRE) covering mailboxes
to exclude: any (UTF-7 encoded and unquoted) mailbox listed in the initial
LIST responses is ignored if it matches the given
expression. Note that the MAILBOXes given as
command-line arguments bypass the check and are always considered
for synchronization. This option is only available in the default
section.
- logfile
- A file name to use to log debug and informational messages. (By default
these messages are written to the error output.) This option is only
available in the default section.
- type
- One of imap, imaps or tunnel. type=imap and
type=imaps are respectively used for IMAP and IMAP over SSL/TLS
connections over a INET socket. type=tunnel causes interimap
to open a pipe to a command instead of a raw
socket. Note that specifying type=tunnel in the
[remote] section makes the default database to
be localhost.db. (Default: imaps.)
- host
- Server hostname, for type=imap and type=imaps. (Default:
localhost.)
- port
- Server port. (Default: 143 for type=imap, 993 for
type=imaps.)
- proxy
- An optional SOCKS proxy to use for TCP connections to the IMAP server
(type=imap and type=imaps only), formatted as
PROTOCOL://[USER:PASSWORD@]PROXYHOST[:PROXYPORT]. If
PROXYPORT is omitted, it is assumed at port 1080. Only SOCKSv5 is
supported (with optional username/password authentication), in two
flavors: socks5:// to resolve hostname locally,
and socks5h:// to let the proxy resolve
hostname.
- command
- Command to use for type=tunnel. Must speak the IMAP4rev1
protocol on its standard output, and understand it on its standard
input.
- STARTTLS
- Whether to use the STARTTLS directive to upgrade to a secure
connection. Setting this to YES for a server not advertising
the STARTTLS capability causes interimap to immediately abort the
connection. (Ignored for types other than imap. Default:
YES.)
- auth
- Space-separated list of preferred authentication mechanisms. interimap
uses the first mechanism in that list that is also advertised
(prefixed with AUTH=) in the server's capability list. Supported
authentication mechanisms are PLAIN and LOGIN. (Default:
PLAIN LOGIN.)
- username,
password
- Username and password to authenticate with. Can be required for non
pre-authenticated connections, depending on the chosen authentication
mechanism.
- compress
- Whether to use the IMAP COMPRESS extension for servers
advertising it. (Default: NO for the [local] section,
YES for the [remote] section.)
- null-stderr
- Whether to redirect command's standard error to /dev/null for
type type=tunnel. (Default: NO.)
- SSL_protocols
- A space-separated list of SSL protocols to enable or disable (if prefixed
with an exclamation mark !. Known protocols are SSLv2,
SSLv3, TLSv1, TLSv1.1, TLSv1.2, and TLSv1.3.
Enabling a protocol is a short-hand for disabling all other
protocols. (Default:
!SSLv2 !SSLv3 !TLSv1 !TLSv1.1, i.e., only enable
TLSv1.2 and above.)
- SSL_cipher_list
- The cipher list to send to the server. Although the server determines
which cipher suite is used, it should take the first supported cipher in
the list sent by the client. See ciphers(1ssl) for more
information.
- SSL_fingerprint
- Fingerprint of the server certificate's Subject Public Key Info, in the
form [ALGO$]DIGEST_HEX where ALGO is the used algorithm (by default
sha256). Attempting to connect to a server with a non-matching
certificate SPKI fingerprint causes interimap to abort the
connection during the SSL/TLS handshake.
You can use the following command to compute the SHA-256 digest of
certificate's Subject Public Key Info.
-
openssl x509 -in /path/to/server/certificate.pem -pubkey \
| openssl pkey -pubin -outform DER \
| openssl dgst -sha256
- SSL_verify
- Whether to verify the server certificate chain. Note that using
SSL_fingerprint to specify the fingerprint of the server
certificate is an orthogonal authentication measure as it ignores the CA
chain. (Default: YES.)
- SSL_CApath
- Directory to use for server certificate verification if
SSL_verify=YES. This directory must be in “hash
format”, see verify(1ssl) for more information.
- SSL_CAfile
- File containing trusted certificates to use during server certificate
authentication if SSL_verify=YES.
interimap takes advantage of servers supporting the
following extensions to the IMAP4rev1 protocol (those marked as
“recommended” give the most significant performance
gain):
- •
- LITERAL+ (RFC 2088, recommended);
- •
- MULTIAPPEND (RFC 3502, recommended);
- •
- COMPRESS=DEFLATE (RFC 4978, recommended);
- •
- NOTIFY (RFC 5465, recommended);
- •
- SASL-IR (RFC 4959); and
- •
- UNSELECT (RFC 3691).
- •
- Using interimap on two identical servers with a non-existent or
empty database will duplicate each message due to the
absence of local ↔ remote UID association. Hence one
needs to manually empty the mail store on one end when migrating to
interimap from another synchronisation solution.
- •
- interimap is single threaded and doesn't use IMAP command
pipelining. Synchronization could be boosted up by sending
independent commands (such as the initial LIST and STATUS
commands) to both servers in parallel, and for a given server, by
sending independent commands (such as flag updates) in a
pipeline.
- •
- Because the IMAP protocol doesn't have a specific response code for when a
message is moved to another mailbox (either using the MOVE
command from RFC 6851, or via COPY + STORE +
EXPUNGE), moving a message causes interimap to believe
that it was deleted while another one (which is replicated again)
was added to the other mailbox in the meantime.
- •
- PLAIN and LOGIN are the only authentication mechanisms currently
supported.
- •
- interimap will probably not work with non RFC-compliant
servers. In particular, no work-around is currently implemented
beside the tunables in the configuration file. Moreover, few
IMAP servers have been tested so far.
- •
- M. Leech, M. Ganis, Y. Lee, R. Kuris, D. Koblas and L. Jones, SOCKS
Protocol Version 5, RFC 1928, March 1996.
- •
- M. Leech, Username/Password Authentication for SOCKS V5, RFC 1929,
March 1996.
- •
- J. Myers, IMAP4 non-synchronizing literals, RFC 2088, January
1997.
- •
- D. Goldsmith and M. Davis, A Mail-Safe Transformation Format of
Unicode, RFC 2152, May 1997.
- •
- C. Newman, Using TLS with IMAP, POP3 and ACAP, RFC 2595, June
1999.
- •
- M. Crispin, Internet Message Access Protocol - Version 4rev1, RFC
3501, March 2003.
- •
- M. Crispin, Internet Message Access Protocol (IMAP) -
MULTIAPPEND
Extension, RFC 3502, March 2003.
- •
- A. Melnikov, Internet Message Access Protocol (IMAP)
UNSELECT
command, RFC 3691, February 2004.
- •
- M. Crispin, Internet Message Access Protocol (IMAP) -
UIDPLUS
extension, RFC 4315, December
2005.
- •
- A. Melnikov, Synchronization Operations for Disconnected IMAP4
Clients, RFC 4549, June 2006.
- •
- A. Gulbrandsen, The IMAP COMPRESS
Extension, RFC 4978,
August 2007.
- •
- R. Siemborski and A. Gulbrandsen, IMAP Extension for Simple
Authentication and Security Layer (SASL) Initial Client
Response, RFC 4959, September 2007.
- •
- A. Gulbrandsen and A. Melnikov, The IMAP
ENABLE
Extension, RFC 5161, March
2008.
- •
- B. Leiba and A. Melnikov, Internet Message Access Protocol version 4
- LIST Command
Extensions, RFC 5258, June 2008.
- •
- A. Gulbrandsen, C. King and A. Melnikov, The IMAP
NOTIFY
Extension, RFC 5465,
February 2009.
- •
- A. Melnikov and T. Sirainen, IMAP4 Extension for Returning
STATUS Information in
Extended LIST, RFC 5819, March
2010.
- •
- A. Gulbrandsen and N. Freed, Internet Message Access Protocol (IMAP) -
MOVE
Extension, RFC 6851, January
2013.
- •
- A. Melnikov and D. Cridland, IMAP Extensions: Quick Flag Changes
Resynchronization (CONDSTORE)
and Quick Mailbox Resynchronization
(QRESYNC),
RFC 7162, May 2014.
Bugs or feature requests for interimap should
be filed with the Debian project's bug
tracker at <https://www.debian.org/Bugs/>.
Guilhem Moulin (mailto:guilhem@fripost.org).