Gaia/Email/Implementation/MailSynchronization

Overview

We use completely different synchronization implementations for IMAP and ActiveSync. This is an outgrowth of differences in the protocols.

Key IMAP / ActiveSync Differences

• Fixed sync windows versus flexible sync windows
  • For ActiveSync, we are constrained to sync along exact, pre-defined time boundaries. We can sync 1 day, 3 days, 1 week, 2 weeks, 1 month, or all messages. That's it.
  • For IMAP we do synchronize whole days in their entirety, but otherwise what we sync until we have 'enough' messages for our display needs.

IMAP

Possibilities

IMAP sync algorithms can be classified into two major classes: whole folder sync, and partial folder sync.

Whole folder sync is what most desktop mail clients like Thunderbird do. This is simpler but has the potential for much greater resource usage and a greater delay before the client is first usable. The greater resource usage is traditionally not a problem for desktop-class devices with good bandwidth.

Partial folder sync is what the Gaia e-mail app does, and probably most mobile mail clients.

All mail sync can be optimized or made better by the various optional IMAP extensions that exist out there, like CONDSTORE and QRESYNC. The big problem is that because these are optional extensions, many servers don't implement them and we absolutely cannot depend on them. For example, GMail and Yahoo implement the absolute minimum required by the IMAP spec.

Whole Folder Sync Basics

Whole-folder sync is potentially simpler because it avoids a lot of edge cases related to partial sync.

A simple implementation finds out what the highest unique id is in the folder and then asks for all of the envelopes for messages from UID 1 through the high UID. Relying on a correlation between UIDs and recent messages allows a client to start with higher UIDs and (usually) end up fetching the envelopes of the most recent messages first. Incremental synchronization after this initial synchronization is made much easier because

Partial Folder Sync Basics

The tricky thing about partial folder sync is that usually what you want is the most recent messages by date but the way messages are numbered in an IMAP folder is based on when they were put in the folder. Since new messages must have been put in a folder recently, there is a correlation between the message UID (the number / identifier it is given when it is added to the folder), but it is just a correlation. The most recent messages in a folder by UID may be message that are six months old because the user just moved them into that folder.

This not-good-enough correlation is what complicates things. So rather than synchronizing just based on UID, the Gaia E-Mail client synchronizes based on the date the message was received, also known as its INTERNALDATE. This is a (theoretically) immutable value based on when the SMTP server received the message, but it might also be when the message was fed into the IMAP server. Either way, the value is distinct from the 'Date' the author of the message indicates.

We use the SINCE and BEFORE SEARCH operations to get a list of messages falling within a given time window.

Partial Sync Implications

Because we synchronize based on time-windows, we can always synchronize more messages. There is no hard boundary stopping us from synchronizing more messages, unlike for ActiveSync.

ActiveSync

ActiveSync was more explicitly designed for the constrained mobile device sync device idiom. In fact, it's somewhat over-constrained in this regard. When we ask to synchronize a folder, our choices are to synchronize a time range stretching: 1 day, 3 days, 1 week, 2 weeks, or 1 month into the past or ALL of the messages in the folder.

Partial Sync Possibilities

It's conceivable that we could be clever and try and implement something similar to our IMAP partial sync strategy for ActiveSync. We haven't looked into it.