Leverage Offline capabilities to make online more responsive

The basic idea is to use the offline capabilities of Thunderbird while online to make the UI more responsive, when operating on IMAP messages, out of the box, without the user having to tweak any settings.

There are several different things we can do in this area to make the user experience better. One is to download imap messages for offline use by default, so that the message bodies are already stored locally when the user goes to read the message. An other is to perform message moves and deletes "pseudo-offline", which means we do the operations as if we were offline (storing the operations as offline ops in the db), which is very quick, and then playing back the operations to the imap server later. So, for example, deleting an imap message could change from issuing the protocol to copy the message to the trash, and to store the deleted flag on it, and when that's done, load the next message from the imap server, to storing the move as an offline operation in the db, and then loading the next message from the offline store, and issuing the protocol to copy and delete the message while the user is reading the next message.

Why store offline operations in the db instead of (in the case of delete) doing something like removing the row(s) from the view, loading the next message, and then issuing the delete to the server? The short answer is that not losing these operations is a good thing, and storing them persistently helps a lot there. We have to remember them somewhere, and we might as well leverage the existing offline code. There's also a fair amount of code that knows about offline deleted/moved messages and handles them gracefully (for example, we know that when we connect to the server and try to download headers for messages not in the local db, we should ignore offline deleted messages, because we're just going to delete them from the server after we've done the header sync). Netscape 4.x used this approach, and it worked reasonably well.

Two obvious questions are:

1. When do we download the message bodies?
2. When do we playback the offline operations?

There are also issues having to do with automatically compacting the offline stores every so often, because the offline store is in Berkeley mailbox format, which means that deleting messages doesn't shrink the file, until you do a compact. Since there's a hard 4GB limit to the size of the offline store folder, we need to compact at some point.

The low hanging fruit approach is to do the following:

• switch the default for imap folders so that they're configured for offline use out of the box.
• switch mail.server.default.autosync_offline_stores to true, by default
• change imap move/deletes though the UI so that they're done offline, and the offline operation is played back after the next message is displayed, or immediately if there's no next message to display.

This will make it so whenever the user opens an imap folder, we download the message bodies at the same time as we download the message headers. If there are a lot of message bodies to download (e.g., the new profile case), this will take a long time and not be a great experience, but it's what a lot of IMAP clients do (Outlook Express, Outlook, Mail.app).

Question: One common complaint with Mail.app is that it downloads them in chronological order, rather than in reverse chronological order, meaning that the emails most likely to be needed right now are the last to be downloaded. Would doing a reverse chronological download be possible, or is there an IMAP reason why it wouldn't work?

Answer: The most succinct form of the command to fetch multiple messages (bodies or headers) fetches them in UID, roughly chronological order, using ranges, e.g., 1-20. But you certainly could fetch 20,19,18,17 etc. Are you talking about the new profile case, or the every day case? For the new profile case, a simple thing to do would be to fetch the bodies of the unread messages first, then the bodies of the read messages. Or we could fetch ranges of messages, but start at the end, not the top, e.g., 80-100, 60-79, etc.

One other important thing to realize is that when we're doing something with an IMAP folder, like fetching message bodies, we can't do anything else with that IMAP folder, like fetch a single message when the user clicks on a header, because having two connections to the same imap folder can be bad (especially with the UW server). So in this situation, the user will be blocked from reading a message until we've either downloaded that message body as part of our big download process, or the download process finishes.

Question: Couldn't we timeslice the connection, so that we're blocked on the folder at most one message body download's worth of time? (which can be big, but usually isn't).

Answer: That would require fetching the message bodies one at a time, which generates a lot more protocol round trips, and is quite a bit slower.

There are lots of potential improvements to the low hanging fruit approach:

• We could use the Idle service to download message bodies when the user is idle.
• We could breakup the download of message bodies into multiple passes, which would allow the user to sneak in and start reading messages before we've synced the whole folder.
• We could compact offline stores when the user is idle.
• We could playback offline operations when the user is idle, though I think we'd run into less issues if we tried to do the offline playback closer to the UI event.

Other issues

• What UI do we present to allow users to override the auto-download of all message bodies?
• Do we give the user feedback when the delete actually happens?
• Do we make the Trash an offline folder like all other folders?
• Are there other imap operations that we would want to do "offline", like renaming a folder, or should we stick with the most often used commands, like reading a message, move, delete, etc?
• Do we want to move the Unsent Messages folder from under the Local Folders hierarchy and somehow make it more visible in the folder tree, maybe calling it Outbox like Mail.app? [davida]

I [bienvenu] would think one big advantage of offline stores by default is it could make search quite a bit faster.

Current Offline Mode Behavior

• If autosync_offline_stores pref is selected, TB downloads all messages proactively.
• If this pref is not selected, then TB only downloads when the user reads them.
• If the user does a file | offline sync command, TB downloads all the messages it doesn't have.
• It is also possible to select some messages, and say file | offline | download now.