IMAP

Emre's Notes

Emre has put together some good diagrams that show code flow. Not all of these diagrams are specific to IMAP but he did significant work on the IMAP portions. Emre's diagrams can be found here.

Code Flow

Here is the basic flow of an IMAP operation. (This probably cries out for a graphical representation, but for now text will have to suffice.)

The path from user request to communication with the server is broken into two parts. First, the UI action is translated into a URL which is stored and then the call stack exits out. Next, the IMAP protocol begins processing its URLs and communicates with the server. This is actually reminiscent of Classic Eudora's command processing queue, though currently TB does not do any local processing before placing items on the queue.

There are two entry points into the IMAP protocol handling code: nsImapIncomingServer (which is derived from nsMsgIncomingServer -- the entry point for various incoming protocols) and nsImapMailFolder. I should note that sometimes the JavaScript calls into nsMessenger which calls into nsMsgFolderDataSource which eventually reaches nsImapMailFolder -- I do not understand enough of this yet to explain why sometimes the extra levels are needed, the crucial thing being that the entry point into IMAP is nsImapMailFolder.

nsImapIncomingServer - This class serves at least three purposes. The first is to be the starting point for IMAP operations that are independent of a specific mailbox. (The other purposes are discussed below.)

nsImapMailFolder - This class represents an individual IMAP mail folder. It is the starting point for IMAP operations on that mail folder.

Both of the above call into nsImapService.

nsImapService - This class has methods for all IMAP operations. First it sets up an nsImapUrl to indicate what IMAP action is being performed then calls into nsImapIncomingServer.

nsImapUrl - This class is purely a helper class to manage the URL's specifying the IMAP requests. The nsImapService class above appends a string to the URL which indicates the IMAP action, then shortly after that uses a series of 43 (yes, 43) strcmp()'s to convert the string into an int. Furthermore, it doesn't look like anyone even looks at that portion of the URL at any point in the future. This is mind-boggling. While I suspect this is a good candidate for future performance improvements, it is much too scary to even contemplate changing this at this time.

nsImapIncomingServer now serves its second purpose by informing the appropriate nsImapProtocol object of the new URL. After this entire call stack then returns out, ending part 1 of the process. This would be the equivalent of Classic Eudora queuing an action (minus doing the local portion first).

Here is the stack involved in part 1 (UI to URL) of doing a mail check:

 nsMsgIncomingServer::GetNewMessages()
   nsImapMailFolder::GetNewMessages()
     nsImapMailFolder::UpdateFolder()
       nsImapMailFolder::UpdateFolder()
         nsImapService::SelectFolder()
           nsImapService::GetImapConnectionAndLoadUrl()
             nsImapIncomingServer::GetImapConnectionAndLoadUrl()

Part 2 of the process begins when on a subsequent call to nsThread::ProcessNextEvent() the nsImapProtocol::Run() entry point is called, indicating to the nsImapProtocol class that it is being given time to do some processing. This is equivalent to Classic Eudora taking an item off the queue and doing the online work (again, the key difference being that the offline portion has not already been done in TB).

nsImapProtocol - This class handles the details of the IMAP protocol. It has methods to perform each IMAP action (e.g., Store(), Expunge()). The basic call stack for this object is:

 nsImapProtocol::Run()
   nsImapProtocol::ImapMainThreadLoop()
     nsImapProtocol::ProcessCurrentURL()

The ProcessCurrentURL() method acts on the URL created above and handles the specified IMAP action. To determine if there is already an IMAP connection it does the following:

 if (!TestFlag(IMAP_RECEIVED_GREETING))
   EstablishServerConnection();

If the protocol has not received a greeting from the server, a connection to the server is established at that time. Further down the line the appropriate commands are sent to the server via the SendData() method.

nsImapIncomingServer - This class now serves its third purpose, managing the connection with the server and communicating with the all important nsImapProtocol class.

nsImapServerResponseParser - This class parses the data returned by the IMAP server.

Here is the stack involved in part 2 (URL to actual server communication) of doing a mail check:

 nsImapProtocol::Run()
   nsImapProtocol::ImapThreadMainLoop()
     nsImapProtocol::ProcessCurrentURL()
       nsImapProtocol::ProcessSelectedStateURL()
         nsImapProtocol::ProcessMailboxUpdate()
           nsImapProtocol::FetchMessage()
             nsImapProtocol::SendData()
             nsImapProtocol::ParseIMAPandCheckForNewMail()
               nsImapServerResponseParser::ParseIMAPServerResponse()

Here are a few other key classes that come into play:

nsIMAPHostInfo - This is essentially a data structure containing the info for a given server: capabilities, password, trash mode, etc. Note that you will probably never access this object directly, but rather through the nsIImapHostSessionList class below.

nsIImapHostSessionList - This is a list of the hosts. More than just keeping track of the hosts it also provides access to the data specific to the hosts. One key distinction between this class and the Eudora equivalent is that this class is not used to help route actions. When a user requests an action on an IMAP mailbox the request is routed directly through the relevant nsImapMailFolder object. This list is actually consulted later in the process.

nsIMAPHostSessionList - This class derives from nsIImapHostSessionList as well as two other classes. This class apparently exists simply to provide the functionality provided by these other classes. Code that needs to access the host list does so via nsIImapHostSessionList above, not directly using this class.

nsImapMailboxSpec - This is basically just a data structure containing information about an IMAP mailbox. I haven't taken the time yet to fully understand the relationship between this and the nsImapMailFolder class.

nsIMAPMailboxInfo - While the class name suggests this might be an important class, it is only used in the case of a one-time upgrade from an old model to the new so we can safely ignore this class.

Establishing Connection to Server

A lot happens inside the function nsImapProtocol::ProcessCurrentURL() and I will not attempt to cover all of it here. However, I will detail how a connection to an IMAP server is established in this function.

Every time nsImapProtocol::ProcessCurrentURL() is called it (among other things) checks to see if a connection is already in place that can handle the URL in question. If a connection is not in place nsImapProtocol::EstablishServerConnection() is called to establish the connection and receive the greeting. After that nsImapProtocol::Capability() is called (assuming the CAPABILITY list wasn't returned in the greeting) to read and parse the server capabilities. If SSL is needed or desired nsImapProtocol::StartTLS() is called. Once all of this is done nsImapProtocol::TryToLogon() is called to attempt to login.

Once the connection is up this function moves on to calling either nsImapProtocol::ProcessAuthenticatedStateURL() or nsImapProtocol::ProcessSelectedStateURL() depending on the URL in question.

Message Downloading

The decision regarding which message bodies to download occurs in the following stack:

 nsImapMailFolder::HeaderFetchCompleted()
   nsImapMailFolder::GetBodysToDownload()

The code has the following to say about HeaderFetchCompleted(): 1) nsImapMailFolder::HeaderFetchCompleted: is triggered when TB notices that there are pending messages on the server -- via IDLE command from the server, via explicit select from the user, or via automatic Update during idle time. If it turns out that there are pending messages on the server, it adds them into nsAutoSyncState's download queue.

The function nsImapMailFolder::GetBodysToDownload() iterates over the headers for the mailbox and determines which message bodies should be downloaded.

The following describes what happens when you click on an undownloaded IMAP message.

The initial steps taken are identical to all other IMAP actions as detailed above. We pick up the story when the nsImapProtocol processes the URL indicating that a message should be downloaded. Here is the call stack which the nsImapProtocol object uses:

 nsImapProtocol::ProcessCurrentURL()
   nsImapProtocol::ProcessSelectedStateURL()
     nsImapProtocol::FetchTryChunking()
       nsImapProtocol::FetchMessage()

This is where the interesting stuff begins to happen. Inside FetchMessage() the FETCH command is formatted and nsImapProtocol::SendData() is called to send this command to the server. Next, the confusingly named nsImapProtocol::ParseIMAPandCheckForNewMail() method is called to download and parse the server's response.

The nsImapProtocol object has an associated nsImapServerResponseParser object whose purpose is (obviously enough) to parse the server's responses. The inner workings of nsImapServerResponseParser are still somewhat mysterious to me and I make no effort to unravel the many interwoven method calls here. The final stack that concerns us is the following:

 nsImapServerResponseParser::ParseIMAPServerResponse()
   nsImapServerResponseParser::response_data()
     nsImapServerResponseParser::numeric_mailbox_data()
       nsImapServerResponseParser::msg_fetch()
         nsImapServerResponseParser::msg_fetch_content()
           nsImapServerResponseParser::msg_fetch_literal()

Inside here the actual message data is downloaded and passed back to the nsImapProtocol via the method nsImapProtocol::HandleMessageDownLoadLine(). It is worth noting that the use of the word "Line" in this method name is potentially misleading. The documentation for this method notes that an actual line of text may be broken up over several calls to this method and experience shows that the entire header is passed in as a single line. The upshot is that this method is called with various chunks of the message data. This method then calls the object's nsMsgImapLineDownloadCache object to have it cache the "line": nsMsgImapLineDownloadCache::CacheLine().

When nsImapServerResponseParser::msg_fetch_content() sees that the complete response has been received it launches the following stack:

 nsImapProtocol::NormalMessageEndDownload()
   nsImapProtocol::PostLineDownLoadEvent()

If the message is being stored for offline use the nsImapProtocol will call into its "message sink" (in this case a nsImapMailFolder) and tell it to store the data. In this case nsImapProtocol::PostLineDownLoadEvent() calls into nsImapMailFolder::ParseAdoptedMsgLine().

At this point we disappear into the details of output streams which is beyond the scope of this discussion.

Threading

See Thread Handling for general details on Thunderbird thread handling.

Multiple IMAP threads can be in the queue and multiple threads can run simultaneously. (There is a user-settable limit to the number of simultaneous connections to a given IMAP server, the default being 5.) The thread handling page shows the following call stack for placing a thread on a queue (I have trimmed off some of the later stack items which are not relevant to this discussion):

 NS_InvokeByIndex()
   nsImapMailFolder::UpdateFolder()
     nsImapService::SelectFolder()
       nsImapService::GetImapConnectionAndLoadUrl()
         nsImapIncomingServer::GetImapConnectionAndLoadUrl()
           nsImapIncomingServer::GetImapConnection()
             nsImapIncomingServer::CreateProtocolInstance()

This is actually the most basic case where a new nsImapProtocol object is created and added to the queue. If a new nsImapProtocol object were created for every action you could potentially have multiple actions queued that act on the same objects and thus should be sequential, not simultaneous.

The nsImapIncomingServer::GetImapConnection() method actually checks to see if there is an existing connection which can be reused and if so it creates a new thread referencing that nsImapProtocol instance. If no usable connection is found then a new one is created. The nsImapProtocol object has a flag to indicate that it is currently processing an action assuring that actions on the same object will be sequential.

Offline IMAP

Offline operation centers around the nsIMsgOfflineImapOperation interface. When a user requests an action while offline that action is placed in a queue. When Thunderbird next goes online it iterates over the queue and then queues threads to perform those actions online.

I haven't yet figured out all the details of the offline queue and will fill this section in more when I know more.

Here are some preliminary notes generated by some investigation I did. I encountered a case where a bogus item was in the action queue -- the action item was for appending a message to the Drafts folder but the message was 0 length. The result of this was a continuous loop of the error message "The current command did not succeed. The mail server responded: Command Argument Error. 12." The action item was never cleared from the queue despite the fact that it could never succeed. The problem disappeared as mysteriously as it appeared before I could resolve it but I did generate the following notes while the problem was happening.

 nsImapOfflineSync::OnStopRunningUrl()
   ProcessNextOperation();
     This recurses (don't completely understand that) until coming up with a currentOp which is then processed.
       Recurses by comparing actions against the target type:
         mCurrentPlaybackOpType
           nsIMsgOfflineImapOperation::kFlagsChanged
           nsIMsgOfflineImapOperation::kAddKeywords
           ...
           nsIMsgOfflineImapOperation::kMsgMoved
       nsMailDatabase::GetOfflineOpForKey()
         *offlineOp = new nsMsgOfflineImapOperation(this, offlineOpRow);