Thunderbird:Thunderbird Post 3.1 BackendPlans
1. Pluggable Mail Store
See https://bugzilla.mozilla.org/show_bug.cgi?id=402392. There are two main goals here - replacing our Berkeley Mailbox store with a MailDir implementation by default, to avoid all the issues that arise from having all messages in a folder in a single file (compaction, mailbox contention, virus checker deleting file, inability to do incremental backup, ease of integration with desktop search, etc), and allowing other stores to be implemented (e.g., a sqlite store).
This entails making the backend code use the pluggable store interface instead of assuming berkeley mailbox, implementing a berkeley mailbox store plugin, implementing a maildir store plugin, and writing some migration code to migrate from berkeley mailbox format to maildir.
My thinking is to have maildir be the default store going foward, in the release after Miramar. We will most likely migrate profiles by default, instead of just using MailDir for new accounts/profiles. We may have UI to allow the user to stick with Berkeley Mailbox format, if only to optionally maintain compatibility with previous versions of Thunderbird.
2. Make it easier to get new communication data into Thunderbird.
Currently, this is very hard to do, as jcranmer and rkent have discovered, trying to add web-forums and Exchange messages respectively, into Thunderbird. It entails adding a new nsIMsgIncomingServer object, new nsIMsgFolder object, new nsIMsgService service, etc. The first two contain dozens and dozens of attributes and methods, most of which are implemented by common base classes (e.g., nsMsgDBFolder, nsMsgIncomingServer). Doing a new data source in js requires implementing all these methods by hand. Doing it in C++ is only marginally easier, if you inherit from the common base classes, or delegate to them.
Current Approaches, as I understand them:
RSS approach - RSS inherits from nsLocalMailFolder, nsMailboxServer, and implements nsRssService. It stores messages in the local mail folder, and each message essentially contains a link to the rss message. For simple datasources, like Twitter, this would be a fairly simple approach.
RKent's Exchange approach - tweak the backend code to allow instantiation of the common base classes, and implement new objects that delegate most methods to the core classes, overriding only the ones that it needs to specialize.
Jcranmer's Web-forum approach - Allow js objects to extend the core c++ objects, and implement the web-forum specific things in the js objects.
Dmose's facescrape aproach - Extension sticks new messages in existing account (e.g., Local Folders) Inbox or other folder, neatly sidestepping almost all the difficulties and concentrating on the main value add. The folder could be an IMAP folder fairly easily as well.
Possible new approaches:
Implement a library that does most of the common implementation, with a pluggable mechanism for doing the data source-specific things (e.g., enumerating containers/folders, pulling in messages, etc).
Hook in at the folder pane level - extension would add a new ftvItem sub-class, and handle everything at the UI level. The select handler would load a new deck into what was the thread pane, and handle clicks on items loading something to a new deck in the message pane, etc. In other words, the extension would have to implement everything, including command handling.
Different data sources have different requirements, both at a technical level, and a UX level. Some data sources need changes propagated back to the server (e.g, Exchange) but most do not. Some need to fully sync with the server in an IMAP-like way (e.g., if a message is deleted from the server, remove it from the client), but most do not. Some may lend themselves to the folder metaphor, like web forums or MAPI; others, like twitter, may not. So there can't be a one size fits-all solution.
Replacing Mork with Sqlite
Although we can now open Mork databases asynchronously, Mork's all in-memory design, among many other things, makes it desirable to switch to a more scalable database. Asuth has suggested using a very simple schema, where each row consists of a key and a json-like blob representation of a msg hdr. I would probably add message-id and subject columns&indexes to make threading easier, if we want to continue doing threading in the db (and I suspect we do, or else we'll end up sucking in all the messages in order to display the current thread pane). One big challenge will be to figure out how to avoid blocking the UI waiting for db lookups. If we do lookups on the UI thread, we'll need to be very careful. If we use sqlite asynchronously, we'll have to redo a lot of code that expects to be able to get msg hdrs synchronously. Caching can deal with the simple cases, but the edge cases (e.g., select all + delete) could be problematic.
We'd probably have one sqlite db per account, though gloda seems to be able to get everything into one db.