From MozillaWiki
Jump to: navigation, search

Implement mbox->maildir converter for Thunderbird, Google Summer of Code 2016.


Currently Thunderbird uses mbox as primary storage format for mails (locally). All the mails in a folder are stored in the same file with the same name as the folder and no extension. The messages are separated from each other by a "From - <something>" line. In the maildir format each message is stored in a separate file. The maildir format is also supported, but there is no converter for it yet and the storage format of an account cannot be changed from the user interface. The menu on server settings of an account to change the storage format is currently disabled. This project enables the menu to select the desired storage format and implements a component to migrate a store from the mbox format to maildir format or from the maildir format to the mbox format.

Link to the bitbucket repository

comm-central-v2 is my repository on bitbucket which I cloned from - Mozilla's repository for Thunderbird, SeaMonkey, and Lightning code - to work on during GSoC.

Link to the commits made during GSoC period

The default branch on comm-central-v2 is in sync with Mozilla's comm-central default and the 'converter' branch is the work that I did during GSoC. So the 'converter' branch holds the commits that were made to implement the mbox->maildir converter. b1fa56a is the id of the final commit.

Link to bug 856087 on bugzilla

The final diff ie the diff between the converter branch and the default branch of comm-central-v2 is submitted to bug 856087 (which is about implementing the converter) in bugzilla where it's being reviewed.

Screenshots of the converter in action

Weekly Updates

Week 1:

  1. Enabled the store type drop down menu.
  2. Detect change in store type. Whenever there is a change in the store type a migration dialog modal appears.
  3. The migration dialog modal currently has a temporary title and text.
  4. There are 2 buttons - Yes ( to proceed with the migration ) and No ( to cancel the migration ).
  5. For now, pressing Yes creates a folder "Converter" in the tmp directory with the folder hierarchy of the account root folder and copies over the files in their corresponding places from the account root folder.
  6. Pressing No closes the migration dialog modal.

Week 2:

  1. On pressing 'No' or the close button, the migration dialog modal closes and the store type is reverted to the original store type.
  2. On pressing 'Yes' a folder "Converter" is created in the tmp directory with the folder hierarchy of the account root folder and the mbox files are parsed and corresponding maildir files are created in proper places in the "Converter" folder.
  3. Check if migration is complete and on completion, the migration dialog modal closes with an alert message 'done'.

Week 3:

  1. On pressing 'Yes' a progress bar appears and the warning messages disappear.
  2. The max value of progress bar is equal to the number of messages in the account.
  3. The value of progress bar increases as messages are copied over.
  4. When the value of progress bar is equal to the max value ie the migration is complete, the migration dialog modal closes with an alert message 'done'.

Week 4:

  1. The .msf and the .dat files get copied over as part of migration.
  2. For imap accounts max value of progress bar is equal to the no. of files and folders within the account root folder.
  3. For imap accounts value of progress bar increases by 1 for each file or folder that is encountered as the converter moves through the account root folder.

Week 5:

  1. Before creating the "Converter" folder, if there is already a folder with the same name in the tmp directory, then the already existing folder is first removed.
  2. The name of the "Converter" folder is "Converter-" followed by the account root folder name.
  3. After the migration is complete, if there is already a folder with the same name in the parent directory of account root folder, then the already existing folder is first removed.
  4. Then the "Converter" folder is moved to the parent directory of account root folder.
  5. The following prefs are appropriately changed to make the "Converter" folder the new account root folder -

6.Thunderbird restarts and the new messages are added to the new account root folder.

Week 6:

  1. Fixed the style issues.
  2. Created a test for the converter.
  3. The test currently works for a pop account.
  4. Made changes to the converter modal dialog box.

Week 7:

  1. Created a jsm module to carry out the conversion and separated the conversion process from the user interface and the test.
  2. The jsm file exports function convertMailStoreTo that expects mailstoreContractId, account nsIMsgIncomingServer and an event target and returns a promise which gets resolved with the path of the new account root folder when the conversion is complete.
  3. convertMailStoreTo function dispatches event "progress" on the event target every time a message is copied for a pop account or every time a file or folder is copied for an imap account.
  4. The test and the user interface js file import the jsm and call convertMailStoreTo function.
  5. Made changes to variable and function names and comments.
  6. Made changes to converter modal dialog box.

Week 8:

  1. Made Thunderbird go offline before starting the conversion and go online after completion.
  2. Compact folders for an imap account before starting the conversion.
  3. Made changes to entity names in converterDialog.dtd.
  4. Made finish button appear and cancel button disappear on the modal on completion instead of window.alert("done").
  5. Disabled store type menu list after completion.
  6. Made converterWorker.js post proper messages on error and rejected promise with proper reasons in case of an error.
  7. Made converterDialog.js revert the prefs and rootFolder file path to original values in case of error.
  8. Made the test check if files and folders are actually created in the right places.

Week 9:

  1. Thunderbird restarts automatically after conversion after displaying a conversion done and restart needed message on the modal.
  2. Added test for imap account.
  3. Error handling with localized error messages.
  4. Passing percent conversion done from jsm to the tests and to converterDialog.js.

Week 10:

  1. Handled conversion of deferred accounts.
  2. Handled conversion of accounts to which other accounts are deferred.
  3. Added a test for deferred accounts.
  4. Fixed modal scrollbar issues.

Week 11:

  1. Enable store type menu for local folders.
  2. Do not close modal on error.
  3. Changed error codes.
  4. Made changes to tests.
  5. Made changes to messages displayed on modal.
  6. Changed log level to DEBUG.
  7. Implemented maildir to mbox conversion.
  8. Fixed size issues with modal.

Week 12:

  1. Implement conversion of nntp and movemail accounts.
  2. Changed Converter folder name from "Converter-accountname" to "accountname-maildir".
  3. Use worker's onerror error handler for error handling and remove try catch blocks from worker.
  4. Terminate all workers on error.
  5. Use OS.Path.join for path manipulations.
  6. Use OS.File.makeDir with from option to create all directories required.