User talk:Rtanglao/DRAFT-Compacting-article

From MozillaWiki
Jump to: navigation, search

Compacting

Background

Compacting is a Thunderbird housekeeping process, required because of the way the mbox (Berkley mailbox format) message storage system works that Thunderbird uses to store mails. Compacting should really not be noticeable, it should do its work quietly in the background and never be seen or mentioned. But unfortunately it is a process that impacts Thunderbird quite heavily while it is occurring. So instead of it being a task we never hear about, it gets mentioned far too often, both as a process that slows the computer and one that can cause mail to be lost.

What is this house keeping process?

When you delete or move a mail in Thunderbird some things that most people assume happen don't. First the mail is not moved. It is copied to the new location and marked as deleted. Second deleted mail is not deleted, it is marked as deleted and a copy made in the deleted mail folder.

What this means is that after you delete a message you now have an folder with a message in it that Thunderbird does not show you. It is still there occupying space and slowing searches and folder access, you just can't see it. You also have a nice new copy of that mail in the Deleted messages folder. While this is not a major issue in the short term, after all what is one message more of less, it does become a huge issue over time as the number of hidden messages starts to approach or even exceed the ones you have in the folder. Compacting is simply Thunderbird taking out the trash and getting rid of that hidden message.

What does compacting actually do?

Compacting is a garbage removal routine. It identifies the garbage that has been left in files through normal usage and removes it. Compacting does not compress or Zip your existing mail. It does nothing to save space other than act as a garbage collector and remove messages from the message store that you have already disposed of.

How is this done.

To compact a folder, Thunderbird opens your existing folder (say inbox.) Based on the rules for the MBOX mail format it reads the file one message at a time. If the message is still current it writes to to a new file, if it is marked as deleted it skips that message and moves onto the next message. This process is repeated one message at a time until the end of the file is reached. Then and only then the original message storage file is deleted and the new one replaces it. This is followed by the generation of an index for this message file. As many of the messages in the file have moved location, the original index is out of date and must be replaced .

If for some reason this process is interrupted, Thunderbird will add the compacted version of the file to the list of folders under local folder as a folder named NSTMP (if there is already one there a sequential number is added. So NSTMP1,NSTMP2 etc could occur. These files are saved as a backup in the event that whatever caused the interruption also lead to some sort of data loss. They are almost always a duplicate of the original folder and can be safely deleted at any time.

Problems following compacting

The compacting process does a number of things that do not occur in normal usage of Thunderbird, The most important from the user perspective is that it actually rebuilds the index for the mail stored in that file. Rebuilding the index file may be slow and may makes using the program difficult while it occurs. The result is that this rebuilding is something that is really only done at the time of compacting. This rebuilding from scratch sometimes returns a list of mail that is not what we saw before compacting.

Folder is blank

Problem: This is caused when the process of building the new index is interrupted and stopped. Before any valid index is generated

Cause: The usual cause is the file being locked by the anti virus program which is trying to scan this new file

Solution: Right click the folder, select properties and then the repair button. This will force Thunderbird to build an index for the folder and once it has done so it will display the contents correctly

Folder missing lots of message

Problem: This to is caused by the index building process being interrupted, but in this case the process results in a partial index

Cause: Attempting to open the folder before the index has been complete. While clicking directly on the folder does not display content sometime changing to this folder from another will display the partial index and abort the completion of the process.

Solution: Right click the folder, select properties and then the repair button. This will force Thunderbird to build an index for the folder and once it has done so it will display the contents correctly.

Messages after a certain date are missing

Problem: This is almost always a complete loss of messages after a certain date due to folder corruption.

Cause: This has a number of possible causes:

  • Underlying file has been corrupted by the incorrect removal of an email that your anti virus program has identified as containing a virus.
  • File system error in your hard disk
  • Failing hard disk

Solution: There is no recovering from this kind of issue, short of restoring from a backup. Excluding the Thunderbird profile, or at least the mail folders from the folders scanned by your anti-virus will significantly reduce the risks of this occurring.

Do not use system restore

Making Compaction less frequent.

Until version 5 of Thunderbird, compaction was set to occur when the space occupied by junk in the mail store exceeded 100Kb, with the release of Version 5 Thunderbird's default was changed to 20Mb. This huge change was prompted by the fact that a very large percentage of mails actually exceed the limit set at 100Kb so deleting a single mail triggered a compaction process, and this had serious effects on performance.

This default can easily be modified, and most people should consider doing so. Having the threshold set high enough to make compacting a daily event would probably be ideal. If you get lots of mails which you then file to another folder that contain videos you might need to increase this 20Mb to a higher figure.

If Thunderbird appears determined to compact every time you delete something or every few deletions it might be that a compact is not actually getting to completion. This can occur if you have an IMAP account that pushes mail to you when it is received (as opposed to Thunderbird going and asking if there is new mail), or you have the timing to collect mail set very low. As mentioned earlier scanning of the files by your anti virus program can also cause an abort of the process. In this case you may need to follow the following procedure to ensure that a compact does actually occur

On the file menu, select offline and then work offline. Allow any download or sync to occur. On the View menu, select folders and then ALL From the file menu select Compact Folders Go make coffee, this might take a while. Particularly if you have lots of mail In the bottom left hand corner of your Thunderbird screen eventually a Done Compacting message will appear. On the View menu, select folders and set it back to the original value

On the file menu, select offline and then work offline to bring Thunderbird back into it's online state.

I don't want to compact anything

You can certainly turn off the compaction process in the options for Thunderbird, but this does have it's own inherent problems. These are two fold:

  • Old deleted mail may contain malware that your anti virus program does not see as such today, but a full system scan reveals. This could see your mail folder corrupted by the anti virus program as it attempts to excise the mail from the store, or it could see the whole folder disappear completely as the anti virus program quarantines the whole file
  • Huge mail storage files that grow up over a very long time will eventually degrade the performance of Thunderbird as the manipulation of these files and the seeking within them takes more and more time the larger they grow.

Problems recommending Compacting

Problem: The folder inbox [or other named folder] is full

Cause: Historically Thunderbird had a limit of 4Gb on the size of folder. This limit has been removed in recent versions, but those who have older versions of Thunderbird may see this error message.

Solution: Upgrade to a current version of Thunderbird which does not have this limitation.