Breakpad:Current Implementation
This is a summary of the current state of Breakpad integration into Mozilla.
Client Side
Breakpad crash reporting is currently enabled by default on trunk builds of Firefox on Windows, Mac OS X and Linux.
Symbol Extraction/Upload
Symbol extraction and upload is done at build time. There are two targets in the top level Makefile, buildsymbols and uploadsymbols.
Exception Handler
The exception handler is set in XRE_Main, so any app using that as an entry point can use it simply by setting some values in application.ini. See the Firefox application.ini for an example.
There is an API for adding extra data to the crash report, implemented using the nsICrashReporter interface. Note that the server must be modified to handle additional data.
Upon handling an exception, a minidump is written, and then the data added using annotateCrashReport is written to a file with a .extra extension alongside the minidump. The crash reporter client is then launched with the minidump filename as the sole parameter.
Crash Reporter Client
The crash reporter client will display the contents of the .extra file for the user to view, and allow the user to submit a report. The client can also restart the crashed application. The client stores reports in a directory structure explained here. Reports that have not yet been submitted are stored in pending, and successful reports have a text file with information about the report placed in submitted.
Server
Information about the server can be found at the Socorro Google Code page. The Mozilla instance is located at http://crash-stats.mozilla.com/ .
Server Side
The server side consists of several parts.
Symbol Store
The symbol store is simply a flat file store with a directory structure mirroring that of the Microsoft Symbol Server. See above for a link to the Breakpad source describing the structure. The benefit of this is that if we preserve PDB files in this same directory structure, and make the symbol store accessible via HTTP, we can make Windows release builds debuggable. Building on this, in the future, we may even be able to provide source level debugging of release builds.
You can see an example symbol store here. Note that there is no provision for deleting outdated symbols currently, although the symbol upload script does create a text file listing the symbols contained in each build.
Crash Report Collector
Our server code is named Socorro and lives in its own Google code project. It is implemented in Python on top of Pylons. Rob Sayre has some more info on his blog.
The collector currently uses a MySQL database to store crash reports. Crash reports are processed as they are submitted, which is probably a poor choice as they may take over 10 seconds when provided with full symbols. Minidumps are passed to the minidump_stackwalk program provided by Breakpad, which supports a "-m" command line flag to get machine-readable output.
Crash Report Viewer
The crash report viewer is probably the least-well-implemented portion of the entire system. It is very minimal, allowing one to see all crash reports that have been submitted, and to click on a particular crash report to see the details, including a stack trace. You can see a sample here. Socorro does something similar right now.