Breakpad:Current Implementation
This is a summary of the current state of Breakpad integration into Mozilla.
Client Side
Currently Breakpad work in Mozilla has only progressed on Win32. An OS X implementation is available in the Breakpad source. Linux patches are in progress, but not yet available.
Symbol Extraction/Upload
Symbol extraction and upload is done at build time. There are two targets in the top level Makefile, buildsymbols and uploadsymbols.
The |buildsymbols| target finds PDB files associated with EXE and DLL files in $(DIST), and processes them with dump_syms.exe to get a SYM file for each, using the make_symbol_store.pl script. The SYM files and PDB files are both placed in $(DIST)/airbag-symbols/$(BUILDID) in the directory structure used for the Microsoft Symbol Server. A description of this directory structure can be found in the Breakpad source. The symbols are then compressed into a .tar.bz2 file, in a dated directory alongside the top source directory.
The |uploadsymbols| target simply calls the upload_symbols.sh script to upload the symbols using rsync. This script relies on certain environment variables being set:
- AIRBAG_SYMBOL_SERVER : host to upload symbols to
- AIRBAG_SYMBOL_USER : username on that host
- AIRBAG_SYMBOL_PATH : path on that host to put symbols in
The symbols are copied to the symbol server and untarred into the specified path.
Exception Handler
The exception handler is set in XRE_Main by calling SetAirbagExceptionHandler(). The exception handler is not actually installed unless the MOZ_AIRBAG environment variable is set to a non-zero value. Once a profile has been found, a "minidumps" directory is located or created in the profile directory, and this path is passed to SetAirbagMinidumpPath() as the destination for minidump files from the exception handler.
Upon catching an exception, the handler attempts to execute crashreporter.exe from the same directory as the current executable. The full path to the minidump file is passed in as a command line parameter. The application then exits.
Crash Reporter Client
The crash reporter client is currently capable of enabling/disabling crash reporting, and submitting minidumps to a server for processing.
If run with no parameters, the client simply allows the user to enable or disable crash reporting.
If run with parameters, the client first checks the Windows registry to see if it has been disabled. It looks for the value "Enabled" under the key "Software\Mozilla\Crash Reporter" in both the HKEY_LOCAL_MACHINE and HKEY_CURRENT_USER roots. If the value exists and is not set to "1", then the client simply exits. If the value exists and is set to "1" the client proceeds to upload the minidump file. If the minidump is uploaded successfully, then the file is deleted locally. The client then displays a message box indicating whether the minidump was successfully submitted or not.
The crash reporter client follows the lead of the updater software and is potentially localizable by translating strings in the crashreporter.ini file. Unfortunately, currently the URL of the server to send crash reports to is also stored in the crashreporter.ini file.
Server Side
The server side consists of several parts.
Symbol Store
The symbol store is simply a flat file store with a directory structure mirroring that of the Microsoft Symbol Server. See above for a link to the Breakpad source describing the structure. The benefit of this is that if we preserve PDB files in this same directory structure, and make the symbol store accessible via HTTP, we can make Windows release builds debuggable. Building on this, in the future, we may even be able to provide source level debugging of release builds.
You can see an example symbol store here. Note that there is no provision for deleting outdated symbols currently, although the symbol upload script does create a text file listing the symbols contained in each build.
Crash Report Collector
Our server code is named Socorro and lives in its own Google code project. It is implemented in Python on top of Pylons. Rob Sayre has some more info on his blog.
The collector currently uses a MySQL database to store crash reports. Crash reports are processed as they are submitted, which is probably a poor choice as they may take over 10 seconds when provided with full symbols. Minidumps are passed to the minidump_stackwalk program provided by Breakpad, which supports a "-m" command line flag to get machine-readable output.
Crash Report Viewer
The crash report viewer is probably the least-well-implemented portion of the entire system. It is very minimal, allowing one to see all crash reports that have been submitted, and to click on a particular crash report to see the details, including a stack trace. You can see a sample here.