Breakpad:Current Implementation

« back to Breakpad main page

This is a summary of the current state of Breakpad integration into Mozilla.

Client Side

Currently Breakpad work in Mozilla has only progressed on Win32. An OS X implementation is available in the Breakpad source. Linux patches are in progress, but not yet available.

Symbol Extraction/Upload

Symbol extraction and upload is done at build time. There are two targets in the top level Makefile, buildsymbols and uploadsymbols.

The |buildsymbols| target finds PDB files associated with EXE and DLL files in $(DIST), and processes them with dump_syms.exe to get a SYM file for each, using the make_symbol_store.pl script. The SYM files and PDB files are both placed in $(DIST)/crashreporter-symbols/$(BUILDID) in the directory structure used for the Microsoft Symbol Server. A description of this directory structure can be found in the Breakpad source. The symbols are then compressed into a zip file, in a dated directory alongside the top source directory. A list of the symbol files that were created are stored in a text file inside the archive, named $(MOZ_APP_NAME)-$(MOZ_APP_VERSION)-$(OS_ARCH)-$(BUILDID)-symbols.txt, for example "firefox-3.0a4-WINNT-2007032801-symbols.txt".

The |uploadsymbols| target simply calls the upload_symbols.sh script to upload the symbols using scp. This script relies on certain environment variables being set:

• SYMBOL_SERVER_HOST  : host to upload symbols to
• SYMBOL_SERVER_USER  : username on that host
• SYMBOL_SERVER_PATH  : path on that host to put symbols in

The symbols are copied to the symbol server and unzipped into the specified path.

Exception Handler

The exception handler is set in XRE_Main by calling SetExceptionHandler(). The exception handler is not actually installed unless the MOZ_AIRBAG environment variable is set to a non-zero value. The code in XRE_Main then sets some additional data (currently Vendor, ProductName, Version and BuildID) by calling AnnotateCrashReport. Once a profile has been found, a "minidumps" directory is located or created in the profile directory, and this path is passed to SetMinidumpPath() as the destination for minidump files from the exception handler.

Upon catching an exception, the handler writes the additional data set using AnnotateCrashReport out to a file with the same base name as the minidump, but with a ".extra" extension. attempts to execute crashreporter.exe from the same directory as the current executable. The full path to the minidump file is passed in as a command line parameter. The application then exits.

Crash Reporter Client

The crash reporter client is currently capable of enabling/disabling crash reporting, and submitting minidumps to a server for processing.

If run with no parameters, the client simply allows the user to enable or disable crash reporting.

If run with parameters, the client first checks the Windows registry to see if it has been disabled. It looks for the value "Enabled" under the key "Software\Mozilla\Crash Reporter" in both the HKEY_LOCAL_MACHINE and HKEY_CURRENT_USER roots. If the value exists and is not set to "1", then the client simply exits. If the value exists and is set to "1" the client proceeds to upload the minidump file. If the minidump is uploaded successfully, then the file is deleted locally. The client then displays a message box indicating whether the minidump was successfully submitted or not.

The crash reporter client follows the lead of the updater software and is potentially localizable by translating strings in the crashreporter.ini file. Unfortunately, currently the URL of the server to send crash reports to is also stored in the crashreporter.ini file.

Win32 trunk nightlies currently include the airbag code, but it's disabled at runtime by default. To test it, you will need to set the environment variable MOZ_AIRBAG=1. Crash reports are currently collected at: http://mavra.perilith.com/~luser/airbag-collector/list.py

Server Side

The server side consists of several parts.

Symbol Store

The symbol store is simply a flat file store with a directory structure mirroring that of the Microsoft Symbol Server. See above for a link to the Breakpad source describing the structure. The benefit of this is that if we preserve PDB files in this same directory structure, and make the symbol store accessible via HTTP, we can make Windows release builds debuggable. Building on this, in the future, we may even be able to provide source level debugging of release builds.

You can see an example symbol store here. Note that there is no provision for deleting outdated symbols currently, although the symbol upload script does create a text file listing the symbols contained in each build.

Crash Report Collector

Our server code is named Socorro and lives in its own Google code project. It is implemented in Python on top of Pylons. Rob Sayre has some more info on his blog.

The collector currently uses a MySQL database to store crash reports. Crash reports are processed as they are submitted, which is probably a poor choice as they may take over 10 seconds when provided with full symbols. Minidumps are passed to the minidump_stackwalk program provided by Breakpad, which supports a "-m" command line flag to get machine-readable output.

Crash Report Viewer

The crash report viewer is probably the least-well-implemented portion of the entire system. It is very minimal, allowing one to see all crash reports that have been submitted, and to click on a particular crash report to see the details, including a stack trace. You can see a sample here. Socorro does something similar right now.