In order for Socorro to produce useful stack traces for crash reports, it needs debug symbols matching the libraries that are loaded in each report. Currently these symbols are all in the Breakpad text symbol format, but there is work in progress (bug 607831, symboltests, minidump-stackwalk, binary-symbols branch) to make them use the new binary format that is implemented in upstream Breakpad.
- 1 How Socorro locates symbols
- 2 How symbols get to the symbol server
- 3 Symbol Cleanup
- 4 Extra Details
How Socorro locates symbols
Currently, Socorro uses the stock minidump_stackwalk tool from upstream Breakpad to process minidumps into usable crash reports. minidump_stackwalk uses SimpleSymbolSupplier to locate symbols. SimpleSymbolSupplier simply looks for symbol files in a known directory structure given a list of symbol directories. Each library in a crash report has a "debug filename", which is simply the library filename everywhere except Windows, where it is the PDB filename, and a "debug identifier", which is a GUID+1 extra character that uniquely identifies a compiled version of a library. On Windows this is a GUID+age field that is baked into the binary by the linker. On other platforms this is a hash of certain parts of the file. The SimpleSymbolSupplier looks for symbol files in file paths of the form:
<debug filename>/<debug identifier>/<debug filename>.sym
How symbols get to the symbol server
Mozilla nightly/release builds
The buildsymbols target runs the platform-specific dump_syms tool (Linux, Mac, Windows) via the symbolstore.py script in order to generate a symbol package with the correct directory structure. In addition to Breakpad-formatted symbols, symbolstore.py includes platform-native debug symbols (PDB files on Windows, .dSYM files on Mac, stripped debug sections on Linux) so that users can use the symbol server to debug nightly and release builds.
The uploadsymbols target calls the upload_symbols.sh script, which scp's a symbol package in zip format to the symbol server, unpacks it, and runs a post-symbol-upload command.
Windows system symbols
PDB symbols containing function names and stack walking information are available for Windows system libraries from Microsoft's symbol server. The Breakpad source contains a utility library that can fetch these PDB files and convert them to the Breakpad symbol format. There is a small utility binary built using this library which we use in a Python script. The Python script gets its input from a Socorro cron job which runs a map/reduce job to gather a list of modules from Windows crash reports.
The script is somewhat complex, because the symbol fetching and conversion needs to happen on a Windows machine, and in Mozilla's production environment, this machine cannot write directly to the symbol store, so it has a read-only mount of the symbol store, and uploads the symbols using SCP.
Note: This is not currently being run in production, but only on an ad-hoc basis. It is expected to move to production in the Socorro 1.7.7 release See bug 598908.
OS X system symbols
There is no central source for OS X debug symbols, so I wrote a Python script that runs dump_syms on system libraries to get the exported function names and stack walking information, and uploads them to a small RESTful symbol server. This symbol server lives on a separate machine from the actual production symbol server, so there's also another script involved that uploads symbols from there to production.
The script itself is intended to crowdsource the gathering of symbols, but at present it's likely that I'm the only one running it on a regular basis.
Android system symbols
Because of the large amount of Android crashes happening in system libraries, I wrote an extension to gather symbols from Android system libraries: https://addons.mozilla.org/en-US/mobile/addon/android-symbol-sender/
The extension code is here: http://hg.mozilla.org/users/tmielczarek_mozilla.com/androidsymbols/
but it also uses a bunch of the Breakpad code for symbol dumping, compiled into a shared library loaded via js-ctypes. It also relies on a few Breakpad patches in my patch queue: http://hg.mozilla.org/users/tmielczarek_mozilla.com/mq/file/3d8e40b12dfe
most notably one that uses some libunwind code to parse the ARM unwind tables: http://hg.mozilla.org/users/tmielczarek_mozilla.com/mq/file/3d8e40b12dfe/arm-unwind
The extension submits symbols to the same REST symbol server used for OS X system symbols.
Linux system symbols
We currently have no plans for collecting Linux system symbols. The huge variety of distributions and library versions makes this a difficult problem.
Flash Player symbols
Currently Adobe sends us symbols for Windows and Mac Flash Player releases. More information is available on the intranet wiki.
There is a symbol cleanup script that runs as a cron job on the #Symbol upload server. Currently it never removes symbols from alpha/beta/release builds, and tries to keep 30 days worth of nightly symbols per product-branch-os grouping. The cleanup script works by parsing the #Symbol Index Files, so symbols not listed in index files will not be cleaned up.
Symbol upload server
All symbols are currently uploaded via SSH to symbolpush.mozilla.org, which is a server that exists for this express purpose. The actual symbol storage is a NFS mount that is exposed to that server. This hardware is located in Mozilla's Phoenix data center.
Symbol Index Files
Every symbol upload includes an index file which lists all the symbols that are contained within this upload. The format is:
The post-symbol-upload script
All upload processes run a post-symbol-upload script after uploading symbols. This script is currently a no-op, but it may change in the future if we decide to change the format in which symbols are stored.