Snappy Symbolication Server

From MozillaWiki
Jump to: navigation, search

The Snappy Symbolication Server is a Web server for symbolicating Firefox stacks. It matches PC addresses to modules in memory and looks up the corresponding function names in server-side symbol files (.SYM files).


The source code for Snappy lives in the Mozilla Github repository. Snappy runs at Python 2.7 and depends on Tornado and concurrent.futures packages. The best way to install them is through pip:

pip install tornado futures

To run snappy on your machine, just type:

python <configuration-file>

The snappy repository contains two sample configuration files for Linux and Windows.

Use the sample-windows.ini file for symbolicating local builds, and sample-linux.ini when deploying to the official symbol server!

Summary of the config fields:

Section Fields
Field Description
General hostname Server address or name
portNumber Server port number
remoteSymbolServer The address of a secondary remote server to forward requests to
MemoryCache maxMemCacheFiles Number of libraries to cache in RAM
DiskCache diskCachePath Directory to store cached symbol files
maxDiskCacheFiles Maximum #of files in the disk cache directory
Log maxFiles Maximum number of log files
maxFileSize Maximum size for a log file
logPath Path to log files
SymbolPaths Firefox Path to search for symbols in the local disk. Set Firefox=obj-dir/dist/crashreporter-symbols/
Windows Leave this commented out (unless somehow you're using NFS symbols)
Thunderbird Point to your local .sym file path if you're doing Thunderbird development
SymbolURLs MozillaS3 Each entry in SymbolURLs represents a remote path to search for symbols


It's a Github project, just fork it and send a PR. If you want to ask something, you can find people involved with Snappy Server in the #perf channel of (e.g. message user "vladan").

Project ideas

  • Snappy parses symbol files in another process (which we informally call "Symbolication Process") due to the well known GIL problems. We could have a Symbolication Process per CPU core, but we have a tricky problem here. The Symbolication Process maintains a in memory symbols cache, with the most recently used symbols. The problem is how we handle this, we could have a single shared cache among all Symbolication Processes, which would bring contention, hurting the code parallelism. Other approach is that each Symbolication Process could have its own memory cache, but we potentially could waste memory due to duplicated symbols among all processes, and we could duplicate work because all subprocesses would parse the same symbol file in case of several similar symbolication requests. One good solution is to maintain the memory cache in the parent process.
  • The Symbolication Process requests symbol files from S3. This is a I/O bound task, so this should happen in the main process, and then Snappy would use asynchronous I/O for that. The problem is that we have to send the symbol file to the Symbolication Process through IPC. The IPC overhead could kill the performance gain with asynchronous requests. We need performance numbers here to make a decision on what's the best approach.
  • Too bad we don't have unit tests.

Regressions tests

Some regressions tests to perform before sending a PR:

  • Perform a get request to the server
  • Check if the IP in X-Forward-For is logged
  • Test requests with /gecko-profiler/ path
  • Test for /debug and /nodebug special paths
  • Test with compressed symbols files
  • Exiting with Control-C should kill the server smoothly
  • Check if the host handles ill-formed requests
  • Check if symbol server can forward requests
  • Parse a symbol file, exit and start the server again. The server should fetch the sym file from cache
  • Check if the server writes logs to files
  • Test requests forward