Snappy Symbolication Server
The Snappy Symbolication Server is a Web server for symbolicating Firefox stacks. It matches PC addresses to modules in memory and looks up the corresponding function names in server-side symbol files (.SYM files).
pip install tornado futures
To run snappy on your machine, just type:
python symbolicationWebService.py <configuration-file>
The snappy repository contains two sample configuration files for Linux and Windows.
Use the sample-windows.ini file for symbolicating local builds, and sample-linux.ini when deploying to the official symbolapi.mozilla.org symbol server!
Summary of the config fields:
|General||hostname||Server address or name|
|portNumber||Server port number|
|remoteSymbolServer||The address of a secondary remote server to forward requests to|
|MemoryCache||maxMemCacheFiles||Number of libraries to cache in RAM|
|DiskCache||diskCachePath||Directory to store cached symbol files|
|maxDiskCacheFiles||Maximum #of files in the disk cache directory|
|Log||maxFiles||Maximum number of log files|
|maxFileSize||Maximum size for a log file|
|logPath||Path to log files|
|logLevel||NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL|
|SymbolPaths||Firefox||Path to search for symbols in the local disk. Set Firefox=obj-dir/dist/crashreporter-symbols/|
|Windows||Leave this commented out (unless somehow you're using NFS symbols)|
|Thunderbird||Point to your local .sym file path if you're doing Thunderbird development|
|SymbolURLs||MozillaS3||Each entry in SymbolURLs represents a remote path to search for symbols|
It's a Github project, just fork it and send a PR. If you want to ask something, you can find people involved with Snappy Server in the #perf channel of irc.mozilla.org (e.g. message user "vladan").
- Snappy parses symbol files in another process (which we informally call "Symbolication Process") due to the well known GIL problems. We could have a Symbolication Process per CPU core, but we have a tricky problem here. The Symbolication Process maintains a in memory symbols cache, with the most recently used symbols. The problem is how we handle this, we could have a single shared cache among all Symbolication Processes, which would bring contention, hurting the code parallelism. Other approach is that each Symbolication Process could have its own memory cache, but we potentially could waste memory due to duplicated symbols among all processes, and we could duplicate work because all subprocesses would parse the same symbol file in case of several similar symbolication requests. One good solution is to maintain the memory cache in the parent process.
- The Symbolication Process requests symbol files from S3. This is a I/O bound task, so this should happen in the main process, and then Snappy would use asynchronous I/O for that. The problem is that we have to send the symbol file to the Symbolication Process through IPC. The IPC overhead could kill the performance gain with asynchronous requests. We need performance numbers here to make a decision on what's the best approach.
- Too bad we don't have unit tests.
Some regressions tests to perform before sending a PR:
- Perform a get request to the server
- Check if the IP in X-Forward-For is logged
- Test requests with /gecko-profiler/ path
- Test for /debug and /nodebug special paths
- Test with compressed symbols files
- Exiting with Control-C should kill the server smoothly
- Check if the host handles ill-formed requests
- Check if symbol server can forward requests
- Parse a symbol file, exit and start the server again. The server should fetch the sym file from cache
- Check if the server writes logs to files
- Test requests forward