Snappy Symbolication Server - Revision history

Ksteuber: Removed deprecated information and added a link to the up-to-date information

2016-12-09T00:54:39Z

Removed deprecated information and added a link to the up-to-date information

← Older revision		Revision as of 00:54, 9 December 2016
Line 1:		Line 1:
	The Snappy Symbolication Server is a Web server for symbolicating Firefox stacks. It matches PC addresses to modules in memory and looks up the corresponding function names in server-side symbol files (.SYM files).		The Snappy Symbolication Server is a Web server for symbolicating Firefox stacks. It matches PC addresses to modules in memory and looks up the corresponding function names in server-side symbol files (.SYM files).

	~~== Running ==~~		The documentation for Snappy no longer lives here. It can be found in the [https://github.com/mozilla/Snappy-Symbolication-Server/blob/master/README.md Readme on Snappy's GitHub]

	The ~~source code~~ for Snappy lives in the [https://github.com/mozilla/Snappy-Symbolication-Server ~~Mozilla Github repository]. Snappy runs at [http://www.python.org Python] 2.7 and depends on [http://www.tornadoweb.org Tornado] and~~
	~~[https://pypi.python.org/pypi~~/~~futures concurrent.futures] packages. The best way to install them is through [https:~~//~~pip~~.~~pypa.io pip]:~~

	~~:<code>pip install tornado futures</code>~~

	~~To run snappy~~ on ~~your machine, just type:~~

	~~:<code>python symbolicationWebService.py <configuration-file></code>~~

	~~The snappy repository contains two sample configuration files for Linux and Windows.~~

	~~<div class="warning">~~
	~~'''Use the sample-windows.ini file for symbolicating local builds, and sample-linux.ini when deploying to the official symbolapi.mozilla.org symbol server!'''~~
	~~</div>~~

	~~Summary of the config fields:~~

	~~{\| class="wikitable"~~
	~~! Section !! colspan=3 \| Fields~~
	\|-
	~~! !! Field !! Description~~
	\|-
	~~\| rowspan=3 \| General \|\| hostname \|\| Server address or name~~
	\|-
	~~\| portNumber \|\| Server port number~~
	\|-
	~~\| remoteSymbolServer \|\| The address of a secondary remote server to forward requests to~~
	\|-
	~~\| rowspan=1 \| MemoryCache \|\| maxMemCacheFiles \|\| Number of libraries to cache in RAM~~
	\|-
	~~\| rowspan=2 \| DiskCache \|\| diskCachePath \|\| Directory to store cached symbol files~~
	\|-
	~~\| maxDiskCacheFiles \|\| Maximum #of files in the disk cache directory~~
	\|-
	~~\| rowspan=4 \| Log \|\| maxFiles \|\| Maximum number of log files~~
	\|-
	~~\| maxFileSize \|\| Maximum size for a log file~~
	\|-
	~~\| logPath \|\| Path to log files~~
	\|-
	~~\| logLevel \|\| NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL~~
	\|-
	~~\| rowspan=3 \| SymbolPaths \|\| Firefox \|\| Path to search for symbols in the local disk. Set Firefox=obj-dir/dist/crashreporter-symbols/~~
	\|-
	~~\| Windows \|\| Leave this commented out (unless somehow you're using NFS symbols)~~
	\|-
	~~\| Thunderbird \|\| Point to your local .sym file path if you're doing Thunderbird development~~
	\|-
	~~\| rowspan=1 \| SymbolURLs \|\| MozillaS3 \|\| Each entry in SymbolURLs represents a remote path to search for symbols~~
	\|}

	~~== Contributing ==~~

	It's ~~a [https://github.com/mozilla/Snappy-Symbolication-Server/ Github~~ ] ~~project, just fork it and send a PR. If you want to ask something, you can find people involved with Snappy Server in the #perf channel of irc.mozilla.org (e.g. message user "vladan").~~

	~~== Project ideas ==~~
	* Snappy parses symbol files in another process (which we informally call "Symbolication Process") due to the well known [https://www.youtube.com/watch?v=Obt-vMVdM8s GIL problems]. We could have a Symbolication Process per CPU core, but we have a tricky problem here. The Symbolication Process maintains a in memory symbols cache, with the most recently used symbols. The problem is how we handle this, we could have a single shared cache among all Symbolication Processes, which would bring contention, hurting the code parallelism. Other approach is that each Symbolication Process could have its own memory cache, but we potentially could waste memory due to duplicated symbols among all processes, and we could duplicate work because all subprocesses would parse the same symbol file in case of several similar symbolication requests. One good solution is to maintain the memory cache in the parent process.

	* The Symbolication Process requests symbol files from S3. This is a I/O bound task, so this should happen in the main process, and then Snappy would use asynchronous I/O for that. The problem is that we have to send the symbol file to the Symbolication Process through IPC. The IPC overhead could kill the performance gain with asynchronous requests. We need performance numbers here to make a decision on what's the best approach.

	* Too bad we don't have unit tests.

	~~== Regressions tests ==~~

	~~Some regressions tests to perform before sending a PR:~~

	* Perform a get request to the server
	* Check if the IP in X-Forward-For is logged
	* Test requests with /gecko-profiler/ path
	* Test for /debug and /nodebug special paths
	* Test with compressed symbols files
	* Exiting with Control-C should kill the server smoothly
	* Check if the host handles ill-formed requests
	* Check if symbol server can forward requests
	* Parse a symbol file, exit and start the server again. The server should fetch the sym file from cache
	* Check if the server writes logs to files
	* Test requests forward

Vladan: /* Running */

2015-08-26T00:18:26Z

Running

← Older revision		Revision as of 00:18, 26 August 2015
Line 45:		Line 45:
	\| logLevel \|\| NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL		\| logLevel \|\| NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL
	\|-		\|-
	\| rowspan=3 \| SymbolPaths \|\| Firefox \|\| Path to search for symbols in the local disk. Set Firefox=~~<path to your local .sym files>~~		\| rowspan=3 \| SymbolPaths \|\| Firefox \|\| Path to search for symbols in the local disk. Set Firefox=obj-dir/dist/crashreporter-symbols/
	\|-		\|-
	\| Windows \|\| Leave this commented out (unless somehow you're using NFS symbols)		\| Windows \|\| Leave this commented out (unless somehow you're using NFS symbols)

Vladan: /* Contributing */

2015-08-25T23:35:15Z

Contributing

← Older revision		Revision as of 23:35, 25 August 2015
Line 56:		Line 56:
	== Contributing ==		== Contributing ==

	It is a [https://github.com Github ] project, just fork it and send a PR. If you want to ask something, you can find people involved with Snappy Server in the #perf channel at irc.mozilla.org.		It's a [https://github.com/mozilla/Snappy-Symbolication-Server/ Github ] project, just fork it and send a PR. If you want to ask something, you can find people involved with Snappy Server in the #perf channel of irc.mozilla.org (e.g. message user "vladan").

	== Project ideas ==		== Project ideas ==

Vladan: /* Running */

2015-08-25T23:34:23Z

Running

← Older revision		Revision as of 23:34, 25 August 2015
Line 12:		Line 12:
	:<code>python symbolicationWebService.py <configuration-file></code>		:<code>python symbolicationWebService.py <configuration-file></code>

	The snappy repository contains two sample configuration files for Linux and Windows. ~~Here is a summary~~ for the config fields:		The snappy repository contains two sample configuration files for Linux and Windows.

			<div class="warning">
			'''Use the sample-windows.ini file for symbolicating local builds, and sample-linux.ini when deploying to the official symbolapi.mozilla.org symbol server!'''
			</div>

			Summary of the config fields:

	{\| class="wikitable"		{\| class="wikitable"
	! Section !! colspan=2 \| Fields		! Section !! colspan=3 \| Fields
	\|-		\|-
	! !! Field !! Description		! !! Field !! Description
	\|-		\|-
	\| rowspan=6 \| General \|\| hostname \|\| Server address or name		\| rowspan=3 \| General \|\| hostname \|\| Server address or name
	\|-		\|-
	\| portNumber \|\| Server port number		\| portNumber \|\| Server port number
Line 25:		Line 31:
	\| remoteSymbolServer \|\| The address of a secondary remote server to forward requests to		\| remoteSymbolServer \|\| The address of a secondary remote server to forward requests to
	\|-		\|-
	\| ~~maxCacheEntries~~ \|\| ~~Number of entries for RAM symbol cache~~		\| rowspan=1 \| MemoryCache \|\| maxMemCacheFiles \|\| Number of libraries to cache in RAM
	\|-
	\| ~~mruSymbolStateFile~~ \|~~\| Json file with RAM cache info~~
	\|-
	~~\| maxMRUSymbolPersist \|\| Maximum number~~ of ~~symbols for~~ cache
	\|-		\|-
	\| rowspan=2 \| DiskCache \|\| ~~cachePath~~ \|\| ~~Path~~ to cached symbol files		\| rowspan=2 \| DiskCache \|\| diskCachePath \|\| Directory to store cached symbol files
	\|-		\|-
	\| ~~maxCacheFiles~~ \|\| Maximum ~~number~~ of files in the cache ~~dir~~		\| maxDiskCacheFiles \|\| Maximum #of files in the disk cache directory
	\|-		\|-
	\| rowspan=4 \| Log \|\| ~~MaxFiles~~ \|\| Maximum number of log files		\| rowspan=4 \| Log \|\| maxFiles \|\| Maximum number of log files
	\|-		\|-
	\| maxFileSize \|\| Maximum size for a log file		\| maxFileSize \|\| Maximum size for a log file
Line 43:		Line 45:
	\| logLevel \|\| NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL		\| logLevel \|\| NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL
	\|-		\|-
	\| SymbolPaths \|\| \|\| ~~Each entry represents a path~~ to search for symbols in the local disk		\| rowspan=3 \| SymbolPaths \|\| Firefox \|\| Path to search for symbols in the local disk. Set Firefox=<path to your local .sym files>
			\|-
			\| Windows \|\| Leave this commented out (unless somehow you're using NFS symbols)
			\|-
			\| Thunderbird \|\| Point to your local .sym file path if you're doing Thunderbird development
	\|-		\|-
	\| SymbolURLs \|\| \|\| Each entry represents a remote path to search for symbols		\| rowspan=1 \| SymbolURLs \|\| MozillaS3 \|\| Each entry in SymbolURLs represents a remote path to search for symbols
	\|}		\|}

Wcosta at 13:51, 17 August 2015

2015-08-17T13:51:38Z

← Older revision		Revision as of 13:51, 17 August 2015
Line 73:		Line 73:
	* Parse a symbol file, exit and start the server again. The server should fetch the sym file from cache		* Parse a symbol file, exit and start the server again. The server should fetch the sym file from cache
	* Check if the server writes logs to files		* Check if the server writes logs to files
			* Test requests forward

Wcosta at 11:25, 17 August 2015

2015-08-17T11:25:57Z

← Older revision		Revision as of 11:25, 17 August 2015
Line 58:		Line 58:

	* Too bad we don't have unit tests.		* Too bad we don't have unit tests.

	~~=== Symbol file cache proposal ===~~

	Snappy searches for symbol files on S3 when it doesn't find them locally. The proposal is to cache these files locally, with a LRU eviction policy. Instead of storing them in the original sym file, we store in the parsed format using [https://docs.python.org/2/library/pickle.html Pickle] module.

	The cache size is counted by the number of files stored, and must be accounted to keep some extra free space, since the eviction code runs in a timely fashion. Here is some pseudo Python code for file cache:

	~~<code>~~
	~~def Initialize():~~
	~~:for each file in the cache directory:~~
	~~::accessTime = AccessTime(filePath)~~
	~~::ListofFiles.append(FilePath)~~
	~~:LRU = ListOfFiles reverse sorted by access time~~
	~~</code>~~

	~~<code>~~
	~~def StoreCache(SymbolData, path):~~
	~~:writeFile(SymbolData, path)~~
	~~:LRU.append(path)~~
	~~</code>~~

	~~<code>~~
	~~def FetchCache(path):~~
	~~:SymbolData = readFile(path)~~
	~~:LRU.remove(path)~~
	~~:LRU.append(path)~~
	~~</code>~~

	~~<code>~~
	~~def Evict():~~
	~~:lruSize = len(LRU)~~
	~~:idealSize = maxSize * 0.6~~
	~~:while lruSize >= idealSize:~~
	~~::path = LRU.popleft()~~
	~~::removeFile(path)~~
	~~</code>~~

	== Regressions tests ==		== Regressions tests ==

Wcosta: /* Symbol file cache proposal */

2015-07-31T19:39:33Z

Symbol file cache proposal

← Older revision		Revision as of 19:39, 31 July 2015
Line 59:		Line 59:
	* Too bad we don't have unit tests.		* Too bad we don't have unit tests.

	= Symbol file cache proposal =		=== Symbol file cache proposal ===

	Snappy searches for symbol files on S3 when it doesn't find them locally. The proposal is to cache these files locally, with a LRU eviction policy. Instead of storing them in the original sym file, we store in the parsed format using [https://docs.python.org/2/library/pickle.html Pickle] module.		Snappy searches for symbol files on S3 when it doesn't find them locally. The proposal is to cache these files locally, with a LRU eviction policy. Instead of storing them in the original sym file, we store in the parsed format using [https://docs.python.org/2/library/pickle.html Pickle] module.

Wcosta: /* File cache proposal */

2015-07-31T19:39:03Z

File cache proposal

← Older revision		Revision as of 19:39, 31 July 2015
Line 58:		Line 58:

	* Too bad we don't have unit tests.		* Too bad we don't have unit tests.

			= Symbol file cache proposal =

			Snappy searches for symbol files on S3 when it doesn't find them locally. The proposal is to cache these files locally, with a LRU eviction policy. Instead of storing them in the original sym file, we store in the parsed format using [https://docs.python.org/2/library/pickle.html Pickle] module.

			The cache size is counted by the number of files stored, and must be accounted to keep some extra free space, since the eviction code runs in a timely fashion. Here is some pseudo Python code for file cache:

			<code>
			def Initialize():
			:for each file in the cache directory:
			::accessTime = AccessTime(filePath)
			::ListofFiles.append(FilePath)
			:LRU = ListOfFiles reverse sorted by access time
			</code>

			<code>
			def StoreCache(SymbolData, path):
			:writeFile(SymbolData, path)
			:LRU.append(path)
			</code>

			<code>
			def FetchCache(path):
			:SymbolData = readFile(path)
			:LRU.remove(path)
			:LRU.append(path)
			</code>

			<code>
			def Evict():
			:lruSize = len(LRU)
			:idealSize = maxSize * 0.6
			:while lruSize >= idealSize:
			::path = LRU.popleft()
			::removeFile(path)
			</code>

	== Regressions tests ==		== Regressions tests ==

Wcosta: /* Project ideas */

2015-07-30T11:41:17Z

Project ideas

← Older revision		Revision as of 11:41, 30 July 2015
Line 53:		Line 53:

	== Project ideas ==		== Project ideas ==
	* Snappy parses symbol files in another process (which we informally call "Symbolication Process") due to the well known [https://www.youtube.com/watch?v=Obt-vMVdM8s GIL problems].		* Snappy parses symbol files in another process (which we informally call "Symbolication Process") due to the well known [https://www.youtube.com/watch?v=Obt-vMVdM8s GIL problems]. We could have a Symbolication Process per CPU core, but we have a tricky problem here. The Symbolication Process maintains a in memory symbols cache, with the most recently used symbols. The problem is how we handle this, we could have a single shared cache among all Symbolication Processes, which would bring contention, hurting the code parallelism. Other approach is that each Symbolication Process could have its own memory cache, but we potentially could waste memory due to duplicated symbols among all processes, and we could duplicate work because all subprocesses would parse the same symbol file in case of several similar symbolication requests. One good solution is to maintain the memory cache in the parent process.
	We could have a Symbolication Process per CPU core, but we have a tricky problem here. The Symbolication Process maintains a in memory symbols cache, with the most recently used
	symbols. The problem is how we handle this, we could have a single shared cache among all Symbolication Processes, which would bring contention, hurting the code parallelism.
	Other approach is that each Symbolication Process could have its own memory cache, but we potentially could waste memory due to duplicated symbols among all processes, and we could
	duplicate work because all subprocesses would parse the same symbol file in case of several similar symbolication requests. One good solution is to maintain the memory cache in the parent process.

	* The Symbolication Process requests symbol files from S3. This is a I/O bound task, so this should happen in the main process, and then Snappy would use asynchronous I/O for that.		* The Symbolication Process requests symbol files from S3. This is a I/O bound task, so this should happen in the main process, and then Snappy would use asynchronous I/O for that. The problem is that we have to send the symbol file to the Symbolication Process through IPC. The IPC overhead could kill the performance gain with asynchronous requests. We need performance numbers here to make a decision on what's the best approach.
	The problem is that we have to send the symbol file to the Symbolication Process through IPC. The IPC overhead could kill the performance gain with asynchronous requests. We need performance
	numbers here to make a decision on what's the best approach.

	* Too bad we don't have unit tests.		* Too bad we don't have unit tests.

Wcosta: /* Project ideas */

2015-07-30T11:39:34Z

Project ideas

← Older revision		Revision as of 11:39, 30 July 2015
Line 53:		Line 53:

	== Project ideas ==		== Project ideas ==
			* Snappy parses symbol files in another process (which we informally call "Symbolication Process") due to the well known [https://www.youtube.com/watch?v=Obt-vMVdM8s GIL problems].
			We could have a Symbolication Process per CPU core, but we have a tricky problem here. The Symbolication Process maintains a in memory symbols cache, with the most recently used
			symbols. The problem is how we handle this, we could have a single shared cache among all Symbolication Processes, which would bring contention, hurting the code parallelism.
			Other approach is that each Symbolication Process could have its own memory cache, but we potentially could waste memory due to duplicated symbols among all processes, and we could
			duplicate work because all subprocesses would parse the same symbol file in case of several similar symbolication requests. One good solution is to maintain the memory cache in the parent process.

	~~TODO~~		* The Symbolication Process requests symbol files from S3. This is a I/O bound task, so this should happen in the main process, and then Snappy would use asynchronous I/O for that.
			The problem is that we have to send the symbol file to the Symbolication Process through IPC. The IPC overhead could kill the performance gain with asynchronous requests. We need performance
			numbers here to make a decision on what's the best approach.

			* Too bad we don't have unit tests.

	== Regressions tests ==		== Regressions tests ==