947
edits
Auto-tools/Projects/Signal From Noise/Execution2012: Difference between revisions
Jump to navigation
Jump to search
Auto-tools/Projects/Signal From Noise/Execution2012 (view source)
Revision as of 00:09, 18 December 2012
, 18 December 2012→Problems We Aimed to Solve with Datazilla: correct link
(→Execution of Signal from Noise: reword, paragraph) |
(→Problems We Aimed to Solve with Datazilla: correct link) |
||
| Line 19: | Line 19: | ||
=== Problems We Aimed to Solve with Datazilla === | === Problems We Aimed to Solve with Datazilla === | ||
* Preserve and capture raw performance numbers. The Talos test framework is a bad place to do statistics, because if you do any averaging before uploading the results then the ability to retrieve the original data is forever lost. Instead, datazilla should take in all raw values from talos and provide a central platform for regression/improvement detection and statistical study | * Preserve and capture raw performance numbers. The Talos test framework is a bad place to do statistics, because if you do any averaging before uploading the results then the ability to retrieve the original data is forever lost. Instead, datazilla should take in all raw values from talos and provide a central platform for regression/improvement detection and statistical study | ||
* Reduce the granularity of Talos from a page set to a single page: http://k0s.org/mozilla/blog | * Reduce the granularity of Talos from a page set to a single page: http://k0s.org/mozilla/blog/20120425093346 ; statistics and regressions should be dealt with on a per-page basis, as pages may have wildly different performance values; see also https://wiki.mozilla.org/Metrics/Talos_Investigation#Unrolling_Talos | ||
* Establish a full, extensible RESTful interface to the data: Datazilla's data and statistical methods should be accessible by all developers and the tools they wish to create and modify to use the data. | * Establish a full, extensible RESTful interface to the data: Datazilla's data and statistical methods should be accessible by all developers and the tools they wish to create and modify to use the data. | ||
* Statistics should be self-evident: often, Talos+Graphserver and other statistical systems have been approached as a "black box": A number comes out that is "good" or "bad". However, this effectively leaves an interested developer in the dark as to where this number came from and discourages understanding the system and playing with data. Datazilla was designed to expose the statistics being used so that there are no mysteries here. | * Statistics should be self-evident: often, Talos+Graphserver and other statistical systems have been approached as a "black box": A number comes out that is "good" or "bad". However, this effectively leaves an interested developer in the dark as to where this number came from and discourages understanding the system and playing with data. Datazilla was designed to expose the statistics being used so that there are no mysteries here. | ||