947
edits
Auto-tools/Projects/Signal From Noise/Execution2012: Difference between revisions
Jump to navigation
Jump to search
Auto-tools/Projects/Signal From Noise/Execution2012 (view source)
Revision as of 00:08, 18 December 2012
, 18 December 2012→Execution of Signal from Noise: reword, paragraph
(→Execution of Signal from Noise: punctuation) |
(→Execution of Signal from Noise: reword, paragraph) |
||
| Line 11: | Line 11: | ||
== Execution of Signal from Noise == | == Execution of Signal from Noise == | ||
In general, it was initially expected that some back-of-the-envelope analysis of existing Talos numbers and some (again) back-of-the-envelope engineering implementations of statistics that at least appeared (though | |||
In general, it was initially expected that some back-of-the-envelope analysis of existing Talos numbers and some (again) back-of-the-envelope engineering implementations of statistics that at least appeared (though non-rigorously) less noisy would satisfy the goals of the Signal from Noise project for the time being, with optimistically ongoing effort being invested in analysis of performance data. The initial SfN effort was scheduled for a single quarter. This proved rather optimistic. (See also: http://k0s.org/mozilla/blog/20120829151007 .) | |||
In practice, because of the way averaging was split between talos and graphserver, it was effectively impossible to utilize the current system to develop more robust statistics. It quickly became apparent that we needed a system that preserved the raw measurements in order to allow us to use and compare the fidelity of different statistical models on the Talos data. The Talos test harness itself should not be crunching numbers: it should limit its role to measuring them and reporting them. | In practice, because of the way averaging was split between talos and graphserver, it was effectively impossible to utilize the current system to develop more robust statistics. It quickly became apparent that we needed a system that preserved the raw measurements in order to allow us to use and compare the fidelity of different statistical models on the Talos data. The Talos test harness itself should not be crunching numbers: it should limit its role to measuring them and reporting them. | ||
Work was devoted to the creation of a graphserver replacement that would make it possible to perform regression and improvement detection per-push. This is the datazilla project: https://wiki.mozilla.org/Auto-tools/Projects/Datazilla . The decision to write a new piece of infrastructure versus refining graphserver was not taken lightly. While for any new software there will be unknown (but non-incidental) sinks of time, there was little existing code in graphserver to use as a foundation for the problems we cared about solving. | Work was devoted to the creation of a graphserver replacement that would make it possible to perform regression and improvement detection per-push. This is the datazilla project: https://wiki.mozilla.org/Auto-tools/Projects/Datazilla . The decision to write a new piece of infrastructure versus refining graphserver was not taken lightly. While for any new software there will be unknown (but non-incidental) sinks of time, there was little existing code in graphserver to use as a foundation for the problems we cared about solving. | ||