947
edits
Auto-tools/Projects/Signal From Noise/StatusNovember2012: Difference between revisions
Jump to navigation
Jump to search
m
Auto-tools/Projects/Signal From Noise/StatusNovember2012 (view source)
Revision as of 21:48, 26 December 2012
, 26 December 2012→Conclusion: formatting
(→Performance Testing Roadmap: 2013: bug #s) |
m (→Conclusion: formatting) |
||
| Line 62: | Line 62: | ||
= Conclusion = | = Conclusion = | ||
In the last year, we've dug into every part of the performance testing automation at Mozilla. We have analyzed the test harness, the reporting tools, and the statistical soundness of the results that were being generated. Over the course of that year, we used what we learned to make the Talos framework easier to maintain, easier to run, simpler to set up, easier to test on try, and less error prone.We have created Datazilla, an extensible system for storing and retriving all our performance metrics from Talos and any future performance automation. We have rebooted our performance statistical analysis and created statistically viable, per-push regression/improvement detection. We have made all these systems easier to use and more open so that any contributor anywhere can take a look at our code and even experiment with new methods of statistical analysis on our performance data. | In the last year, we've dug into every part of the performance testing automation at Mozilla. We have analyzed the test harness, the reporting tools, and the statistical soundness of the results that were being generated. Over the course of that year, we used what we learned to make the Talos framework easier to maintain, easier to run, simpler to set up, easier to test on try, and less error prone. | ||
We have created Datazilla, an extensible system for storing and retriving all our performance metrics from Talos and any future performance automation. We have rebooted our performance statistical analysis and created statistically viable, per-push regression/improvement detection. We have made all these systems easier to use and more open so that any contributor anywhere can take a look at our code and even experiment with new methods of statistical analysis on our performance data. | |||
But, we're not finished yet. There are more fixes to be done to the Talos framework itself. And the most critical piece of the infrastructure move still has to take place. We have to shift to using Datazilla in production and deprecating our use of Graphserver for new versions of Firefox. As we do that, we can clean out the remaining cruft in the Talos test framework, and focus our efforts on new ground breaking performance automation.Stay tuned. Or better yet, get involved: https://wiki.mozilla.org/Auto-tools#Want_to_Help.3F | But, we're not finished yet. There are more fixes to be done to the Talos framework itself. And the most critical piece of the infrastructure move still has to take place. We have to shift to using Datazilla in production and deprecating our use of Graphserver for new versions of Firefox. As we do that, we can clean out the remaining cruft in the Talos test framework, and focus our efforts on new ground breaking performance automation.Stay tuned. Or better yet, get involved: https://wiki.mozilla.org/Auto-tools#Want_to_Help.3F | ||