947
edits
Auto-tools/Projects/Signal From Noise/Execution2012: Difference between revisions
Jump to navigation
Jump to search
Auto-tools/Projects/Signal From Noise/Execution2012 (view source)
Revision as of 00:17, 18 December 2012
, 18 December 2012→Problems We Aimed to Solve with Datazilla
(→Execution of Signal from Noise: more paragraphy and rewording) |
|||
| Line 26: | Line 26: | ||
* Statistics should be self-evident: often, Talos+Graphserver and other statistical systems have been approached as a "black box": A number comes out that is "good" or "bad". However, this effectively leaves an interested developer in the dark as to where this number came from and discourages understanding the system and playing with data. Datazilla was designed to expose the statistics being used so that there are no mysteries here. | * Statistics should be self-evident: often, Talos+Graphserver and other statistical systems have been approached as a "black box": A number comes out that is "good" or "bad". However, this effectively leaves an interested developer in the dark as to where this number came from and discourages understanding the system and playing with data. Datazilla was designed to expose the statistics being used so that there are no mysteries here. | ||
* No need to update the database every time a test or machine changes. Unlike the maintenance nightmare that is the current data.sql in graphserver, the Datazilla schema should be dynamic in response to uploaded data. | * No need to update the database every time a test or machine changes. Unlike the maintenance nightmare that is the current data.sql in graphserver, the Datazilla schema should be dynamic in response to uploaded data. | ||
* Allow experimentation with statistics: While in practice, there will be a canonical manner to determine regressions and improvements, alternatives should be investigatable and swappable.And this can only be done by creating a system that stores all the raw data from the performance system. | * Allow experimentation with statistics: While in practice, there will be a canonical manner (or conceivably manners) to determine regressions and improvements, alternatives should be investigatable and swappable.And this can only be done by creating a system that stores all the raw data from the performance system. | ||
* Ability to utilize data from arbitrary performance suites, not just talos. Whatever we create next for performance analysis should be able to use Datazilla as a data storage and retrieval system. This way we can use Datazilla as a building block in our next performance automation task. | * Ability to utilize data from arbitrary performance suites, not just talos. Whatever we create next for performance analysis should be able to use Datazilla as a data storage and retrieval system. This way we can use Datazilla as a building block in our next performance automation task. | ||
* Datazilla should be able to be scalable enough to accumulate data per-push and generate a "regression/improvement" analysis for that push in real time. | * Datazilla should be able to be scalable enough to accumulate data per-push and generate a "regression/improvement" analysis for that push in real time. | ||