Auto-tools/Projects/Signal From Noise/Q2: Difference between revisions
< Auto-tools | Projects | Signal From Noise
(Created page with "= Overview = Continuing the https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise work done in Q1 to complete Signal From Noise. Now that we have an idea of how th...") |
|||
| (5 intermediate revisions by the same user not shown) | |||
| Line 11: | Line 11: | ||
* Send raw results from all talos runs to new graph server | * Send raw results from all talos runs to new graph server | ||
** Optimize data loading procedures, add bullet proof error handling for JSON format errors, and tune indexes if needed. | |||
** Start using my.cnf from the production db in the dev database | |||
** Validate the data, make sure no fields are truncated and the data types are correct | |||
* Have high level views in new graph server to achieve same functionality as existing graph server | * Have high level views in new graph server to achieve same functionality as existing graph server | ||
** To address this milestone, lets focus on three data view collections for the Developer, Sheriff, and Investigator use cases. These three use cases will be entries in the Navigation Menu | |||
*** Developer's Delight - Data view collection for developers, mimics the production graph server but adds data views for analyzing a pageset and its associated raw data values. | |||
*** Sheriff Duty - Data view collection for the Sheriff use case | |||
*** Investigator Indiscretions - Data view collection for an investigator, provides a comprehensive set of data views for analyzing talos data in a variety of ways. | |||
* Find 6 stakeholders who can participate for 30 minutes weekly to look at the new graph server and ensure it meets their needs | * Find 6 stakeholders who can participate for 30 minutes weekly to look at the new graph server and ensure it meets their needs | ||
* Do initial investigations of tsvg, tdhtml, ally, tscroll | * Do initial investigations of tsvg, tdhtml, ally, tscroll | ||
* Understand compare talos | * Understand compare talos | ||
** Decide on set of features we care about for this | ** Decide on set of features we care about for this | ||
*dev.tree-planning email | |||
** Decide on set of features we care about for this | |||
** We need to make sure we mimic all url based entry points or modify the software that generates urls to the new graph server. | |||
** The existing url structure looks like it uses database id's to identify product branch/talos test/platform combinations. The new database will have different ids for this reference information. We may need to map from one to the other for a period of time. | |||
== Milestone 2 == | == Milestone 2 == | ||
| Line 21: | Line 32: | ||
* Deploy tsvg and tdhtml with any recommendations | * Deploy tsvg and tdhtml with any recommendations | ||
* Deploy new graph server with basic functionality | * Deploy new graph server with basic functionality | ||
** Add TBPL like browsing | |||
** Add changeset filtering | |||
* Have new views in graph server for additional functionality | * Have new views in graph server for additional functionality | ||
** Add a page centric data view for analyzing a particular page's performance across all changesets for a given time interval. | |||
* Document variance calculation and how to track it | * Document variance calculation and how to track it | ||
** ideally we would want to track this monthly | ** ideally we would want to track this monthly | ||
Latest revision as of 17:21, 5 April 2012
Overview
Continuing the [work done in Q1] to complete Signal From Noise. Now that we have an idea of how this stuff works, lets do this for all tests and create the tools for people to find regressions easier.
Goals
- Deploy the new graph server in production
- Send raw results to graph server
- Create tools to reproduce and analyze local and buildbot runs
Milestone 1
- Send raw results from all talos runs to new graph server
- Optimize data loading procedures, add bullet proof error handling for JSON format errors, and tune indexes if needed.
- Start using my.cnf from the production db in the dev database
- Validate the data, make sure no fields are truncated and the data types are correct
- Have high level views in new graph server to achieve same functionality as existing graph server
- To address this milestone, lets focus on three data view collections for the Developer, Sheriff, and Investigator use cases. These three use cases will be entries in the Navigation Menu
- Developer's Delight - Data view collection for developers, mimics the production graph server but adds data views for analyzing a pageset and its associated raw data values.
- Sheriff Duty - Data view collection for the Sheriff use case
- Investigator Indiscretions - Data view collection for an investigator, provides a comprehensive set of data views for analyzing talos data in a variety of ways.
- To address this milestone, lets focus on three data view collections for the Developer, Sheriff, and Investigator use cases. These three use cases will be entries in the Navigation Menu
- Find 6 stakeholders who can participate for 30 minutes weekly to look at the new graph server and ensure it meets their needs
- Do initial investigations of tsvg, tdhtml, ally, tscroll
- Understand compare talos
- Decide on set of features we care about for this
- dev.tree-planning email
- Decide on set of features we care about for this
- We need to make sure we mimic all url based entry points or modify the software that generates urls to the new graph server.
- The existing url structure looks like it uses database id's to identify product branch/talos test/platform combinations. The new database will have different ids for this reference information. We may need to map from one to the other for a period of time.
Milestone 2
- Deploy tsvg and tdhtml with any recommendations
- Deploy new graph server with basic functionality
- Add TBPL like browsing
- Add changeset filtering
- Have new views in graph server for additional functionality
- Add a page centric data view for analyzing a particular page's performance across all changesets for a given time interval.
- Document variance calculation and how to track it
- ideally we would want to track this monthly
- Have compare talos and similar tools working with patches and in staging
Milestone 3
- Deploy ally and tscroll
- Deploy automatic variance calculation
- Deploy new compare talos
- Deploy new features in graph server