Auto-tools/Projects/Signal From Noise/Q2

From MozillaWiki
Jump to: navigation, search

Overview

Continuing the [work done in Q1] to complete Signal From Noise. Now that we have an idea of how this stuff works, lets do this for all tests and create the tools for people to find regressions easier.

Goals

  • Deploy the new graph server in production
  • Send raw results to graph server
  • Create tools to reproduce and analyze local and buildbot runs

Milestone 1

  • Send raw results from all talos runs to new graph server
    • Optimize data loading procedures, add bullet proof error handling for JSON format errors, and tune indexes if needed.
    • Start using my.cnf from the production db in the dev database
    • Validate the data, make sure no fields are truncated and the data types are correct
  • Have high level views in new graph server to achieve same functionality as existing graph server
    • To address this milestone, lets focus on three data view collections for the Developer, Sheriff, and Investigator use cases. These three use cases will be entries in the Navigation Menu
      • Developer's Delight - Data view collection for developers, mimics the production graph server but adds data views for analyzing a pageset and its associated raw data values.
      • Sheriff Duty - Data view collection for the Sheriff use case
      • Investigator Indiscretions - Data view collection for an investigator, provides a comprehensive set of data views for analyzing talos data in a variety of ways.
  • Find 6 stakeholders who can participate for 30 minutes weekly to look at the new graph server and ensure it meets their needs
  • Do initial investigations of tsvg, tdhtml, ally, tscroll
  • Understand compare talos
    • Decide on set of features we care about for this
  • dev.tree-planning email
    • Decide on set of features we care about for this
    • We need to make sure we mimic all url based entry points or modify the software that generates urls to the new graph server.
    • The existing url structure looks like it uses database id's to identify product branch/talos test/platform combinations. The new database will have different ids for this reference information. We may need to map from one to the other for a period of time.

Milestone 2

  • Deploy tsvg and tdhtml with any recommendations
  • Deploy new graph server with basic functionality
    • Add TBPL like browsing
    • Add changeset filtering
  • Have new views in graph server for additional functionality
    • Add a page centric data view for analyzing a particular page's performance across all changesets for a given time interval.
  • Document variance calculation and how to track it
    • ideally we would want to track this monthly
  • Have compare talos and similar tools working with patches and in staging

Milestone 3

  • Deploy ally and tscroll
  • Deploy automatic variance calculation
  • Deploy new compare talos
  • Deploy new features in graph server