Auto-tools/Projects/Signal From Noise/Meetings/2012-05-10

From MozillaWiki
Jump to: navigation, search

Previous Action Items

  • [ctalbert,carljm,jhammel,jmaher] - look at datazilla VM
  • [jmaher] - identify tp5 tests that hit the internet, give christina set of numbers with those pages missing
  • [christina] - analyze reduced pageset for SfN metric

Metric Calculations

  • 70 pages still are not fitting in the model :(
    • will provide top offenders to a*team


Page Specific Views

Compare Talos Functionality

Round Table

  • proposal for moving forward
    • focus on tdhtml and tsvg as page centric
      • find a model for pages that can detect regressions with a single new data point
      • display data on graphs for investigation purposes
      • determine a metric to report for pass fail purposes and tracking as a page set (mochitests have pass/fail/todo, and if 1 test fails, we turn the job orange, maybe we do something similar here)
    • if a page doesn't fit a model then lets not run it and file a bug to investigate the page (80/20 rule here)
    • consider a maybe flag (probably orange) which doesn't match our model, but we cannot determine programatically that it is a regressions or measurable improvement.
  • new color for tbpl? (You might have regressed?)
  • killing pages for network access: (could just use datazilla)

Action Items

  • find what pages we can consider reliable
    • define what "reliable" is
  • triangulate
  • [christina] - give jeads/ctalbert/jmaher a set of pages that are just too noisy and operating systems
  • local TBPL (contact: edmorley)