Auto-tools/Projects/Signal From Noise/StatusNovember2012: Difference between revisions

* New method for regression detection: https://wiki.mozilla.org/images/d/dd/Talos_Statistical_Analysis_Writeup.pdf : Working with Datazilla results for tp5 test pages, Metrics developed a regression detection algorithm.To compare the mean of each page to of the new push to the mean of each page to the current push, hypothesis tests are conducted http://en.wikipedia.org/wiki/Statistical_hypothesis_testing. Welch's t-test is used to determine whether a page has regressed for the given new push. Moving to page-centric testing led to multiple hypothesis testing problem, and to correct for the inflation of false positives, False Discovery Rate Procedure (FDR) is used: http://www.stat.cmu.edu/~genovese/talks/hannover1-04.pdf.  Due to the natural variation between consecutive pushes, exponential smoothing was implemented before performing FDR procedure. Code for this is available in https://github.com/mozilla/datazilla-metrics

* we use more replicates per page

* Datazilla utilizes improved statistical methodologies. Datazilla uses the welch's ttest, the FDR stuff, and the exponential smoothing.