947
edits
Auto-tools/Projects/Signal From Noise/StatusNovember2012: Difference between revisions
Jump to navigation
Jump to search
m
Auto-tools/Projects/Signal From Noise/StatusNovember2012 (view source)
Revision as of 20:48, 28 December 2012
, 28 December 2012→State of Statistics: November 2012: caps
(→State of Statistics: November 2012: consolidate wrt ignore_first:5) |
m (→State of Statistics: November 2012: caps) |
||
| Line 56: | Line 56: | ||
* Determining the number of observations per testpage: It is crucial that we have a good balance machine time for a talos test and having enough replicates for statisical viability of the test results. The optimal number of replicates for each test page for statistical testing is about 30 (J devore, Probability & Statistics for Engineering & Sciences 8th ed. p. 226). However, due to the time constraints, we decided to collect 25 replicates (still a big improvement from previous, when we collected 10 replicates but not optimal). | * Determining the number of observations per testpage: It is crucial that we have a good balance machine time for a talos test and having enough replicates for statisical viability of the test results. The optimal number of replicates for each test page for statistical testing is about 30 (J devore, Probability & Statistics for Engineering & Sciences 8th ed. p. 226). However, due to the time constraints, we decided to collect 25 replicates (still a big improvement from previous, when we collected 10 replicates but not optimal). | ||
* | * Pageload tests ignore the first 5 data points (''Ignore_first:5'') : Metrics determined that ignoring the first 5 data points on pageload tests increased statistical viability because most of the variation was coming from the first few data points ; see https://bugzilla.mozilla.org/show_bug.cgi?id=731391 , https://wiki.mozilla.org/images/d/dd/Talos_Statistical_Analysis_Writeup.pdf. | ||
* Audit of tp5 pageset : The tp5 (see https://wiki.mozilla.org/Buildbot/Talos#tp5 ) pageset was audited as some of the pages had significantly large variation within replicates: https://wiki.mozilla.org/images/3/38/Tp5_Good_Pages.pdf (Metrics recommend decreasing the size of test pages in tp5 and increasing the number of replicates). | * Audit of tp5 pageset : The tp5 (see https://wiki.mozilla.org/Buildbot/Talos#tp5 ) pageset was audited as some of the pages had significantly large variation within replicates: https://wiki.mozilla.org/images/3/38/Tp5_Good_Pages.pdf (Metrics recommend decreasing the size of test pages in tp5 and increasing the number of replicates). | ||