Confirmed users
513
edits
Platform/Games/GameFocusedBenchmarking: Difference between revisions
Jump to navigation
Jump to search
Platform/Games/GameFocusedBenchmarking (view source)
Revision as of 21:10, 12 December 2013
, 12 December 2013→Current Results
Klahnakoski (talk | contribs) |
Klahnakoski (talk | contribs) |
||
| Line 22: | Line 22: | ||
Each individual mode has low variance; the weight given to each of the modes is equal to the number of samples that we observe; and the average provides us with number that best reflects that balance. | Each individual mode has low variance; the weight given to each of the modes is equal to the number of samples that we observe; and the average provides us with number that best reflects that balance. | ||
'''Caveate''' - The time series ([https://metrics.mozilla.com/bugzilla-analysis/Perfy-TimeSeries.html#platform=Platform.Linux&benchmark=Benchmark.octane-2.0 example]) does '''NOT''' use average, it uses median! The purpose of the time-series is to detect regressions in performance, which means we are only interested in one of the two modes and whether it changes over time. The median naturally chooses the most popular mode giving us consistent results over time. There are dangers to using median; for one, it is not sensitive to change in the balance of modes over time. Yet, median is much better | '''Caveate''' - The time series ([https://metrics.mozilla.com/bugzilla-analysis/Perfy-TimeSeries.html#platform=Platform.Linux&benchmark=Benchmark.octane-2.0 example]) does '''NOT''' use average, it uses median! The purpose of the time-series is to detect regressions in performance, which means we are only interested in one of the two modes and whether it changes over time. The median naturally chooses the most popular mode giving us consistent results over time. There are dangers to using median; for one, it is not sensitive to change in the balance of modes over time. Yet, median is much better than average given the small number of tests we perform. Average is sensitive to the random fluctuations between each battery of tests, not large enough when viewed on it's own, but distracting when looking at variation between batteries. We do not want to be chasing regressions when GC happened to hit thrice this round, and only twice last time. | ||
==Scope== | ==Scope== | ||