Buildbot/Talos/Data
How are the numbers calculated?
To ensure that the base profile is correctly installed for every test the browser is opened/closed once before test execution. This first cold open is excluded for the test result calculation.
For "cold" tests the caches are cleared after this initial open/close - in this way the browser is configured and ready but returned to a "cold" state.
All tests are run with newly installed profiles - profiles are not shared across test runs.
Pageload style tests (tp5, tdhtml, etc)
The overall test number is determined by first calculating the median page load time for each page in the set (excluding the max page load per individual page). The max median from that set is then excluded and the average is taken; that becomes the number reported to the tinderbox waterfall.
Ts style tests (ts, twinopen, ts_cold, etc)
The overall test number is calculated by excluding the max opening time and taking an average of the remaining numbers.
Where are the numbers stored?
The results of every talos test are reported to the Perfomatic graph server. When running locally, you can specify output to a file using the --results_url argument to PerfConfigurator, e.g.
PerfConfigurator --activeTests tsvg -e `which firefox` -o tsvg.yml --results_url file://${PWD}/tsvg.txt