|
|
| Line 21: |
Line 21: |
|
| |
|
| For each test a new profile is installed in the browser (either an empty base profile or a profile with an existing places.sqlite in case of the dirty tests). Profiles are not shared across test runs. To initialize the profile an initial open/close is done to the browser. This initial open/close is not included in the test results and is only for configuration purposes. | | For each test a new profile is installed in the browser (either an empty base profile or a profile with an existing places.sqlite in case of the dirty tests). Profiles are not shared across test runs. To initialize the profile an initial open/close is done to the browser. This initial open/close is not included in the test results and is only for configuration purposes. |
|
| |
| == Regressions ==
| |
|
| |
| To determine whether a good point is "good" or "bad", we take 20-30 points of historical data, and 5 points of future data.
| |
| We compare these using a [http://en.wikipedia.org/wiki/Student%27s_t-test t-test]. See https://wiki.mozilla.org/images/c/c0/Larres-thesis.pdf#page=74 .
| |
| Regressions are mailed to the [https://lists.mozilla.org/listinfo/dev-tree-management dev-tree-management] mailing list. Regressions are calculated by the [http://hg.mozilla.org/graphs/file/tip/server/analysis/analyze_talos.py analyze_talos.py] script which uses a configuration file based on http://hg.mozilla.org/graphs/file/tip/server/analysis/analysis.cfg.template
| |
|
| |
|
| == Talos Tests == | | == Talos Tests == |
| Line 209: |
Line 203: |
| * Indepenedent of Observation? Will this test produce the same number regardless of what was run before it? | | * Indepenedent of Observation? Will this test produce the same number regardless of what was run before it? |
| * What considerations are there for how this test should be run and what tools are required? | | * What considerations are there for how this test should be run and what tools are required? |
|
| |
| == How are the numbers calculated? ==
| |
|
| |
| To ensure that the base profile is correctly installed for every test the browser is opened/closed once before test execution. This first cold open is excluded for the test result calculation.
| |
|
| |
| For "cold" tests the caches are cleared after this initial open/close - in this way the browser is configured and ready but returned to a "cold" state.
| |
|
| |
| All tests are run with newly installed profiles - profiles are not shared across test runs.
| |
|
| |
| === Pageload style tests (tp5, tdhtml, etc) ===
| |
|
| |
| The overall test number is determined by first calculating the median page load time for each page in the set (excluding the max page load per individual page). The max median from that set is then excluded and the average is taken; that becomes the number reported to the tinderbox waterfall.
| |
|
| |
| === Ts style tests (ts, twinopen, ts_cold, etc) ===
| |
|
| |
| The overall test number is calculated by excluding the max opening time and taking an average of the remaining numbers.
| |
|
| |
| == Where are the numbers stored? ==
| |
| The results of every talos test are reported to the [https://wiki.mozilla.org/Perfomatic Perfomatic] graph server. When running locally, you can specify output to a file using the --results_url argument to PerfConfigurator, e.g.
| |
|
| |
| PerfConfigurator --activeTests tsvg -e `which firefox` -o tsvg.yml --results_url file://${PWD}/tsvg.txt
| |
|
| |
| [[Buildbot/Talos/DataFormat|Talos data formatting]]
| |
|
| |
|
| == Bugs == | | == Bugs == |