Confirmed users
2,197
edits
Buildbot/Talos/Data: Difference between revisions
Redirected page to TestEngineering/Performance/Talos/Data
m (s/replicates/data points/) |
(Redirected page to TestEngineering/Performance/Talos/Data) |
||
| (3 intermediate revisions by one other user not shown) | |||
| Line 1: | Line 1: | ||
#REDIRECT [[TestEngineering/Performance/Talos/Data]] | |||
= Talos Data = | = Talos Data = | ||
Raw data is generated by Talos. We apply some filters to summarize and reduce the data, then we post it to a server: | Raw data is generated by Talos. We apply some filters to summarize and reduce the data, then we post it to a server: | ||
| Line 27: | Line 29: | ||
== Subtest Filters == | == Subtest Filters == | ||
We have a variety of [ | We have a variety of [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/filter.py filters] defined for Talos. I will explain what each filter is, and you can see the exact settings used for each filter by looking at the individual [https://wiki.mozilla.org/index.php?title=Buildbot/Talos/Tests tests]. | ||
=== ignore_first === | === ignore_first === | ||
| Line 33: | Line 35: | ||
* input: an array of subtest data points | * input: an array of subtest data points | ||
* returns: an array of data points | * returns: an array of data points | ||
* source: [ | * source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/filter.py#l127 filter.py] | ||
* used in most tests with X=1, X=2, and X=5 (5 is the normal case) | * used in most tests with X=1, X=2, and X=5 (5 is the normal case) | ||
| Line 40: | Line 42: | ||
* input: an array of subtest data points | * input: an array of subtest data points | ||
* returns: a single value | * returns: a single value | ||
* source: [ | * source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/filter.py#l58 filter.py] | ||
* used in most tests | * used in most tests | ||
| Line 47: | Line 49: | ||
* input: an array of subtest data points | * input: an array of subtest data points | ||
* returns: a single value | * returns: a single value | ||
* source: [ | * source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/filter.py#l50 filter.py] | ||
* used in kraken for subtests | * used in kraken for subtests | ||
| Line 54: | Line 56: | ||
* input: an array of dromaeo (DOM|CSS) subtest data points | * input: an array of dromaeo (DOM|CSS) subtest data points | ||
* returns: a single number (geometric_mean of the metric summarization) | * returns: a single number (geometric_mean of the metric summarization) | ||
* source: [ | * source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/filter.py#l92 filter.py] | ||
* used in dromaeo_dom and dromaeo_css to build a single value for the subtests | * used in dromaeo_dom and dromaeo_css to build a single value for the subtests | ||
=== v8_subtest === | === v8_subtest === | ||
* input: an array of v8_7 subtest data points | * input: an array of v8_7 subtest data points | ||
* returns: a single value representing the benchmark weighted score for the subtest | * returns: a single value representing the benchmark weighted score for the subtest (see [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/filter.py#l168 filter.py] for details) | ||
* source: [ | * source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/filter.py#l168 filter.py] | ||
* used in v8_7 for the subtests | * used in v8_7 for the subtests | ||
| Line 72: | Line 74: | ||
* inputs: array of subtest summarized data points (one point per subtest) | * inputs: array of subtest summarized data points (one point per subtest) | ||
* returns: a single value representing the geometric mean of all the subtests | * returns: a single value representing the geometric mean of all the subtests | ||
* source: [ | * source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/filter.py#l114 filter.py] | ||
* used for most tests | * used for most tests | ||
| Line 79: | Line 81: | ||
* inputs: array of v8 subtest summaries | * inputs: array of v8 subtest summaries | ||
* returns: a single v8 score | * returns: a single v8 score | ||
* source: [ | * source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/output.py#l102 output.py] | ||
* used for v8 version 7 only | * used for v8 version 7 only | ||
=== Canvasmark_metric === | === Canvasmark_metric === | ||
This is the metric used to calculate the Canvasmark score from the subtest summarized results. Essentially it is a sum of the subtests. | This is the metric used to calculate the Canvasmark score from the subtest summarized results. Essentially it is a sum of the subtests. This is identical to the [https://wiki.mozilla.org/Buildbot/Talos/Data#js_metric js_metric] | ||
=== js_metric === | === js_metric === | ||
| Line 93: | Line 91: | ||
* inputs: array of Kraken subtest results | * inputs: array of Kraken subtest results | ||
* returns: a single Kraken score | * returns: a single Kraken score | ||
* source: [ | * source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/output.py#l108 output.py] | ||
* used for Kraken only | * used for Kraken only | ||