MozillaWiki

TestEngineering/Performance/Talos/Data: Difference between revisions

Page Discussion

TestEngineering/Performance/Talos/Data (view source)

Revision as of 12:36, 13 August 2019

68 bytes removed ,  13 August 2019
no edit summary
No edit summary
No edit summary
Line 1: Line 1:
= Talos Data =
= At a glance =
Raw data is generated by Talos. We apply some filters to summarize and reduce the data, then we post it to a server:
Raw data is generated by Talos. We apply some filters to summarize and reduce the data, then we post it to [[#Perfherder]].
* Graphserver
* Perfherder


== Terminology ==
= Terminology =


=== Job ===
== Job ==
When a build is completed we run a series of test (unittest and performance "talos") jobs.  Each job reserves a machine for itself, then runs the script which sets up, installs, executes the test, generates the results, and cleans up after itself.  In general we try to ensure the jobs complete in 30 minutes or less.
When a build is completed we run a series of test (unittest and performance "talos") jobs.  Each job reserves a machine for itself, then runs the script which sets up, installs, executes the test, generates the results, and cleans up after itself.  In general we try to ensure the jobs complete in 30 minutes or less.


For Talos we have a series of jobs and each job runs 1 or more test suites (with the exception of tp and xperf, jobs run 2-4 suites at a time). A suite would be something like 'ts_paint', 'Canvasmark', or 'tp5'.  Each Suite will run its respective subtests, and provide a summary number which is representative of a meaningful aggregation of the individual subtest results.  When all the suites in a job have completed, the results will be output (uploaded in some cases) and we will be able to look for regressions, view data in a graph, and query for the summarized data.
For Talos we have a series of jobs and each job runs 1 or more test suites (with the exception of tp and xperf, jobs run 2-4 suites at a time). A suite would be something like 'ts_paint', 'Canvasmark', or 'tp5'.  Each Suite will run its respective subtests, and provide a summary number which is representative of a meaningful aggregation of the individual subtest results.  When all the suites in a job have completed, the results will be output (uploaded in some cases) and we will be able to look for regressions, view data in a graph, and query for the summarized data.


=== Suite ===
== Suite ==
A collection of subtests which run. Often these are referred to as 'tests'. Some examples are "tresize", "TART", "tp5", "ts_paint".
A collection of subtests which run. Often these are referred to as 'tests'. Some examples are "tresize", "TART", "tp5", "ts_paint".
* in graph server this is the lowest level of granularity available in the UI
* in graph server this is the lowest level of granularity available in the UI
* in Perfherder suite-level results are called a 'summary' (e.g "tp5o summary opt")
* in Perfherder suite-level results are called a 'summary' (e.g "tp5o summary opt")


=== Subtest ===
== Subtest ==
A specific test (usually a webpage to load) which we collect data points from.  Typically we run many cycles of each subtest to build up a representative collection of data points to make sure the data is meaningful.
A specific test (usually a webpage to load) which we collect data points from.  Typically we run many cycles of each subtest to build up a representative collection of data points to make sure the data is meaningful.
* in graph server Talos upload a single number for each subtest, the data points are summarized by Talos prior to uploading.
* in graph server Talos upload a single number for each subtest, the data points are summarized by Talos prior to uploading.
* in Perfherder the subtest data is preserved as raw data points as well as summarized by Talos.  We use the summarizations when showing a graph.
* in Perfherder the subtest data is preserved as raw data points as well as summarized by Talos.  We use the summarizations when showing a graph.


=== Data Points (aka Replicates) ===
== Data points (aka replicates) ==
Data Points refer to the single numbers or replicates we collect while executing a talos test.  In this regard, we collect a series of numbers (usually 20 or more) for each subtest.  Each of these 20+ numbers are called data points.
Data Points refer to the single numbers or replicates we collect while executing a talos test.  In this regard, we collect a series of numbers (usually 20 or more) for each subtest.  Each of these 20+ numbers are called data points.


We do filtering on the data points, mainly because the first few data points are not a representative sample of the remaining data points we collect.  The one exception would be [[../Tests#Internal_Benchmarks]] (generally suites which measure something other than time).  For Benchmarks, there is usually a special formula applied to the data points.
We do filtering on the data points, mainly because the first few data points are not a representative sample of the remaining data points we collect.  The one exception would be [[../Tests#Internal_Benchmarks]] (generally suites which measure something other than time).  For Benchmarks, there is usually a special formula applied to the data points.


== Subtest Filters ==
= Subtest filters =
We have a variety of [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/filter.py filters] defined for Talos.  I will explain what each filter is, and you can see the exact settings used for each filter by looking at the individual [[../Tests|tests]].
We have a variety of [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/filter.py filters] defined for Talos.  I will explain what each filter is, and you can see the exact settings used for each filter by looking at the individual [[../Tests|tests]].


=== ignore_first ===
== ignore_first ==
This filter ignores the first 'X' data points allowing us to ignore warmup runs.
This filter ignores the first 'X' data points allowing us to ignore warmup runs.
* input: an array of subtest data points
* input: an array of subtest data points
Line 36: Line 34:
* used in most tests with X=1, X=2, and X=5 (5 is the normal case)
* used in most tests with X=1, X=2, and X=5 (5 is the normal case)


=== median ===
== median ==
This filter takes in an array of data points and returns the median of the data points (a single value).
This filter takes in an array of data points and returns the median of the data points (a single value).
* input: an array of subtest data points
* input: an array of subtest data points
Line 43: Line 41:
* used in most tests
* used in most tests


=== mean ===
== mean ==
This filter takes in an array of data points and returns the mean value of the data points (a single value).
This filter takes in an array of data points and returns the mean value of the data points (a single value).
* input: an array of subtest data points
* input: an array of subtest data points
Line 50: Line 48:
* used in kraken for subtests
* used in kraken for subtests


=== dromaeo ===
== dromaeo ==
This filter is a specific filter defined by dromaeo and respects the data points as every 5 data points represents a different metric being measured.
This filter is a specific filter defined by dromaeo and respects the data points as every 5 data points represents a different metric being measured.
* input: an array of dromaeo (DOM|CSS) subtest data points
* input: an array of dromaeo (DOM|CSS) subtest data points
Line 57: Line 55:
* used in dromaeo_dom and dromaeo_css to build a single value for the subtests
* used in dromaeo_dom and dromaeo_css to build a single value for the subtests


=== v8_subtest ===
== v8_subtest ==
* input: an array of v8_7 subtest data points
* input: an array of v8_7 subtest data points
* returns: a single value representing the benchmark weighted score for the subtest (see [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/filter.py#l168 filter.py] for details)
* returns: a single value representing the benchmark weighted score for the subtest (see [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/filter.py#l168 filter.py] for details)
Line 65: Line 63:
NOTE: this deviates from the exact definition of v8 as we retain the Encrypt and Decrypt as subtests (instead of combining them into Crypto) as well as keeping Earley and Boyer (instead of combining them into EarleyBoyer).  There is a slight tweak in the final suite score, but it is <1% different.
NOTE: this deviates from the exact definition of v8 as we retain the Encrypt and Decrypt as subtests (instead of combining them into Crypto) as well as keeping Earley and Boyer (instead of combining them into EarleyBoyer).  There is a slight tweak in the final suite score, but it is <1% different.


== Suite Summarization Filters ==
= Suite summarization filters =
Once we have a single number from each of the subtests, we need to generate a single number for the suite. There are 4 specific calculations used.
Once we have a single number from each of the subtests, we need to generate a single number for the suite. There are 4 specific calculations used.


=== geometric_mean ===
== geometric_mean ==
This is a standard geometric mean of the data:
This is a standard geometric mean of the data:
* inputs: array of subtest summarized data points (one point per subtest)
* inputs: array of subtest summarized data points (one point per subtest)
Line 75: Line 73:
* used for most tests
* used for most tests


=== v8_metric ===
== v8_metric ==
this is a custom metric which take the geometric_mean of the subtests and multiplies it by 100.
this is a custom metric which take the geometric_mean of the subtests and multiplies it by 100.
* inputs: array of v8 subtest summaries
* inputs: array of v8 subtest summaries
Line 82: Line 80:
* used for v8 version 7 only
* used for v8 version 7 only


=== Canvasmark_metric ===
== Canvasmark_metric ==
This is the metric used to calculate the Canvasmark score from the subtest summarized results.  Essentially it is a sum of the subtests.  This is identical to the [#js_metric]]
This is the metric used to calculate the Canvasmark score from the subtest summarized results.  Essentially it is a sum of the subtests.  This is identical to the [#js_metric]]


=== js_metric ===
== js_metric ==
This is the metric used to calculate the Kraken score from the subtest summarized results.  Essentially it is a sum of the subtests.
This is the metric used to calculate the Kraken score from the subtest summarized results.  Essentially it is a sum of the subtests.
* inputs: array of Kraken subtest results
* inputs: array of Kraken subtest results
Line 92: Line 90:
* used for Kraken only
* used for Kraken only


= Perfherder =
Perfherder ingests data from Talos by parsing the raw log, then it stores the data in a database while preparing it for regression detection and displaying on graphs.


== Perfherder ==
== Raw data ==
Perfherder ingests data from talos by parsing the raw log, then it stores the data in a database while preparing it for regression detection and displaying on graphs.
 
=== Raw Data ===
In the log files, we look for "TALOSDATA: " text followed by a valid json blob.  An example TALOSDATA blob looks like:
In the log files, we look for "TALOSDATA: " text followed by a valid json blob.  An example TALOSDATA blob looks like:
<pre>
<pre>
Line 102: Line 99:
</pre>
</pre>


=== Filtering & Calculations ===
== Filtering & calculations ==
When the raw data comes in, we look for the summary tag in the json.   
When the raw data comes in, we look for the summary tag in the json.   
<pre>
<pre>
Line 108: Line 105:
</pre>
</pre>


In this case we would use 23.22 for the value inside of perfherder (perfherder rounds to two decimal places). This is the value that will be used for calculating alerts, displaying points on the graph, and for data when comparing two revisions.
In this case we would use 23.22 for the value inside of Perfherder (Perfherder rounds to two decimal places). This is the value that will be used for calculating alerts, displaying points on the graph, and for data when comparing two revisions.


In all cases there should be a 'subtests' field as well that lists out each page loaded along with a set of values:
In all cases there should be a 'subtests' field as well that lists out each page loaded along with a set of values:
Line 115: Line 112:
</pre>
</pre>


These values are used in the sub test specific view (not the suite summary). When viewing a graph, you can switch between different values for each data point to see what the mean, median, etc. are.  This is where we get the fields. In addition, the default value is the 'filtered' value, this takes into account filters (ignore first 'x' data points, median|mean, etc.) on the raw data so we have summarized data being calculated at a single point.
These values are used in the subtest specific view (not the suite summary). When viewing a graph, you can switch between different values for each data point to see what the mean, median, etc. are.  This is where we get the fields. In addition, the default value is the 'filtered' value, this takes into account filters (ignore first 'x' data points, median|mean, etc.) on the raw data so we have summarized data being calculated at a single point.
 
Each suite has the ability to set custom filters and keeping this logic inside of talos ensures that it is always done in a single place, in source code, where developers can easily look and find it.
 


== browser_output.txt ==
Each suite has the ability to set custom filters and keeping this logic inside of Talos ensures that it is always done in a single place, in source code, where developers can easily look and find it.


= browser_output.txt =
The data is harvested from browser_output.txt:
The data is harvested from browser_output.txt:


Davehunt
Confirmed users
2,197

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1216417"