Buildbot/Talos/Data: Difference between revisions

A specific test (usually a webpage to load) which we collect replicates (numbers) from.  Typically we run many cycles of each subtest to build up a representative collection of replicates to make sure the data is meaningful.

* in graph server Talos upload a single number for each subtest, the replicates are summarized by Talos prior to uploading.

* in Perfherder the subtest data is preserved as raw replicates as well as summarized by Talos.  We use the summarizations when showing a graph.

=== Replicates (aka Data Points) ===

Replicates refer to the single numbers or data points we collect while executing a talos test.  In this regard, we collect a series of numbers (usually 20 or more) for each subtest.  Each of these 20+ numbers are called replicates.

We do filtering on the replicates, mainly because the first few replicates are not a representative sample of the remaining replicates we collect.  The one exception would be [https://wiki.mozilla.org/Buildbot/Talos/Tests#Internal_Benchmarks internal benchmarks] (generally suites which measure something other than time).  For Benchmarks, there is usually a special formula applied to the replicates.

This filter ignores the first 'X' replicates allowing us to ignore warmup runs.

* input: an array of subtest replicates

* returns: an array of replicates

This filter takes in an array of replicates and returns the median of the replicates (a single value).

* input: an array of subtest replicates

This filter takes in an array of replicates and returns the mean value of the replicates (a single value).

* input: an array of subtest replicates

This filter is a specific filter defined by dromaeo and respects the replicates as every 5 replicates represents a different metric being measured.

* input: an array of dromaeo (DOM|CSS) subtest replicates

* input: an array of v8_7 subtest replicates