Confirmed users
3,376
edits
Buildbot/Talos/DataFormat: Difference between revisions
Jump to navigation
Jump to search
m
→Talos Data Formatting: - explain terminology
m (→Filtering: - format stuff) |
m (→Talos Data Formatting: - explain terminology) |
||
| Line 1: | Line 1: | ||
= Talos Data | = Talos Data = | ||
Raw data is generated by Talos. We apply some filters to summarize and reduce the data, then we post it to a server: | Raw data is generated by Talos. We apply some filters to summarize and reduce the data, then we post it to a server: | ||
* Graphserver | * Graphserver | ||
* Perfherder | * Perfherder | ||
== | == Terminology == | ||
=== Job === | |||
When a build is completed we run a series of test (unittest and performance "talos") jobs. Each job reserves a machine for itself, the runs the script which sets up, installs, executes the test, generates the results, and cleans up after itself. | |||
For Talos we have a series of jobs and each job runs 1 or more tests. For the purposes of discussing data we will refer to each test as a suite. A suite would be something like 'ts_paint', 'Canvasmark', or 'tp5'. Each Suite will run it's respective subtests, and summarize itself properly. When all the suites in a job have completed, the results will be output (uploaded in some cases) and we will be able to look for regressions, view data in a graph, and query for the summarized data. | |||
=== Suite === | |||
A collection of subtests which run, the subtest results are summarized to the suite level. Often these are referred to as 'tests'. Some examples are "tresize", "TART", "tp5", "ts_paint". | |||
* in graph server this is the lowest level of granularity available in the UI | |||
* in Perfherder a suite is referenced as a 'Summary' (e.g "tp5o summary opt") | |||
=== Subtest === | |||
A specific test (usually a webpage to load) which we collect replicates (numbers) from. Typically we run many cycles of each subtest to build up a representative collection of replicates to make sure the data is meaningful. | |||
* in graph server Talos upload a single number for each subtest, the replicates are summarized by Talos prior to uploading. | |||
* in Perfherder the subtest data is preserved as raw replicates as well as summarized by Talos. We use the summarizations when showing a graph. | |||
=== Replicates === | |||
Replicates mean the single numbers or data points we collect while executing a talos test. In this regard, we collect a series of numbers (usually 20 or more) for each subtest. Each of these 20+ numbers are called replicates. | |||
We do filtering on the replicates, mainly because the first few replicates are not a representative sample of the remaining replicates we collect. The one exception would be [https://wiki.mozilla.org/Buildbot/Talos/Tests#Internal_Benchmarks internal benchmarks] (generally suites which measure something other than time). For Benchmarks, there is usually a special formula applied to the replicates. | |||
== Perfherder == | == Perfherder == | ||