Confirmed users
955
edits
Buildbot/Talos/Data: Difference between revisions
→Terminology: try to clarify terminology a bit more
m (→dromaeo) |
(→Terminology: try to clarify terminology a bit more) |
||
| Line 5: | Line 5: | ||
== Terminology == | == Terminology == | ||
=== Job === | === Job === | ||
When a build is completed we run a series of test (unittest and performance "talos") jobs. Each job reserves a machine for itself, then runs the script which sets up, installs, executes the test, generates the results, and cleans up after itself. In general we try to ensure the jobs complete in 30 minutes or less. | When a build is completed we run a series of test (unittest and performance "talos") jobs. Each job reserves a machine for itself, then runs the script which sets up, installs, executes the test, generates the results, and cleans up after itself. In general we try to ensure the jobs complete in 30 minutes or less. | ||
For Talos we have a series of jobs and each job runs 1 or more tests (with the exception of tp and xperf, jobs run 2-4 suites at a time). For the purposes of discussing data we will refer to each test as a suite. A suite would be something like 'ts_paint', 'Canvasmark', or 'tp5'. Each Suite will run its respective subtests, and | For Talos we have a series of jobs and each job runs 1 or more tests (with the exception of tp and xperf, jobs run 2-4 suites at a time). For the purposes of discussing data we will refer to each test as a suite. A suite would be something like 'ts_paint', 'Canvasmark', or 'tp5'. Each Suite will run its respective subtests, and provide a summary number which is representative of a meaningful aggregation of the individual subtest results. When all the suites in a job have completed, the results will be output (uploaded in some cases) and we will be able to look for regressions, view data in a graph, and query for the summarized data. | ||
=== Suite === | === Suite === | ||
A collection of subtests which run. | A collection of subtests which run. Often these are referred to as 'tests'. Some examples are "tresize", "TART", "tp5", "ts_paint". | ||
* in graph server this is the lowest level of granularity available in the UI | * in graph server this is the lowest level of granularity available in the UI | ||
* in Perfherder | * in Perfherder these suite-level results are called a 'summary' (e.g "tp5o summary opt") | ||
=== Subtest === | === Subtest === | ||