Difference between revisions of "Sheriffing/Job Visibility Policy"
From MozillaWiki
(Add "Requesting changes in visibility" section) |
(→4) Scheduled on every push: Clarify PGO reasoning) |
||
| (One intermediate revision by the same user not shown) | |||
| Line 22: | Line 22: | ||
* Otherwise job failures when tree X merges into mozilla-central will not be attributable to a single changeset, resulting in either tree closure or backout of the entire merge (see requirement #2). | * Otherwise job failures when tree X merges into mozilla-central will not be attributable to a single changeset, resulting in either tree closure or backout of the entire merge (see requirement #2). | ||
| − | === 4) | + | === 4) Scheduled on every push === |
* Otherwise job failures will not be attributable to a single changeset, resulting in either tree closure or backout of multiple pushes (see requirement #2). | * Otherwise job failures will not be attributable to a single changeset, resulting in either tree closure or backout of multiple pushes (see requirement #2). | ||
| − | * An exception is made for nightly builds with an virtually equivalent non-nightly variant that is built on every push. | + | * An exception is made for nightly builds with an virtually equivalent non-nightly variant that is built on every push & for tests run on PGO builds (given that PGO builds take an inordinate amount of time, we still schedule them every 3/6 hours depending on tree, and relatively speaking there any not too many PGO-only test failures). |
| + | * Note also that coalescing (buildbot queue collapsing when there is more than one queued job) may mean that not all scheduled jobs actually get run. Whilst coalescing makes sheriffing harder, it's a necessary evil given that automation infrastructure demand frequently outstrips supply. | ||
=== 5) Easily run on try server === | === 5) Easily run on try server === | ||
Revision as of 15:23, 26 March 2013
Contents
- 1 Requirements for being shown in the default TBPL view
- 1.1 1) Has an active owner
- 1.2 2) Breakage is expected to be followed by tree closure or backout
- 1.3 3) Runs on all trees that merge into mozilla-central
- 1.4 4) Scheduled on every push
- 1.5 5) Easily run on try server
- 1.6 6) Outputs failures in a TBPL-starrable format
- 1.7 7) Per job intermittent failure rate of less than 5%
- 1.8 8) Supports the disabling of individual tests
- 1.9 9) Must not rely on resources outside of the build network
- 1.10 10) Has sufficient documentation
- 1.11 11) Easy for a dev to run locally
- 2 Requesting changes in visibility
- 3 My platform/test-suite does not meet the requirements, what now?
- 4 The future
Requirements for being shown in the default TBPL view
This page was created to clarify the existing requirements that a platform/test-suite has to meet, before its jobs can be shown in the default TBPL view. To propose changes to this policy, please speak to the sheriffs and/or post to dev.platform.
1) Has an active owner
- Who is committed to ensuring the other requirements are met not just initially, but over the long term.
- Who will ensure the new job type is switched off to save resources, should we stop finding it useful in the future.
2) Breakage is expected to be followed by tree closure or backout
- Failures visible in the default view (other than those that are known intermittents/transient), must have their cause backed out in a timely fashion or else the tree closed until diagnosed.
- Why? If tier != 1 jobs were instead made visible in the default view, they would:
- Interfere with ability to sheriff the tree:
- Indistinguishable from tier-1 failures.
- Appear in the failure count/cause the tab to glow.
- Slow down navigation of failures when using keyboard shortcuts.
- Cause extra workload for sheriffs by making them perform initial diagnosis/bug filing & then starring of the failure on every push until it is fixed an indeterminate amount of time later.
- Cause confusion for non-sheriffs using project branches/try-server, as well as on all trees at the weekends when there are no employed sheriffs.
- Interfere with ability to sheriff the tree:
- If your platform/test falls under the category of "someone should just file a bug and it will be investigated by our team later", then it unfortunately does not meet this requirement. From past requests this normally translates to "group X think this job type is important but we want to delegate the task of monitoring it to someone else".
3) Runs on all trees that merge into mozilla-central
- Otherwise job failures when tree X merges into mozilla-central will not be attributable to a single changeset, resulting in either tree closure or backout of the entire merge (see requirement #2).
4) Scheduled on every push
- Otherwise job failures will not be attributable to a single changeset, resulting in either tree closure or backout of multiple pushes (see requirement #2).
- An exception is made for nightly builds with an virtually equivalent non-nightly variant that is built on every push & for tests run on PGO builds (given that PGO builds take an inordinate amount of time, we still schedule them every 3/6 hours depending on tree, and relatively speaking there any not too many PGO-only test failures).
- Note also that coalescing (buildbot queue collapsing when there is more than one queued job) may mean that not all scheduled jobs actually get run. Whilst coalescing makes sheriffing harder, it's a necessary evil given that automation infrastructure demand frequently outstrips supply.
5) Easily run on try server
- Otherwise developers who have had their landing backed out for breaking the job type will be unable to easily debug/fix the failures, particularly if they only reproduce on our infrastructure.
- Developers should not be expected to guess try chooser options, so http://trychooser.pub.build.mozilla.org/ must have been updated.
6) Outputs failures in a TBPL-starrable format
- Failures must appear in the TBPL annotated summary (ie: use the standard TEST-UNEXPECTED-{FAIL,PASS}, PROCESS-CRASH, ... format), otherwise sheriffs & devs have to open the full logs.
- Failures must output the test names correctly, so TBPL can perform the BzAPI intermittent-failure searches for bug suggestions.
- Exceptions & timeouts must be caught and handled with a TBPL compatible failure message.
- The sheriffs will be happy to help advise how to meet this requirement.
7) Per job intermittent failure rate of less than 5%
- A high failure rate:
- Causes unnecessary sheriff workload.
- Affects the ability to sheriff the trees as a whole, particularly during times of heavy coalescing.
- Undermines devs confidence in the platform/test-suite - which as demonstrated by Firefox for Android, permanently affects their willingness to believe any future failures, even once the intermittent-failure rate is lowered.
- A mozilla-central push results in ~400 jobs. A 5% failure rate would mean 20 failures on that push - ie: an OrangeFactor of 20. The typical OrangeFactor across all trunk trees is normally 3-5, so a 5% failure rate is extremely generous.
8) Supports the disabling of individual tests
- It must be possible for sheriffs to disable an individual test per platform or entirely, by either annotating the test or editing a manifest/moz.build/Makefile. (See also requirement #10).
9) Must not rely on resources outside of the build network
- Since these will cause failures when the external site is unavailable, as well as impacting end to end times & adding noise to performance tests.
- eg: Emulator/driver binaries direct from a vendor's site, package downloads from PyPi or page assets for unit/performance tests.
10) Has sufficient documentation
- Has a wiki page with:
- An overview of the test-suite.
- Instructions for running locally.
- How to disable an individual failing test.
- The current owner/who to contact for help.
- That wiki page is linked to from https://developer.mozilla.org/docs/Mozilla/QA/Automated_testing
11) Easy for a dev to run locally
- Is supported by mach.
- Ideally part of mozilla-central (legacy exception being Talos).
Requesting changes in visibility
- Please file a bug using this template, so that changes in visibility are more discoverable (vs IRC or asking as part of a bug in another product/component).
My platform/test-suite does not meet the requirements, what now?
- Your platform/test-suite will still be being run, just not shown on the default view. This model has worked well for many projects/build types (eg jetpack, xulrunner, spidermonkey).
- To see it, append '&showall=1' to the URL (bug 748833 will add a checkbox for this to the TBPL UI).
- To filter the jobs displayed, under the 'Filters' menu use the 'job name' field (which supports regex).
- eg: to see both ASan & Valgrind jobs on mozilla-central (neither of which are shown by default), use: https://tbpl.mozilla.org/?showall=1&jobname=(asan|valgrind)
The future
- Planned improvements to our tooling will likely mean that some of these requirements can be relaxed in the future, as well as making it easier for maintainers of non-default-view job types to track their success/failure without having to monitor TBPL continuously.
- The successor to TBPL (Auto-tools/Projects/TBPL2) will support:
- Multiple dashboards/views for different use cases/teams (giving us more flexibility than just "default-view" or "&showall=1".
- Opt-in notifications (email, IRC, dashboard, ...?) of failures for desired job types (see proposal in bug 851061).
- Auto-tools/Projects/Bisect_in_the_cloud will allow sheriffs to more easily narrow regression ranges for job types that do not run on every push, making it more viable to accept them into certain views/dashboards.
