User:RyanVM/Job Visibility Policy Updates: Difference between revisions

Jump to navigation Jump to search

User:RyanVM/Job Visibility Policy Updates (view source)

Revision as of 13:03, 1 June 2015

749 bytes added ,  1 June 2015
First pass of obvious Tier 2 changes
(Import existing policy as-is)
 
(First pass of obvious Tier 2 changes)
Line 2: Line 2:
== Requirements for jobs shown in the default Treeherder view ==
== Requirements for jobs shown in the default Treeherder view ==


This page was created to clarify the requirements that a platform/test-suite has to meet, before its jobs can be shown in the default [https://treeherder.mozilla.org/ Treeherder] view for the main development trees. Owners of non-sheriff managed project/disposable repos do not need to meet these requirements before requesting visibility changes.
This page was created to clarify the requirements that a platform/test-suite has to meet before its jobs can be shown in the default [https://treeherder.mozilla.org/ Treeherder] view for the main development trees. Owners of non-sheriff managed project/disposable repos do not need to meet these requirements before requesting visibility changes. The below section applies to both Tier 1 and Tier 2 jobs.


Common sense will apply in cases where some of the requirements are not applicable for a particular platform/build/test type.
Common sense will apply in cases where some of the requirements are not applicable for a particular platform/build/test type.


To propose changes to this policy, please speak to the sheriffs and/or post to [https://lists.mozilla.org/listinfo/dev-platform dev.platform].
To propose changes to this policy, please speak to the sheriffs and/or post to [https://lists.mozilla.org/listinfo/dev-platform dev.platform]. <b><big>xxxRyan: SHOULD THIS BE DEV.TREE-MANAGEMENT NOW?</big></b>


==== Has an active owner ====
==== Has an active owner ====
* Who is committed to ensuring the other requirements are met not just initially, but over the long term.
* Who is committed to ensuring the other requirements are met not just initially, but over the long term.
* Who will ensure the new job type is switched off to save resources, should we stop finding it useful in the future.
* Who will ensure the new job type is switched off to save resources should we stop finding it useful in the future.


==== Usable job logs ====
==== Usable job logs ====
Line 26: Line 26:
** Exceptions & timeouts must be handled with appropriate log output (eg: the failure line must state in which test the timeout occurred, not just that the entire run has timed out).
** Exceptions & timeouts must be handled with appropriate log output (eg: the failure line must state in which test the timeout occurred, not just that the entire run has timed out).
* The sheriffs will be happy to advise regarding the above.
* The sheriffs will be happy to advise regarding the above.
==== Low intermittent failure rate ====
* A high failure rate:
** Causes unnecessary sheriff workload.
** Affects the ability to sheriff the trees as a whole, particularly during times of heavy coalescing.
** Undermines devs confidence in the platform/test-suite - which as demonstrated by Firefox for Android, permanently affects their willingness to believe any future failures, even once the intermittent-failure rate is lowered.
* A mozilla-central push results in ~400 jobs. The typical OrangeFactor across all trunk trees is normally (excluding the recent spike) 3-4, ie: a failure rate of ~1%.
* Therefore as a rough guide a new platform/testsuite must have at most a 5% per job failure rate initially, and ideally <1% longer term.
* However, sheriffs will make the final determination of whether a job type has too many intermittent failures. This will be a based on a combination of factors including failure rate, length of time the failures have been occurring, owner interest in fixing them & whether Treeherder is able to make bug suggestions.
==== Must avoid patterns known to cause non deterministic failures ====
* Must avoid pulling the tip of external repositories as part of the build - since landings there can cause non-obvious failures. If an external repository is absolutely necessary, instead reference the desired changeset from a manifest in mozilla-central (like talos or gaia do).
* Must not rely on resources from sites whose content we do not control/have no SLA:
** Since these will cause failures when the external site is unavailable, as well as impacting end to end times & adding noise to performance tests.
** eg: Emulator/driver binaries direct from a vendor's site, package downloads from PyPi or page assets for unit/performance tests.
** Ensure MOZ_DISABLE_NONLOCAL_CONNECTIONS is defined in the automation environment (see {{bug|995417}}) & use a list of automation prefs for switching off undesirable behaviour (eg automatic updates, telemetry pings; see {{bug|1023483}} for where these are set).
* Must not contain time bombs, e.g. tests that will fail after a certain date or when run at certain times (e.g., the day summer time starts or ends, or when the test starts before midnight and finishes after midnight).
* See the [https://developer.mozilla.org/en-US/docs/Mozilla/QA/Avoiding_intermittent_oranges best practices for avoiding intermittent failures (oranges)].


==== Has sufficient documentation ====
==== Has sufficient documentation ====
Line 54: Line 36:
* That wiki page is linked to from https://developer.mozilla.org/docs/Mozilla/QA/Automated_testing
* That wiki page is linked to from https://developer.mozilla.org/docs/Mozilla/QA/Automated_testing


== Additional requirements for tier 1 jobs ==
== Additional requirements for Tier 1 jobs ==


Tier 1 jobs are those that ... #todo (also reference https://developer.mozilla.org/en-US/docs/Supported_build_configurations). In addition to the requirements above, tier 1 jobs must also meet the following.
Tier 1 jobs are those that run on a [https://developer.mozilla.org/docs/Supported_build_configurations Tier-1 platform] and are monitored by the sheriffing team for new failures, bustage, etc. Patches that cause builds to break or tests to fail will be backed out immediately and trees will be closed if necessary. New intermittent failures will be filed and can potentially lead to a backout or [https://wiki.mozilla.org/Sheriffing/Test_Disabling_Policy test disabling]. In addition to the requirements above, Tier 1 jobs must also meet the following.


==== Breakage is expected to be followed by tree closure or backout ====
==== Breakage is expected to be followed by tree closure or backout ====
Line 75: Line 57:
==== Scheduled on every push ====
==== Scheduled on every push ====
* Otherwise job failures will not be attributable to a single changeset, resulting in either tree closure or backout of multiple pushes (see requirement #2).
* Otherwise job failures will not be attributable to a single changeset, resulting in either tree closure or backout of multiple pushes (see requirement #2).
* An exception is made for nightly builds with an virtually equivalent non-nightly variant that is built on every push & for tests run on PGO builds (given that PGO builds take an inordinate amount of time, we still schedule them every 3/6 hours depending on tree, and relatively speaking there are not too many PGO-only test failures).
* An exception is made for nightly builds with an virtually equivalent non-nightly variant that is built on every push & for tests run on PGO builds (given that PGO builds take an inordinate amount of time, we still schedule them every 3/6 hours depending on tree, and relatively speaking there are not too many PGO-only test failures). Periodic builds have also been granted an exception as they don't run tests and have sufficient coverage on other platforms such that the odds of unique bustage are small and relatively easy to diagnose.
* Note also that coalescing (buildbot queue collapsing when there is more than one queued job of the exact same tree/type) may mean that not all scheduled jobs actually get run. Whilst coalescing makes sheriffing harder, it's a necessary evil given that automation infrastructure demand frequently outstrips supply.
* Note also that coalescing (buildbot queue collapsing when there is more than one queued job of the exact same tree/type) may mean that not all scheduled jobs actually get run. Whilst coalescing makes sheriffing harder, it's a necessary evil given that automation infrastructure demand frequently outstrips supply.
==== Must avoid patterns known to cause non deterministic failures ====
* Must avoid pulling the tip of external repositories as part of the build - since landings there can cause non-obvious failures. If an external repository is absolutely necessary, instead reference the desired changeset from a manifest in mozilla-central (like talos or gaia do).
* Must not rely on resources from sites whose content we do not control/have no SLA:
** Since these will cause failures when the external site is unavailable, as well as impacting end to end times & adding noise to performance tests.
** eg: Emulator/driver binaries direct from a vendor's site, package downloads from PyPi or page assets for unit/performance tests.
** Ensure MOZ_DISABLE_NONLOCAL_CONNECTIONS is defined in the automation environment (see {{bug|995417}}) & use a list of automation prefs for switching off undesirable behaviour (eg automatic updates, telemetry pings; see {{bug|1023483}} for where these are set).
* Must not contain time bombs, e.g. tests that will fail after a certain date or when run at certain times (e.g., the day summer time starts or ends, or when the test starts before midnight and finishes after midnight).
* See the [https://developer.mozilla.org/en-US/docs/Mozilla/QA/Avoiding_intermittent_oranges best practices for avoiding intermittent failures (oranges)].
==== Low intermittent failure rate ====
* A high failure rate:
** Causes unnecessary sheriff workload.
** Affects the ability to sheriff the trees as a whole, particularly during times of heavy coalescing.
** Undermines devs confidence in the platform/test-suite - which as demonstrated by Firefox for Android, permanently affects their willingness to believe any future failures, even once the intermittent-failure rate is lowered.
* A mozilla-central push results in ~400 jobs. The typical OrangeFactor across all trunk trees is normally (excluding the recent spike) 3-4, ie: a failure rate of ~1%.
* Therefore as a rough guide a new platform/testsuite must have at most a 5% per job failure rate initially, and ideally <1% longer term.
* However, sheriffs will make the final determination of whether a job type has too many intermittent failures. This will be a based on a combination of factors including failure rate, length of time the failures have been occurring, owner interest in fixing them & whether Treeherder is able to make bug suggestions.


==== Easily run on try server ====
==== Easily run on try server ====
Line 100: Line 100:
* For Try specifically, you can request that the job type by made non-default (ie requires explicit opt-in when using trychooser syntax, and won't be scheduled with '-u all' or similar), in order to be shown in the default view on Try - [http://hg.mozilla.org/build/buildbot-configs/file/27dff21cb799/mozilla/config.py#l332 example].
* For Try specifically, you can request that the job type by made non-default (ie requires explicit opt-in when using trychooser syntax, and won't be scheduled with '-u all' or similar), in order to be shown in the default view on Try - [http://hg.mozilla.org/build/buildbot-configs/file/27dff21cb799/mozilla/config.py#l332 example].


== My platform/test-suite does not meet the additional tier 1 requirements, what now? ==
== My platform/test-suite does not meet all of the additional Tier 1 requirements, what now? ==
* Once Treeherder UI changes are made, we'll be able to have tier 2 jobs shown in the default view, without them impacting sheriffing workflows. However before these changes are made, tier 2 jobs will need to either meet the requirements for tier 1 jobs, or else be hidden from the default view
* Treeherder's UI now supports Tier 2 jobs as well. These will still be visible in the default UI. However, they do not come with the guarantee of continuous sheriff monitoring, immediate backouts, etc. If a Tier 2 job begins to fail, a bug will be filed with the suite owner notified to resolve the regression.


== The future ==
== The future ==
<b><big>xxxRyan: IS THIS ACCURATE ANYMORE?</big></b>
* Planned improvements to our tooling will likely mean that some of these requirements can be relaxed in the future, as well as making it easier for maintainers of non-default-view job types to track their success/failure without having to monitor Treeherder continuously.
* Planned improvements to our tooling will likely mean that some of these requirements can be relaxed in the future, as well as making it easier for maintainers of non-default-view job types to track their success/failure without having to monitor Treeherder continuously.
* Planned features for [[Auto-tools/Projects/Treeherder]] include:
* Planned features for [[Auto-tools/Projects/Treeherder]] include:
RyanVM
Confirmed users
1,052

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1077592"

Navigation menu