User:RyanVM/Job Visibility Policy Updates: Difference between revisions

Jump to navigation Jump to search

User:RyanVM/Job Visibility Policy Updates (view source)

Revision as of 20:25, 26 June 2015

6 bytes added ,  26 June 2015
no edit summary
No edit summary
No edit summary
Line 1: Line 1:
This page exists to clarify the policy towards how jobs reporting to [https://treeherder.mozilla.org/ Treeherder] are managed. Common sense will apply in cases where some of the requirements are not applicable for a particular platform/build/test type.


== Requirements for jobs shown in the default Treeherder view ==
To propose changes to this policy, please speak to the sheriffs and/or send a message to [https://lists.mozilla.org/listinfo/dev-tree-management dev.tree-management].


This page was created to clarify the requirements that a platform/test-suite has to meet before its jobs can be shown in the default [https://treeherder.mozilla.org/ Treeherder] view for the main development trees. Owners of non-sheriff managed project/disposable repos do not need to meet these requirements before requesting visibility changes. The below section applies to both Tier 1 and Tier 2 jobs.
== Overview of the Job Visibility Tiers ==
Jobs reporting to Treeherder can fall into three tiers.
* <b>Tier 1:</b> Jobs that run on a [https://developer.mozilla.org/docs/Supported_build_configurations Tier-1 platform], are shown by default on Treeherder, and are sheriff-managed. Bustage will cause a tree closure and is expected to result in a quick follow-up push or a backout (at the discretion of the sheriff on duty). Bugs will be filed for new intermittent test failures and are subject to the [https://wiki.mozilla.org/Sheriffing/Test_Disabling_Policy Test Disabling Policy] if not addressed in a timely fashion.
* <b>Tier 2:</b> Jobs are shown by default on Treeherder, but are not sheriff-managed. Results will be shown on Treeherder "for information only". New test failures/bustage will not result in a backout, but a tracking bug will be filed when observed.
* <b>Tier 3:</b> Jobs are not shown by default on Treeherder. All responsibilities for monitoring the results will fall upon the owner of the job.


Common sense will apply in cases where some of the requirements are not applicable for a particular platform/build/test type.
== Requirements for jobs shown in the default Treeherder view ==
 
The below section applies to <b>both Tier 1 and Tier 2 jobs</b>. Owners of non-sheriff managed project/disposable repos do not need to meet these requirements. However, they must be satisfied prior to being enabled in production.
To propose changes to this policy, please speak to the sheriffs and/or post to [https://lists.mozilla.org/listinfo/dev-tree-management dev.tree-management].


==== Has an active owner ====
==== Has an active owner ====
Line 37: Line 41:


== Additional requirements for Tier 1 jobs ==
== Additional requirements for Tier 1 jobs ==
Tier 1 jobs are those that run on a [https://developer.mozilla.org/docs/Supported_build_configurations Tier-1 platform] and are monitored by the sheriffing team for new failures, bustage, etc. Patches that cause builds to break or tests to fail will be backed out immediately and trees will be closed if necessary. New intermittent failures will be filed and can potentially lead to a backout or [https://wiki.mozilla.org/Sheriffing/Test_Disabling_Policy test disabling]. In addition to the requirements above, Tier 1 jobs must also meet the following.


==== Breakage is expected to be followed by tree closure or backout ====
==== Breakage is expected to be followed by tree closure or backout ====
* Failures visible in the default view (other than those that are known intermittents/transient), must have their cause backed out in a timely fashion or else the tree closed until diagnosed.
* Failures visible in the default view (other than those that are known intermittents/transient), must have their cause backed out in a timely fashion or else the tree closed until diagnosed.
* Why? If tier != 1 jobs were instead made visible in the default view, they would:
* Why? If Tier != 1 jobs were treated the same, they would:
** Interfere with ability to sheriff the tree:
** Interfere with ability to sheriff the tree:
*** Indistinguishable from tier-1 failures.
*** Indistinguishable from Tier 1 failures.
*** Appear in the failure count/cause the tab to glow.
*** Appear in the failure count/cause the tab to glow.
*** Slow down navigation of failures when using keyboard shortcuts.
*** Slow down navigation of failures when using keyboard shortcuts.
Line 52: Line 54:


==== Runs on mozilla-central and all trees that merge into it ====
==== Runs on mozilla-central and all trees that merge into it ====
* Otherwise job failures when tree X merges into mozilla-central will not be attributable to a single changeset, resulting in either tree closure or backout of the entire merge (see requirement #2).
* Otherwise job failures when tree X merges into mozilla-central will not be attributable to a single changeset, resulting in either tree closure or backout of the entire merge (see the previous requirement).
* When filing the release engineering bug to enable your job on all the required trees, ask to enable it on "mozilla-central based trees" and release engineering will enable it in the default config from which all trunk trees inherit (unless the various tree owners have explicitly opted out). As a rough guide, mozilla-central based trees include mozilla-inbound, fx-team, b2g-inbound as well as many of the other project/disposable repositories.
* When filing the release engineering bug to enable your job on all the required trees, ask to enable it on "mozilla-central based trees" and release engineering will enable it in the default config from which all trunk trees inherit (unless the various tree owners have explicitly opted out). As a rough guide, mozilla-central based trees include mozilla-inbound, fx-team, b2g-inbound as well as many of the other project/disposable repositories.


Line 96: Line 98:
== My platform/test-suite does not meet the base requirements, what now? ==
== My platform/test-suite does not meet the base requirements, what now? ==
* Your platform/test-suite will still be being run, just not shown on the default view. This model has worked well for many projects/build types (eg jetpack, xulrunner, spidermonkey).
* Your platform/test-suite will still be being run, just not shown on the default view. This model has worked well for many projects/build types (eg jetpack, xulrunner, spidermonkey).
* To see it, click the "show/hide hidden jobs" checkbox to the left of the quick filter input field in the Treeherder UI. ({{bug|1078081}} will add URL parameter support for this soon).
* To see it, click the "show/hide hidden jobs" checkbox to the left of the quick filter input field in the Treeherder UI. Alternatively, |&exclusion_profile=false| can be added to the URL to show all hidden jobs.
* To filter the jobs displayed, under the 'Filters' menu use the 'job name' field (which supports regexp).
* To filter the jobs displayed, under the 'Filters' menu use the 'job name' field.
* For Try specifically, you can request that the job type by made non-default (ie requires explicit opt-in when using trychooser syntax, and won't be scheduled with '-u all' or similar), in order to be shown in the default view on Try - [http://hg.mozilla.org/build/buildbot-configs/file/27dff21cb799/mozilla/config.py#l332 example].
* For Try specifically, you can request that the job type by made non-default (ie requires explicit opt-in when using trychooser syntax, and won't be scheduled with '-u all' or similar), in order to be shown in the default view on Try - [http://hg.mozilla.org/build/buildbot-configs/file/27dff21cb799/mozilla/config.py#l332 example].
== My platform/test-suite does not meet all of the additional Tier 1 requirements, what now? ==
* Treeherder's UI now supports Tier 2 jobs as well. These will still be visible in the default UI. However, they do not come with the guarantee of continuous sheriff monitoring, immediate backouts, etc. If a Tier 2 job begins to fail, a bug will be filed with the suite owner notified to resolve the regression.
RyanVM
Confirmed users
1,052

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1081682"

Navigation menu