- 1 [stockwell disable-recommended]
- 1.1 Finding bugs to disable
- 1.2 Finding the manifest
- 1.3 Determining which platforms are affected
- 1.4 Updating a manifest to disable a test
- 1.5 Making a patch
- 1.6 Requesting review
- 1.7 After the test is disabled
- 1.8 Special cases / getting help
When an intermittent-failure bug has recorded 150 or more failures in the last 21 days, the Orange Factor robot will change the bug's stockwell whiteboard tag to "[stockwell disable-recommended]". These bugs should be reviewed at least twice per week (current schedule is Monday and Thursday).
This page describes how to review [stockwell disable-recommended] bugs.
Finding bugs to disable
- Use bugzilla to find all bugs with whiteboard tag "[stockwell disable-recommended]".
- For each disable-recommended bug, determine if the bug can be addressed by disabling tests.
- Ignore "meta" bugs or any bugs not related to a particular test, some examples:
- Eliminate bugs that have been recently fixed.
- Eliminate bugs with a patch under review, or coming very soon.
- Review recent bug history.
- If there is a patch under review or comments indicate that a fix is coming soon, consider postponing action on this bug.
Finding the manifest
For each test to be disabled, find the associated test manifest:
- Search for the test name in searchfox or dxr, or,
- Run 'mach test-info --show-info <test-name>' in a mozilla-central check-out.
$ ./mach test-info --show-info browser/base/content/test/general/browser_ctrlTab.js ===== browser/base/content/test/general/browser_ctrlTab.js ===== Found browser\base\content\test\general\browser_ctrlTab.js in source control. Build configuration changed. Regenerating backend. browser\base\content\test\general\browser_ctrlTab.js found in manifest browser/base/content/test/general/browser.ini flavor: browser-chrome
In the above cases you would want to edit browser/base/content/test/general/browser.ini
$ pwd $ /home/mozilla/mozilla-central $ nano browser/base/content/test/general/browser.ini
Determining which platforms are affected
On OrangeFactor, check the detail view for the bug for the last 30 days. Look down the list of bugs for affected Platforms and Build Types. Does this test need to be disabled on all platforms, or only on some? For opt or debug builds, or both?
There are many reasons why a test might only fail on certain configs:
- we don't run that test on a specific platform (for example devtools on android)
- the test is skipped already on a configuration or platform
We typically have 4 platforms:
- Linux (opt|debug|pgo|asan|ccov) - also bits (32|64)
- OSX - (opt|debug) - only run on 10.10
- Windows - (opt|debug|pgo|ccov) - also Windows 7|10
- Android - (opt|debug) - x86|arm
- no browser-chrome, devtools, web-platform-tests, and a few others
If the test is failing at least 5 times in the last 7 days on any given config, lets skip it. We cannot skip on pgo specifically, so that is opt (!debug). Often I try to make the skip syntax as simple as possible for simplicity sake and future editing. If we have:
[browser_sanity.js] skip-if = (os == 'linux' && debug && bits == 64) || (os == 'linux' && !debug && bits == 64) || (os == 'osx' && !debug)
I would prefer to see:
[browser_sanity.js] skip-if = (os == 'linux' && bits == 64) || (os == 'osx' && !debug)
If the test already had a skip if:
[browser_sanity.js] skip-if = (os == 'win') # Bug 3141592 - timeout
we would end up with:
[browser_sanity.js] skip-if = true # Bug 3141592, 271289
The reason for this is we would be skipping on linux/windows/osx, but not linux32 or android. Since this test doesn't run on Android, we would only be skipping on linux32 - and a test that only runs on 1 platform doesn't provide a lot of value unless it is platform specific.
Updating a manifest to disable a test
There are 3 different types of manifest files, each with a different format:
- "ini" manifests like mochitest.ini, chrome.ini, browser.ini, xpcshell.ini
- reftest or "list" manifests like reftest.list, crashtest.list
- web-platform manifests in testing/web-platform/meta
- Manifest example: https://searchfox.org/mozilla-central/source/browser/base/content/test/about/browser.ini
- In this type of manifest, add a "skip-if" statement under the test name -- on the next line.
[some_test.html] skip-if = ...
- Some examples of "skip-if":
skip-if = true # skip this test everywhere / always (test never runs) skip-if = os == "android" # skip on Android only (continues to run on Linux, Windows, etc.) # other os strings: "win", "linux", "mac" skip-if = debug # skip on all debug builds skip-if = !debug # skip on all non-debug builds (opt, pgo) skip-if = os == "android" || os == "linux" # skip on Android and skip on Linux skip-if = os == "android" && debug # skip on Android/Debug only (continues to run on Android/Opt)
- For any test, there can only be one "skip-if" line. Often it is necessary to use || to expand the scope of a skip-if. For example, if a manifest already has:
[some_test.html] skip-if = os == "android" || debug
but some_test.html is still failing on Windows/opt, it can be changed to:
[some_test.html] skip-if = os == "android" || debug || os == "win"
- other keywords: os_version, bits, e10s, stylo, webrender, asan
- Windows os versions: https://msdn.microsoft.com/en-us/library/windows/desktop/ms724832(v=vs.85).aspx
- Manifest example: https://searchfox.org/mozilla-central/source/dom/canvas/test/reftest/filters/reftest.list
- In this type of manifest, add a skip-if() statement on the same line as the test:
skip-if(...) == some_test.html some_ref.html
- Some examples of "skip-if":
skip-if(gtkWidget) # skip on Linux skip-if(cocoaWidget) # skip on Mac skip-if(winWidget) # skip on Windows skip-if(Android) # skip on Android skip-if(isDebugBuild) # skip on all Debug builds skip-if(!isDebugBuild) # skip on all non-Debug builds skip-if(Android&&isDebugBuild) # skip on Android/Debug
- Other keywords: stylo, webrender, oscpu
- Reference: https://searchfox.org/mozilla-central/source/layout/tools/reftest/README.txt#42
web-platform test manifests
web-platform tests are found in the mozilla-central repo under testing/web-platform/tests. These tests have their own, unique manifest format. Also, not all tests are listed in a manifest: A manifest is only created when a test needs to be skipped or otherwise annotated.
Manifests for web-platform tests are found in testing/web-platform/meta. The /meta directory structure parallels the /tests structure. For example testing/web-platform/meta/2dcontext/building-paths has manifests related to the tests in testing/web-platform/tests/2dcontext/building-paths.
When disabling a web-platform test, check to see if there is an existing manifest in the /meta directory for that test: If there is, modify the existing manifest as required; if not, add a new file (remember to add the new file to your patch with 'hg add').
Here's an example of a patch disabling testing/web-platform/meta/content-security-policy/reporting/multiple-report-policies.html, on Linux and Windows 10/debug:
diff --git a/testing/web-platform/meta/content-security-policy/reporting/multiple-report-policies.html.ini b/testing/web-platform/meta/content-security-policy/reporting/multiple-report-policies.html.ini new file mode 100644 --- /dev/null +++ b/testing/web-platform/meta/content-security-policy/reporting/multiple-report-policies.html.ini @@ -0,0 +1,4 @@ +[multiple-report-policies.html] + disabled: + if (os == "linux"): https://bugzilla.mozilla.org/show_bug.cgi?id=1435526 + if debug and (os == "win") and (version == "10.0.15063"): https://bugzilla.mozilla.org/show_bug.cgi?id=1435526
Making a patch
To actually disable a test, we need to update the manifest file in mozilla-central. Before doing that, create a mercurial patch containing the change, and upload the patch to bugzilla for review.
Some useful mercurial commands for creating a patch with "mq":
hg qseries # see what patches are in your queue / what is applied hg qdiff # see the file changes in the current patch hg qnew # create a new patch hg qrefresh # update your patch with local file changes hg qrefresh -e # edit the commit message associated with your patch hg add <filename> # add a new file to source control
I typically follow this workflow:
hg qnew bug1234567.patch <edit> <manifest>.ini hg qrefresh hg qdiff # self review hg qrefresh -e <inside editor add comment |Bug 1234567 - disable <test> for frequent failures. r=gbrown| and save> hg qrefresh -u "Joel Maher <email@example.com>" # adds my username to the patch cp .hg/patches/bug1234567.patch ~/ hg qpop
NOTE: the self review is a great time to make sure:
- there are not extra spaces on a blank line or at the end of a line, if there are they will be highlighted in red
- you are only editing one line in a file, not multiple lines on accident
- a chance to review your syntax and any comments
NOTE: the comment for the commit needs to follow the general pattern above: "Bug xxx - <patch description>. r=yyy"
In bugzilla, in the bug with [stockwell disable-recommended], use "Attach File" to attach your patch. Browse to find the patch in <your repo directory>/.hg/patches. Make sure you select "patch" for the Content Type. Then request review, "r?", and select a reviewer. In most cases, ask :gbrown or :jmaher for review.
In the above section it was suggested to copy your patch to ~/ via
cp .hg/patches/bug1234567.patch ~/
In doing this, you could easily find your patch when in bugzilla and looking to attach a file since it will be ~/bug1234567.patch.
A few extra notes here:
- when uploading a patch the checkbox for "assign this bug to myself" is checked by default, uncheck this as you are just disabling a test
- use a description, not the patch name. Something like: |disable test on <platforms>|
There are additional, generic instructions for getting Mozilla reviews at:
After the test is disabled
- "Watch" your patch landing to see if the patch was effective:
- Are there any new failures on your push, or on the next few pushes, of the disabled test?
- Or, check OrangeFactor (maybe the next day) to see if any new failures are reported.
Special cases / getting help
There are lots of "special cases" -- different types of tests, platform variations, and who knows what! If something seems odd, if you need more information, we're here to help: Ping/email :gbrown / :jmaher.