Confirmed users
3,376
edits
Auto-tools/Projects/Stockwell/disable-recommended: Difference between revisions
Auto-tools/Projects/Stockwell/disable-recommended (view source)
Revision as of 16:20, 9 February 2018
, 9 February 2018→Determining which platforms are affected: - example of all
(→Finding the manifest: - summary) |
m (→Determining which platforms are affected: - example of all) |
||
| Line 47: | Line 47: | ||
=== Determining which platforms are affected === | === Determining which platforms are affected === | ||
On OrangeFactor, check the detail view for the bug for the last 30 days. Look down the list of bugs for affected Platforms and Build Types. Does this test need to be disabled on all platforms, or only on some? For opt or debug builds, or both? | On OrangeFactor, check the detail view for the bug for the last 30 days. Look down the list of bugs for affected Platforms and Build Types. Does this test need to be disabled on all platforms, or only on some? For opt or debug builds, or both? | ||
There are many reasons why a test might only fail on certain configs: | |||
* we don't run that test on a specific platform (for example devtools on android) | |||
* the test is skipped already on a configuration or platform | |||
We typically have 4 platforms: | |||
* Linux (opt|debug|pgo|asan|ccov) - also bits (32|64) | |||
* OSX - (opt|debug) - only run on 10.10 | |||
* Windows - (opt|debug|pgo|ccov) - also Windows 7|10 | |||
* Android - (opt|debug) - x86|arm | |||
** no browser-chrome, devtools, web-platform-tests, and a few others | |||
If the test is failing at least 5 times in the last 7 days on any given config, lets skip it. We cannot skip on pgo specifically, so that is opt (!debug). Often I try to make the skip syntax as simple as possible for simplicity sake and future editing. If we have: | |||
<pre> | |||
[browser_sanity.js] | |||
skip-if = (os == 'linux' && debug && bits == 64) || (os == 'linux' && !debug && bits == 64) || (os == 'osx' && !debug) | |||
</pre> | |||
I would prefer to see: | |||
<pre> | |||
[browser_sanity.js] | |||
skip-if = (os == 'linux' && bits == 64) || (os == 'osx' && !debug) | |||
</pre> | |||
If the test already had a skip if: | |||
<pre> | |||
[browser_sanity.js] | |||
skip-if = (os == 'win') # Bug 3141592 - timeout | |||
</pre> | |||
we would end up with: | |||
<pre> | |||
[browser_sanity.js] | |||
skip-if = true # Bug 3141592, 271289 | |||
</pre> | |||
The reason for this is we would be skipping on linux/windows/osx, but not linux32 or android. Since this test doesn't run on Android, we would only be skipping on linux32- and a test that only runs on 1 platform doesn't provide a lot of value unless it is platform specific. | |||
=== Updating a manifest to disable a test === | === Updating a manifest to disable a test === | ||