Confirmed users
611
edits
Sheriffing/How To/Intermittent bugs: Difference between revisions
Sheriffing/How To/Intermittent bugs (view source)
Revision as of 16:09, 29 October 2025
, 29 October 2025add handling of wpt backlog failures on autoland
(→How to file a security bug: mention Treeherder's support for filing security bugs and that manual process is a fallback) |
(add handling of wpt backlog failures on autoland) |
||
| (9 intermediate revisions by the same user not shown) | |||
| Line 6: | Line 6: | ||
[[File:Treeherder suggestion.png|Treeherder suggestions]] | [[File:Treeherder suggestion.png|Treeherder suggestions]] | ||
If a test failure has a suggestion with the test name and "single tracking bug" appended, it should be used for the failure classification if the failure is expected to be intermittent. Example: <code>Intermittent devtools/client/aboutdebugging/test/browser/browser_aboutdebugging_addons_debug_popup.js | single tracking bug</code> | |||
==wpt failures on autoland== | |||
Backlog tasks for web platform tests run on autoland but only on Linux debug. On mozilla-central, the opt versions runs for all three desktop operating systems. If a permanent unexpected result is noticed and shall be handled with an update to the test expectation, the other "wpt" backlog tasks (= 1-2 per platform) shall be requested first to check if more platforms than Linux need to be updated. | |||
==TV (test-verify) failure to test suite mapping== | ==TV (test-verify) failure to test suite mapping== | ||
| Line 36: | Line 31: | ||
If you are not sure if a failure line is a new frequent failure or even a perma-failure and require a backout of the change which causes the failure, you can request more runs of the job: | If you are not sure if a failure line is a new frequent failure or even a perma-failure and require a backout of the change which causes the failure, you can request more runs of the job: | ||
* On the same push ("retrigger") by pressing the '''r''' key or clicking the button (circular arrow). | * On the same push ("retrigger") by pressing the '''r''' key or clicking the button (circular arrow). | ||
** The glove icon [[File:Treeherder - intermittent failed and successful run indicator.png|left]] for a failed task indicates there have been successful runs of the task with the same configuration (e.g. platform) on the same push - the failure is verified to be intermittent. | |||
* On previous pushes by opening the actions menu in the bottom toolbar ("...") and and calling "Backfill". This will run the job on the 5 previous pushes, independent if it already ran for those pushes or not. If the job depends on a build which is missing, that build will be generated before the test runs. | * On previous pushes by opening the actions menu in the bottom toolbar ("...") and and calling "Backfill". This will run the job on the 5 previous pushes, independent if it already ran for those pushes or not. If the job depends on a build which is missing, that build will be generated before the test runs. | ||
** A quicker way - if supported - to check if this fails frequently is test-verify backfill. | ** A quicker way - if supported - to check if this fails frequently is test-verify backfill. | ||
| Line 56: | Line 52: | ||
* As filter value, enter the path of the folder which contains the test but without test name. E.g. if you want to find toolkit/components/passwordmgr/test/browser/browser_doorhanger_generated_password.js the search should be for toolkit/components/passwordmgr/test/browser/ | * As filter value, enter the path of the folder which contains the test but without test name. E.g. if you want to find toolkit/components/passwordmgr/test/browser/browser_doorhanger_generated_password.js the search should be for toolkit/components/passwordmgr/test/browser/ | ||
* Press Enter. | * Press Enter. | ||
= Mobile Failures = | |||
If tasks for mobile products like Firefox for Android failure, they need to retriggered until they succeed or the changeset causing the permanent issue has been backed out if the task group ends with "-debug", "-beta" or "-nightly". Without the successful run, the mobile Nightly does not ship to users. | |||
= General failure messages - deciding if new bug needed = | = General failure messages - deciding if new bug needed = | ||
| Line 67: | Line 66: | ||
If a task fails with a message mentioning "ADBError" or something which starts with "ADB" and ends with "Error", it's a failure returned by the phone which runs the task to the controlling computer. Each failure message needs its own bug (e.g. for "init failed", "failed to create directory" etc.) | If a task fails with a message mentioning "ADBError" or something which starts with "ADB" and ends with "Error", it's a failure returned by the phone which runs the task to the controlling computer. Each failure message needs its own bug (e.g. for "init failed", "failed to create directory" etc.) | ||
== Crash reports without human-readable function name == | |||
If a crash mentions a file and a hexadecimal address, e.g. "application crashed [@ libxul.so + 0x17eb4ff]", instead of a function name like "application crashed [@ {virtual override thunk({offset(-16)}, mozilla::net::Http2Session::TakeHttpConnection())} + 0x2d]", the translation of the crash where the crash occurred from machine address to human-readable address ("symbolication") failed. Usually the files "libxul.so" (Linux and Android) or "xul.dll" (Windows) are mentioned and can be observed mostly on systems which limited memory, e.g. Android and Windows 7 (= 32-bit). In these cases, assume such a crash matches a bug suggestion about a crash if there is only one suggestion about a crash. | |||
= How to file a bug for an intermittent failure = | = How to file a bug for an intermittent failure = | ||
| Line 147: | Line 149: | ||
* Run <code>export TASKCLUSTER_ROOT_URL=https://firefox-ci-tc.services.mozilla.com</code> | * Run <code>export TASKCLUSTER_ROOT_URL=https://firefox-ci-tc.services.mozilla.com</code> | ||
* Sign in: <code>eval | * Sign in: <code>eval $(taskcluster signin --scope 'worker-manager:*')</code> | ||
* Log in in the browser, close the browser after you clicked 'Create Client'. | * Log in in the browser, close the browser after you clicked 'Create Client'. | ||
* The syntax for terminating a worker is <code>taskcluster api workerManager removeWorker <workerPoolId> <workerGroup> <workerId></code><br/> | * The syntax for terminating a worker is <code>taskcluster api workerManager removeWorker <workerPoolId> <workerGroup> <workerId></code><br/> | ||
| Line 201: | Line 203: | ||
* The task generation is stored in Firefox Build System :: Task Configuration. | * The task generation is stored in Firefox Build System :: Task Configuration. | ||
If it is necessary to check if an issue is related to Treeherder or Taskcluster, open an affected job by selecting it and then clicking on the task link at the bottom left. Is the job showing as expected in the Taskcluster page, Treeherder might not receive the (correct) data and the bug should be filed against Treeherder, else into Taskcluster. | If it is necessary to check if an issue is related to Treeherder or Taskcluster, open an affected job by selecting it and then clicking on the task link at the bottom left. Is the job showing as expected in the Taskcluster page, Treeherder might not receive the (correct) data and the bug should be filed against Treeherder, else into Taskcluster. | ||
== Reverting classifications ("Unclassification") == | |||
* Classifications of individual tasks can be removed from the "Annotations" tab of the task; the bug number should be removed first. | |||
* The classifications of multiple tasks can be removed by code sheriffs. The tasks have to be pinned to the pinboard and "Unclassify all" from the "save" button's dropdown menu has to be called. | |||
== Nightly / Release task failure == | |||
Builds which shall ship to users of Nightly, Beta, Release and ESR use an existing push and schedule missing and additional tasks, e.g. startup tests and localized builds. These must all complete for the build to be regarded as "good to ship" (the tasks are grouped by platform and if they are for English en-US builds or other locales and shipping gets activated for each of those groups when its tasks complete). | |||
Sometimes it is not obvious if a task must be rerun or in a few special cases won't be shown by Treeherder. To validate all tasks in the release graph completed, open the Taskcluster page which shows all tasks in the release graph. For every non-Nightly release, the <code>taskcluster-firefoxci</code> bot posts it in the [https://chat.mozilla.org/#/room/#releaseduty:mozilla.org #releaseduty] channel (for beta, there are 2 release graphs: 1 for the Developer Edition and 1 for Beta). | |||
To get the corresponding Taskcluster page for a Nightly: | |||
# Open the [https://treeherder.mozilla.org/jobs?repo=mozilla-central Treeherder page for mozilla-central]. | |||
# Find the cron task which scheduled the Nightly release graph. For desktop, it has the symbol `cron(Nd)`. | |||
# Select the task. | |||
# At the bottom left, open the link to the Taskcluster url for the task. | |||
# In the Taskcluster page, click the "Task Group" link near the top left. | |||
If all tasks are listed under the green "Completed" category and the count for all other states is 0, the release should have shipped. | |||