Sheriffing/How To/Retrigger Jobs
Sometimes builds and tests need to be retriggered. For some classes of automation/infrastructure failures, this happens automatically and the job is marked in the Treeherder UI as dark blue. For other cases, or if you're doing investigative work e.g. testing for an intermittent failure, you'll need to retrigger the job manually. Retriggering a job will cause the certain test to be run again.
Tasks which belong to the release process to actually ship a build to users must not be retriggered or backfilled else the task chain will break. These tasks can be found e.g. on beta and release trees. Examples are the "UV" or "Snap" tasks. If a task name at the bottom left of treeherder starts with "release-", it's a release task. If you are unsure about it, ask in Firefox CI on Matrix.
- Select a job result in Treeherder and click on it.
- This will display a results pane in the left bottom corner with information like Job, Machine, Task, etc.
- To retrigger this job/test, click on the circular arrow with the mouseover text of "Repeat the selected job" at the top of the results pane and the job will be retriggered. You can accomplish the same thing by simply pressing "r" in the results pane when you are logged in to Treeherder.
- If you want to manually retrigger multiple jobs, you can add them to the pinboard and click on retrigger all:
Backfilling runs the job on the previous pushes (at the moment: 5) where it often didn’t run (regarded as not necessary or to save resources). To backfill, you select a job and from the panel you click on the “...” next to the retrigger button, and choose the first option:
This comes in handy when you want to determine from which push a certain failure has started (ex: for backouts).
If you want to backfill on a certain number of pushes, click on “...” and then on Custom Action:
- Choose backfill and change the depth to your number of choice:
- And trigger.
Retriggering Nightly Builds
If new Nightlies have to be requested - be it for a backout or because the merge had to be later than expected - wait for the normal 'Gecko Decision Task' to finish and request the Nightlies after that. Due to the new 'shippable' builds, the Nightlies will create far less jobs and reuseh the already running shippable builds.
Nightly builds run at 12:00 / 01:00 AM/PM RO time so if we don't succeed in doing merges to central before that, nightly builds will need to be restarted on the last merge otherwise they will run again on the previous merge.
Note: We only respin nightlies if we miss them by a few minutes or if we need to get something into the next nightly( for example - fixes for crashes). If they have been running for more than half an hour, we won’t retrigger them.
Note: If you request new Nightlies, only request them for the platforms you canceled. Firefox for Desktop (Windows, Linux, Mac) gets Nightlies twice per day (10am and 10pm UTC) while Firefox for Android only at 10am UTC. If the night shift needs new Nightlies, it shall only request them for desktop.
- Open mozilla-central and type "nightly" in the upper right search box, also select the running jobs (gray) and deselect the rest
- Scroll down to the last merge, you will see "N" builds running
- Pin all the jobs and cancel them
- Pin all nightlies and select “Clear”
- Open https://tools.taskcluster.net and login if necessary
- Click on the following links:
1.To retrigger nightlies for all desktop platforms (there is no hook for just Linux nightlies) https://tools.taskcluster.net/hooks/#project-releng/nightly-desktop%252fmozilla-central 2.To retrigger nightlies for Android https://tools.taskcluster.net/hooks/#project-releng/nightly-fennec%252fmozilla-central
- At the top you will see when the task has been ran (picture is for reference)
- Scroll down and click Trigger hook, a pop-up will be displayed, click Trigger Hook again
Do the same for both links and we've successfully started nightly builds, they will run on the last merge, please check the latest merge for this.
As of Jul 26, 2017 nightly builds for all platforms on m-c are now running in taskcluster so more instructions on how to respin them have been updated for these platforms See https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Other_Duties#Nightlies
CAVEAT: there are implications to triggering too many Nightly builds in a single day or in quick succession. Please talk with a sheriff first before retriggering Nightly builds.
How to bulk retrigger build bustages a push at a time
Please note that this will run all failed jobs again, not only build bustages!
Step 1: Run pip install taskcluster to install a taskcluster component from pip install taskcluster
If you get the message "The program 'pip' is currently not installed" then you have to install it by running: a.
sudo apt install -y python-pipb.
when python-pip install is completed, run pip install taskcluster
Step 2: Save the file and make it executable by running the command: sudo wget https://hg.mozilla.org/build/braindump/raw-file/default/taskcluster/tc-filter.py -P /usr/local/sbin/ && chmod +x /usr/local/sbin/tc-filter.py
Step 3: Re-sign in with the taskcluster tool if you were already signed in
How to use:
- Set the url of the taskcluster instance in which the failing tasks ran (there is also a community taskcluster instance which doesn't get sheriffed):
- sign in with the taskcluster tool ( eval $(taskcluster signin) | if you were not already signed in)
- run tc-filter.py --state failed --action rerun --graph-id geckoDecisionTaskTaskId
Note: Replace geckoDecisionTaskTaskId with the task id being shown on the bottom left when you click on the gecko decision task for the push with the failures.
Rerunning build bustages
How to install taskcluster CLI
This tool is needed in order to retrigger some build jobs, especially nightly builds. Download Taskcluster CLI on Ubuntu from https://github.com/taskcluster/taskcluster-cli
From your /home/user folder (or the location where mozilla-unified is stored), run the following commands:
- sudo wget https://index.taskcluster.net/v1/task/project.taskcluster.taskcluster-cli.latest/artifacts/public/linux-amd64/taskcluster -P /usr/local/sbin/
- sudo chmod +x /usr/local/sbin/taskcluster
The tool is now installed and made executable in /usr/local/sbin/.
How to use Taskcluster CLI
- From the terminal, run the command: eval $(taskcluster signin). This tool will only work as long as the terminal remains open.
- When the browser page opens, login using LDAP
- Click Create a new clientId and go to the end of the page, then click Create Client.
- Wait a few seconds, then close the browser.
- In the console, the following message should appear: Credentials output as environment variables.
- Run taskcluster task rerun TASK_ID (take the TASK_ID from the job summary – go to Treeherder, click the job and on the left side of the window you have Task)
- After following these steps, the console should output either running or pending.