Sheriffing/How To/Getting started as a sheriff< Sheriffing
So you want to be a Sheriff? Here's how to start.
Things You'll Need
Not all of these have to be ready on Day One, but eventually you'll need the following.
- IRC account, for communicating with developers, fellow sheriffs, and others.
- Clone of the Mercurial repositories, for committing changes to the repositories.
- A properly configured mercurial setup. You can use
./mach mercurial-setupfrom your clone of the repository to run through a setup wizard. I recommend setting up the qbackout extension to aid in easily backing out patches.
- LDAP account, for signing in to Treeherder and pushing commits to the source code repositories.
- A modern browser, to access Treeherder, Bugzilla, and other Mozilla web tools.
Set up Mozilla SSH Key, mercurial and mozilla-unified repo, test push to try
Mozilla has a server for testing patches before they are pushed to the integration repos. Practically, every developer can test their patch without breaking the repo.
You can test the functionality of a patch by creating the job in treeherder using Mercurial.
For that the following need to be done:
IF YOU DO NOT HAVE A SSH KEY in https://login.mozilla.com/ssh_keys_index then:
- Generate a new ssh key for Mozilla. Preferably, the key should be generated in a new Ubuntu virtual machine (VM).
- After you created the VM User in Ubuntu, log in that account with putty and generate the private and public keys using the following command:
ssh-keygen -t rsa -b 4096 -C "Mozilla key". The above command will generate id_rsa and id_rsa.pub in ~user/.ssh/
- At Enter passphrase you need to type in a password which you will not forget!
- After generating the key, the content of id_rsa.pub needs to be added here: https://login.mozilla.com/ssh_keys_index . If you already have a key, make sure the public one is in https://login.mozilla.com/ssh_keys_index and the private one in the new Ubuntu user, in id_rsa.
- After creating/setting the key, create the following file ~user/.ssh/config and add the following lines (in the email section add your Mozilla email address):
- Host hg.mozilla.org
- User email@example.com
- For additional debug info, ssh -vvv hg.mozilla.org is often helpful.
Mozilla recommends using the latest Mercurial version. We’ll need to install it. If you have any other version installed you can uninstall it with apt-get remove mercurial and install the new version using the following commands (with root):
apt-get install python-pip -y
pip install Mercurial
Setting up the mozilla-unified repo
Now that you have Mercurial installed, we will make a clone of mozilla-unified.
Log out of root, remain on your new user account and run the following command in the home of your current user:
hg clone --uncompressed https://hg.mozilla.org/mozilla-unified .
Wait until the download is finished (about 5-10 min) and the mozilla-unified folder will be created.
Setting up Mercurial for Mozilla
- After Mozilla-Unified has been downloaded, access the folder (
cd mozilla-unified) and run the below command:
- In the name and email sections, type your Mozilla account details, for all other questions just answer Y.
- After you’ve finished the above setup, download .hgrc.txt file and save it in /home/user under the name .hgrc . Overwrite the file with the existing one and edit it: type your name and surname and your Mozilla user in the username section. Also, in the lower part of the file, type only your username.
- Access the mozilla-unified folder and download autoland repo using this command:
hg pull autoland
- After the above command finished downloading autoland, edit .hg/firefoxtrees and delete all the lines except autoland, inbound, central, active esrs (check the ones available on Treeherder), beta and release.
Note: If only AUTOLAND is in that file, you need to run the following command:
hg pull beta && hg pull central && hg pull inbound && hg pull esr
If you’ve come this far, you only need to run the following commands:
hg pull fxtrees
hg update <repo>//repo = autoland/inbound/central/esr/release/beta
Check the content of the .hgrc file in the home folder, it shall looks similar to this:
[ui] username = Your Name <firstname.lastname@example.org> merge = internal:merge interface = curses [alias] trymc = push -f ssh://hg.mozilla.org/try [defaults] commit = -v diff = -U 8 qdiff = -U 8 qnew = -U [diff] git = 1 showfunc = 1 unified = 8 [extensions] blackbox = bzexport = /home/user/.mozbuild/version-control-tools/hgext/bzexport firefoxtree = /home/user/.mozbuild/version-control-tools/hgext/firefoxtree hgext.churn = histedit = mozext = /home/user/.mozbuild/version-control-tools/hgext/mozext progress = purge = push-to-try = /home/user/.mozbuild/version-control-tools/hgext/push-to-try rebase = strip = transplant = absorb = # js-format = /home/user/.mozbuild/version-control-tools/hgext/js-format # format-source = /home/user/.mozbuild/version-control-tools/hgext/format-source [hooks] [experimental] clonebundles = true [bugzilla] username = email@example.com
Test push to try repo
After the setup is finished, run the following test command:
./mach try -b o -p linux -u mochitest-1 -t none
!after a change of ssh keys, if you hit the following error when pushing to try: Remote: sign_and_send_pubkey: signing failed: agent refused operation then you’ll need to delete the current ssh agent and add the new one using these commands:
ssh -add -d
Afterwards you can run
./mach try -b o -p linux -u mochitest-1 -t none
This command will create a push in treeherder, and if all goes well, that push will appear here: https://treeherder.mozilla.org/#/jobs?repo=try . The output of the test should look like this:
./mach try -b o -p linux -u mochitest-1 -t none Creating temporary commit for remote... pushing to ssh://hg.mozilla.org/try Enter passphrase for key '/home/arny/.ssh/id_rsa': searching for changes remote: adding changesets remote: adding manifests remote: adding file changes remote: added 1 changesets with 0 changes to 0 files (+1 heads) remote: recorded push in pushlog remote: remote: View your change here: remote: https://hg.mozilla.org/try/rev/a9e732c8d4df66b2961f476652765ab451dcdeae remote: remote: Follow the progress of your build on Treeherder: remote: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a9e732c8d4df66b2961f476652765ab451dcdeae remote: recorded changegroup in replication log in 0.008s push complete temporary commit removed, repository restored
The Sheriffs How-To
A lot of knowledge about what sheriffs do is built up by day-to-day experiences on the job. We've created a number of wiki pages documenting a lot of this information, indexed here.
Some of the more helpful (and up to date) pages:
- Sheriffing/How:To:Treeherder - Explains some useful terminology, what each part of Treeherder's UI means/does, and how to use Treeherder to classify failures
- Sheriffing/How:To:SheriffingFromUnifiedRepos - This is useful, though some of the referenced repositories are no longer sheriffed, and it assumes you have knowledge of a lot of Mozilla/mercurial terms.
Sheriffs hang out in the sheriffs channel on Matrix - also it's useful to be in the following Channels:
- CI Automation: For questions about test automation.
- developers: To find developers or e.g. provide status updates on longer tree closures
- Firefox CI: Escalate e.g. issues with hardware backlog here - the ones in the 'Hardware' table of the backlog dashboard
- release-drivers: Find the Release Managers (the people who approve beta uplifts) here and ask questions or escalate issues related to that
- releaseduty: For issues with tasks belonging to a release, e.g. the 'UV' ones on beta etc.
- releng: For general issues with the release build infrastructure
- Taskcluster - best to check #taskclouster-cloudops on Slack before if the Taskcluster system got updated
- treeherder: The website to manage build, test and performance results
A more-complete list of channels is listed here.
Also join the sheriffs mailing list.
Mozilla has dozens of different source code repositories for various projects and releases, but sheriffs only need to care about a few of them:
- mozilla-central, the main development repository.
- mozilla-inbound, one of mozilla-central's integration branches where new commits initially land for testing.
- autoland, another integration branch for mozilla-central.
- mozilla-beta, one of the release stabilization branches for code that has already gone through several weeks of testing.
- mozilla-release, the official release branch, where Firefox code goes to become part of an official build of Firefox.
- mozilla-esrXX, one or more branches set up to track patches for the current (and at times when the ESR releases overlap, the previous) version of Firefox ESR. (At the moment, this is mozilla-esr52.)
Once you're up to speed, it's best to leave a tab open to each of these repositories throughout your work day so you can track failures as they come in.
Sheriffs do a lot of things throughout the day.
- Watch the sheriffed repositories via Treeherder for failures.
- Classify/Star known intermittent failures.
- File tracking bugs for newly discovered intermittent failures.
- Resolve issues with code that has landed. This can be done with some of the following:
- Backout the bad code.
- Ensure that developers land followup patches to fix issues that are discovered.
- Monitor the repositories and tests for any issues related to infrastructure issues.
- Mark repositories as being unable to accept commits from developers (also called "close the trees") via Treestatus, allowing you to fix issues without a bunch of new potentially-bad commits from landing in the interim.
- Merge known-good commits from the mozilla-inbound and autoland repositories over to mozilla-central, and then back around to mozilla-inbound and autoland.
- Once the actual merge is performed, you need to run the Bugherder tool against the merge on mozilla-central to get the information properly updated in Bugzilla. There's a link to Bugherder in the per-push menu item in Treeherder.
- Landing approved patches that are marked as "checkin-needed" onto mozilla-inbound or autoland.
- Backporting approved patches (also called "uplifting") from mozilla-central to one or more of the release stabilization branches (mozilla-beta, mozilla-release, mozilla-esr52, etc) to get the patch into users' hands sooner.
The following permission have to be set for the mentioned actions:
- Subscribe to sheriffs mailing list
- Get access to document about escalating infrastructure issues from Aryx
- commit access level 1 (scm_level_1 LDAP group):
- save job classifications
- push to Try server (test patches, do beta simulations etc.)
- commit access level 3 (scm_level_3 LDAP group):
- push to mozilla-central etc. (not autoland or modify servo/, see below)
- land patches from within the browser with Lando/Phabricator
- retrigger jobs
- request to run job on previous pushes where it might have been skipped ("Backfilling")
- request to add selected jobs to a push
- scm_allow_direct_push LDAP group
- push directly to trees without using Phabricator/Lando to land backouts
- scm_autoland LDAP group:
- push directly to autoland (needed for merges and some backouts)
- scm_servo_vendor LDAP group:
- allow to changes files in the servo/ folder (needed for merges and some backouts)
- sheriff LDAP group:
- terminate machines running tests
- cancel all jobs on non-Try pushes
- manually request Nightlies (?)
- vpn_treestatus LDAP group:
- change the status of the trees (e.g. close on test failures or infrastructure issues)
- is_staff set for user account by Treeherder admins:
- request to run all missing jobs from menu for a push
- request to run all missing Talos jobs from menu for a push
- document edit access
- bugs not suggested by Treeherder
- beta simulation document containing past beta simulations, issues found and list of issues still active