ReleaseEngineering/Day 1 Checklist
- 1 Overview
- 2 Development Best Practices
- 3 Access
- 4 Communication
- 4.1 Mail
- 4.2 Calendar
- 4.3 Bugzilla
- 4.4 Filing bugs against Release Engineering
- 4.5 IRC
- 4.6 Slack
- 4.7 Vidyo Services
- 4.8 Wiki
- 4.9 Pastebin
- 4.10 Google Drive
- 4.11 Mana
- 4.12 Other Resources
- 4.13 Future Access as you need it
Welcome to Release Engineering!
This page is meant to get new hires, interns, or interested community members up to speed with the right software, configurations, and communication channels to contribute effectively to the release engineering pipeline.
- Release Engineering home page
- Video Resources - this page has a mix of introductions, tutorials, and presentations.
Development Best Practices
- Read and keep up to date with: Development Best Practices
Generally, we rely on auth0 across Mozilla for authentication and LDAP for authorization. Once given LDAP and you have created a permanent password, you can use that to login to the SSO portal. From SSO, you should have links to various services from email, irc, calendar, slack, mana, etc. More on each of those later on this page
login.mozilla.com is where you can change a number of authentication/authorization access bits that you have control over. Each todo in this section assumes you have access to this page.
LDAP password reset
If you were given a temporary ldap password or you haven't created your own password yet, you should do this now.
Warning for people who already have an LDAP account: Change your password. Otherwise, adding you to the releng group may lock your account without further notice.
Note:There is a stronger password policy in releng: users must change their password every 3 months. If you don't change your password, the only symptom will be that one day or another (already observed after 8 days), your regular password won't work anymore. If this happens to you, contact people in #servicedesk, they will be able to reset your password.
This is mostly applicable only to employees and interns, although it *is* possible for other contributors to acquire some limited LDAP access. Speak to someone you work with on the releng team if you would like to investigate this.
Upload your public ssh key. It is a good idea to generate a separate ssh keypair from your personal one or any other that you have created in the past and use that explicitly for Releng and upload that. Follow this SSH guidelines doc on how to generate, configure, and use your ssh key.
note: example ssh config for accessing our systems given below in Jumphost section
We use pgp keys to share private information, secrets, and verify that the source came from someone we trust. Generate a keypair for this and upload your public key so others can find it. It would be really good if you could have other people sign your key, adding more trust that this key really belongs to you.
Many of our systems are behind a private network in addition to auth0. Follow the prompts to generate and download an openVPN certificate that you can use to import to your vpn client.
See the instructions on how to install and configure your VPN client and help choosing the right client for your platform.
note: macOS and Windows users should use Viscosity. This application comes with a free 30 day trial. During your trial, your manager can help you create a ServiceNow ticket to get a Viscosity full license.
This MFA account is specific to login.mozilla.com and is used for LDAP/auth0 based logins. Follow the instructions to download the Duo Mobile app and create a Mozilla account.
note: later on in this page we will create more MFA accounts for various systems like Github and accessing our Jumphost
Most development in releng (and at Mozilla writ-large) is stored in version control using hg.
There is an excellent step-by-step guide for setting up and using hg: Mercurial for Mozillians
The root webview of the Mozilla hg repositories is here: https://hg.mozilla.org/
Most releng code lives in repos under https://hg.mozilla.org/build
There are 3 levels of commit access:
- Level 1 access allows you to use the Try Server and setup user repos. As a new contributor, you should request this on day one.
- Level 2 access is required to land code in the build and project repos. Once you have a proven track record of successful patches, you can ask your manager/mentor to vouch for your Level 2 access. Your manager/mentor can also land patches for you until you receive Level 2 access.
- Level 3 access is required to land code in mozilla-central and its derived integration & release branches. At some point in your Mozilla contribution story, you may need Level 3 access but many contributors never do. Talk to your manager/mentor if you think you need this access. You should already have Level 2 access when you request Level 3.
You need to file an IT bug to get hg commit access. Follow the instructions for Becoming a Mozilla Committer, and for Level 2, specify you need access to (at least) hg.mozilla.org/build/* (Product/Component: mozilla.org/Repository Account Requests).
- example request: bug 703351
Git & Github
There are git mirrors of many popular Mozilla repositories. One of the Mozilla github admins (catlee, kmoir, jlund) can add you to the following GitHub groups:
There are also a handful of git repos hosted directly by Mozilla. Your manager/mentor will let you know if you need access to one of these. (See also)
Mozilla mail is handled by Gmail now.
You should be added to the email@example.com google group as a new hire/intern. This mailing list is managed by Google groups. Owners of this group will be able to add you. Send a test message to firstname.lastname@example.org to verify that your address has been added/subscribed. Talk to your manager if it is not working.
WARNING: email@example.com can contain security-sensitive information. Do not automatically forward your email to a system that is not under Mozilla's control.
You'll need to manually subscribe to:
With all that new email, you will want to set up some filters in Gmail (https://mail.google.com/mail/u/0/#settings/filters) to filter some of the higher-volume automated mail into a folder. You may eventually want to handle this information, but on day one hundreds of nagios notifications are not going to be educational.
Here is an imperfect set of Gmail filters that you can import to get you started.
A list of new (and some older) automated emails are indexed by subject, along with relevant actions, here.
If you are going to working on puppet, you should also look at this page on how to read releng shared emails.
Like mail, we now use Google calendar.
You'll want to subscribe to the following public calendars:
Talk to your manager/mentor to get added to the various other private calendars as appropriate.
Almost everything at Mozilla goes through Bugzilla. Create a Bugzilla account if you have not already.
You'll need a few tweaks to your account to get access to everything releng-related:
- Add privileges for bugzilla group "build" (Mozilla Build Team) (Can be done by catlee or bugzilla admin.)
- Add your irc nickname & ldap username as "aliases" for your account
- log into bugzilla & follow links "Preferences" -> "Account Information"
- append the aliases, with a leading ':' and enclosed in brackets ('') to the "Real Name" field
- e.g.: "Chris AtLee [:catlee]"
- QuickSearch help
Filing bugs against Release Engineering
The product to use is, unsurprisingly, "Release Engineering." There are multiple possible components under that product, so take your best guess or ask for guidance in IRC.
The majority of day-to-day communication in releng happens on IRC, and many people use the locally-hosted irccloud instance. Servicedesk has some great getting started tips for IRC.
If you run a traditional IRC client:
- https://wiki.mozilla.org/IRC irc://irc.mozilla.org
- irc.mozilla.org:6697 #mozbuild (use SSL, ask for the channel key; your nick needs to be registered)
- private team channel
* /attach ircs://irc.mozilla.org:6697/?pass=[your_irc_pass] * /join #mozbuild [access_key]
- irc.mozilla.org #releng
- public channel for discussion with sheriffs, and continuous integration bot
- keywords to hilite on:
- "!squirrel" is "I need eyes on this now" keyword in #mozbuild & #releng- please set your client to alert on it.
- "r?" - team member looking for a review, perhaps not in but (irc/pastebin)
- also useful to join
- #developers, #airmozilla, #sf, #mobile, #planning, #release-drivers, #ateam,
- #moco access_key is on mana
- #firebot for hiliting when you're mentioned in a bug, review request, etc.
IT now also provides a hosted IRCCloud cloud instance you can partake of.
- NOTE: if you're using IRCCloud, and are invited to a password protected channel, IRCCloud will not have the password available to re-connect you. Do the following as soon as you enter the channel:
- Click the "Options" button
- Copy the password
- Click the "Leave" button
- Click the "Rejoin" button
- Paste the password into the text box.
Some parts of Mozilla prefer Slack to IRC, more info on mana.
Our primary two way video meeting platform is Vidyo. Basic usage instructions are here. Especially if you are running linux, it is highly recommended that you install the client and make test calls prior to any meeting. Many of our team meetings are held in the ReleaseEngineering room.
- Pro tip: many folks have found the mobile client useful to have preinstalled as a backup device.
- If you're going to record a meeting, practice first. (Instructions are linked from mana page.)
- Ask team members for details on recording in the ReleaseEngineering room.
If you're reading this now, you found the wiki! ;)
Please add yourself and details to the list of team members.
Don't be shy about making improvements to releng pages based on your experiences. Getting someone in releng to review your changes first is good practice. Just ask in #releng.
- Useful templates (aka "macros")
- it creates a new page from stdin, and outputs the url to stdout
- an argument will be interpretted as the syntax highlighting format to use
- sample usage: hg diff | pastebin python | pbcopy (last part mac specific)
- set & export PASTEBIN_NAME if you don't want to post as anonymous.
(Many folks now use the pastebin integrated into IRCCloud.)
Google Drive (formerly Google docs) is a preferred way to share things these days. This includes spreadsheets and documents that will change a great deal over time.
Google Drive access should be enabled with your email account when you start. If you need access to a particular document, talk to the document owner or your manager/mentor.
- RelEng crowd-sourced Glossary of Terms
- Join https://mozillians.org/ (public, and yet-another-set-of-credentials)
- join the "release engineering" group
- Mozilla intranet
- https://mana.mozilla.org/wiki/display/RelEng/Release+Engineering+Home (for sensitive information only) (LDAP Note: Req attribute RelEng)
- https://trello.com/b/KwwYSXE1/release-engineering-status-board (Projects and deliverables status)
- Sign up for O'Reilly's online library of their boooks - a great resource
- Overview of dev cycle: http://k0s.org/mozilla/workflow.svg (slightly dated). And browse http://k0s.org/mozilla
- Releng Tips and TricksFirefox Desktop + Firefox Mobile release process docs:
- Egencia account: this should be accessible via SSO
Future Access as you need it
Talk to your mentor/manager to see which of these make sense. For each section below, request for these as you need them.
File a bug in bugzilla under 'MOC: Service Requests'
We have a Releng AWS account. To get access file a Release Engineering: General ticket and request for a user account with a policy that grants you access to what you need in each service. Also enable MFA
We have a secrets vault that holds access to various passwords and keys. As you need access to various parts of infra, you will need to get access to the vault and then ask for your gpg key be added to the encrypted secret. Talk to your manager as this comes up.
You may have access to the ldap admin page and see your own groups that you have on your record. This page is behind vpn and auth0.
Although you can read your current groups, you will not be able to modify them. To extend with Releng groups that you need. You and your manager will need to file a ticket for them under "MOC: Service Requests"
example ldap groups they may have by default:
cn=corp-vpn,ou=groups,dc=mozilla cn=IntranetWiki,ou=groups,dc=mozilla cn=irccloud,ou=groups,dc=mozilla cn=mfa,ou=groups,dc=mozilla cn=phonebook_access,ou=groups,dc=mozilla cn=team_moco,ou=groups,dc=mozilla cn=vpn_corp,ou=groups,dc=mozilla cn=vpn_default,ou=groups,dc=mozilla
TODO audit these and break them up by security level (Bug 1465535)
Example ldap groups you may need to file for and request added (example, Bug 1434168):
cn=releng,ou=groups,dc=mozilla cn=RelEngWiki,ou=groups,dc=mozilla cn=vpn_releng,ou=groups,dc=mozilla cn=vpn_releng_loan,ou=groups,dc=mozilla cn=vpn_relengwiki,ou=groups,dc=mozilla cn=vpn_tooltooleditor,ou=groups,dc=mozilla cn=nagiosadmin,ou=groups,dc=mozilla cn=GraphsAdmin,ou=groups,dc=mozilla cn=vpn_genericrhel6,ou=groups,dc=mozilla
To access any of Release Engineering, Taskcluster, and Release Operations hosts directly, you will need to go through VPN -> a Jumphost machine -> Separate MFA -> your target host.
To do that, you and your manager will need to file a ticket against Release Operations and have them send you an invite to add an MFA account on your Duo App.
Then once you have your Jumphost MFA setup correctly, you will need to have your ssh config to correctly route through the jumphost before trying the target host you want.
example ssh config:
# Ensure KnownHosts are unreadable if leaked - it is otherwise easier to know which hosts your keys have access to. HashKnownHosts yes # Host keys the client accepts - order here is honored by OpenSSH HostKeyAlgorithms firstname.lastname@example.org,email@example.com,ssh-ed25519,ssh-rsa,firstname.lastname@example.org,email@example.com,firstname.lastname@example.org,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256 Host hg.mozilla.org git.mozilla.org User USERNAME@mozilla.com Compression yes ServerAliveInterval 300 Host *.mozilla.com User USERNAME IdentityFile ~/.ssh/id_rsa_mozilla_2017-05-12 Compression yes ServerAliveInterval 300 Host *.build.mozilla.org Compression yes User cltbld ServerAliveInterval 300 Host rejh?.srv.releng.????.mozilla.com ControlMaster auto ControlPath ~/.ssh/ssh-%C ControlPersist 10m ForwardAgent no Host *.releng.mdc1.mozilla.com !rejh?.srv.releng.mdc1.mozilla.com ProxyJump rejh1.srv.releng.mdc1.mozilla.com Host *.releng.us??.mozilla.com *.releng.mdc2.mozilla.com !rejh?.srv.releng.mdc2.mozilla.com !*.private.releng.????.mozilla.com ProxyJump rejh1.srv.releng.mdc2.mozilla.com