MozillaWiki - User contributions [en]

ReleaseEngineering/How To/Heroku CLI

2019-06-13T06:51:36Z

Dlabici: minor edits

== How to create the Heroku account ==
# Register with Heroku (use your mozilla email!)
# Go to: https://mozillians.org and login your Mozilla LDAP account.
# Go to: https://mozillians.org/en-US/group/heroku-members/ and click Request Access.
# Wait for the request to join the group to be approved. Usually takes a day or less.
# DO THE NEXT STEPS ONLY AFTER YOU ARE APPROVED!
# Activate SSO via this [https://sso.heroku.com/saml/mozillacorporation/init HEROKU Link]
# Bookmark [https://sso.heroku.com/saml/mozillacorporation/init THIS LINK]. Once you activate SSO, you won't be able to login using the normal login page! So make sure the bookmark is in place!
# Install Heroku for your OS.
# Open your favorite terminal and type '''heroku login'''. The page should tell you that you can safely logout and the terminal should tell you that you have logged in on your account.
# When you are done with the previous steps, login on heroku's website/dashboard and see if you are member of 'mozillacorporation' team.

== Heroku CLI commands ==
You can check your DUO session using the following command:
heroku auth:token
To login use this command:
heroku login
You can update heroku with the following command:
heroku update
If you need to reboot a TC Heroku dyno(s) you can do it with the following command:
heroku dyno:restart --app <heroku-service-name>
Between dyno:restart and --app, you can add a specific dyno name which will restart only the selected dyno:
heroku dyno:restart <dyno-name> --app <heroku-service-name>

== When do we restart the dynos (processes)? ==
'''''Whenever you feel something is not right.'''''

It is okay, and recommended, to restart all dynos (eg don't provide a name between dyno:restart and --app).
If you need to restart more services, lets say 3 of them, there is no order that the restart needs to follow. Simply reboot them, all should go smooth.

How do you know that a services is responsible for a problem?
#Usually if treeherder is not running, well we know we need to reboot '''taskcluster-treeherder'''
#If tests have issues with cache, we need to reboot '''taskcluster-purge-cache'''
#If tests have issues with secrets, we need to reboot '''taskcluster-secrets'''
#If there are problems with queue (machines not picking up tests, or the tests are "unscheduled" or decision tasks not running) well, we will reboot '''queue-taskcluster-net'''.

As you might have noticed, the issue will show you exactly what service you need to restart.

CIDuty/How To

2019-01-15T17:11:12Z

Dlabici: Updated information on page.

===== Alerts =====
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Dealing_with_high_pending_counts High Pending Counts]
: '''Processes'''
: [https://mana.mozilla.org/wiki/display/NAGIOS/Pending+Scriptworker+Tasks Pending Scriptworker Tasks]
: [https://mana.mozilla.org/wiki/display/NAGIOS/Puppet+Freshness Puppet Freshness]
:: Virtual Machines
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Add_Disk_to_an_EC2_Instance Add a new disk to an EC2 instance]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_AWS_slaves Manage AWS instances]
:: AWS loaners
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave Loan an AWS Machine]
: [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Slave_Management#Slave_Tracking Machine Tracking]

===== TaskCluster =====
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave#t-yosemite-r7_taskcluster Loan an OSX Worker]
: [https://wiki.mozilla.org/ReleaseEngineering:TestingTechniques Test a Patch]
: [https://wiki.mozilla.org/BuildDuty:TaskClusterCli Connect to Taskcluster CLI]
: [https://wiki.mozilla.org/BuildDuty:QuarantineInstances Quarantine instances]
: [https://wiki.mozilla.org/CIDuty/How_To/QuarantineMultipleInstances Quarantine multiple instances using python code]
: [https://wiki.mozilla.org/BuildDuty/How_To/Take_actions_to_RelEng_Hardware_from_TaskCluster_UI Restart/Ping from taskcluster]
: [https://mana.mozilla.org/wiki/display/TAS/How+To+Set+Up+or+Update+a+Taskcluster+Project Set up or Update a Taskcluster Project]
: [https://mana.mozilla.org/wiki/display/TAS/How+To+Query+Historical+Task+Analytics Query Historical Task Analytics]
: [https://mana.mozilla.org/wiki/display/TAS/How+To+Investigate+Audit+Logs Investigate Audit Logs]
: [https://mana.mozilla.org/wiki/display/TAS/How+to+set+up+a+new+Heroku+service Set up a new Heroku service]
: [https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=61941455 Have a Postmortem/Retrospective]
: [https://wiki.mozilla.org/Connect_and_Troubleshoot_workers_in_CI Connect and Troubleshoot workers in CI]

===== Tree Management =====
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Close_or_Open_the_Tree Close or Open the Tree]
: [https://wiki.mozilla.org/ReleaseEngineering/TryServer Push to Try Server]

===== Troubleshooting =====
: [https://wiki.mozilla.org/CIDuty/How_To/Troubleshoot_Hardware Hardware Workers]
: [https://wiki.mozilla.org/CIDuty/How_To/Troubleshoot_AWS AWS Instances]
: [https://wiki.mozilla.org/CIDuty/How_To/AWS_Pending_Test Escalate AWS Pending Tests]

===== Other Tasks =====
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Upload_to_internal_Pypi Upload a Python Package to the Internal Pypi]
: [https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool Upload a Package/Binary to ToolTool]
: [https://wiki.mozilla.org/Buildduty/How_To/Add_new_secrets_to_Puppet_Hiera Add new secrets to Puppet Hiera]
: [https://wiki.mozilla.org/Buildduty/How_To/PyUp_Dependency_PR_Updates Work with PyUp Dependency Update PRs]

===== See Also =====
: Relevant [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Training_-_Relevant_Buildduty_Bugs ciduty bugs] that would be suitable for training

===== Deprecated / Archived =====
: Loaners
:: [https://wiki.mozilla.org/Buildduty/How_To/How_To_Loan_BD How to loan servers - New step-by-step guide ]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave Loan a Machine]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Update_VPN_ACL Grant VPN Access]
: [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Reconfigs Perform a Reconfig]
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Work_with_Golden_AMIs Work with Golden AMIs]
: Puppet
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Build_DEBs Build DEBs]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Build_DMGs Build DMGs]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Build_RPMs Build RPMs]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Hack_on_PuppetAgain Hack on PuppetAgain]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Push_changes_to_Production Push Changes to Production]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Modules/users Rotate Passwords across Releng Infra]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Set_up_a_user_environment Setup a User Environment]
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Update_Mercurial Update Mercurial]
: TaskCluster
: [https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=65349374 Restore Azure Table/Blob from Backup]
: [https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=77896889 Modify Gecko Branch Configuration (including twig resets)]
: [https://mana.mozilla.org/wiki/display/TAS/How+To+Provision+taskcluster-worker+on+Packet Provision taskcluster-worker on Packet]

CIDuty:QuarantineInstances

2019-01-15T16:30:22Z

Dlabici: Updated information on the page.

= When and How to quarantine taskcluster instances =

: 1. Choose the worker types you wish to investigate. You can find them [https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/ here.]

: 2. Check which instances have exception(orange) or failed(red) at task state and investigate each of them separately.

[[File:Worker List.png|left]]

: 3. If the last 5-6+ tests are problematic, be sure to check out a few of them.

[[File:Test name.png]]

:: Check public/logs/live_backing.log for errors on a few of the latest tests by going to <Test-Name> -> Run Artifacts -> public/logs/live_backing.log as shown below:

[[File:Log location.png]]

: 4. Judging by the error logs we will know if the machine is faulty (quarantine if this is true) or not. There is no black or white answer for this only that we will know it from experience. Thus far we know if the above conditions are met and the error log terminated with error code -1 and a message like :

[[File:Error log.png]]

: 5. Quarantine all the instances for which all of the above is true by pressing the Quarantine button and leaving the default 1000 years as expiration date, as shown in this [https://irccloud.mozilla.com/file/JTN97Erw/image.png image.]

: 6. File a bug in Bugzilla under RelOps e.g.: https://bugzilla.mozilla.org/show_bug.cgi?id=1441820

: 7. Update the [https://docs.google.com/spreadsheets/d/1IPTmppvqDw0PQV-O1LgXLJg_7TC-H_IAAnSxcur8c7I/edit?ts=5ad7748a#gid=562893333 Master Moonshot Inventory] spreadsheet with the details for the bug (usually BUG:<NUMBER>]

CIDuty/How To/How To Loan BD

2019-01-15T16:21:48Z

Dlabici: Added deprecated text box.

{{warning|This page is deprecated!}}

= Introduction =
Use this page if you want to loan a machine to a developer. This guide has been updated on '''26 March 2018''' and reflects the new changes/requirements on how you should approach the process.

The format will be as follows:
* Finding a server
* Cleaning it up
* Activating and Testing the VNC connection to the server
* Sending the credentials to the requester
Everything you need to do when loaning one of the two type of servers is on this page. You can safely click on the links when instructed too as you will be automatically scrolled to the next step (eg: Server Steps -> VNC -> Email).
= Windows Machines=

TaskCluster Windows machines are self-served! We have to let the requester know how to obtain the server(s) himself. Provide the following link to the requester: https://wiki.mozilla.org/ReleaseEngineering/How_To/Self_Provision_a_TaskCluster_Windows_Instance

= OSX Yosemite r7 =
== Find a server ==
Head to TaskCluster, following this link: [https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010] and search for a worker that doesn't have a running job and it's not quarantined.

If all workers are running jobs come back in 30 minutes and check again.

== Connect to the server ==
Now that we have a server number, we need to SSH into it, you can do that by typing this into the terminal:
ssh root@t-yosemite-r7-462.test.releng.xxxx.mozilla.com
xxxx- represents the datacenter the worker is located into.

Now run the following to commands:
passwd root
passwd cltbld
And change the passwords to the ones offered by your team leader.

== Terminate TC Worker and Clean up TC files ==
At this step we gonna run, again, 3 commands, to clean up some files from server. Like last time, run this commands one at a time.
<nowiki> rm -rf /var/lib/puppet/ssl/private_keys /Library/LaunchDaemons/com.mozilla.puppet.plist \
/Library/LaunchDaemons/org.mozilla.puppetize.plist ~cltbld/.bash_history \
~root/.bash_history ~root/.sh_history /etc/kcpassword ~cltbld/.ssh \
/opt/runner /etc/taskcluster-worker.yml /usr/local/bin/taskcluster-worker \
/usr/local/bin/generic-worker/ \
/Library/LaunchDaemons/org.mozilla.pf.plist \
/Library/LaunchDaemons/org.mozilla.pflog.plist \
/Library/LaunchAgents/net.generic.worker.plist

find /builds -maxdepth 1 -type f -print -delete
</nowiki>
<nowiki>rm -f /usr/local/bin/{taskcluster,generic}-worker
rm -f /etc/generic-worker.config
pkill -f '(generic|taskcluster)-worker'

sudo reboot</nowiki>

== Activate VNC on the server ==
Copy and Paste the following commands, run them one by one and give it a few seconds for the output to come.
<nowiki>vncpw=moz1234

/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart \
-configure -allowAccessFor -allUsers -privs -all

/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart \
-configure -clientopts -setvnclegacy -vnclegacy yes

/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart \
-configure -clientopts -setvncpw -vncpw ${vncpw}

/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart \
-restart -agent -console</nowiki>

Add the requester to the LDAPAdmin -> Group -> List. (https://wiki.mozilla.org/ReleaseEngineering/How_To/Update_VPN_ACL)

If you don't have access in order to grant access to requester and add the ip & host of the loaned machine, you should create a bug for MOC: Service Requests. You can use this bug as reference. To speedup the process, check on #moc for the admin on duty and tell him about your request and the bug that you've created.
At this point all thats left if to send the credentials to the requester.

== Grant access to keychain ==
If the loaner needs to access the keychain, you can reset the password issuing the following command:
<nowiki>security set-keychain-password /Users/cltbld/Library/Keychains/login.keychain</nowiki>
( enter the cltbld password from private repo as old password then use the loaner cltbld password as new password value. )

Finally reboot the machine.
= Send the credentials for the server to the requester =

Follow this link and copy the first code block, as your email template: https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave#Notifying

Change the needed information and send it.

The second code block, from the above link, copy it, paste it inside the Bugzilla Bug, modify it and then assign the ticket to the requester, as shown below.
Pro-Tip: If you don't see all those field, click on the Blue "Edit" button. Don't forget to Save changes!

CIDuty

2019-01-15T15:09:34Z

Dlabici: Moved 2 links.

__TOC__

= What is CIDuty? =
CiDuty (formerly BuildDuty) is a team dedicated to helping out developers with Firefox continuous integration infra issues and enquiries. We currently have six people based in Romania that provide 24/7 support. CiDuty complement the [[Sheriff|sheriffing team]] where sheriffs respond to Firefox code regressions, CiDuty respond to the infrastructure that builds and tests Firefox code.

Have a question or issue with Firefox, build and test infrastructure? ciduty can help and ensure your inquiry gets answered.

= Communication =

As a 24/7 support team, ciduty are available via irc, email, and bugzilla.

irc:
* #ci - look for 'ciduty' in nick (monitors other channels as well)

bugzilla:
* needinfo or assign ciduty@mozilla.com
* file under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Infrastructure%20%26%20Operations&component=CIDuty CIDuty] component if you are not sure where to file your CI related ticket

email:
* ciduty@mozilla.com

= Manifesto =
The [[ReleaseEngineering/Buildduty_manifesto| CIDuty manifesto]] describes the team responsibilities in a nutshell.

= Team =

{| border=1
| '''Name'''
| '''Profile'''
| '''Social'''
| '''Blog'''
|-
| Jordan Lund
| [https://mozillians.org/u/jlund jlund]
| [https://github.com/lundjordan github]
| [http://jordan-lund.ghost.io/ blog]
|-
| Zsolt Fay
| [https://mozillians.org/en-US/u/zfay/ zfay]
| [https://github.com/Rivulu5 github]
| N/A
|-
| Radu Iman
| [https://mozillians.org/en-US/u/riman/ riman]
| [https://github.com/raduiman github]
| N/A
|-
| Bogdan Crisan
| [https://mozillians.org/en-US/u/bcrisan/ bcrisan]
| [https://github.com/bccrisan github]
| N/A
|-
| Danut Labici
| [https://mozillians.org/en-US/u/dlabici/ dlabici]
| [https://github.com/akhliskun github]
| N/A
|-
| Roland Mutter
| [https://mozillians.org/en-US/u/rmutter/ rmutter]
| [https://github.com/mutterroland github]
| N/A
|-
| Adrian Pop
| [https://mozillians.org/en-US/u/apop/ apop]
| [https://github.com/popadrianc github]
| N/A
|}

= CiDuty priorities =
The [[ReleaseEngineering/Buildduty_actionable| CiDuty actionable]] enumerates their daily/weekly sanity job.

= Documentation =
There's a [https://wiki.mozilla.org/CIDuty/How_To HowTo wiki page] that aggregates useful info related to the tasks CiDuty is taking care of (as of January 2019).

= Useful Links =
* [[ReleaseEngineering/Buildduty/day_1_checklist|Day 1 checklist]]
* [https://tools.taskcluster.net/provisioners Provision Explorer]
* [https://wiki.mozilla.org/Buildduty/How_To Public "How To" documents]
* [https://mana.mozilla.org/wiki/dosearchsite.action?queryString=title%3A%22How%20To%22&where=RelEng Private "How To" documents]

= Deprecated / Archived =
The following links and pages are out-of-date or not used anymore. They are still here for historical reasons.

== Others ==
* '''[[CIDuty/Other_Duties|other, less-frequent duties]]''' that CiDuty can assist with.
* [[ReleaseEngineering/How_To|Old/Deprecated Public "How To" documents]]

== Meeting Notes ==
Old meeting docs from BuildDuty era.
* [https://etherpad.mozilla.org/buildduty-notes Daily buildduty stand-up notes]
* [[ReleaseEngineering/Buildduty/Meetings|Old buildduty weekly meetings notes]]
* [[ReleaseEngineering/Buildduty/SVMeetings| SoftVision buildduty stand-up notes]]

Buildduty/day 1 checklist

2019-01-15T15:07:23Z

Dlabici: Removed Nagios entry

__TOC__

Welcome to CiDuty!

This page is meant to serve as a resource for new CiDuty team-members getting up to speed and ensuring they have access to the appropriate systems that they are expected to use.

= CiDuty and the Manifesto =

* Before you start it would be a good idea to make sure you know what CiDuty is all about. Please have a read through of [[ReleaseEngineering/Buildduty_manifesto|the manifesto]]

= Access =

== SSO ==

Generally, we rely on [https://auth0.com/ auth0] across Mozilla for authentication and [https://mana.mozilla.org/wiki/display/SYSADMIN/LDAP+Architecture LDAP] for authorization. Once given LDAP and you have created a temporary password (about 3 months), you can use that to login to the [https://sso.mozilla.com SSO portal]. From SSO, you should have links to various services from email, irc, calendar, slack, mana, etc. More on each of those later on this page

== login.mozilla.com ==

[https://login.mozilla.com/ login.mozilla.com] is where you can change a number of authentication/authorization access bits that you have control over. Each todo in this section assumes you have access to this page.

=== LDAP password reset ===

If you were given a temporary ldap password or you haven't created your own password yet, you should do this now.

=== SSH ===

Upload your public ssh key. It is a good idea to generate a separate ssh keypair from your personal one or any other that you have created in the past and use that explicitly for CiDuty and upload that. Follow this [[Security/Guidelines/OpenSSH#OpenSSH_client|SSH guidelines doc]] on how to generate, configure, and use your ssh key.

note: example ssh config for accessing our systems given below in Jumphost section

=== PGP ===

We use pgp keys to share private information, secrets, and verify that the source came from someone we trust. Generate a keypair for this and upload your public key so others can find it. It would be really good if you could have other people sign your key, adding more trust that this key really belongs to you.

You can use the the [https://mana.mozilla.org/wiki/display/SD/Generating+a+GPG+Public+Key pgp quickstart guide on mana] or you can use the The [https://www.gnupg.org/gph/en/manual.html GNU Privacy Handbook] for reference.

=== VPN ===

Many of our systems are behind a private network in addition to auth0. Follow the prompts to generate and download an openVPN certificate that you can use to import to your vpn client.

See the instructions on how to [https://mana.mozilla.org/wiki/display/SD/VPN install and configure your VPN client] and help choosing the right client for your platform.

note: macOS users should use [https://www.sparklabs.com/viscosity/ Viscosity]. This application comes with a free 30 day trial. During your trial, your manager can help you create a ServiceNow ticket to get a Viscosity full license. While Windows users can use OpenVPN GUI that is free.

=== MFA ===

This MFA account is specific to login.mozilla.com and is used for LDAP/auth0 based logins. Follow the instructions to download the Duo Mobile app and create a Mozilla account.

note: later on in this page we will create more MFA accounts for various systems like Github and accessing our Jumphost

== Jumphost ==

To access any of Release Engineering, Taskcluster, and Release Operations hosts directly, you will need to go through VPN -> a Jumphost machine -> Separate MFA -> your target host.

To do that, you and your manager will need to file a ticket against Release Operations and have them send you an invite to add an MFA account on your Duo App.

Then once you have your Jumphost MFA setup correctly, you will need to have your ssh config to correctly route through the jumphost before trying the target host you want.

example ssh config:
<source lang="ruby">
# Ensure KnownHosts are unreadable if leaked - it is otherwise easier to know which hosts your keys have access to.
HashKnownHosts yes
# Host keys the client accepts - order here is honored by OpenSSH
HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256

Host hg.mozilla.org git.mozilla.org
User USERNAME@mozilla.com
Compression yes
ServerAliveInterval 300

Host *.mozilla.com
User USERNAME
IdentityFile ~/.ssh/id_rsa_mozilla_2017-05-12
Compression yes
ServerAliveInterval 300

Host *.build.mozilla.org
Compression yes
User cltbld
ServerAliveInterval 300

Host rejh?.srv.releng.????.mozilla.com
ControlMaster auto
ControlPath ~/.ssh/ssh-%C
ControlPersist 10m
ForwardAgent no

Host *.releng.mdc1.mozilla.com !rejh?.srv.releng.mdc1.mozilla.com !*.private.releng.????.mozilla.com
ProxyJump rejh1.srv.releng.mdc1.mozilla.com

Host *.releng.us??.mozilla.com *.releng.scl3.mozilla.com !rejh?.srv.releng.????.mozilla.com !*.private.releng.scl3.mozilla.com
ProxyJump rejh1.srv.releng.scl3.mozilla.com
</source>

= Communications =

== Mail ==
Mozilla mail is handled by [https://mail.google.com/ Gmail].

Have your manager subscribe you to this list if you are not already.

=== Mailing lists ===

Needs permission:
* [https://groups.google.com/a/mozilla.com/forum/?hl=en#!forum/ciduty ciduty@mozilla.com] - this is our team email. Ask owner (jlund) for access
* [https://groups.google.com/a/mozilla.com/forum/?hl=en#!forum/releng-puppet-mail Puppet Mail] (warning: you will want to filter this as it can send a lot of mail)

You'll also need to manually subscribe to the following public lists:
* [https://lists.mozilla.org/listinfo/release-engineering release-engineering] public mailing list
* [https://lists.mozilla.org/listinfo/dev-planning mozilla.dev.planning]
* [https://lists.mozilla.org/listinfo/dev-tree-management mozilla.dev.tree-management]
* https://mail.mozilla.org/listinfo/taskcluster-announce - announcements of events, major changes (low volume, no discussion)
* https://lists.mozilla.org/listinfo/tools-taskcluster - general taskcluster discussion
* [https://groups.google.com/a/mozilla.com/forum/#!forum/firefox-ci firefox-ci] mailing list

These are available as [news://news.mozilla.org newsgroups], google groups, and [https://lists.mozilla.org/listinfo Mailman lists]

== Calendar ==
Like mail, we now use [https://www.google.com/calendar/ Google calendar].

You'll want to subscribe to the following public calendars:
* [https://calendar.google.com/calendar/embed?src=mozilla.com_2d32343333353036312d393737%40resource.calendar.google.com Release Engineering - Public]

== Bugzilla ==

Almost everything at Mozilla goes through Bugzilla. [https://bugzilla.mozilla.org/createaccount.cgi Create a Bugzilla account] if you have not already. You should setup MFA and can reuse your LDAP based MFA account.

You'll need a few tweaks to your account to get access to everything releng-related:
* Use your LDAP email if you would like. You can use a personal one or bugzilla specific one if you prefer to filter that way
* Add privileges for bugzilla group "build" (Mozilla Build Team) (Can be done by catlee or bugzilla admin.)
* Add your irc nickname & ldap username as "aliases" for your account
** log into bugzilla & follow links "Preferences" -> "Account Information"
** append the aliases, with a leading ':' and enclosed in brackets ('[]') to the "Real Name" field
** e.g.: "<tt>Chris AtLee [:catlee]</tt>"
* [https://bugzilla.mozilla.org/page.cgi?id=quicksearch.html QuickSearch help]

== Vidyo ==

Our primary two way video meeting platform is Vidyo. Basic usage instructions are [https://mana.mozilla.org/wiki/display/SD/Vidyo+Desktop+3 here]. Especially if you are running linux, it is highly recommended that you install the client and make test calls prior to any meeting.

Special [https://mana.mozilla.org/wiki/display/~gene@mozilla.com/How+to+get+Vidyo+working+on+Ubuntu+16.04+and+newer Ubuntu instructions] that may help

Add to your contact list the CiDuty room

== IRC ==

Historically, IRC is the primary place for chat based communication. Many people use the [https://mana.mozilla.org/wiki/display/SD/IRCCloud+Account+Setup%20IRCCloud%20cloud locally-hosted irccloud] instance. SSO should have a link to the irccloud instance. Servicedesk has some great [https://mana.mozilla.org/wiki/display/SD/Internet+Relay+Chat+-+IRC getting started tips] for IRC.

Useful channels
* #ci, #releaseduty, #releng-bots #taskcluster, #developers, #mobile, #ateam, #moc

Protected channels
* #platform-ops-soc, #platform-ops-alerts
* password in secrets repo
* you don't need to join all of these but some may be useful: https://mana.mozilla.org/wiki/display/SYSADMIN/IRC+use+within+IT
* passwordw in mana

== Slack ==

Some parts of Mozilla prefer Slack to IRC, more info on [https://mana.mozilla.org/wiki/display/CCT/Slack mana].

== Wiki ==

wiki.mozilla.org (here) is the main source for public documentation

== Mana ==

Some internal Mozilla systems (IT, HR) are documented on [https://mana.mozilla.org mana]. File a [https://mozilla.service-now.com/ ServiceNow] ticket if you don't have access when you start.

== Google Drive ==

Google Drive (formerly Google docs) is a preferred way to share things these days. This includes spreadsheets and documents that will change a great deal over time.

Google Drive access should be enabled with your email account when you start. If you need access to a particular document, talk to the document owner or your manager/mentor.

= Development =

== Mercurial (hg) ==

Most development in releng (and at Mozilla writ-large) is stored in version control using [http://mercurial.selenic.com/ hg].

There is an excellent step-by-step guide for setting up and using hg: [https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html Mercurial for Mozillians]

The root webview of the Mozilla hg repositories is here: https://hg.mozilla.org/

There are 3 levels of commit access:
* Level 1 access allows you to use the [[ReleaseEngineering/TryServer|Try Server]] and setup user repos. As a new contributor, you should request this on day one.
* Level 2 access is required to land code in the build and project repos. Once you have a proven track record of successful patches, you can ask your manager/mentor to vouch for your Level 2 access. Your manager/mentor can also land patches for you until you receive Level 2 access.
* Level 3 access is required to land code in [https://hg.mozilla.org/mozilla-central mozilla-central] and its derived integration & release branches. At some point in your Mozilla contribution story, you may need Level 3 access but many contributors never do. Talk to your manager/mentor if you think you need this access. You should already have Level 2 access when you request Level 3.

You need to file an IT bug to get hg commit access. Follow the instructions for [https://www.mozilla.org/en-US/about/governance/policies/commit/ Becoming a Mozilla Committer], and for Level 2, specify you need access to (at least) hg.mozilla.org/build/* (Product/Component: mozilla.org/Repository Account Requests).
* example request: {{bug|703351}}

== Git & Github ==

There are git mirrors of many popular Mozilla repositories. One of the Mozilla github admins (jlund) can add you to the following GitHub groups:
* [https://github.com/orgs/mozilla-releng/teams/buildduty/members CiDuty Github Team]

There are also a handful of git repos hosted directly by Mozilla. Your manager/mentor will let you know if you need access to one of these. ([[Github|See also]])

= Secrets =

Releng and Relops have a shared repo of secrets. Please reach out to jlund or another manager for instructions on how to access and ask to be add yourself as a recipient to every secret that the rest of the CiDuty team (users/buildduty-fingerprints) has access to

= Releng/TC AWS account =

File a release engineering "general" ticket and needinfo jlund or another releng manager to create a user account and add you to the CiDuty group

= Other Services =

For access to other services, you'll need file a couple of bugs:
* Access to Papertrail
** ask jlund or another manager to be added to the encrypted file in the secrets repo

Buildduty/day 1 checklist

2019-01-15T15:06:07Z

Dlabici: Updated information on page.

__TOC__

Welcome to CiDuty!

This page is meant to serve as a resource for new CiDuty team-members getting up to speed and ensuring they have access to the appropriate systems that they are expected to use.

= CiDuty and the Manifesto =

* Before you start it would be a good idea to make sure you know what CiDuty is all about. Please have a read through of [[ReleaseEngineering/Buildduty_manifesto|the manifesto]]

= Access =

== SSO ==

Generally, we rely on [https://auth0.com/ auth0] across Mozilla for authentication and [https://mana.mozilla.org/wiki/display/SYSADMIN/LDAP+Architecture LDAP] for authorization. Once given LDAP and you have created a temporary password (about 3 months), you can use that to login to the [https://sso.mozilla.com SSO portal]. From SSO, you should have links to various services from email, irc, calendar, slack, mana, etc. More on each of those later on this page

== login.mozilla.com ==

[https://login.mozilla.com/ login.mozilla.com] is where you can change a number of authentication/authorization access bits that you have control over. Each todo in this section assumes you have access to this page.

=== LDAP password reset ===

If you were given a temporary ldap password or you haven't created your own password yet, you should do this now.

=== SSH ===

Upload your public ssh key. It is a good idea to generate a separate ssh keypair from your personal one or any other that you have created in the past and use that explicitly for CiDuty and upload that. Follow this [[Security/Guidelines/OpenSSH#OpenSSH_client|SSH guidelines doc]] on how to generate, configure, and use your ssh key.

note: example ssh config for accessing our systems given below in Jumphost section

=== PGP ===

We use pgp keys to share private information, secrets, and verify that the source came from someone we trust. Generate a keypair for this and upload your public key so others can find it. It would be really good if you could have other people sign your key, adding more trust that this key really belongs to you.

You can use the the [https://mana.mozilla.org/wiki/display/SD/Generating+a+GPG+Public+Key pgp quickstart guide on mana] or you can use the The [https://www.gnupg.org/gph/en/manual.html GNU Privacy Handbook] for reference.

=== VPN ===

Many of our systems are behind a private network in addition to auth0. Follow the prompts to generate and download an openVPN certificate that you can use to import to your vpn client.

See the instructions on how to [https://mana.mozilla.org/wiki/display/SD/VPN install and configure your VPN client] and help choosing the right client for your platform.

note: macOS users should use [https://www.sparklabs.com/viscosity/ Viscosity]. This application comes with a free 30 day trial. During your trial, your manager can help you create a ServiceNow ticket to get a Viscosity full license. While Windows users can use OpenVPN GUI that is free.

=== MFA ===

This MFA account is specific to login.mozilla.com and is used for LDAP/auth0 based logins. Follow the instructions to download the Duo Mobile app and create a Mozilla account.

note: later on in this page we will create more MFA accounts for various systems like Github and accessing our Jumphost

== Jumphost ==

To access any of Release Engineering, Taskcluster, and Release Operations hosts directly, you will need to go through VPN -> a Jumphost machine -> Separate MFA -> your target host.

To do that, you and your manager will need to file a ticket against Release Operations and have them send you an invite to add an MFA account on your Duo App.

Then once you have your Jumphost MFA setup correctly, you will need to have your ssh config to correctly route through the jumphost before trying the target host you want.

example ssh config:
<source lang="ruby">
# Ensure KnownHosts are unreadable if leaked - it is otherwise easier to know which hosts your keys have access to.
HashKnownHosts yes
# Host keys the client accepts - order here is honored by OpenSSH
HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256

Host hg.mozilla.org git.mozilla.org
User USERNAME@mozilla.com
Compression yes
ServerAliveInterval 300

Host *.mozilla.com
User USERNAME
IdentityFile ~/.ssh/id_rsa_mozilla_2017-05-12
Compression yes
ServerAliveInterval 300

Host *.build.mozilla.org
Compression yes
User cltbld
ServerAliveInterval 300

Host rejh?.srv.releng.????.mozilla.com
ControlMaster auto
ControlPath ~/.ssh/ssh-%C
ControlPersist 10m
ForwardAgent no

Host *.releng.mdc1.mozilla.com !rejh?.srv.releng.mdc1.mozilla.com !*.private.releng.????.mozilla.com
ProxyJump rejh1.srv.releng.mdc1.mozilla.com

Host *.releng.us??.mozilla.com *.releng.scl3.mozilla.com !rejh?.srv.releng.????.mozilla.com !*.private.releng.scl3.mozilla.com
ProxyJump rejh1.srv.releng.scl3.mozilla.com
</source>

= Communications =

== Mail ==
Mozilla mail is handled by [https://mail.google.com/ Gmail].

Have your manager subscribe you to this list if you are not already.

=== Mailing lists ===

Needs permission:
* [https://groups.google.com/a/mozilla.com/forum/?hl=en#!forum/ciduty ciduty@mozilla.com] - this is our team email. Ask owner (jlund) for access
* [https://groups.google.com/a/mozilla.com/forum/?hl=en#!forum/releng-puppet-mail Puppet Mail] (warning: you will want to filter this as it can send a lot of mail)

You'll also need to manually subscribe to the following public lists:
* [https://lists.mozilla.org/listinfo/release-engineering release-engineering] public mailing list
* [https://lists.mozilla.org/listinfo/dev-planning mozilla.dev.planning]
* [https://lists.mozilla.org/listinfo/dev-tree-management mozilla.dev.tree-management]
* https://mail.mozilla.org/listinfo/taskcluster-announce - announcements of events, major changes (low volume, no discussion)
* https://lists.mozilla.org/listinfo/tools-taskcluster - general taskcluster discussion
* [https://groups.google.com/a/mozilla.com/forum/#!forum/firefox-ci firefox-ci] mailing list

These are available as [news://news.mozilla.org newsgroups], google groups, and [https://lists.mozilla.org/listinfo Mailman lists]

== Calendar ==
Like mail, we now use [https://www.google.com/calendar/ Google calendar].

You'll want to subscribe to the following public calendars:
* [https://calendar.google.com/calendar/embed?src=mozilla.com_2d32343333353036312d393737%40resource.calendar.google.com Release Engineering - Public]

== Bugzilla ==

Almost everything at Mozilla goes through Bugzilla. [https://bugzilla.mozilla.org/createaccount.cgi Create a Bugzilla account] if you have not already. You should setup MFA and can reuse your LDAP based MFA account.

You'll need a few tweaks to your account to get access to everything releng-related:
* Use your LDAP email if you would like. You can use a personal one or bugzilla specific one if you prefer to filter that way
* Add privileges for bugzilla group "build" (Mozilla Build Team) (Can be done by catlee or bugzilla admin.)
* Add your irc nickname & ldap username as "aliases" for your account
** log into bugzilla & follow links "Preferences" -> "Account Information"
** append the aliases, with a leading ':' and enclosed in brackets ('[]') to the "Real Name" field
** e.g.: "<tt>Chris AtLee [:catlee]</tt>"
* [https://bugzilla.mozilla.org/page.cgi?id=quicksearch.html QuickSearch help]

== Vidyo ==

Our primary two way video meeting platform is Vidyo. Basic usage instructions are [https://mana.mozilla.org/wiki/display/SD/Vidyo+Desktop+3 here]. Especially if you are running linux, it is highly recommended that you install the client and make test calls prior to any meeting.

Special [https://mana.mozilla.org/wiki/display/~gene@mozilla.com/How+to+get+Vidyo+working+on+Ubuntu+16.04+and+newer Ubuntu instructions] that may help

Add to your contact list the CiDuty room

== IRC ==

Historically, IRC is the primary place for chat based communication. Many people use the [https://mana.mozilla.org/wiki/display/SD/IRCCloud+Account+Setup%20IRCCloud%20cloud locally-hosted irccloud] instance. SSO should have a link to the irccloud instance. Servicedesk has some great [https://mana.mozilla.org/wiki/display/SD/Internet+Relay+Chat+-+IRC getting started tips] for IRC.

Useful channels
* #ci, #releaseduty, #releng-bots #taskcluster, #developers, #mobile, #ateam, #moc

Protected channels
* #platform-ops-soc, #platform-ops-alerts
* password in secrets repo
* you don't need to join all of these but some may be useful: https://mana.mozilla.org/wiki/display/SYSADMIN/IRC+use+within+IT
* passwordw in mana

== Slack ==

Some parts of Mozilla prefer Slack to IRC, more info on [https://mana.mozilla.org/wiki/display/CCT/Slack mana].

== Wiki ==

wiki.mozilla.org (here) is the main source for public documentation

== Mana ==

Some internal Mozilla systems (IT, HR) are documented on [https://mana.mozilla.org mana]. File a [https://mozilla.service-now.com/ ServiceNow] ticket if you don't have access when you start.

== Google Drive ==

Google Drive (formerly Google docs) is a preferred way to share things these days. This includes spreadsheets and documents that will change a great deal over time.

Google Drive access should be enabled with your email account when you start. If you need access to a particular document, talk to the document owner or your manager/mentor.

= Development =

== Mercurial (hg) ==

Most development in releng (and at Mozilla writ-large) is stored in version control using [http://mercurial.selenic.com/ hg].

There is an excellent step-by-step guide for setting up and using hg: [https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html Mercurial for Mozillians]

The root webview of the Mozilla hg repositories is here: https://hg.mozilla.org/

There are 3 levels of commit access:
* Level 1 access allows you to use the [[ReleaseEngineering/TryServer|Try Server]] and setup user repos. As a new contributor, you should request this on day one.
* Level 2 access is required to land code in the build and project repos. Once you have a proven track record of successful patches, you can ask your manager/mentor to vouch for your Level 2 access. Your manager/mentor can also land patches for you until you receive Level 2 access.
* Level 3 access is required to land code in [https://hg.mozilla.org/mozilla-central mozilla-central] and its derived integration & release branches. At some point in your Mozilla contribution story, you may need Level 3 access but many contributors never do. Talk to your manager/mentor if you think you need this access. You should already have Level 2 access when you request Level 3.

You need to file an IT bug to get hg commit access. Follow the instructions for [https://www.mozilla.org/en-US/about/governance/policies/commit/ Becoming a Mozilla Committer], and for Level 2, specify you need access to (at least) hg.mozilla.org/build/* (Product/Component: mozilla.org/Repository Account Requests).
* example request: {{bug|703351}}

== Git & Github ==

There are git mirrors of many popular Mozilla repositories. One of the Mozilla github admins (jlund) can add you to the following GitHub groups:
* [https://github.com/orgs/mozilla-releng/teams/buildduty/members CiDuty Github Team]

There are also a handful of git repos hosted directly by Mozilla. Your manager/mentor will let you know if you need access to one of these. ([[Github|See also]])

= Secrets =

Releng and Relops have a shared repo of secrets. Please reach out to jlund or another manager for instructions on how to access and ask to be add yourself as a recipient to every secret that the rest of the CiDuty team (users/buildduty-fingerprints) has access to

= Releng/TC AWS account =

File a release engineering "general" ticket and needinfo jlund or another releng manager to create a user account and add you to the CiDuty group

= Other Services =

For access to other services, you'll need file a couple of bugs:
* Access to [http://nagios.mozilla.org/nagios/ Nagios]
** File a bug in bugzilla under 'MOC: Service Requests'
* Access to Papertrail
** ask jlund or another manager to be added to the encrypted file in the secrets repo

CIDuty

2019-01-15T14:43:19Z

Dlabici: Updated "as of Date"

__TOC__

= What is CIDuty? =
CiDuty (formerly BuildDuty) is a team dedicated to helping out developers with Firefox continuous integration infra issues and enquiries. We currently have six people based in Romania that provide 24/7 support. CiDuty complement the [[Sheriff|sheriffing team]] where sheriffs respond to Firefox code regressions, CiDuty respond to the infrastructure that builds and tests Firefox code.

Have a question or issue with Firefox, build and test infrastructure? ciduty can help and ensure your inquiry gets answered.

= Communication =

As a 24/7 support team, ciduty are available via irc, email, and bugzilla.

irc:
* #ci - look for 'ciduty' in nick (monitors other channels as well)

bugzilla:
* needinfo or assign ciduty@mozilla.com
* file under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Infrastructure%20%26%20Operations&component=CIDuty CIDuty] component if you are not sure where to file your CI related ticket

email:
* ciduty@mozilla.com

= Manifesto =
The [[ReleaseEngineering/Buildduty_manifesto| CIDuty manifesto]] describes the team responsibilities in a nutshell.

= Team =

{| border=1
| '''Name'''
| '''Profile'''
| '''Social'''
| '''Blog'''
|-
| Jordan Lund
| [https://mozillians.org/u/jlund jlund]
| [https://github.com/lundjordan github]
| [http://jordan-lund.ghost.io/ blog]
|-
| Zsolt Fay
| [https://mozillians.org/en-US/u/zfay/ zfay]
| [https://github.com/Rivulu5 github]
| N/A
|-
| Radu Iman
| [https://mozillians.org/en-US/u/riman/ riman]
| [https://github.com/raduiman github]
| N/A
|-
| Bogdan Crisan
| [https://mozillians.org/en-US/u/bcrisan/ bcrisan]
| [https://github.com/bccrisan github]
| N/A
|-
| Danut Labici
| [https://mozillians.org/en-US/u/dlabici/ dlabici]
| [https://github.com/akhliskun github]
| N/A
|-
| Roland Mutter
| [https://mozillians.org/en-US/u/rmutter/ rmutter]
| [https://github.com/mutterroland github]
| N/A
|-
| Adrian Pop
| [https://mozillians.org/en-US/u/apop/ apop]
| [https://github.com/popadrianc github]
| N/A
|}

= CiDuty priorities =
The [[ReleaseEngineering/Buildduty_actionable| CiDuty actionable]] enumerates their daily/weekly sanity job.

= Documentation =
There's a [https://wiki.mozilla.org/CIDuty/How_To HowTo wiki page] that aggregates useful info related to the tasks CiDuty is taking care of (as of January 2019).

= Useful Links =
* [[ReleaseEngineering/Buildduty/day_1_checklist|Day 1 checklist]]
* [https://tools.taskcluster.net/provisioners Provision Explorer]
* [https://wiki.mozilla.org/Buildduty/How_To Public "How To" documents]('''new''')
* [[ReleaseEngineering/How_To|Old/Deprecated Public "How To" documents]]
* [https://mana.mozilla.org/wiki/dosearchsite.action?queryString=title%3A%22How%20To%22&where=RelEng Private "How To" documents]

= Deprecated / Archived =
The following links and pages are out-of-date or not used anymore. They are still here for historical reasons.

== Others ==
There is a long list of '''[[CIDuty/Other_Duties|other, less-frequent duties]]''' that CiDuty can assist with.

== Meeting Notes ==
Old meeting docs from BuildDuty era.
* [https://etherpad.mozilla.org/buildduty-notes Daily buildduty stand-up notes]
* [[ReleaseEngineering/Buildduty/Meetings|Old buildduty weekly meetings notes]]
* [[ReleaseEngineering/Buildduty/SVMeetings| SoftVision buildduty stand-up notes]]

CIDuty

2019-01-15T14:42:05Z

Dlabici: Removed duplicated entry of "Others"

__TOC__

= What is CIDuty? =
CiDuty (formerly BuildDuty) is a team dedicated to helping out developers with Firefox continuous integration infra issues and enquiries. We currently have six people based in Romania that provide 24/7 support. CiDuty complement the [[Sheriff|sheriffing team]] where sheriffs respond to Firefox code regressions, CiDuty respond to the infrastructure that builds and tests Firefox code.

Have a question or issue with Firefox, build and test infrastructure? ciduty can help and ensure your inquiry gets answered.

= Communication =

As a 24/7 support team, ciduty are available via irc, email, and bugzilla.

irc:
* #ci - look for 'ciduty' in nick (monitors other channels as well)

bugzilla:
* needinfo or assign ciduty@mozilla.com
* file under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Infrastructure%20%26%20Operations&component=CIDuty CIDuty] component if you are not sure where to file your CI related ticket

email:
* ciduty@mozilla.com

= Manifesto =
The [[ReleaseEngineering/Buildduty_manifesto| CIDuty manifesto]] describes the team responsibilities in a nutshell.

= Team =

{| border=1
| '''Name'''
| '''Profile'''
| '''Social'''
| '''Blog'''
|-
| Jordan Lund
| [https://mozillians.org/u/jlund jlund]
| [https://github.com/lundjordan github]
| [http://jordan-lund.ghost.io/ blog]
|-
| Zsolt Fay
| [https://mozillians.org/en-US/u/zfay/ zfay]
| [https://github.com/Rivulu5 github]
| N/A
|-
| Radu Iman
| [https://mozillians.org/en-US/u/riman/ riman]
| [https://github.com/raduiman github]
| N/A
|-
| Bogdan Crisan
| [https://mozillians.org/en-US/u/bcrisan/ bcrisan]
| [https://github.com/bccrisan github]
| N/A
|-
| Danut Labici
| [https://mozillians.org/en-US/u/dlabici/ dlabici]
| [https://github.com/akhliskun github]
| N/A
|-
| Roland Mutter
| [https://mozillians.org/en-US/u/rmutter/ rmutter]
| [https://github.com/mutterroland github]
| N/A
|-
| Adrian Pop
| [https://mozillians.org/en-US/u/apop/ apop]
| [https://github.com/popadrianc github]
| N/A
|}

= CiDuty priorities =
The [[ReleaseEngineering/Buildduty_actionable| CiDuty actionable]] enumerates their daily/weekly sanity job.

= Documentation =
There's a [https://wiki.mozilla.org/CIDuty/How_To HowTo wiki page] that aggregates useful info related to the tasks CiDuty is taking care of (as of December 2017).

= Useful Links =
* [[ReleaseEngineering/Buildduty/day_1_checklist|Day 1 checklist]]
* [https://tools.taskcluster.net/provisioners Provision Explorer]
* [https://wiki.mozilla.org/Buildduty/How_To Public "How To" documents]('''new''')
* [[ReleaseEngineering/How_To|Old/Deprecated Public "How To" documents]]
* [https://mana.mozilla.org/wiki/dosearchsite.action?queryString=title%3A%22How%20To%22&where=RelEng Private "How To" documents]

= Deprecated / Archived =
The following links and pages are out-of-date or not used anymore. They are still here for historical reasons.

== Others ==
There is a long list of '''[[CIDuty/Other_Duties|other, less-frequent duties]]''' that CiDuty can assist with.

== Meeting Notes ==
Old meeting docs from BuildDuty era.
* [https://etherpad.mozilla.org/buildduty-notes Daily buildduty stand-up notes]
* [[ReleaseEngineering/Buildduty/Meetings|Old buildduty weekly meetings notes]]
* [[ReleaseEngineering/Buildduty/SVMeetings| SoftVision buildduty stand-up notes]]

CIDuty

2019-01-15T14:41:22Z

Dlabici: Updated information on page.

__TOC__

= What is CIDuty? =
CiDuty (formerly BuildDuty) is a team dedicated to helping out developers with Firefox continuous integration infra issues and enquiries. We currently have six people based in Romania that provide 24/7 support. CiDuty complement the [[Sheriff|sheriffing team]] where sheriffs respond to Firefox code regressions, CiDuty respond to the infrastructure that builds and tests Firefox code.

Have a question or issue with Firefox, build and test infrastructure? ciduty can help and ensure your inquiry gets answered.

= Communication =

As a 24/7 support team, ciduty are available via irc, email, and bugzilla.

irc:
* #ci - look for 'ciduty' in nick (monitors other channels as well)

bugzilla:
* needinfo or assign ciduty@mozilla.com
* file under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Infrastructure%20%26%20Operations&component=CIDuty CIDuty] component if you are not sure where to file your CI related ticket

email:
* ciduty@mozilla.com

= Manifesto =
The [[ReleaseEngineering/Buildduty_manifesto| CIDuty manifesto]] describes the team responsibilities in a nutshell.

= Team =

{| border=1
| '''Name'''
| '''Profile'''
| '''Social'''
| '''Blog'''
|-
| Jordan Lund
| [https://mozillians.org/u/jlund jlund]
| [https://github.com/lundjordan github]
| [http://jordan-lund.ghost.io/ blog]
|-
| Zsolt Fay
| [https://mozillians.org/en-US/u/zfay/ zfay]
| [https://github.com/Rivulu5 github]
| N/A
|-
| Radu Iman
| [https://mozillians.org/en-US/u/riman/ riman]
| [https://github.com/raduiman github]
| N/A
|-
| Bogdan Crisan
| [https://mozillians.org/en-US/u/bcrisan/ bcrisan]
| [https://github.com/bccrisan github]
| N/A
|-
| Danut Labici
| [https://mozillians.org/en-US/u/dlabici/ dlabici]
| [https://github.com/akhliskun github]
| N/A
|-
| Roland Mutter
| [https://mozillians.org/en-US/u/rmutter/ rmutter]
| [https://github.com/mutterroland github]
| N/A
|-
| Adrian Pop
| [https://mozillians.org/en-US/u/apop/ apop]
| [https://github.com/popadrianc github]
| N/A
|}

= CiDuty priorities =
The [[ReleaseEngineering/Buildduty_actionable| CiDuty actionable]] enumerates their daily/weekly sanity job.

== Others ==
There is a long list of '''[[CIDuty/Other_Duties|other, less-frequent duties]]''' that CiDuty can assist with.

= Documentation =
There's a [https://wiki.mozilla.org/CIDuty/How_To HowTo wiki page] that aggregates useful info related to the tasks CiDuty is taking care of (as of December 2017).

= Useful Links =
* [[ReleaseEngineering/Buildduty/day_1_checklist|Day 1 checklist]]
* [https://tools.taskcluster.net/provisioners Provision Explorer]
* [https://wiki.mozilla.org/Buildduty/How_To Public "How To" documents]('''new''')
* [[ReleaseEngineering/How_To|Old/Deprecated Public "How To" documents]]
* [https://mana.mozilla.org/wiki/dosearchsite.action?queryString=title%3A%22How%20To%22&where=RelEng Private "How To" documents]

= Deprecated / Archived =
The following links and pages are out-of-date or not used anymore. They are still here for historical reasons.

== Others ==
There is a long list of '''[[CIDuty/Other_Duties|other, less-frequent duties]]''' that CiDuty can assist with.

== Meeting Notes ==
Old meeting docs from BuildDuty era.
* [https://etherpad.mozilla.org/buildduty-notes Daily buildduty stand-up notes]
* [[ReleaseEngineering/Buildduty/Meetings|Old buildduty weekly meetings notes]]
* [[ReleaseEngineering/Buildduty/SVMeetings| SoftVision buildduty stand-up notes]]

CiDuty/actionable

2019-01-15T14:37:09Z

Dlabici: Updated information on page.

== Intro ==
CiDuty is responsible for routine keep-the-lights-on (KTLO) waterline tasks. These include but are not limited to providing loaners to developers, triaging urgent break-fix issues, handling trees closures, and maintaining documentation. If any of these come up early in CiDuty morning, they take priority over anything else. CiDuty should continue performing checks on a regular basis throughout the day to make sure things are in a good state.

== Continuously ==
'''1.''' Nagios and SNS Alerts - Monitor alerts from [https://nagios1.private.releng.mdc1.mozilla.com/releng-mdc1/ MDC1] / [https://nagios1.private.releng.mdc2.mozilla.com/releng-mdc2/ MDC2] Nagios instances and SNS alerts from [https://papertrailapp.com/dashboard/ papertrail] in the #ci IRC channel. Triage unacknowledged alerts and file/fix bugs as necessary according to the [https://wiki.mozilla.org/CIDuty/How_To CiDuty How-Tos]. Make sure that all CiDuty bugs have the correct priority set according to the [[ReleaseEngineering/Buildduty_actionable#Buildduty_Bugzilla_Priority_levels| priority list]] below. A few examples of such alerts include:
* High CI wait times
* CI pending job backlogs
* Unresponsive machines
* Disk/RAM/CPU issues
* Failed processes

'''2.''' Monitor the #ci, #platform-ops-alerts, #releaseduty and #taskcluster irc channels for requests/questions from developers and other ops teams.

'''3.''' Bug triaging - The [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=CiDuty&list_id=14513240 CiDuty report] should be your starting point for bug triaging. Click on the bug number link in the CiDuty report to see the specified bug in bugzilla.

== Daily ==
'''1.''' Review email from AWS looking for announced maintenance, degraded instances, etc. Resolve any issues with instances specified in the AWS email, and notify the appropriate groups of any planned maintenance.

'''2.''' Check for and terminate long-running/outdated AWS instances.

'''3.''' Sanity checks in [https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/ TaskCluster Provisioner]making sure that the workers are working as expected.

== Weekly ==
'''1.''' Review AWS instances. All those AWS that have 'Unknown State/Type' or have been 'stopped for a while'.
* check the AWS sanity logs in [https://papertrailapp.com/dashboard/ papertrail]
* for each host under heading "''[https://papertrailapp.com/systems/74950614/events?q=Unknown%20State Unknown State]''", "''[https://papertrailapp.com/systems/74950614/events?q=Unknown%20Type Unknown Type]''", follow this [https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_AWS_slaves#Unknown_Type_Or_State_Instances/ guide]
* for each host under heading "''[https://papertrailapp.com/systems/74950614/events?q=Stopped%20For%20A%20While Stopped For A While]''", follow this [https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_AWS_slaves#Stopped_For_A_While_Instances/ guide]

'''2.''' Run the aws_manage_routingtables script to make sure that the AWS routing tables are up to date and error free. Fix any reported errors

'''3.''' Review the loaner email report and verify that loaners are still required. Reimage/terminate returned loaners as necessary

== CiDuty Bugzilla Priority levels ==

'''P1:''' Waterline KTLO work.
This includes developer loaners, urgent break-fix, trees closures, etc.

'''P2:''' Projects above waterline that take high priority and have hard deadlines.
This includes various fixes that we need to take care of once, not on a recurring basis. Examples of this are [https://bugzilla.mozilla.org/show_bug.cgi?id=1363897/ 1363897] (legacy extensions disabling prior to 57) and [https://bugzilla.mozilla.org/show_bug.cgi?id=1315977/ 1315977] (upgrading python to 2.7.6 on mac builders)

'''P3.''' Training/Daily Documentation.
This includes onboarding and runbook documentation. Creating/expanding this documentation will help us deliver faster on other bugs.

'''P4.''' Operational development.
This includes adding new scripts or improving existing scripts, e.g. to reduce running times, improve logging and notifications system, etc.

Buildduty/manifesto

2019-01-15T13:44:36Z

Dlabici: Added info about PyUp work.

== Intro ==

CiDuty is an operational support team dedicated to monitoring and maintaining the health of Firefox’s continuous integration (CI) infrastructure. Employees are contractors located in Romania that provide 24/7 support. The team's '''''responsibilities include''''' but are not limited to:
* [[ReleaseEngineering/Buildduty_manifesto#Firefox_CI_infrastructure_outage_coordination_and_investigation| Firefox CI infrastructure outage coordination and investigation]]
* [[ReleaseEngineering/Buildduty_manifesto#Firefox_CI_support_and_case_management| Firefox CI support and case management]]
* [[ReleaseEngineering/Buildduty_manifesto#Track_Firefox_CI_infrastructure_changes| TrackFirefox CI infrastructure changes]]
* [[ReleaseEngineering/Buildduty_manifesto#Monitoring,_investigating,_and_debugging_issues_with_the_Linux,_Windows,_and_OS_X_Firefox_CI_infrastructure| Monitoring, investigating, and debugging issues with the Linux, Windows, and OS X Firefox CI infrastructure]]
* [[ReleaseEngineering/Buildduty_manifesto#Monitoring_Firefox_CI_backlog/pending_counts| Monitoring Firefox CI backlog/pending counts]]
* [[ReleaseEngineering/Buildduty_manifesto#Tree_closing_and_opening| Tree closing and opening]]
* [[ReleaseEngineering/Buildduty_manifesto#Loaning_Firefox_build/test_instances_to_developers| Loaning Firefox build/test instances to developers]]
* [[ReleaseEngineering/Buildduty_manifesto#Upload_new_packages_or_Python_modules_to_our_internal mirrors| Upload new packages or Python modules to our internal mirrors]]
* [[ReleaseEngineering/Buildduty_manifesto#Routine_maintenance_of_the_Firefox_CI_configuration| Routine maintenance of the Firefox CI configuration]]
* [[ReleaseEngineering/Buildduty_manifesto#PyUp_PR_work.| PyUp PR work]]
The team’s responsibilities cover a wide variety of tasks, however they are not deeply knowledgeable about any particular tool, worker, or task running in our infra. Therefore, They should be treated as quick res ponders who are able to assess state in a timely manner, and escalate issues and inquiries to the appropriate person.

== Things CIDuty can help with ==

===== Track Firefox CI infrastructure changes =====

CIDuty track and help publish all CI related changes. While this would be public, they can point people to this and help correlate regressions

===== Firefox CI support and case management =====

First and foremost, ciduty are "case managers" to your CI developer needs. They have escalation paths and a well defined knowledge of the CI system as a whole. Given that, they are excellent at responding to issues and inquiries, and making sure anything Firefox CI related is triaged and managed appropriately.

===== Firefox CI infrastructure outage coordination and investigation =====
When the Firefox CI system fails, getting services online again is ciduty's top priority. They are the initial point of contact for outages, but will likely escalate to additional teams with subject matter experts for resolution.

===== Monitoring, investigating, and debugging issues with the Linux, Windows, and OS X Firefox CI infrastructure =====
CiDuty monitors the Firefox CI infrastructure using the Nagios GUI and irc alerts in the #ci irc channel. They routinely look for system issues, resolve them using our automation tooling, or work with datacenter staff to repair offline or degraded hardware. They also monitor email from AWS about infrastructure that is degraded or requires maintenance.

===== Monitoring Firefox CI backlog/pending counts =====
CiDuty is the first point of contact for monitoring the load on the Firefox CI system and determining the cause of any high backlog or pending job counts. If they are unable to determine the root cause and solve the issue, CiDuty escalates to other teams who have subject matter experts.

===== Tree closing and opening =====
Closing and opening the trees (denying and allowing code checkins to our mercurial repos) are typically handled by the Mozilla Code Sheriffs, but CiDuty can also help out with this if needed.

===== Loaning Firefox build/test instances to developers =====
CiDuty processes bugzilla requests from developers for Firefox CI build or test loaners. To obtain a loaner, submit a request to bugzilla under [https://mzl.la/2L2izjE CiDuty] and expect a response in less than one working day (UTC+2).

===== Upload new packages or Python modules to our internal mirrors =====
CiDuty can help a developer who needs a new software package uploaded to tooltool or a Python package uploaded to our internal PyPi mirror. They can also grant other developers access to upload packages to tooltool, for a given paths subset, to allow for future self-service.

===== Routine maintenance of the Firefox CI configuration =====
While most of the Taskcluster configuration is handled by the end-developer, we still have infrastructure using the Buildbot CI infrastructure as well. CiDuty has the knowledge and capability to modify the Firefox buildbot-configs and perform general maintenance of the Buildbot systems. Maintenance includes tasks such as retasking machines from one platform to another as capacity requirements demand, decommissioning machines, updating keys and secrets, etc.

===== PyUp PR work =====
CiDuty is in charge of keeping few repos, such as build-puppet, addonscript and treescript, always up-to-date with PRs that come from PyUp. In a general sense, they know how to investigate why tests are failing and know how to do the appropriate changes.

== Things ciduty are not responsible for ==

===== Fixing Firefox build and test tasks =====

While ciduty have the skills to diagnose CI infra health and make sure that the workers are in a good state, they are not knowledgeable about build and test internal logic. They do however know who owns what and can help you escalate to the appropriate team

CIDuty/How To/PyUp Dependency PR Updates

2018-09-13T12:21:46Z

Dlabici: WIP - How to work with PyUp PRs

== Overview ==
PyUp is an automated service, that at it's most basic level, helps developers keep their Python Dependencies always Up-To-Date.
CiDuty will be helping RelOps/RelEng with this weekly actionable (every Wednesday) and making sure that the PR is all green, before merging.
This is an easy process that requires attention and knowledge on how pip is working.

== The 4 pillars of the Task ==
# Review and Fix possible conflicts in the PR.
# Make a list of critical dependencies and/or servers that will be affected.
# Communicate with the repository maintainer(s)
# Merge the PyUp PR and Monitor incoming Puppet mails for issues.

== Important notes to take into account ==
While working on a PyUp PR you always have to take notes and/or ask yourself the following questions:
- Did any signing-server (NOT scriptworkers!) requirements file(s) got changed?
= Yes? ANNOUNCE bhearsum because the signing servers might need to be restarted!
= We don't do merges till Ben gives the green light.
- Did any important deps. got updated?
= Example: Crypthograpy, certifi, anything to do with OAuth, etc.
- Did any MAC machines got requirements file(s) changed?
= Yes? ANNOUNCE bhearsum because the deps might need to be manually downloaded!
- Did any deps upgrades ask for lower version packages?
= Dependiong on which repo, you will have to talk with jlorenzo or bhearsum and see which packages get pinned (locked version)
- When everything is ready to be merged, ALWAYS "Squash and Merge".

== Repositories which we need to activate on ==
# [https://github.com/mozilla-releng/mozilla-version/ Mozilla Version]
# [https://github.com/mozilla-releng/build-puppet/ Build Puppet]

== How To: Mozilla-Version ==
We always start with this repository first because the repository itself is a dependency of Build-Puppet!
# Head to https://github.com/mozilla-releng/mozilla-version/pulls and check if any "Scheduled weekly dependency update for week X" is currently open.
# Verify if the tests are green. Keep in mind that all three checks need to be green:
## Travis CI PR
## Travis CI Push
## Coverage/Coveralls
# If tests are RED head to the build/test and look in the logs for problems.
# Implement the needed changes + commit them (you can do it via github webui). Wait for the test results to come (about 10-15 minutes)
# Squash and Merge the PR.
# Announce JLorenzo that the weekly PR is complete.

=== Common Issues ===
This repository usually has 2 distinct problems which will stop you from merging the PR:
# Requirements of a dependency. Example: flake8 3.5.0 has requirement pycodestyle<2.4.0,>=2.0.0, but you have pycodestyle 2.4.0.
With this issue, you have to keep which deps you have to either manually revert back at the older version (keep in mind you need the old sha512!) or if you pin the version of the dependency.

# Wrong Sha
When a error like this comes:

ERROR: could not install deps [-rrequirements-test.txt]; v = InvocationError('/home/travis/build/mozilla-releng/mozilla-version/.tox/py27/bin/python -m pip install -rrequirements-test.txt (see /home/travis/build/mozilla-releng/mozilla-version/.tox/py27/log/py27-1.log)', 1)

it means the hash=sha512 is wrong. Please contact JLorenzo for a fix/sha string.
== How To: Build-Puppet ==
After you are done with Mozilla-Version, its time for Build-Puppet!
# Head to https://github.com/mozilla-releng/build-puppet/pulls and check if any "Scheduled weekly dependency update for week X" is currently open.
# Verify if the tests are green. Keep in mind that all three checks need to be green:
## Travis CI PR
## Travis CI Push
## TaskCluster (pull_request)
# If tests are RED head to the build/test and look in the logs for problems. In taskcluster, usually the errors are at the bottom.
# Implement the needed changes + commit them (you can do it via github webui). Wait for the test results to come (about 40-60 minutes)
# Squash and Merge the PR.
# Announce Bhearsum that the weekly PR is complete.
=== Common Issues ===
# Requirements of a dependency. Example: flake8 3.5.0 has requirement pycodestyle<2.4.0,>=2.0.0, but you have pycodestyle 2.4.0.
With this issue, you have to keep which deps you have to either manually revert back at the older version (keep in mind you need the old sha512!) or if you pin the version of the dependency.

CIDuty/How To

2018-09-13T11:08:34Z

Dlabici: Added link to new How To

===== Alerts =====
: [https://wiki.mozilla.org/ReleaseEngineering/Queue_directories Command queues]
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Dealing_with_high_pending_counts High Pending Counts]
: '''Processes'''
:: [https://wiki.mozilla.org/ReleaseEngineering/Applications/BuildbotBridge Buildbot Bridge]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Restart_Buildbot_Masters Buildbot Master - buildbot age]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_spot_AMIs Golden AMI]
: [https://mana.mozilla.org/wiki/display/NAGIOS/Pending+Scriptworker+Tasks Pending Scriptworker Tasks]
: [https://mana.mozilla.org/wiki/display/NAGIOS/Puppet+Freshness Puppet Freshness]
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Fix_Build4h_not_updating Build4h not updating]

===== Buildbot =====
: '''Manage Buildbot Masters'''
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Setup_buildbot_masters_in_AWS Add a New Buildbot Master in AWS]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Bring_Buildbot_Masters_Up_After_an_Outage Bring Buildbot Masters Up After an Outage]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Decommission_buildbot_masters Decommission a Master]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Land_Buildbot_Master_Changes Land Buildbot Master Changes]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Restart_Buildbot_Masters Restart Buildbot Masters]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Setup_Personal_Development_Master Setup a Personal Development Master]
: '''Manage Buildbot Slaves'''
:: Hardware Machines
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Connect_To_IPMI Connect to IPMI/PDU]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Decommission_Slave Decommission a Slave]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Handle_an_Idle_Slave Handle an Idle Slave ]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Reboot_Multiple_Slaves_Using_Slaveapi Reboot Multiple Machines]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Request_That_a_Machine_Be_Reimaged Re-image machines]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Set_Up_a_Freshly_Imaged_Slave Return a re-imaged machine to production]
::: [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Slave_Management Slave_Management]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Move_slaves_between_pools Move slaves between pools]
:: Virtual Machines
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Add_Disk_to_an_EC2_Instance Add a new disk to an EC2 instance]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_AWS_slaves Manage AWS instances]
:: Loaners
::: [https://wiki.mozilla.org/Buildduty/How_To/How_To_Loan_BD How to loan servers - New step-by-step guide ]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave Loan a Machine]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Update_VPN_ACL Grant VPN Access]
: [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Slave_Management#Slave_Tracking Machine Tracking]
: [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Reconfigs Perform a Reconfig]
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Work_with_Golden_AMIs Work with Golden AMIs]

===== Puppet =====
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Build_DEBs Build DEBs]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Build_DMGs Build DMGs]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Build_RPMs Build RPMs]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Hack_on_PuppetAgain Hack on PuppetAgain]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Push_changes_to_Production Push Changes to Production]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Modules/users Rotate Passwords across Releng Infra]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Set_up_a_user_environment Setup a User Environment]
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Update_Mercurial Update Mercurial]

===== TaskCluster =====
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave#t-yosemite-r7_taskcluster Loan an OSX Worker]
: [https://wiki.mozilla.org/ReleaseEngineering:TestingTechniques Test a Patch]
: [https://wiki.mozilla.org/BuildDuty:TaskClusterCli Connect to Taskcluster CLI]
: [https://wiki.mozilla.org/BuildDuty:QuarantineInstances Quarantine instances]
: [https://wiki.mozilla.org/CIDuty/How_To/QuarantineMultipleInstances Quarantine multiple instances using python code]
: [https://wiki.mozilla.org/BuildDuty/How_To/Take_actions_to_RelEng_Hardware_from_TaskCluster_UI Restart/Ping from taskcluster]
: [https://mana.mozilla.org/wiki/display/TAS/How+To+Set+Up+or+Update+a+Taskcluster+Project Set up or Update a Taskcluster Project]
: [https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=65349374 Restore Azure Table/Blob from Backup]
: [https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=77896889 Modify Gecko Branch Configuration (including twig resets)]
: [https://mana.mozilla.org/wiki/display/TAS/How+To+Query+Historical+Task+Analytics Query Historical Task Analytics]
: [https://mana.mozilla.org/wiki/display/TAS/How+To+Provision+taskcluster-worker+on+Packet Provision taskcluster-worker on Packet]
: [https://mana.mozilla.org/wiki/display/TAS/How+To+Investigate+Audit+Logs Investigate Audit Logs]
: [https://mana.mozilla.org/wiki/display/TAS/How+to+set+up+a+new+Heroku+service Set up a new Heroku service]
: [https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=61941455 Have a Postmortem/Retrospective]
: [https://wiki.mozilla.org/Connect_and_Troubleshoot_workers_in_CI Connect and Troubleshoot workers in CI]

===== Tree Management =====
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Close_or_Open_the_Tree Close or Open the Tree]
: [https://wiki.mozilla.org/ReleaseEngineering/TryServer Push to Try Server]

===== Other Tasks =====
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Upload_to_internal_Pypi Upload a Python Package to the Internal Pypi]
: [https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool Upload a Package/Binary to ToolTool]
: [https://wiki.mozilla.org/Buildduty/How_To/Add_new_secrets_to_Puppet_Hiera Add new secrets to Puppet Hiera]
: [https://wiki.mozilla.org/Buildduty/How_To/PyUp_Dependency_PR_Updates Work with PyUp Dependency Update PRs]

===== See Also =====
: Relevant [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Training_-_Relevant_Buildduty_Bugs buildduty bugs] that would be suitable for training

CIDuty/How To/Update Python dependencies

2018-08-16T14:39:32Z

Dlabici: Added more information on signing servers

= Introduction =

Use this page if you want to update Python dependencies.

The action list is as follows:
# Check the puppet repo for patch made by pyup bot
# Check the dependencies and how they affect the systems
# Check if there are any releases on the go also check with the release team.
# If you have the Ok, start the update
# Monitoring the update on puppet mail and papertrail
# In case of failure - rollback
# Weekly admin task
# Questions and Answers

= Check the puppet repo for patch made by pyup bot =

Go to https://github.com/mozilla-releng/build-puppet and click on Pull requests and look for "Scheduled weekly dependency update for week X" ( where X is the current week )

[[File:Repo.png|none]]

= Check the dependencies and how they affect the systems =

If you don't know what it does, you can click on it name and find out also on most of them you can also check the change log.
For example : ptyprocess 0.6.0

[[File:Dep 1.png|none]] [[File:Change log.png|none]]

Also to check how are the files changed : Go to Files change tab to see what will be changed and how :

[[File:Ewpo change.png]]

= Check if there are any releases on the go also check with the release team =

The update action shouldn't be done during a release. Always ask the release duty team before running the update

= Running the update =

After checking all the dependencies and if there are no releases, you can start the update, by merging the pyup update into master

= Monitoring the update on puppet mail and papertrail =

In order to monitor the updates you should subscribe to puppet mail : [https://groups.google.com/a/mozilla.com/forum/#!forum/releng-puppet-mail Releng Puppet Mail]

You should also check and monitor using papertrail : [https://papertrailapp.com Papertrail]
Ex of search : [https://papertrailapp.com/groups/1141234/events?q=balrog_scriptworker&focus=958572280949137435 Balrog scriptworker]

= In case of failure - Rollback =

In case of any failure it is needed to create a patch that reverts the changes.
For this is needed to be created a ticket.

= In case of update to Signing Servers =

Please note if there's signingserver changes so someone from the short list (https://github.com/mozilla-releng/build-puppet/blob/master/manifests/moco-config.pp#L239) will need to restart those instances within a few hours of this being merged.

= Weekly admin task =

Every Monday, the admin who is on duty needs to check the build puppet, for the dependencies .
If there is a "Scheduled weekly dependency update for week X" then he will need to update them following the action list

= Questions and Answers =

Q : What do we need to do before update ?

A : We need to check all the dependencies that will be updated, to figure out what could be affected in case of any surprises.

Q : How do we do the update ?

A : We need to merge pyup-bot patch to the master.

Q: When it is recommended to run the update ?

A : This should be done, when there are no releases and also it should be wise to ask the release duty team before starting any action.

Q : Where and what do we need to monitor when we triggered the update ?

A : The monitoring should be done the Puppet mail and also on papertrail

Q : Who are affected when an update or more goes wrong ?

A : Possibly, anyone!

ReleaseEngineering/How To/Self Provision a TaskCluster Windows Instance

2018-07-30T22:45:11Z

Dlabici: Added note that CiDuty should be announced when MS loaners are taken.

{{Release Engineering|How To|Self Provision a TaskCluster Windows Instance}}

__TOC__

= For generic-worker 10.5.0 onwards =

'''Integration with taskcluster one-click loaner workflow will be done in [https://bugzil.la/1368961 bug 1368961].'''

1) If you will take a MoonShot, make sure to
announce in #ci and/or create a bug under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Infrastructure%20%26%20Operations&component=CIDuty CiDuty - Buzilla] announcing the fact that you will take a loaner.

2) Find a the task you want to play with in treeherder, and follow the link to the Taskcluster Task Inspector. 
3) Go to ''Actions -> Edit Task''. 
4) Add <code>rdpInfo</code> to the payload section:

payload:
rdpInfo: 'login-identity/<login-identity>/rdpinfo.txt'

For example:

payload:
rdpInfo: 'login-identity/mozilla-auth0/ad|Mozilla-LDAP|pmoore/rdpinfo.txt'

(check https://tools.taskcluster.net/credentials to see what your login identity is, e.g. you should have the scope <code>assume:login-identity:<login-identity></code>).

4) Check which <code>workerType</code> the task uses from the task definition, and then add the following scope to the list of task scopes:

scopes:
- 'generic-worker:allow-rdp:aws-provisioner-v1/<workerType>'

For example:

scopes:
- 'generic-worker:allow-rdp:aws-provisioner-v1/gecko-t-win7-32'

5) If you will require administrator privileges:

5.1) Ensure you have scope <code>generic-worker:os-group:Administrators</code> in https://tools.taskcluster.net/credentials/

If you do not have the scope, request it using a [https://bugzilla.mozilla.org/enter_bug.cgi?product=Taskcluster&component=Service%20Request Service Request] or by asking in <code>#taskcluster</code> IRC channel.

5.2) Add the following to the task payload:

scopes:
- generic-worker:os-group:Administrators
payload:
osGroups:
- Administrators

6) Make sure that if your payload mounts any artifacts, that the task ID(s) are included as a list in top level property <code>dependencies</code> ('''not''' under <code>payload</code>)

So if it says

mounts:
- directory: .
content:
taskId: HUqhbXTMTXSXYEM6K2H3wA

Add (at the top level indent)

dependencies:
- HUqhbXTMTXSXYEM6K2H3wA

7) Submit the task. If the task resolves as exception and doesn't run, '''check the task log file'''. It should provide more information about the cause. 
8) When the task successfully starts, go to ''Run Artifacts'' to see the <code>rdpInfo.txt</code> file appear with rdp connection information. 
9) Enter the connection information into your RDP client of choice. 
10) Connect with screen resolution 1280x1024 ! Note, it is '''important''' to use this resolution for gecko tests, since this is the screen size used by the tests, and the screen size cannot change once you have made a connection.

== Performing operations as Administrator ==

Until [https://bugzil.la/1465374 bug 1465374] is resolved this can be a little tricky.

1) Make sure you followed step 5 above! 
2) Open a regular command shell (e.g. ''Start Menu'' -> ''Run'' -> <code>cmd.exe</code>) 
3) From there check which users are in the Administrators group:

C:\Windows\System32>net localgroup Administrators
Alias name Administrators
Comment Administrators have complete and unrestricted access to the computer/domain

Members

-------------------------------------------------------------------------------
Administrator
task_1527672240
The command completed successfully.

4) Check the user you are logged in as is one of the above listed users:

C:\Windows\System32>whoami
i-015fe55bb8553\task_1527672240

5) Open a new UAC elevated command shell:

C:\Windows\System32>powershell.exe Start-Process cmd.exe -Verb runAs

6) This will require you enter Administrative credentials. You will be presented with a prompt similar to this:

[[File:Screen Shot 2018-05-30 at 12.13.51.png|none|Screenshot of dialogue box asking for Administrative credentials]]

Click on ''More choices'' and select the task user, and copy/paste the task password (<code>Ctrl-V</code>).

You should now have a command shell running as Administrator!

== Rerunning tasks after they have completed ==

If you want to rerun a step after the task has completed, there is a script for each command in the task payload. They are named like this:

Z:\task_XXXXXXX\command_000000_wrapper.bat
Z:\task_XXXXXXX\command_000001_wrapper.bat
Z:\task_XXXXXXX\command_000002_wrapper.bat
....

Simply run them by hand to reproduce any of the task steps. If you wish to make changes, feel free to edit them. After the loan expires, the task directory, including all your changes, will be deleted. Therefore make sure to keep track of any changes that need to be put back in the gecko build system.

= For earlier versions of generic-worker (<10.5.0) =

== How does it work? ==

TaskCluster Windows instances can be borrowed for debugging at will. There is no need to raise a bug. The process is triggered by running a special task that tells the instance you intend to commandeer it.

When you trigger this task, a recurring scheduled job on the instance will:
* perform a little cleanup, deleting data and configuration that is only pertinent for automation
* change the instance credentials, so that only you will have access to the instance and so that the instance credentials can be shared with you
* place encrypted credentials in the task artifacts for you
* delete the generic-worker and associated user and configuration

== How is it triggered? ==

Visit https://tools.taskcluster.net/task-creator and create a new task that looks something like below.

provisionerId: aws-provisioner-v1
workerType: gecko-t-win7-32
retries: 0
created: '2017-05-24T12:05:16.556Z'
deadline: '2017-05-24T13:05:16.556Z'
expires: '2018-05-24T09:08:08.128Z'
scopes: []
payload:
maxRunTime: 300
artifacts:
- path: ..\loan
name: public/test_info
type: directory
expires: '2018-05-24T08:49:41.238Z'
env:
REQUESTER_EMAIL: grenade@mozilla.com
REQUESTER_PUBLIC_KEY_URL: https://keybase.io/grenade/pgp_keys.asc?fingerprint=1c09ac24c113c7f080dd4aa5b3c5a958508a43f2
command:
- >-
c:\mozilla-build\python\python.exe -c "import os, json;
json.dump({'requester': { 'email': os.environ['REQUESTER_EMAIL'],
'publickeyurl': os.environ['REQUESTER_PUBLIC_KEY_URL'], 'taskid':
os.environ['TASK_ID'], 'taskFolder': os.getcwd()}},
open(r'z:\loan-request.json', 'wb'))"
- >-
c:\mozilla-build\python\python.exe -c "exec(\"import os.path, time\nwhile
(not os.path.exists(r'z:\loan\credentials.txt.gpg')):\n time.sleep(1)\"")
metadata:
name: Windows Loan Request
description: Self service windows instance loan request
owner: grenade@mozilla.com
source: https://wiki.mozilla.org/index.php?title=ReleaseEngineering/How_To/Self_Provision_a_TaskCluster_Windows_Instance
tags: {}
extra: {}

* Change '''workerType''' to one of:
** '''gecko-1-b-win2012''': Windows Server 2012 r2 - Use for debugging '''Build''' issues
** '''gecko-t-win10-64''': Windows 10 Enterprise, x86_64 - Use for debugging '''Test''' issues
** '''gecko-t-win10-64-gpu''': Windows 10 Enterprise, x86_64 - Use for debugging '''Test''' issues where a GPU is required
** '''gecko-t-win7-32''': Windows 7 Enterprise, x86 - Use for debugging '''Test''' issues
** '''gecko-t-win7-32-gpu''': Windows 7 Enterprise, x86 - Use for debugging '''Test''' issues where a GPU is required
* Change '''REQUESTER_EMAIL''' to the email address associated with your GPG public key
* Change '''REQUESTER_PUBLIC_KEY_URL''' to a url that will return a plain text (raw) copy of you GPG public key.
* Change '''owner''' to your own email address
* Click the '''Update Timestamps''' button at the bottom
* Click the '''Create Task''' button at the bottom
* When the task completes, navigate to the artifacts tab (select run 0 first) and download the '''public/test_info/credentials.txt.gpg''' artifact to your computer.
* Decrypt it with a command similar to:

gpg2 --decrypt credentials.txt.gpg

== How do I connect to my borrowed instance? ==

Remote Desktop is the only currently supported mechanism. Further, the processes that manage cleanup of abandoned or unused loaner instances check for RDP sessions to determine if the instance is actually in use or if it has been abandoned.

=== Connecting from Windows ===

* GUI: Start > Remote Desktop Connection > enter the IP Address and credentials from the decrypted credentials.txt file and connect
* Command line:

mstsc /w:1024 /h:768 /v:<ip_address>

=== Connecting from Linux ===
==== Fedora ====
# install xfreerdp
sudo dnf install -y xfreerdp

# if you have an American English (en-US) keyboard:
xfreerdp /u:$username /p:"$password" /kbd:409 /w:1024 /h:768 +clipboard /v:$ip_address

# if you have a Queen's English (en-GB) keyboard:
xfreerdp /u:$username /p:"$password" /kbd:809 /w:1024 /h:768 +clipboard /v:$ip_address

For other keyboard layouts, see: https://github.com/FreeRDP/FreeRDP/blob/master/include/freerdp/locale/keyboard.h

=== Connecting from Mac OSX ===
Untested. Please edit this section if you know more.

open rdp://$username:$password@$ip_address:3389?screendepth###24:screenWidth###1024:screenHeight###768

see also: https://apple.stackexchange.com/a/54925/112073

Microsoft also has a Remote Desktop app for macOS (available on the app store at https://itunes.apple.com/us/app/microsoft-remote-desktop-10/id1295203466?mt=12) that works pretty well and is easy to use.

== How do I return it when I'm done? ==

There's no need. Your loaned instance is an Amazon EC2 spot instance whose lifetime is finite. It can even die while it's in the middle of working for you, if the spot price is outbid. Think of it as a disposable minion whose survival depends on being busy and who will expire when exhausted, bored, or will defect to a higher-paying villain at any time. If you don't create an RDP connection to your instance within 30 minutes of requesting it, it will die of boredom. If you disconnect your RDP session for more than 15 minutes, it will die of boredom. If you shut it down. It will die fulfilled. When working on the loaner, it would be advisable to save your intermediate results periodically and script your actions, so that if the loaner disappears you can quickly get back to where you were on a new loaner.

== How do I get help if it's not working? ==

* raise a bug in the [https://bugzilla.mozilla.org/enter_bug.cgi?product=Infrastructure%20%26%20Operations&component=RelOps RelOps component]. cc or assign rthijssen@mozilla.com
* ping :grenade in #taskcluster on IRC (during normal working hours: 9 - 5 GMT+3)

CIDuty/How To/Tree Closure

2018-06-26T16:53:45Z

Dlabici: Fixed typo

==How do I schedule a downtime?==

Whenever Sheriffs/RelEng/IT/WebDev wants a tree-closing downtime because of a infra issue, they should contact the CiDuty person of the day in #ci, by email or preferably by ''putting exact wording they'd like in the downtime notice as it relates to their bug, and then nominating the bug using the "needs-treeclosure?" flag''.

Once we know that we want to close the tree(s), head to: https://mozilla-releng.net/treestatus login select the tree(s) to be closed and add a note with the bug number, then update the tree to the correct status (open/closed).

=== When do we close the trees ===
* Infra related issue:
** Error code 500 (server load)
** HG Issues (cloning issues, timeouts)
** HDD problems (space, I/O)
** Pending Tasks keep raising (Indicator for another problems, such as BBB not working properly)

When planning a downtime, CiDuty should consider:
* the urgency of the work
* what other work, if any, can be safely done in the same downtime

=== Preparing for the downtime ===
* CiDuty will:
** Create a bug with the explanation why the tress get closed
** verify that the bug is assigned to the person who will actually be doing the work in the downtime
*** for security sensitive work, be vague, but still include the bug# and vague description. (this reduces confusion about whether an item of work is in/out of a downtime).

==== Who do I notify, and when? ====
# Just After the trees got closed:
## Updated the #ci Channel topic and add " TREES CLOSED - BUG: XXXX " right after the LDAP username.
## Send notifications to: #ci, #sheriffs (if needed), #developers, about the Trees being closed.
## If new updates come from #sheriffs, #developers, #releng or anyone, make sure #ci is being constantly updated with the current status.

=== After the downtime ===
CiDuty will:
# reopen the trees
# verify with sheriff that trees open, all ok?
# update the bugs with status.
# send "TREE OPEN" announcements in #ci, #sheriffs, #developers

For any questions, or if you're not sure about a particular server, please check with '''|ciduty''' in #ci.

If possible, consolidate RelEng and IT downtimes that need tree closures to avoid the disruption of having two tree closures soon after each other. This is "nice to do", not a "requirement"; if it reduces risk by doing two separate downtimes, that's fine!

CIDuty/How To/Tree Closure

2018-06-26T16:53:18Z

Dlabici: added link to tree status

==How do I schedule a downtime?==

Whenever Sheriffs/RelEng/IT/WebDev wants a tree-closing downtime because of a infra issue, they should contact the CiDuty person of the day in #ci, by email or preferably by ''putting exact wording they'd like in the downtime notice as it relates to their bug, and then nominating the bug using the "needs-treeclosure?" flag''.

Once we know that we want to close the tree(s), head to: https://mozilla-releng.net/treestatus loginm select the tree(s) to be closed and add a note with the bug number, then update the tree to the correct status (open/closed).

=== When do we close the trees ===
* Infra related issue:
** Error code 500 (server load)
** HG Issues (cloning issues, timeouts)
** HDD problems (space, I/O)
** Pending Tasks keep raising (Indicator for another problems, such as BBB not working properly)

When planning a downtime, CiDuty should consider:
* the urgency of the work
* what other work, if any, can be safely done in the same downtime

=== Preparing for the downtime ===
* CiDuty will:
** Create a bug with the explanation why the tress get closed
** verify that the bug is assigned to the person who will actually be doing the work in the downtime
*** for security sensitive work, be vague, but still include the bug# and vague description. (this reduces confusion about whether an item of work is in/out of a downtime).

==== Who do I notify, and when? ====
# Just After the trees got closed:
## Updated the #ci Channel topic and add " TREES CLOSED - BUG: XXXX " right after the LDAP username.
## Send notifications to: #ci, #sheriffs (if needed), #developers, about the Trees being closed.
## If new updates come from #sheriffs, #developers, #releng or anyone, make sure #ci is being constantly updated with the current status.

=== After the downtime ===
CiDuty will:
# reopen the trees
# verify with sheriff that trees open, all ok?
# update the bugs with status.
# send "TREE OPEN" announcements in #ci, #sheriffs, #developers

For any questions, or if you're not sure about a particular server, please check with '''|ciduty''' in #ci.

If possible, consolidate RelEng and IT downtimes that need tree closures to avoid the disruption of having two tree closures soon after each other. This is "nice to do", not a "requirement"; if it reduces risk by doing two separate downtimes, that's fine!

CIDuty/How To/Tree Closure

2018-06-26T16:47:11Z

Dlabici: Initial release

==How do I schedule a downtime?==

Whenever Sheriffs/RelEng/IT/WebDev wants a tree-closing downtime because of a infra issue, they should contact the CiDuty person of the day in #ci, by email or preferably by ''putting exact wording they'd like in the downtime notice as it relates to their bug, and then nominating the bug using the "needs-treeclosure?" flag''.

=== When do we close the trees ===
* Infra related issue:
** Error code 500 (server load)
** HG Issues (cloning issues, timeouts)
** HDD problems (space, I/O)
** Pending Tasks keep raising (Indicator for another problems, such as BBB not working properly)

When planning a downtime, CiDuty should consider:
* the urgency of the work
* what other work, if any, can be safely done in the same downtime

=== Preparing for the downtime ===
* CiDuty will:
** Create a bug with the explanation why the tress get closed
** verify that the bug is assigned to the person who will actually be doing the work in the downtime
*** for security sensitive work, be vague, but still include the bug# and vague description. (this reduces confusion about whether an item of work is in/out of a downtime).

==== Who do I notify, and when? ====
# Just After the trees got closed:
## Updated the #ci Channel topic and add " TREES CLOSED - BUG: XXXX " right after the LDAP username.
## Send notifications to: #ci, #sheriffs (if needed), #developers, about the Trees being closed.
## If new updates come from #sheriffs, #developers, #releng or anyone, make sure #ci is being constantly updated with the current status.

=== After the downtime ===
CiDuty will:
# reopen the trees
# verify with sheriff that trees open, all ok?
# update the bugs with status.
# send "TREE OPEN" announcements in #ci, #sheriffs, #developers

For any questions, or if you're not sure about a particular server, please check with '''|ciduty''' in #ci.

If possible, consolidate RelEng and IT downtimes that need tree closures to avoid the disruption of having two tree closures soon after each other. This is "nice to do", not a "requirement"; if it reduces risk by doing two separate downtimes, that's fine!

ReleaseEngineering/How To/Restart Buildbot Masters

2018-06-22T04:29:59Z

Dlabici: Added noted on reconfig.lock

__TOC__
We occasionally need to restart buildbot masters for various reasons:
* upgrades to the underlying OS
* gradual increase in memory usage over time, leading to reduced master performance

There's a Nagios check in place reminding us when the masters need to be rebooted:
<pre>
<nagios-releng> Wed 23:19:55 UTC [7691] [moc] nagios1.private.releng.scl3.mozilla.com:buildbot-master-machines-buildbot_age cluster is WARNING:CLUSTER WARNING: buildbot-master-machines-buildbot_age cluster: 0 ok, 41 warning, 0 unknown, 0 critical (http://m.mozilla.org/buildbot-master-machines-buildbot_age+cluster)
</pre>

= Manually =
If you need to restart a single master by hand, here's the sequence you should follow:
* disable the master in [https://secure.pub.build.mozilla.org/slavealloc/ui/#masters slavealloc]. This prevents the master from taking more slave connections while you're waiting for it to shutdown.
* click the "Clean Shutdown" button on the web interface for the given master, e.g. http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/
* wait for the jobs currently running on that master to complete. You can track progress in two ways:
** search in-page for "Running" on the master's buildslaves page, e.g. http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/buildslaves?no_builders=1
** check BuildAPI for the [https://secure.pub.build.mozilla.org/buildapi/running list of running jobs] and look for the ones corresponding to that master
* once the master is shutdown, perform whatever upgrades are required, etc.
* restart the master. '''NOTE:''' buildbot masters are configured to restart buildbot automatically on boot, so if you reboot the master, buildbot will restart itself. To restart manually:
xebec:buildduty ccooper$ ssh cltbld@buildbot-master82
Unauthorized access prohibited
[cltbld@buildbot-master82.bb.releng.scl3.mozilla.com ~]$ cd /builds/buildbot/build1/
[cltbld@buildbot-master82.bb.releng.scl3.mozilla.com build1]$ make start
* re-enable the master in [https://secure.pub.build.mozilla.org/slavealloc/ui/#masters slavealloc].
* Don't forget to remove the reconfig.lock file if the script gets interrupted for any reason.

= By script =
The above actions have been encapsulated into a script: https://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/restart_masters.py. The script is setup to run on <tt>buildduty-tools.srv.releng.usw2.mozilla.com</tt>, located at <tt>/home/buildduty/restart_masters/</tt> under buildduty's account.

The wrapper script <tt>/home/buildduty/restart_masters.sh</tt> is used to update repositories and call restart_masters.py. The latter will prompt for the usernames and passwords it needs (ldap for slavealloc, cltbld and root for ssh).

Here is an example invocation:

# ssh -A buildduty@buildduty-tools
$ screen -R restart_masters
$ /home/buildduty/restart_masters.sh

You can enter nonsense for the cltbld and root passwords because key auth is used. Forwarding your ssh agent is required for ssh access to the masters. If you use a timeout make sure it's sufficiently long for the script to repeat, but don't leave it running indefinitely. '''NB: rebooting masters by adding the -r arg to restart_masters.py is non-functional because root logins are disabled.'''

The logs go to [https://papertrailapp.com/systems/255918104/events?q=restart_masters papertrail], outputting a progress report every 60 minutes. For really minimal view (which may hide errors) see this [https://papertrailapp.com/systems/255918104/events?q=restart_masters+-%22Connection+as+cltbld%22+-%22Attempting+to+connect+to%22+-%22Checking+shutdown+status%22+-lockfile+-Disabling+-Initiating+-%22Re-enabling%22+-%22Attempting+to+restart+master%22 filtered view].

You can send a SIGUSR1 to restart_masters.py to prompt an extra progress report; allow time for the current status check to complete first.

= Automated =
The above script requires sensitive credentials that shouldn't be stored on disk. For now, we're still running this script by hand.

CIDuty

2018-05-14T08:48:53Z

Dlabici: Update BuildDuty to CiDuty. Renaming project

__TOC__

= What is CiDuty? =
CiDuty (formerly BuildDuty) is a team dedicated to helping out developers with Firefox continuous integration infra issues. We currently have six people based in Romania that provide 24/7 support. CiDuty complement the [[Sheriff|sheriffing team]] where sheriffs respond to Firefox code regressions, CiDuty respond to the infrastructure that builds and tests Firefox code.

= Manifesto =
The [[ReleaseEngineering/Buildduty_manifesto| CiDuty manifesto]] describes the team responsibilities in a nutshell.

= Team =

{| border=1
| '''Name'''
| '''Profile'''
| '''Social'''
| '''Blog'''
|-
| Jordan Lund
| [https://mozillians.org/u/jlund jlund]
| [https://github.com/lundjordan github]
| [http://jordan-lund.ghost.io/ blog]
|-
| Zsolt Fay
| [https://mozillians.org/en-US/u/zfay/ zfay]
| [https://github.com/Rivulu5 github]
| N/A
|-
| Radu Iman
| [https://mozillians.org/en-US/u/riman/ riman]
| [https://github.com/raduiman github]
| N/A
|-
| Bogdan Crisan
| [https://mozillians.org/en-US/u/bcrisan/ bcrisan]
| [https://github.com/bccrisan github]
| N/A
|-
| Danut Labici
| [https://mozillians.org/en-US/u/dlabici/ dlabici]
| [https://github.com/akhliskun github]
| N/A
|-
| Roland Mutter
| [https://mozillians.org/en-US/u/rmutter/ rmutter]
| [https://github.com/mutterroland github]
| N/A
|-
| Adrian Pop
| [https://mozillians.org/en-US/u/apop/ apop]
| [https://github.com/popadrianc github]
| N/A
|}

= CiDuty priorities =
The [[ReleaseEngineering/Buildduty_actionable| CiDuty actionable]] enumerates their daily/weekly sanity job.

== Others ==
There is a long list of '''[[ReleaseEngineering/Buildduty/Other_Duties|other, less-frequent duties]]''' that CiDuty can assist with.

= Documentation =
There's a [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/How_To wiki page] that aggregates useful info related to the tasks CiDuty is taking care of (as of December 2017).

= Useful Links =
* [[ReleaseEngineering/Buildduty/day_1_checklist|Day 1 checklist]]
* [https://secure.pub.build.mozilla.org/builddata/reports/slave_health/ Slave Health]
* [https://secure.pub.build.mozilla.org/slavealloc/ui/#slaves Slave Alloc]
* [https://tools.taskcluster.net/provisioners Provision Explorer]
* [https://secure.pub.build.mozilla.org/builddata/reports/slave_health/buildduty_report.html CiDuty Report]
* [https://secure.pub.build.mozilla.org/buildapi/ Build Dashboard Main Page]
** You can get JSON dumps for people to analyze by adding <code>&format=json</code>
* [https://wiki.mozilla.org/Buildduty/How_To Public "How To" documents]('''new''')
* [[ReleaseEngineering/How_To|Old/Deprecated Public "How To" documents]]
* [https://mana.mozilla.org/wiki/dosearchsite.action?queryString=title%3A%22How%20To%22&where=RelEng Private "How To" documents]

= Meeting Notes =
* [https://etherpad.mozilla.org/buildduty-notes Daily buildduty stand-up notes]
* [[ReleaseEngineering/Buildduty/Meetings|Old buildduty weekly meetings notes]]
* [[ReleaseEngineering/Buildduty/SVMeetings| SoftVision buildduty stand-up notes]]

Buildduty/manifesto

2018-05-14T08:45:55Z

Dlabici: Update BuildDuty to CiDuty. Renaming project

== Intro ==

CiDuty is an operational support team dedicated to monitoring and maintaining the health of Firefox’s continuous integration (CI) infrastructure. Employees are contractors located in Romania that provide 24/7 support. The team's '''''responsibilities include''''' but are not limited to:
* [[ReleaseEngineering/Buildduty_manifesto#Firefox_CI_infrastructure_outage_coordination_and_investigation| Firefox CI infrastructure outage coordination and investigation]]
* [[ReleaseEngineering/Buildduty_manifesto#Monitoring,_investigating,_and_debugging_issues_with_the_Linux,_Windows,_and_OS_X_Firefox_CI_infrastructure| Monitoring, investigating, and debugging issues with the Linux, Windows, and OS X Firefox CI infrastructure]]
* [[ReleaseEngineering/Buildduty_manifesto#Monitoring_Firefox_CI_backlog/pending_counts| Monitoring Firefox CI backlog/pending counts]]
* [[ReleaseEngineering/Buildduty_manifesto#Tree_closing_and_opening| Tree closing and opening]]
* [[ReleaseEngineering/Buildduty_manifesto#Loaning_Firefox_build/test_instances_to_developers| Loaning Firefox build/test instances to developers]]
* [[ReleaseEngineering/Buildduty_manifesto#Upload_new_packages_or_Python_modules_to_our_internal mirrors| Upload new packages or Python modules to our internal mirrors]]
* [[ReleaseEngineering/Buildduty_manifesto#Routine_maintenance_of_the_Firefox_CI_configuration| Routine maintenance of the Firefox CI configuration]]

While team’s responsibilities cover a wide variety of tasks, people sometimes contact them for (or file bugs in their queue for) '''''issues that don’t''''' fall under their area of expertise. The following are some of the most common examples:
* [[ReleaseEngineering/Buildduty_manifesto#Firefox_release_builds| Firefox release builds]]
* [[ReleaseEngineering/Buildduty_manifesto#Developer_test_failures| Developer test failures]]

== Things CiDuty can help with ==

===== Firefox CI infrastructure outage coordination and investigation =====
When the Firefox CI system fails, getting services online again is ciduty's top priority. They are the initial point of contact for outages, but will likely escalate to additional teams with subject matter experts for resolution.

===== Monitoring, investigating, and debugging issues with the Linux, Windows, and OS X Firefox CI infrastructure =====
CiDuty monitors the Firefox CI infrastructure using the Nagios GUI and irc alerts in the #ci irc channel. They routinely look for system issues, resolve them using our automation tooling, or work with datacenter staff to repair offline or degraded hardware. They also monitor email from AWS about infrastructure that is degraded or requires maintenance.

===== Monitoring Firefox CI backlog/pending counts =====
CiDuty is the first point of contact for monitoring the load on the Firefox CI system and determining the cause of any high backlog or pending job counts. If they are unable to determine the root cause and solve the issue, CiDuty escalates to other teams who have subject matter experts.

===== Tree closing and opening =====
Closing and opening the trees (denying and allowing code checkins to our mercurial repos) are typically handled by the Mozilla Code Sheriffs, but CiDuty can also help out with this if needed.

===== Loaning Firefox build/test instances to developers =====
CiDuty processes bugzilla requests from developers for Firefox CI build or test loaners. To obtain a loaner, submit a request to bugzilla under [https://mzl.la/2L2izjE CiDuty] and expect a response in less than one working day (UTC+2).

===== Upload new packages or Python modules to our internal mirrors =====
CiDuty can help a developer who needs a new software package uploaded to tooltool or a Python package uploaded to our internal PyPi mirror. They can also grant other developers access to upload packages to tooltool, for a given paths subset, to allow for future self-service.

===== Routine maintenance of the Firefox CI configuration =====
While most of the Taskcluster configuration is handled by the end-developer, we still have infrastructure using the Buildbot CI infrastructure as well. CiDuty has the knowledge and capability to modify the Firefox buildbot-configs and perform general maintenance of the Buildbot systems. Maintenance includes tasks such as retasking machines from one platform to another as capacity requirements demand, decommissioning machines, updating keys and secrets, etc.

== Things CiDuty isn’t responsible for ==

===== Firefox release builds =====
CiDuty is not responsible for checking the state or fixing issues with any of the Firefox channel builds (Nightly, DevEdition, Release, etc). Releases of any kind are are handled by the ReleaseDuty team within Release Engineering.

===== Developer test failures =====
CiDuty is the first point of contact for any ongoing, systemic, Firefox CI test failures which appear to be caused by infrastructure-related issues, but they do not triage or fix other types of test failures. Any bugs filed in the CiDuty queue for these general types of developer test failures (e.g. Mochitest, reftest, etc) will be moved during bug triage.

ReleaseEngineering/How To/Process release email

2018-05-14T08:41:05Z

Dlabici: Update BuildDuty to CiDuty. Renaming project

{{Release Engineering How To|Process email to release@}}
This is a list of automatically generated emails you should expect to receive as a release engineer at mozilla. It is not complete.

Note that email is not a good notification methodology, and better systems should always be preferred. However, it often is all that is available for audience which needs the notification. To minimize the pain of email notifications, follow these guidelines:
* email should go to a unique address for the service. This can be achieved by using "plus addresses" (preferred due to positive filtering criterea). (Note: AWS SES refers to these as "labels".)
** if not possible, the message MUST have a unique start to the subject field (brittle).
* email should be documented on this page.

Some email is also routed to archives, which you may prefer to search instead of joining a list to receive emails:
* [https://groups.google.com/a/mozilla.com/forum/#!forum/releng-puppet-mail releng puppet email]
* [https://groups.google.com/a/mozilla.com/forum/?hl=en#!forum/release-automation-notifications release automation email]
* [https://groups.google.com/a/mozilla.com/forum/?hl=en#!forum/releng-ops-trial vcs-sync operations email]

= Index =
''Note:'' The "Wildcard" column gives a suggestion on how to filter for that email.

{| cellpadding="10" cellspacing="0" border="1" class="fullwidth-table sortable"
!Field !! Wildcard !! Further Notes
|-
|Subject || collapse report || [[#Performance Metrics]]
|-
|Subject || Suspected machine issue (* || Not an actionable email at this point. (from: nobody@cruncher - s/a {{bug|825625}}
|-
|Subject ||Talos Suspected machine issue *|| ''if you don't know, you don't care''
|-
|Subject ||Try submission *||to: autolanduser@mozilla.com
|-
|Subject ||[vcs2vcs] alert_major_errors* || '''major processing error''' make sure build duty and/or hwine know [[#vcs2vcs|details]]
|-
|Subject ||[vcs2vcs]: git.m.o push N failed: * || single occurrence related to git.m.o/releases/gecko.git '''if repeated, this is a major processing error''' make sure build duty and/or hwine know [[#vcs2vcs|details]]
|-
|Subject ||[vcs2vcs] process delays* || '''if repeated, this is a major processing error''' make sure build duty and/or hwine know [[#vcs2vcs|details]]
|-
|Subject || [release-runner] failed || [[#What_to_do_when_one_is_received_6| How to investigate release runner failures]]. ''(ignore if a release isn't actively being started)''
|-
|To || release+amp@mozilla.com || Test Google Play store account. Blassey and Snorp have access. RelEng does not have access at this time.
|-
|To || release+aws@mozilla.com || AWS admin email, service notifications & marketing. See [[#Amazon_EC2_Instance_scheduled_for_retirement|list of AWS emails]], contact catlee if unsure how to handle.
|-
|To || release+bitbucket@mozilla.com || Mozilla Bitbucket Admin email (contact hwine for now)
|-
|To || release+vcs2vcs@mozilla.com || Output from vcs2vcs hg<->git conversion ([[#vcs2vcs|details]])
|-
|To || release+aws-sanity-check@mozilla.com || Output from cruncher [http://hg.mozilla.org/build/cloud-tools/file/default/aws/aws_sanity_checker.py aws_sanity_checker.py] (contact rail)
|-
|To || release+ec2.*@mozilla.com || error output from crontab on the indicated machine. FIX ISSUE!
|-
|To || release+sns@mozilla.com || SNS issue notifications from various services. FIX ISSUE!
|-
|To || release+update.b2g.o@mozilla.com || low disk space on dogfood update server, see {{bug|877224}}
|-
|To || release+chromecast@mozilla.com || Developer account for Chromecast app support {{bug|1037018}} ([[#chromecast|details]])
|-
|To || release+v2v-gh@mozilla.com || Primary email for github account moz-v2v-gh. Contact vcs-sync folk
|-
|To || release+roku@mozilla.com || Primary email for Roku account, mfinkle is dev contact
|-
|To || release+signaddons@mozilla.com || Primary email for signing addons in automation via API
|-
|To || release+ubuntu-store@mozilla.com || Primary email for Ubuntu Store
|-
|To || release+mozdef@mozilla.com || Security alerts from infosec's Mozdef server. Alert team&infosec if you find suspicious activity.
|-
|To || release+moc_notifications@mozilla.com || Something from the MOC. Action depends on content. (Cited in [https://mana.mozilla.org/wiki/display/MOC/MOC+Contact+Groups#MOCContactGroups-Teams mana].)
|-
|To || release+appleagent@mozilla.com || Related to Apple ID account -- bring to manager's attention if lots of activity.
|-
|To || release+wcw@mozilla.com & release+wmw@mozilla.com || Requests for Wednesday Change Window (mana link to come). CiDuty or manager should respond.
|-
|To || release+cot@mozilla.com || GPG expiration monitoring. Alert aki, catlee, garndt.
|}

 

__TOC__

=Performance Metrics=
== Why we get them ==
We get various emails containing raw data that relates to a performance bottle neck at some point in time. Typically these are produced by cron jobs, and so received regularly regardless of metric status. (I.e. they may not require any action.)

== What is sending them ==
Since this is a "catch all" category, various tools send them. Check the full headers for information on sender and source machine as needed.

== What to do when one is received ==
If you don't know what it's about, you don't need to deal with it beyond setting up a filter to ignore it.

== How to silence or acknowledge this alert ==
It's not an alert, so they'll keep coming until the end of time. Filter them if you're not involved with them.

== Future plans ==
Adhoc, so varies by email. Theoretically, these ''should'' be transitional, and moved into automation and alerting as soon as the metric is understood.

== How to best filter these emails ==
Since these are adhoc, you'll need adhoc filters. It would be nice if folks used a common prefix on subjects, such as "<tt><nowiki>[releng metrics]</nowiki></tt>".

<div id="vcs2vcs"> </div>
= vcs2vcs System=
== Why we get them ==
These emails are the interim notification for vcs2vcs system, and indicate an error that must be addressed. The b2g project is dependent upon parts of the vcs2vcs system, as are other developers and partners.

== What is sending them ==
All emails are sent (perhaps indirectly) by a script from [http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools/ vcs2vcs tools]. The hosts sending the email will be one of the ones listed in [http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-configs/file/default/status the configs]. Full details of how each script is run, including trouble shooting tips, are in the [http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools/file/default/docs docs] (a formatted copy ''may'' be online [http://people.mozilla.com/~hwine/tmp/vcs2vcs/ here]).

== What to do when one is received ==
* if the subject contains "[vcs2vcs] AUTOFIX process delays", then look for another email within a few minutes and proceed as follows:
** if a second email from the same host follows almost immediately with a subject of "[vcs2vcs] process delays", then the AUTOFIX failed, which is an unexpected condition. Page hwine.
** if no followup email is received, the AUTOFIX worked. Log in {{bug|829025}} & delete (or just delete and leave for hwine to log).
* if the subject contains "[vcs2vcs] process delays" '''''and''''' is repeated every 20 minutes, this is a service outage - one or more repositories are no longer being updated. The email contents will give specific errors. Consult the trouble shooting section of the docs (above) for guidance and/or PAGE hwine.
** Unfortunately, there appear to be a few race conditions between scripts, so a single occurrence of the email may be a false positive. ({{bug|839595}} filed to track this.)

* if the subject contains "[vcs2vcs] alert_major_errors alert", this is a major problem - one or more repositories are no longer being updated. The email contents will give specific errors. Consult the trouble shooting section of the docs (above) for guidance and/or contact hwine.
** The most common cause of this is hg repo corruption, the recovery is scripted, but can take some time. Please add to {{bug|808129}} if you fix, or block that bug with a new bug.
** NOTE: you may receive an additional email after the root cause is resolved. (The alert checks on the hour for problems in the prior hour.)

* if the subject containes "[vcs2vcs]: git.m.o push N failed for gecko.git:", this is a (usually) transient problem with pushing gecko.git (the partner facing gecko repository) to either git.m.o or git staging. Two pushes are tried each iteration - both should succeed. Each push is numbered '1' or '2', if you see only one email report, the other already succeeded, and is ignorable. One or two sets of emails is ignorable, any more needs investigation, starting with the health of git.mozilla.org. (Note that the message is short, as this also pages hwine via sms, where brevity is nice.)

* if the subject is something else, this is likely unexpected output from a cron job. Judge the severity and escalate to hwine appropriately. File a bug to get better diagnosis of this error condition in the future.

== How to silence or acknowledge this alert ==
Resolving the root cause will stop the emails.

== Future plans ==
The system will eventually be transitioned to Developer Productivity (nee Developer Services (nee IT)) for operations. Specific email will be converted to nagios alerts before then.

== How to best filter these emails ==
All of these emails are sent to the addresses of the form: <tt>release+vcs2vcs*@mozilla.com</tt>. Common sub addresses are:
; release+vcs2vcs : mail that will have specifics in the Subject line.
; release+vcs2vcs+forward : mail to vcs2vcs user, forwarded via <tt>~/.forward</tt> file.

= Release runner =
== Why we get them ==
Release runner sends e-mail when it fails in any way. Eg, failing to poll ship it after a long period of time or failing to start a submitted release.

== What is sending them ==
* Source: https://hg.mozilla.org/build/tools/file/default/buildfarm/release
* Sent by: buildbot
* Runs on: buildbot-master36 through supervisord

== What to do when one is received ==
[[Release:Release_Automation_on_Mercurial:Troubleshooting#How_to_investigate_release_runner_failures | How to invesigate release runner failures]]

== How to silence or acknowledge this alert ==
Fix whatever problem release runner has hit. (Sometimes this means waiting out network issues.) There's no way to ack (in the nagios sense) release runner e-mails.

== Future plans ==
They're here to stay.

== How to best filter these emails ==
[release-runner] in the subject.

=Amazon EC2 Instance scheduled for retirement=
==Example==
One or more of your Amazon EC2 instances in the us-east-1 region is scheduled for retirement. The following instance(s) will be shut down after 12:00 AM UTC on 2013-10-22.

i-02cc2669

== Why we get them ==
Amazon needs us to move our virtual instance(s) off of certain physical hardware so they can perform maintenance on it.

== What is sending them ==
Automated notification sent by no-reply-aws@amazon.com

== What to do when one is received ==
* determine what host is running on the specified EC2 instance.
* power the instance down in an orderly manner
* start it back up

== How to silence or acknowledge this alert ==

== Future plans ==

== How to best filter these emails ==
Filter on the sender and subject line.

=SNS Notifications from AWS=
==Example==
Anything with the Subject "AWS Notification Message"

== Why we get them ==
We use SNS to deliver notifications about various Amazon services as well as services like Papertrail. These are generally critical alerts that we've set up and should be dealt with/investigated in a timely fashion. At the moment, only AWS Cloudwatch and Papertrail use this service, but we will likely add more in the future after we get an SNS->irc bot set up because it allows for an easy HTTP/HTTPS endpoint push that other services already integrate with.

== What is sending them ==
The Amazon SNS service notification topic "buildduty"
* arn:aws:sns:us-west-2:314336048151:buildduty
* arn:aws:sns:us-east-1:314336048151:buildduty

== What to do when one is received ==
Determine what the issue is by parsing the output. Make sure someone is working on fixing the issue (if you're not sure how, at least contact ciduty for their input/advice).

== How to silence or acknowledge this alert ==
Fix the underlying issue to stop the alert.

== Future plans ==
In the near future we intend to send SNS notifications to an irc bot instead of via email.

== How to best filter these emails ==
Ideally you should not filter them except into a high priority folder. You can filter on the Subject or the To address.

<div id="chromecast"></div>

= Mail to release+chromecast@mozilla.com =
== Why we get them ==
The mobile team is adding Chromecast support (ability to fling videos/tabs from a device to a TV). They need a persistent account not linked to a single developer who might leave the company at some point.

== What is sending them ==
These emails come from the

== What to do when one is received ==
Traffic should be light. If the email is not simply Google self-promotion, please forward it to lead mobile devs, namely :blassey and :mfinkle.

== How to silence or acknowledge this alert ==

== Future plans ==

== How to best filter these emails ==
You can either filter on the "To:" field for "release+chromecast@mozilla.com" to catch just these emails, or filter on "From:" for "noreply@google.com" and move all mail from Google (we have multiple accounts mailing us intermittently) to a separate Google subfolder (coop).

<hr />


=Security Alerts from Mozdef=
== Why we get them ==
Mozdef is an ELK stack (logging aggregator + parser) run by the infosec team. They're consuming our Papertrail logs, at our request.

2016.09.13: We have asked them to create some preliminary alerts on ssh access to our signing infrastructure. See https://bugzilla.mozilla.org/show_bug.cgi?id=1290261

== What is sending them ==
2016.09.13: the infosec team has a cron job finding ssh activity on the signing infrastructure, and that emails us.

== What to do when one is received ==
2016.09.13: The emails are very new. For now, we most likely want to take a look and see what the 'normal' looks like, so we know when something out of the ordinary happens.

On suspicious email, notify the team and infosec.

== How to silence or acknowledge this alert ==
2016.10.08: These will send once an hour if there is ssh access.

== Future plans ==
2016.09.13: We may change the frequency of the emails to be more immediate, once we know the noise level.

== How to best filter these emails ==
As noted in the table above, these are sent to release+mozdef@mozilla.com

=Mail to release+moc_notifications@mozilla.com=
== Why we get them ==
Unsure when MOC will use this address.

== What is sending them ==
Humans from MOC will use this address.

== What to do when one is received ==
* Read and handle

== How to silence or acknowledge this alert ==
* depends on context

== Future plans ==
Unknown - check [https://mana.mozilla.org/wiki/display/MOC/MOC+Contact+Groups#MOCContactGroups-Teams mana] to see if anything has changed.

== How to best filter these emails ==
Filter by "to" address.

=Mail to release+appleagent@mozilla.com=
== Why we get them ==
* 2 step verification
* fall back account

== What is sending them ==
Apple when folks interact with the release Apple ID agent account.

== What to do when one is received ==
* If you generated it, claim it by reply.
* Unclaimed emails should be escalated to folks with access to release Apple ID accounts

== How to silence or acknowledge this alert ==
* depends on context

== Future plans ==
''none''

== How to best filter these emails ==
Filter by "to" address.

=Sample=
== Why we get them ==
Give a brief explanation of why this email is for, what it helps us do and why it should be watched

== What is sending them ==
Include a link to the source of the program sending the email. Include information on which hosts are sending the email, and give information on how program runs. Is it a daemon? Does it have an init script? Do you run it under screen?

== What to do when one is received ==
* if the title contains "[scl-production-puppet-new] <slavename> is waiting to be signed", this is for information and requires no immediate action
* if the title contains "[scl-production-puppet-new] <slavename> has invalid cert", the script will try once to clean the cert before sending the email. If this is successful, you'll see a matching "<slavename> is waiting to be signed" email. The key will be automatically signed

== How to silence or acknowledge this alert ==
Include information on how to make the emails stop

== Future plans ==
provide any future plans for this email. Is it temporary? Is it going to be replaced by a real dashboard? Are you going to add/change things people filter on?

== How to best filter these emails ==
provide insight on how to filter these emails. Is there a distinguishing header? Is it always from a specifc host, or family of hosts? Is there a distinctive subject?

=Mail to release+cot@mozilla.com=
== Why we get them ==
* gpg pubkey expiration monitoring ([https://bugzilla.mozilla.org/show_bug.cgi?id=1373986])

== What is sending them ==
https://tools.taskcluster.net/hooks/project-releng/cot-gpg-keys%2Fexpiration

== What to do when one is received ==
* Alert the cot-gpg-keys contributors [https://github.com/mozilla-releng/cot-gpg-keys/graphs/contributors]

== How to silence or acknowledge this alert ==
* depends on context

== Future plans ==
''none''

== How to best filter these emails ==
Filter by "to" address.

ReleaseEngineering/SpecialRepositories

2018-05-14T08:39:59Z

Dlabici: Update BuildDuty to CiDuty. Renaming project

= Special branches =
Release Engineering and the A-Team have various [[ReleaseEngineering/DisposableProjectBranches#Indefinite_booking|project branches]] booked permanently for various reasons:
* Cedar - where we enable test jobs that are still failing

= Cedar =
== Merging m-c into Cedar ==

We can merge the latest m-c code into cedar to:

* compare latest unittest results + talos numbers between m-c buildbot tests and cedar mozharness tests
* get new cedar builds for testing

To do this:

<pre>
# Get a copy of cedar
hg clone http://hg.mozilla.org/projects/cedar

cd cedar

# Update to latest, in case someone has updated cedar since you last cloned/updated.
# the hg up -C will blow away any local changes!
hg pull
hg up -C -r default

# Pull latest m-c changes in
hg pull http://hg.mozilla.org/mozilla-central
hg merge
hg commit -m "Merge m-c -> cedar"

# Push to cedar
hg push ssh://hg.mozilla.org/projects/cedar
</pre>

Results will show up [https://tbpl.mozilla.org/?tree=Cedar here].

== Viewing logs ==

Jobs will show up [https://tbpl.mozilla.org/?tree=Cedar on tbpl] as they finish.

To view logs

* click on the job on the right
* find "View full log" at the bottom left of the page; click

Known issues are filed against the two tracking bugs:

* {{bug|713055}} - roll out mozharness desktop talos to m-c and project branches
* {{bug|793022}} - roll out mozharness desktop unittests to m-c and project branches

If you find new issues, please file a bug blocking one or both bugs.

== Retriggering jobs ==

* Go to [https://tbpl.mozilla.org/?tree=Cedar tbpl]
* Click on the job in question
* The plus sign in the bottom left will retry the job.

Each of these will pull the latest-and-greatest from mozharness, so if you checked a fix in, you can re-run the broken tests.

== Adding additional tests to Cedar ==

If we want to readd tpaint once {{bug|794587}} is fixed:

* add "tpaint" to the other suite tests here: http://hg.mozilla.org/projects/cedar/file/38215d0638b0/testing/talos/talos.json#l55
* check that into ssh://hg.mozilla.org/projects/cedar
* the resulting build will trigger an "other" talos suite with tpaint.

If we want to add another test type, e.g. marionette, to cedar:

* add "cedar" to the list of marionette branches here: http://hg.mozilla.org/build/buildbot-configs/file/680736f2dc1f/mozilla-tests/config.py#l1076
* move the [http://hg.mozilla.org/build/buildbot-configs/file/680736f2dc1f/mozilla-tests/config.py#l1075 marionette for loop] below the [http://hg.mozilla.org/build/buildbot-configs/file/680736f2dc1f/mozilla-tests/config.py#l1100 unittest for loop].
** This is because the unittest for loop zeros out the tests for cedar [http://hg.mozilla.org/build/buildbot-configs/file/680736f2dc1f/mozilla-tests/config.py#l1132 here].
*** (this is because we don't want to have both buildbot-based and mozharness-based tests running on cedar.)
* get review for your change
* land
* wait for reconfig, or poke ciduty for an expedited reconfig if needed.

Sheriffing/Deciding To Close A Tree

2018-05-14T08:39:00Z

Dlabici: Update BuildDuty to CiDuty. Renaming project

{{Sheriffing How To|Deciding to close a tree}}
== Deciding to close a tree ==
Many objective and subjective criteria are part of the decision to close a tree. Tree closure means that developers are prevented from pushing or merging code to a codebase. Later, sheriffs will reopen the trees when the problem appears to be resolved.

Some of the criteria used include:
* Broken build on an integration or main tree (e.g. mozilla-inbound, mozilla-central, autoland)
* Excessive backlog for builds or tests in any platform
* Infrastructure or systems failures that affect a significant number of tests or builds (e.g. AWS, data center, networking issues)
* Mass "bustage" that could hide other test failures (this is when code lands and causes multiple tests to fail across multiple chunks of tests or suites of tests, making it harder to catch further failures if something else lands *during* the period in which these tests are failing from the original code landing)
* Infrastructure failure that affects our ability to see what's happening (e.g. Treeherder being down, not ingesting jobs, the data it consumes not being updated, or treestatus being broken so we're closed by default)

In short, if the state of the tree and the surrounding systems is such that things are going to get worse if the trees stay open, it is time to close the tree.

== Actions to take ==
Once you've decided to close the tree, you need to take the following steps:

* Decide which trees are affected
** If the cause is not infrastructure- or load-related, you can probably leave the Try branch and only close the affect tree, e.g. mozilla-inbound
** If the cause *is* infrastructure- or load-related, you should close all trees, including Try.
* Use the [https://mozilla-releng.net/treestatus TreeStatus] tool to close the affected trees.
* If a bug doesn't already exist, create a bug for the tree closure.
* Communicate the tree closure to developers. Announce the closure in IRC in #developers and change the channel topic to point to the tree closure bug. This avoid several unnecessary frustrations:
** developers want to push and notice it's not possible.
** developers spending time investigating failures in pending/running testruns (especially Try) that are not caused by her/his changes
** minimizes repeated inquiries to the sheriffs about why a tree is closed, what's the ETA, etc.
* engage the people needed to fix the issue. This could be:
** the developer(s) who landed the suspected code (if this is known)
** domain experts for the module where the builds/tests are failing. The [[Modules/All|Module owner list]] can help track people down.
** the ciduty, releng, and/or the taskcluster teams if it's an infrastructure issue
* If the tree closure is expected to be a longer problem, post a short mail to the mozilla.dev.platform newsgroup, e.g. https://groups.google.com/forum/#!topic/mozilla.dev.platform/Kzd1es4KiYA

== Next steps ==
In case it wasn't clear from the previous section, '''COMMUNICATION''' is the ''most important thing'' during any outage. When engaging others to help, letting them know that the trees are closed ''usually'' encourages prompt cooperation. Developers or service teams may need your help to test fixes efficiently on Try, or to back out particular changesets once they are implicated. Make yourself available to these people as required.

For everyone else not involved in trying to fix the issue, they're simply waiting for the trees to reopen and are effectively blocked. Developers and service teams may not have the time or experience to reliably update the tree closure bugs (or IRC) with status. Sheriffs should take the lead on this and keep other developers updated, both in the bug and in IRC.

== Hand-off ==
In the event of longer tree closures, you may hit the end of your workday. If this happens, or better still, *before* it happens, find someone who can continue acting overseeing the tree closure. If you've followed all the above steps, there should be an adequate papertrail for someone to follow and continue.

Note: in the event of more systemic failures, e.g. major infrastructure failures or AWS outages, it is best to [[Sheriffing/How:To:Escalate|escalate the issue to the MOC]] (Mozilla Operations Center, #moc on IRC). They have 24/7 support and much experience dealing with outages.

QA SoftVision Team

2018-05-14T08:37:22Z

Dlabici: Update BuildDuty to CiDuty. Renaming project

= SoftVision Team =

== Desktop - Release QA Team ==
{|
|-
|[https://wiki.mozilla.org/QA_SoftVision_Team/Desktop Firefox desktop]
|}

{|
|-
|Join [https://02.chat.mibbit.com/?url=irc%3A%2F%2Firc.mozilla.org%2F%2523firefox #firefox] on irc.mozilla.org
|}
{|
|-
|Join [https://02.chat.mibbit.com/?url=irc%3A%2F%2Firc.mozilla.org%2F%2523qa #qa] on irc.mozilla.org
|}

== Fennec Team ==
{|
|-
|[https://wiki.mozilla.org/QA_SoftVision_Team/Mobile Mobile Workflow]
|}

{|
|-
|Join [https://02.chat.mibbit.com/?url=irc%3A%2F%2Firc.mozilla.org%2F%2523mobile #mobile] on irc.mozilla.org (use SSL)
|}

== Add-ons QA Team ==
{|
|-
|[https://wiki.mozilla.org/QA_SoftVision_Team#Add-ons_QA_Team Add-ons QA Team]
|}

{|
|-
|Join [http://wbe02.mibbit.com/?settings=1b10107157e79b08f2bf99a11f521973&server=irc.mozilla.org&channel=%23addonsqa #addonsqa] on irc.mozilla.org (use SSL)
|}

== CI Duty Team ==
{|
|-
|[https://wiki.mozilla.org/QA_SoftVision_Team#Build_Duty_Team_2 CiDuty Team]
|}

{|
|-
|Join [https://irccloud.mozilla.com/#!/ircs://irc1.dmz.scl3.mozilla.com:6697/%23ci #ci] on irc.mozilla.org (use SSL)
|}

== Firefox Development Team ==
{|
|-
|[https://wiki.mozilla.org/QA_SoftVision_Team#Firefox_Development_Team_2 Firefox Development Team]
|}

== Fennec Development Team ==
{|
|-
|[https://wiki.mozilla.org/QA_SoftVision_Team#Fennec_Development_Team_2 Fennec Development Team]
|}

== Participation Systems Team ==
{|
|-
|[https://wiki.mozilla.org/QA_SoftVision_Team#Participation_Systems_QA_Team Participation Systems Team]
|}

== Web Compatibility Team ==
{|
|-
|[https://wiki.mozilla.org/QA_SoftVision_Team#Web_Compatibility_Team_2 Web Compatibility Team]
|}

== Performance Sheriff ==
{|
|-
|[https://wiki.mozilla.org/QA_SoftVision_Team#Performance_Sheriff_2 Performance Sheriff]
|}

== WebExtensions Review Team ==
{|
|-
|[https://wiki.mozilla.org/QA_SoftVision_Team#WebExtensions_Review_Team_2 WebExtensions Review]
|}

== Code Sheriffs Team ==
{|
|-
|[https://wiki.mozilla.org/QA_SoftVision_Team#Code_Sheriffs_Team Code Sheriffs Team]
|}

{|
|-
|Join #sheriffs on irc.mozilla.org (use SSL)
|}

== Automation QA Team ==
{|
|-
|[https://wiki.mozilla.org/QA_SoftVision_Team#Automation_QA_Team_2 Automation QA Team]
|}

= Photos =

== Desktop - Release QA Team ==
{| width="100%"
|-
|[[File:C_Ionce.png‎‎ ‎‎|thumb|left|300px| '''Cornel Ionce (Team Lead) Desktop irc: cornel_ionce ''']]
|[[File:BogdanMaris.jpg‎ ‎‎|thumb|left|300px| '''Bogdan Maris (Team Lead) Desktop irc: bogdan_maris ''']]
|[[File:Cristian_Baica.png |thumb|left|300px| '''Cristian Baica Desktop irc: ''']]
|-
|[[File:Mihai_Boldan.png |thumb|left|300px| '''Mihai Boldan Desktop irc: mboldan ''']]
|[[File:Oana_Botisan.png ‎‎|thumb|left|300px| '''Oana Botisan Desktop irc: Oana.Botisan''']]
|[[File:Camelia_Badau.jpg |thumb|left|300px| '''Camelia Badau Desktop irc: cbadau''']]
|-
|[[File:Gabi_Cheta.png |thumb|left|300px| '''Gabi Cheta Desktop irc: Gabi''']]
|[[File:Iulia_Cristescu.png ‎‎|thumb|left|300px| '''Iulia Cristescu Desktop irc: IuliaC''']]
|[[File:CiprianGeorgiu.png ‎‎|thumb|left|300px| '''Ciprian Georgiu Desktop irc: ciprian.georgiu''']]
|-
|[[File:Cristian_Comorasu.png |thumb|left|300px| '''Cristian Comorasu Desktop irc: CristiComo''']]
|[[File:Alexandru_Simonca.png ‎‎|thumb|left|300px| '''Alexandru Simonca Desktop irc: asimonca''']]
|[[File:Timea_Zsoldos.png ‎‎|thumb|left|300px| '''Timea Zsoldos Desktop irc: zstimi''']].
|-
|[[File:Emil_Ghitta.png |thumb|left|300px| '''Emil Ghitta Desktop irc: emilghitta''']]
|[[File:Anca_Soncutean.png |thumb|left|300px| '''Anca Soncutean Desktop irc: Anca''']]
|[[File:Catalin_Sasca.png |thumb|left|300px| '''Catalin Sasca Desktop irc: csasca''']]
|-
|[[File:Stefania_Farcas.png |thumb|left|300px| '''Stefania Farcas Desktop irc: sfarcas''']]
|[[File:Cristian_Fogel.jpeg |thumb|left|300px| '''Cristian Fogel Desktop irc: cfogel''']]
|[[File:Vlad_Lucaci.jpeg |thumb|left|300px| '''Vlad Lucaci Desktop irc: vlucaci''']]
|}

== Mobile Firefox Team ==
{| width="100%"
|-
|[[File:Ioana_chiorean.jpg|thumb|left|300px| '''Ioana Chiorean Fennec irc: IoanaChiorean''']]
|[[File:Oana_Horvath.png|thumb|left|300px| '''Oana Horvath Fennec irc: oanah''']]
|[[File:Bogdan_Surd.png|thumb|left|300px| '''Bogdan Surd Fennec irc: BogdanS''']]
|-
|[[File:Sorina_Florean.png|thumb|left|300px| '''Sorina Florean Fennec irc: sorina''']]
|[[File:Levente_Sacal.png|thumb|left|300px| '''Levente Sacal Fennec irc: levi24''']]
|[[File:Catalin_Suciu.jpg|thumb|left|300px| '''Catalin Suciu Fennec iOS irc: csuciu''']]
|-
|[[File:Simon_Basca.jpg ‎‎|thumb|left|300px| '''Simon Basca Fennec iOS irc: SimonB''']]
|
|
|}

== Add-ons QA Team ==
{| width="100%"
|-
|[[File:Victor_Carciu.jpg ‎|thumb|left|300px| '''Victor Carciu - Team Lead Add-ons QA irc: victorC''']]
|[[File:Madalin_Cotetiu.jpg |thumb|left|300px| '''Madalin Cotetiu Add-ons QA irc: madalinc''']]
|[[File:Ioana Rusiczki.png ‎‎|thumb|left|300px| '''Ioana Rusiczki Add-ons QA irc: IoanaR''']]
|-
|[[File:Cosmin_Badescu.png|thumb|left|300px| '''Cosmin Badescu Add-ons QA irc: cosminbadescu''']]
|[[File:Alexandra_Moga.png ‎‎|thumb|left|300px| '''Alexandra Moga Add-ons QA irc: LexaSV''']]
|[[File:Marius_Santa.png|thumb|left|300px| '''Marius Santa Add-ons QA irc: Santa''']]
|}

== CI Duty Team ==
{| width="100%"
|-
|[[File:Danut_Labici.png |thumb|left|300px| '''Danut Labici Ci Duty irc: dlabici ''']]
|[[File:Bogdan_Crisan.PNG |thumb|left|300px| '''Bogdan Crisan Ci Duty irc: bcrisan ''']]
|[[File:Adrian_Pop.png ‎|thumb|left|300px| '''Adrian Pop Ci Duty irc: apop ''']]
|-
|[[File:Radu_Iman.png |thumb|left|300px| '''Radu Iman Ci Duty irc: riman ''']]
|[[File:Roland_Mutter.png |thumb|left|300px| '''Roland Mutter Ci Duty irc: rmutter ''']]
|[[File:Zsolt_Fay.PNG |thumb|left|300px| '''Zsolt Fay Ci Duty irc: zfay ''']]
|-
|[[File:Dragos_Crisan.png |thumb|left|300px| '''Dragos Crisan RelOps irc: dragrom ''']]
|}

== Firefox Development Team ==
{| width="50%"
|-
|[[File:Bogdan_Postelnicu.png ‎|thumb|left|300px| '''Bogdan Postelnicu Gecko C++ irc: ABPostelnicu''']]
|}

== Fennec Development Team ==
{| width="50%"
|-
|[[File:VladBaicu.jpg ‎|thumb|left|300px| '''Vlad Baicu - Team Lead irc: vladbaicu email: vbaicu@mozilla.com''']]
|[[File:PetruLingurar.jpg ‎|thumb|left|300px| '''Petru_Mugurel Lingurar irc: petru email: plingurar@mozilla.com''']]
|[[File:AndreiLazar.jpg ‎|thumb|left|300px| '''Andrei Lazar irc: andreilazar email: alazar@mozilla.com''']]
|}

== Participation Systems QA Team ==
{| width="50%"
|-
|[[File:Viorela_Ioia.png|thumb|left|300px| '''Viorela Ioia Participation Systems irc: viorelaioia''']]
|}

== Web Compatibility Team ==
{| width="50%"
|-
|[[File:Sergiu_Logigan.png|thumb|left|300px| '''Sergiu Logigan Web Compatibility irc: Sergiu''']]
|[[File:OanaArbuzov.jpeg|thumb|left|300px| '''Oana Arbuzov Web Compatibility irc: Oana''']]
|}

== Performance Sheriff ==
{| width="50%"
|-
|[[File:Ionut_Goldan.png|thumb|left|300px| '''Ionut Goldan Performance Sheriff irc: igoldan''']]
|}

== WebExtensions Review Team ==
{| width="50%"
|-
|[[File:Raluca_Sofian.png|thumb|left|300px| '''Raluca Sofian WebExtensions Review irc: ralucaSofian''']]
|[[File:Andreea_Neamtiu.png|thumb|left|300px| '''Andreea Neamtiu WebExtensions Review irc: andreeaNeamtiu''']]
|}

== Engineering QA Team ==
{| width="100%"
|-
|[[File:Brindusa_Tot.jpg ‎‎|thumb|left|300px| '''Brindusa Tot - Desktop, Team Lead irc: brindusat, email: btot@mozilla.com ''']]
|[[File:Stefan_Georgiev.png |thumb|left|300px| '''Stefan Georgiev - Desktop, Team Lead irc: StefanG_QA, email: sgeorgiev@mozilla.com ''']]
|[[File:Ovidiu_Boca.jpg ‎‎|thumb|left|300px| '''Ovidiu Boca - Desktop irc: ovidiu, email: ovidiu.boca@softvision .ro ''']]
|-
|[[File:Justin_Williams.jpg‎‎ ‎‎|thumb|left|300px| '''Justin Williams - Desktop irc: JW_SoftvisionQA, email: jwilliams@softvision .com ''']]
|[[File:Abe_Masresha.png |thumb|left|300px| '''Abe Masresha - Desktop irc: Abe_LV, email: amasresha@softvision .com ''']]
|[[File:Grover_Wimberly.png ‎‎|thumb|left|300px| '''Grover Wimberly IV - Desktop irc: Grover-QA, email: gwimberly@softvision .com ''']]
|-
|[[File:Adrian_Florinescu.png ‎‎|thumb|left|300px| '''Adrian Florinescu - Desktop irc: AdrianSv, email: adrian.florinescu @softvision.ro ''']]
|[[File:Hani_Yacoub.png ‎‎|thumb|left|300px| '''Hani Yacoub - Desktop irc: hani, email: hani.yacoub@softvision .ro ''']]
|[[File:Kanchan_Kumari.jpg |thumb|left|300px| '''Kanchan Kumari - Desktop irc: Kanchan_QA, email: kkumari@softvision .com ''']]
|-
|[[File:Roxana_Leitan.jpg |thumb|left|300px| '''Roxana Leitan - Desktop irc: RoxanaLeitan, email: roxana.leitan@ softvision .ro ''']]
|
|}

== Test Pilot QA Team ==
{| width="100%"
|-
|[[File:Paul_Oiegas.png |thumb|left|300px| '''Paul Oiegas Desktop, Team Lead irc: pauloiegasSV email: poiegas@mozilla.com ''']]
|[[File:Cosmin_Muntean.png |thumb|left|300px| '''Cosmin Muntean Platform: Desktop irc: CosminMCG email: cosmin.muntean @softvision.ro ''']]
|[[File:Ciprian_Muresan.png ‎‎|thumb|left|300px| '''Ciprian Muresan Platform: Desktop irc: cmuresan email: ciprian.muresan @softvision.ro ''']]
|-
|[[File:Carmen_Fat.png ‎‎|thumb|left|300px| '''Carmen Fat Platform: Desktop irc: carmenf email: carmen.fat@ softvision.ro ''']]
|[[File:Emil_Pasca.png ‎‎|thumb|left|300px| '''Emil Pasca Platform: Desktop irc: emilpasca email: emil.pasca@ softvision.ro ''']]
|[[File:Cristina_Badescu.jpg |155px|thumb|left| '''Cristina Badescu Platform: Desktop irc: cbadescu email: cristina.badescu@ softvision.ro ''']]
|-
|[[File:Marius Coman.png|300px|thumb|left| '''Marius Coman Platform: Desktop irc: mcoman email: marius.coman@ softvision.ro ''']]
|[[File:Andreea.jpg |155px|thumb|left| '''Andreea Cupsa Platform: Desktop irc: - email: andreea.cupsa@ softvision.ro ''']]
|
|}

== Automation QA Team ==
{| width="100%"

|[[File:Andrei_Filip.jpg ‎‎|thumb|left|300px| '''Andrei Filip - Automation QA, Team Lead irc: afilip, email: afilip@mozilla.com ''']]
|[[File:Alin_Rus.jpg ‎‎|thumb|left|300px| '''Alin Rus - Automation QA email: arus@mozilla.com''']]
|[[File:George_Echim.jpg ‎‎|thumb|left|300px| '''George Echim - Automation QA email: gechim@mozilla.com''']]
|-
|[[File:Ioana_Crisan.jpg ‎‎|thumb|left|300px| '''Ioana Crisan - Automation QA email: icrisan@mozilla.com''']]
|[[File:Ionut_Budeanu.jpg ‎‎|thumb|left|300px| '''Ionut Budeanu - Automation QA email: ibudeanu@mozilla.com''']]
|[[File:Paul_Silaghi.jpg ‎‎|thumb|left|300px| '''Paul Silaghi - Automation QA email: psilaghi@mozilla.com''']]
|-
|[[File:Roxana_Robotin.jpg ‎‎|thumb|left|300px| '''Roxana Robotin - Automation QA irc: roxanarobotin, email: rrobotin@mozilla.com''']]
|[[File:Silviu_Checherita.jpg ‎‎|thumb|left|300px| '''Silviu Checherita - Automation QA email: schecherita@mozilla.com''']]
|}

== Code Sheriffs Team ==
{| width="100%"
|[[File:Andreea_Pavel.png|thumb|left|300px| '''Andreea Pavel Code Sheriff irc: apavel''']]
|[[File:Andrei_Ciure.png|thumb|left|300px| '''Andrei Ciure Code Sheriff irc: andrei_ciure_''']]
|[[File:Arthur_Iakab.png|thumb|left|300px| '''Arthur Iakab Code Sheriff irc: arthur_iakab''']]
|-
|[[File:Bogdan_Tara.png|thumb|left|300px| '''Bogdan Tara Code Sheriff irc: bogdan_tara''']]
|[[File:Cosmin_Sabou.png|thumb|left|300px| '''Cosmin Sabou Code Sheriff irc: cosmin_s''']]
|[[File:Cristian_Brindusan.png|thumb|left|300px| '''Cristian Brindusan Code Sheriff irc: CristianB''']]
|-
|[[File:Cristina_Coroiu.png|thumb|left|300px| '''Cristina Coroiu Code Sheriff irc: ccoroiu''']]
|[[File:Dorel_Luca.png|thumb|left|300px| '''Dorel Luca Code Sheriff irc: dluca''']]
|[[File:Eliza_Balazs.png|thumb|left|300px| '''Eliza Balazs Code Sheriff irc: ebalazs_''']]
|-
|[[File:Narcis_Beleuzu.png|thumb|left|300px| '''Narcis Beleuzu Code Sheriff irc: NarcisB''']]
|[[File:Natalia_Csoregi.png|thumb|left|300px| '''Natalia Csoregi Code Sheriff irc: nataliaCs''']]
|[[File:Noemi_Erli.png|thumb|left|300px| '''Noemi Erli Code Sheriff irc: noemi_erli''']]
|-
|[[File:Raul_Gurzau.png|thumb|left|300px| '''Raul Gurzau Code Sheriff irc: RaulGurzau''']]
|[[File:Stefan_Hindli.png|thumb|left|300px| '''Stefan Hindli Code Sheriff irc: stefan_hindli''']]
|[[File:Tiberius_Oros.png|thumb|left|300px| '''Tiberius Oros Code Sheriff irc: tiberius_oros''']]
|-
|[[File:Attila_Craciun.png|thumb|left|300px| '''Attila Craciun Code Sheriff irc: arny''']]
|[[File:Henrietta_Maior.png|thumb|left|300px| '''Henrietta Maior Team Lead irc: henrietta_maior email: hmaior@mozilla.com''']]
|[[File:Robert_Horotan.png|thumb|left|300px| '''Robert Horotan Team Lead irc: Robert_Horotan email: rhorotan@mozilla.com''']]
|}

== Softvision Project Management ==
{| width="100%"
|-
|[[File:Rares_Bologa.jpg|thumb|left|300px| '''Rares Bologa Project Manager irc: RaresB email: rbologa@mozilla.com''']]
|[[File:Cristian_Chiorean.jpg|thumb|left|300px| '''Cristian Chiorean Project Manager irc: CristianChiorean email: cchiorean@mozilla.com''']]
|[[File:AndreiVaida.png‎‎ ‎‎|thumb|left|300px| '''Andrei Vaida Project Manager irc: AndreiVaida email:avaida@mozilla.com''']]
|-
|[[File:Fmezei.png|thumb|left|300px| '''Florin Mezei Delivery Manager irc: FlorinMezei''']]
|}

ReleaseEngineering/How To/Setup buildbot masters in AWS

2018-05-14T08:33:15Z

Dlabici: Update BuildDuty to CiDuty. Renaming project

{{Release Engineering How To|Setup buildbot masters in AWS}}
This page describes how to set-up a new Buildbot Master in AWS.

= Production masters =
A production buildbot master will serve as a central authority responsible for all the decisions about what, when and how to build. It sends requests for running various jobs (builds, tests, etc.) to the attached workers and they will execute the commands and return the results.

= Steps to add a new master =
In order to add a new master, there're certain places we'll need to update:

== Support files ==
The new master will need to be added to [http://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/production-masters.json production-masters.json] in tools repo. Also make sure '''not to enable''' the master before updating DNS with the corresponding entries and setting it up in puppet.

== Puppet ==
* Make sure your masters are listed in [http://hg.mozilla.org/build/puppet/file/default/manifests/moco-nodes.pp PuppetAgain manifests].
** this can be done by hand, but you can also use a [http://hg.mozilla.org/build/braindump/file/default/puppet-related/json2pp.py little helper] to generate snippets (use the "<tt>--help</tt>" option to see options to generation other snippets useful below):
python json2pp.py bm118
# or even better:
hg -R tools export $you_commit |grep '"name":' | grep ^+ | awk -F: '{print $2}' | awk -F'"' '{print $2}' | xargs python json2pp.py
* To create a master instance use the following snippet as a template. Don't forget to adjust the name and region accordingly (or use the snippets produced with the "<tt>--bash FILE</tt>" option):
<pre>
ssh buildduty@aws-manager2.srv.releng.scl3.mozilla.com
source /builds/aws_manager/bin/activate
cd /builds/aws_manager

#get an ip in the correct region
ip=`python cloud-tools/scripts/free_ips.py -c cloud-tools/configs/buildbot-master -r us-west-2 -n1`
# double-check that the IP address is not in use by some other machine
host $ip
# create a DNS entry
# use full LDAP e.g. user@mozilla.com
invtool A create --ip $ip --fqdn buildbot-master07.bb.releng.usw2.mozilla.com --private --description "bug #: bm118"
# create a DNS reverse-mapping (required for puppet certs to work properly)
invtool PTR create --ip $ip --target buildbot-master07.bb.releng.usw2.mozilla.com --private --description "bug #: bm118"
#create a CNAME
invtool CNAME create --fqdn buildbot-master07.build.mozilla.org --target buildbot-master07.bb.releng.usw2.mozilla.com --private --description "bug #: bm118"
sleep 20m # wait for DNS to propagate

#create the instance
python cloud-tools/scripts/aws_create_instance.py -c cloud-tools/configs/buildbot-master -r us-west-2 -s aws-releng \
-k secrets/aws-secrets.json --ssh-key ~/.ssh/aws-ssh-key \
-i cloud-tools/instance_data/us-west-2.instance_data_master.json buildbot-master118
</pre>

Puppet will reboot the master. You can follow the log in a different terminal:
tail -F buildbot-master118.log

* You can get the IP address of the master from the log or using AWS web console.
* When master rebooted make sure to login as cltbld and '''stop the master''':
cd /builds/buildbot/*1* && make stop

* For build/try masters, '''add master's ip to secrets::network::masterIPs''' on master-puppet1:/etc/puppet/manifests/secrets.pp. The signing instances will be automatically reloaded.

== Add masters to inventory ==

=== Create System in Inventory ===
As of 2013-05-13, there are two ways, manual data entry and CSV import
==== CSV Import ====
* Go to https://inventory.mozilla.org/en-US/csv/
* Import the file created with the "<tt>--inventory-csv FILE</tt>" option above.
* After import, use the links in the results page to visit each server & set allocation to "release"
* Continue with [[#Common_Setup|common setup]] below
==== Manual Entry ====
* Go to https://inventory.mozilla.org/en-US/systems/new/
* Fill the following fields:
** Hostname: use FQDN
** System Status: production
** System Rack: Releng-AWS-VPC - Releng-AWS-USE1/USW2
** System Type: Virtual Server
** Operating System: Centos 6
** Allocated To: release
* Click "Create"

=== Common Setup ===
* Edit the new created entry
* Switch to "Key/Value Store"
* Click "Add adapter"
* Fill the following fields:
** Adapter Number (0-99): 0
** IP Address: IP address
** Mac Address: output of <tt>ip link show eth0 | tail -n -1 | awk '{print $2}'</tt> on the master
** Adapter Name (nic0-99 or mgmt0-99): nic0
** Host Name: fqdn
** Leave the rest as is
* Press "Create", then close box, to dismiss the adapter dialog
* Press "Save" to save the adapter (even though adapter may not show)

== Add masters to slavealloc ==
See [https://wiki.mozilla.org/ReleaseEngineering/How_To/Setup_Personal_Development_Master#Adding_your_master_to_slavealloc Adding your master to slavealloc]

== Add master's SSH key to known_hosts ==
See {{bug|889992}}
* Add the master's (scheduler and build only) ssh key to known_hosts in [http://hg.mozilla.org/build/puppet/file/38764f93e86b/modules/ssh/templates/known_hosts.erb puppet] (to make release-runner work):
:* dump the key by running the following:
<pre>
ssh-keyscan $master_name
</pre>
:* verify the change (ssh to masters as cltbld from bm81)
<pre>
ssh -i .ssh/release-runner buildbot-master80.bb.releng.usw2.mozilla.com
</pre>

== IT ==
# File a [https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org&component=Server Operations Server Operations] bug to:
#* add the master(s) to Nagios (See {{bug|1207411}} or {{bug|1253601}})
#** take note of the checks that are requested in the sample bug
#** you should be able to see the masters in here: [https://nagios.mozilla.org/releng-scl3/cgi-bin/status.cgi?hostgroup=use1-production-buildbot-masters&style=detail use1] and [https://nagios.mozilla.org/releng-scl3/cgi-bin/status.cgi?hostgroup=usw2-production-buildbot-masters&style=detail usw2]

== Lock a slave and let it take jobs ==
* Locked through slavealloc a slave to the newly setup master.
* Let it run for a couple of hours and check that the jobs worked well.
* Check also in the #ci channel for possible nagios checks going off (e.g. [[ReleaseEngineering/Queue_directories|Queue directories]] checks)

If no issues are found then go ahead and enable the master on slavealloc.

== Final Verification ==
The following steps should be performed to ensure the rest has worked okay (all steps should be run as user <tt>cltbld</tt>):

* SSH verification (and associated netflows).
ssh -i ~/.ssh/tbirdbld_dsa tbirdbld@upload.tbirdbld.productdelivery.prod.mozaws.net id
ssh -i ~/.ssh/trybld_dsa trybld@upload.trybld.productdelivery.prod.mozaws.net id
ssh -i ~/.ssh/ffxbld_rsa ffxbld@upload.ffxbld.productdelivery.prod.mozaws.net id
ssh -i ~/.ssh/ffxbld_rsa ffxbld@pvtbuilds2.dmz.scl3.mozilla.com id

* mySQL verification (associated netflows):
<pre>
nc -zv buildbot-rw-vip.db.scl3.mozilla.com 3306
</pre>

* ensure nagios checks are all green and notifications are enabled (aka not disabled), e.g:
https://nagios1.private.releng.scl3.mozilla.com/releng-scl3/cgi-bin/status.cgi?host=buildbot-master129.bb.releng.usw2.mozilla.com

Sheriffing/How To/Getting started as a sheriff

2018-05-14T08:30:48Z

Dlabici: Update BuildDuty to CiDuty. Renaming project

{{Sheriffing How To|Getting started as a sheriff}}
So you want to be a Sheriff? Here's how to start.

= Things You'll Need =

Not all of these have to be ready on Day One, but eventually you'll need the following.
* [[IRC|IRC account]], for communicating with developers, fellow sheriffs, and others.
* [[Sheriffing/How:To:SheriffingFromUnifiedRepos#Setting_up_the_repo|Clone of the Mercurial repositories]], for committing changes to the repositories.
* A properly configured mercurial setup. You can use <code>./mach mercurial-setup</code> from your clone of the repository to run through a setup wizard. I recommend setting up the qbackout extension to aid in easily backing out patches.
* LDAP account, for signing in to Treeherder and pushing commits to the source code repositories.
* [https://www.mozilla.org/en-US/firefox/new/ A modern browser], to access [[Sheriffing/How:To:Treeherder|Treeherder]], [https://bugzilla.mozilla.org Bugzilla], and other Mozilla web tools.

= The Sheriffs How-To =

A lot of knowledge about what sheriffs do is built up by day-to-day experiences on the job. We've created a number of wiki pages documenting a lot of this information, indexed [[Sheriffing/How:To|here]].

Some of the more helpful (and up to date) pages:
* [[Sheriffing/How:To:Treeherder]] - Explains some useful terminology, what each part of Treeherder's UI means/does, and how to use Treeherder to classify failures
* [[Sheriffing/How:To:SheriffingFromUnifiedRepos]] - This is useful, though some of the referenced repositories are no longer sheriffed, and it assumes you have knowledge of a lot of Mozilla/mercurial terms.

= Communication =

Sheriffs hang out in the #sheriffs channel on IRC - also it's useful to be in the following Channels:

* #developers - most of the developers are around here
* #ci - our counterparts of the build system “sheriffs”
* #releng - '''DECOMMISSIONED''' Use #ci for Release Engineering Channel
* #taskcluster - taskcluster team channel

A more-complete list of channels is listed [[IRC#Commonly_Used_Mozilla_IRC_Channels|here]].

Also join [https://mail.mozilla.org/listinfo/sheriffs the sheriffs mailing list].

= Sheriffed Repositories =

Mozilla has dozens of different source code repositories for various projects and releases, but sheriffs only need to care about a few of them:
* [https://treeherder.mozilla.org/#/jobs?repo=mozilla-central mozilla-central], the main development repository.
* [https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound mozilla-inbound], one of mozilla-central's integration branches where new commits initially land for testing.
* [https://treeherder.mozilla.org/#/jobs?repo=autoland autoland], another integration branch for mozilla-central.
* [https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta mozilla-beta], one of the release stabilization branches for code that has already gone through several weeks of testing.
* [https://treeherder.mozilla.org/#/jobs?repo=mozilla-release mozilla-release], the official release branch, where Firefox code goes to become part of an official build of Firefox.
* [https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr52 mozilla-esrXX], one or more branches set up to track patches for the current (and at times when the ESR releases overlap, the previous) version of [https://www.mozilla.org/en-US/firefox/organizations/ Firefox ESR]. (At the moment, this is mozilla-esr52.)

Once you're up to speed, it's best to leave a tab open to each of these repositories throughout your work day so you can track failures as they come in.

= Sheriffing Tasks =

Sheriffs do a lot of things throughout the day.
* Watch the sheriffed repositories via [[Sheriffing/How_To/Treeherder|Treeherder]] for failures.
** Classify/Star known intermittent failures.
** File tracking bugs for newly discovered intermittent failures.
** Resolve issues with code that has landed. This can be done with some of the following:
*** [[Sheriffing/How_To/Backouts|Backout]] the bad code.
*** Ensure that developers land followup patches to fix issues that are discovered.
** Monitor the repositories and tests for any issues related to infrastructure issues.
** Mark repositories as being unable to accept commits from developers (also called "[[Sheriffing/Deciding_To_Close_A_Tree|close the trees]]") via Treestatus, allowing you to fix issues without a bunch of new potentially-bad commits from landing in the interim.
* [[Sheriffing/How_To/Merges|Merge]] known-good commits from the mozilla-inbound and autoland repositories over to mozilla-central, and then back around to mozilla-inbound and autoland.
** Once the actual merge is performed, you need to run the Bugherder tool against the merge on mozilla-central to get the information properly updated in Bugzilla. There's a link to Bugherder in the per-push menu item in Treeherder.
* Landing approved patches that are marked as "checkin-needed" onto mozilla-inbound or autoland.
* Backporting approved patches (also called "[[Sheriffing/How_To/Uplifts|uplifting]]") from mozilla-central to one or more of the release stabilization branches (mozilla-beta, mozilla-release, mozilla-esr52, etc) to get the patch into users' hands sooner.

== Permissions ==
The following permission have to be set for the mentioned actions:
* commit access level 1 (scm_level_1 LDAP group):
** save job classifications
** retrigger jobs
** push to Try server (test patches, do beta simulations etc.)
** request to add selected jobs to a push
* commit access level 3 (scm_level_3 LDAP group):
** push to mozilla-inbound, mozilla-central etc. (not autoland or modify servo/, see below)
** land patches from within the browser with Review Board ("MozReview")
** request to run job on previous pushes where it might have been skipped ("Backfilling")
* scm_autoland LDAP group:
** push directly to autoland (needed for merges and some backouts)
* scm_servo_vendor LDAP group:
** allow to changes files in the servo/ folder (needed for merges and some backouts)
* vpn_sheriff LDAP group:
** terminate machines running tests
** request to run all missing jobs from menu for a push
** cancel all jobs on non-Try pushes
** manually request Nightlies (?)
* vpn_treestatus LDAP group:
** change the status of the trees (e.g. close on test failures or infrastructure issues)

ReleaseEngineering/TryServer

2018-05-14T08:25:57Z

Dlabici: Update BuildDuty to CiDuty. Renaming project

= Try Server =
The [https://treeherder.mozilla.org/#/jobs?repo=try try server] is an easy way to test a patch without actually checking the patch into the core repository. Your code will go through the same tests as a mozilla-central push, and you'll be able to download builds if you wish.

== Getting access to the Try Server ==
To use the try server, you'll need [http://www.mozilla.org/hacking/commit-access-policy/ Level 1 Commit Access]. You can learn more about Mozilla's commit access policies and start the process of signing up for an account here: [http://www.mozilla.org/hacking/committer/ Becoming a Mozilla Contributor]

After you have level 1 commit access, you'll need to do a bit of setup before you'll be able to push. See [https://developer.mozilla.org/en-US/docs/Mercurial/Using_Mercurial#How_do_I_check_stuff_in.3F this guide] for instructions. Assuming you only have level 1 access, you won't be able to push to mozilla-central, but you can replace 'hg.mozilla.org/mozilla-central' with 'hg.mozilla.org/try'.

== Configuration ==
Before using try, there is some recommended configuration you should set up. This can be accomplished by running:

$ ./mach mercurial-setup

Be sure to at least enable the [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/firefoxtree.html firefoxtree extension] and the push-to-try extension. The push-to-try extension is required if you wish to use the |mach try| command (see below). Firefoxtree will give you a handy 'try' alias you can use for pushing, and prevent you from accidentally pushing multiple heads. If for some reason you prefer not to use firefoxtree, you can set the same alias up manually in your hgrc:

[paths]
try = ssh://hg.mozilla.org/try

Note: Never pull the whole try repo. You'll end up with hundreds of heads containing everyone's half baked and broken pushes.

For more information, see [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html mercurial for mozillians].

== How to push to try ==
There are two ways to schedule jobs on try. You can either push like normal, and select which jobs you want using [https://wiki.mozilla.org/EngineeringProductivity/Projects/Treeherder treeherder], or you can specify a try syntax in your commit message to schedule jobs automatically.

=== Scheduling jobs with Treeherder ===

You can push to try with:

$ ./mach try empty

Once the push succeeds, you can open it in Treeherder and then sign in to schedule jobs manually.

Select the drop-down arrow at the top right of your push, and choose "Add new Jobs". It might take several seconds for the jobs to load ({{bug|1288028}}) :

https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042414/TH_dropdown.png

If it helps, you can use the filter box in the site header to restrict the list of runnable jobs being displayed. Click the job symbols you wish to schedule.

If you select a test job, the required build will automatically be scheduled:

https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042426/TH_with_jobs.png

Finally, click "Trigger New Jobs" near the top right of your push.

NOTE: An action task gets scheduled which will schedule all the tasks you chose.

=== Scheduling jobs with Try Syntax ===
If you know exactly what you want to run, you can use [[Build:TryChooser]] to select which jobs run directly in your commit message. Make sure the commit message containing try syntax is placed in your ''topmost commit''. The [http://trychooser.pub.build.mozilla.org/ TryChooser] web page can help you build a commit message for custom requests. It will generate both a syntax string as well as a mach command you can simply paste into your terminal.

==== Using mach ====
The recommended way to push with a try syntax, is to use |mach try|. It works with both mercurial and git, and you can use the [http://trychooser.pub.build.mozilla.org/ TryChooser] web page to generate the command. For example:

$ mach try -b o -p linux -u mochitest-1 -t none

The |mach try| command also has some advanced features. It can actually change how the job gets run within automation. For example, to run only mochitests under 'dom/indexedDB':

$ mach try -b o -p linux -u mochitests --and dom/indexedDB

For more information see:

$ mach try --help

Note: This method doesn't work well with mq.

==== Using the Trychooser Extension ====
You can also use the [https://bitbucket.org/sfink/trychooser mercurial trychooser extension] from sfink. It has some neat features, like a curses ui that lets you graphically choose a try syntax. Though {{bug|1197868}} tracks implementing this feature in |mach try|. Eventually trychooser will be deprecated in favor of |mach try|.

==== Using mq ====
The aforementioned tools may not play all that nicely with [https://www.mercurial-scm.org/wiki/MqExtension mercurial queues]. If you use mq, you can still push to try manually. First build your try syntax with the [http://trychooser.pub.build.mozilla.org/ TryChooser] web page. Then run:

$ hg qref --message "try: <your-computed-syntax-here>"
$ hg push -f try

== Viewing the results ==
You can see the results of your tryserver build in a number of ways:

* You'll get an email on a successful push with a link to treeherder for your revision as well as optional emails on any non-successful build/test/talos results (this setting can be adjusted using [[Build:TryChooser]] args for email notification)
* You can have the results of your try run posted to bug(s) automatically at the completion of the run using the --post-to-bugzilla flag in your try syntax (see: [[Build:TryChooser]] for examples)
* The link to treeherder will be printed on the command line.
* Look for your changeset on [https://treeherder.mozilla.org/#/jobs?repo=try Treeherder]. You can add '''&author=YOUR.EMAIL''' to only see your pushes.
* Download your completed builds from [http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/?C=M;O=D firefox/tryserver-builds on ftp.m.o].

== Using a custom mozconfig ==
The mozconfigs for recent mozilla-central clones are located in the browser/config/mozconfigs directory. Edit those as you please.

If you want to apply the same mozconfig changes to multiple platforms, you can edit <tt>build/mozconfig.common.override</tt> instead. This file is included at the end of each of the in-tree mozconfig files.

Android mozconfigs are in mobile/android/config/mozconfigs.

Note:
* TryServer purpose is to tell what will happen on Tinderbox, not to check every possible build option/configuration.
** Any non-standard feature is implicitly unsupported. You may try them, but don't complain if they break.

== Getting debug symbols ==
By default native debug symbols are not uploaded for Try server builds because of their size. If you want to debug your builds locally you must add <tt>MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS=1</tt> to the in-tree mozconfigs. You can do this for all platforms by importing [http://hg.mozilla.org/users/tmielczarek_mozilla.com/mq/raw-file/b44faf6cd177/enable-full-symbols this patch] into your mq and pushing it along with your changes to Try. This will cause a ...crashreporter-symbols-full.zip package to be uploaded to the builds directory for each platform built.

== Adding new jobs ==
It's possible to create new jobs (or modify existing ones) directly *in* your try push, provided you use taskcluster. Just edit the relevant configuration in testing/taskcluster. For more information on creating jobs, see the [http://docs.taskcluster.net/ taskcluster docs].

== Desktop l10n jobs ==
You can use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.

The jobs can be customized by modifying files prior to pushing:
* reducing the number of locales by limiting <tt>browser/locales/all-locales</tt> (eg top-locales like de fr ja ja-JP-mac ru zh-TW). Leaving a full list of locales is likely to hit a timeout on Mac and Windows
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#4 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario

The resulting builds are uploaded to the same sub-directory as en-US builds, eg try-linux/ for 32bit linux.

== Desktop l10n jobs (on Taskcluster) ==

To get Desktop l10n jobs on taskcluster on try, simply pass `-b o -p linux64-l10n,linux-l10n` with your try push. You'll also get these jobs automatically with `-p linux64,linux` if you are touching one of the files known to be heavily involved in l10n jobs. (Taskcluster support for other platforms is not yet available)

You can also use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.

The jobs can be customized by modifying files prior to pushing:
* reducing the number of locales by limiting <tt>browser/locales/all-locales</tt> (eg top-locales like de fr ja ja-JP-mac ru zh-TW).
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#4 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario

The resulting builds are uploaded as a task artifact, and are not yet signed.

== Android Single-Locale l10n jobs (on Taskcluster) ==

(NOTE: Only supported on Gecko 51a1 and above)

To get Android l10n jobs on taskcluster on try, simply pass `-p android-api-15-l10n` with your try push. You'll also get these jobs automatically with `-p android-api-15` if you are touching one of the files known to be heavily involved in l10n jobs.

You can also use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.

The jobs can be customized by modifying files prior to pushing:
* reducing the number of locales by limiting <tt>mobile/android/locales/all-locales</tt> (eg top-locales like de fr ru zh-TW).
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try_android-api-15.py#3 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario

The resulting builds are uploaded as a task artifact, and are not yet signed.

== Server Status ==
* Pending builds by revision are at https://secure.pub.build.mozilla.org/buildapi/pending
* In-progress builds by revision are are https://secure.pub.build.mozilla.org/buildapi/running

== Other Notes ==
* Finished builds will live in http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/<your_ldap_email>-<revision> for 14 days before deletion
* Treeherder data is purged after 4 months.
* If you have any problems please [https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org&component=Release%20Engineering&status_whiteboard=tryserver file a bug]

* Suggestions for the future can be made [[Build:TryServer:Suggestions|here]] or file a blocking bug against {{bug|try_enhancements}}

== Other Mozilla Try Servers ==
* [[ReleaseEngineering/ThunderbirdTryServer|Thunderbird Try Server]] for the comm-central repository

== Problem Diagnosis ==
=== Can not access try server ===
Test your account & configuration
* <code>ssh hg.mozilla.org</code>, response: "No Interactive shells allowed here!"
* <code>ssh hg.mozilla.org clone invalid_sandbox</code>, response: menu display and interactive prompting.

<div id="long_try_push"></div>
=== Pushes to try take a very long time ===
Note: if a fellow developer cancelled their try push, they have saddled you with the cost of rebuilding the cache. (See [http://dtor.com/halfire/2014/07/02/2014_06_try_server_update.html#caching caching] details.)

If you're experiencing excessive wait times (> 45min) pushing to try, please file a bug asking IT to reset the try repository using [https://bugzilla.mozilla.org/enter_bug.cgi?comment=please%20schedule%20a%20reset%20of%20the%20try%20repository%20ASAP%20with%20sheriffs%20%26%20releng.&component=WebOps%3A%20Source%20Control&op_sys=All&product=Infrastructure%20%26%20Operations&rep_platform=All&short_desc=Push%20to%20try%20taking%20XXX%20minutes this template] (please include specifics of your experience). They will coordinate with sheriffs and release engineering as needed.

=== Waiting for Lock ===
If you get a message similar to:
remote: waiting for lock on repository /repo/hg/mozilla/try/ held by 'hgssh1.dmz.scl3.mozilla.com:23974'
remote: abort: repository /repo/hg/mozilla/try/: timed out waiting for lock held by hgssh1.dmz.scl3.mozilla.com:30549
It means several developers are trying to push to try at the same time. In the case above, nothing appears to be wrong, as the PID changes between the messages.

=== Waiting for Lock multiple times with the same pid ===
Similar to the above case, but with the same pid when you retry over and over again.

Please retry your push. If you see messages indicating the same process has been pushing for more than 15 minutes, treat as [[#long_try_push|above]].

== CiDuty issues ==
=== How do I trigger additional talos/test runs for a given try build? ===
If your trychooser syntax included the tests you'd like more of, then select the job you want on Treeherder and use the + button. For test suites you didn't request originally you can use the "Add New Jobs" interface in Treeherder.

=== How do I cancel existing jobs? ===
For individual jobs, select the relevant one on Treeherder and use the cancel button. To cancel all jobs, use the menu arrow shown on the header row for each push, and then the "Cancel all" option.

=== TryChooser ===
See the [[ReleaseEngineering/TryChooser#Buildduty_Issues|TryChooser]] wiki page.

== See Also ==
* https://developer.mozilla.org/en/Creating_Mercurial_User_Repositories
* http://trychooser.pub.build.mozilla.org/
* https://wiki.mozilla.org/Sheriffing/How:To:Recommended_Try_Practices
* https://wiki.mozilla.org/Build:TryChooser
* https://wiki.mozilla.org/ReleaseEngineering:Autoland
* [http://build.mozilla.org/buildapi/self-serve Manage Submissions] [[https://build.mozilla.org/buildapi/self-serve/try try]]
* [https://catlee.github.io/highscores/highscores.html Scoreboard]

Sheriffing/How To/Escalate

2018-05-14T08:23:20Z

Dlabici: Update BuildDuty to CiDuty. Renaming project

{{Sheriffing How To|Escalating issues to release engineering and the MOC}}
= Guidelines on escalating outages =

== MOC ==
If you cannot pinpoint the affected system or team responsible, going directly to the MOC can save a lot of time, especially during off-hours. The MOC can be reached in irc #moc. The MOC knows how to contact and escalate any issue.

Other signs of a global issue include:
* you see nagios alerts for the issue in #sysadmins
* there's already a status (https://status.mozilla.org/) (but still let MOC know it's tree-impacting)

== Releng and Ciduty ==
If it's something RelEng can handle (or you also want them to be aware), here's the escalation path:
* ping *|ciduty in #ci
* use the !squirrel stalk word in #ci
* Follow the escalation route at https://wiki.mozilla.org/ReleaseEngineering#Contacting_Release_Engineering

== TaskCluster ==
* ping in #taskcluster (see https://wiki.mozilla.org/TaskCluster for coverage in any timezone)

ReleaseEngineering

2018-05-14T08:18:49Z

Dlabici: Update #releng to #ci

<div style="float: left;">__TOC__</div>
<div style="float: right; width: 60%;">
<div class="firstHeading toc" style="border: thin black solid; text-size: 80%; width: 50%;">
<h2>Quick Links For You</h2>
* [https://treeherder.mozilla.org/ Treeherder]
* [https://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering&component=Other File a new RelEng bug]
</div>
 
<div class="firstHeading toc" style="border: thin black solid; text-size: 80%; width: 50%;">
<h2>Quick Links For Us</h2>
* [[ReleaseEngineering/How To|How Tos]]
* [https://tools.taskcluster.net/ Taskcluster Tools]
* [https://mozilla-releng.net/ Releng Services]
* [https://ship-it.mozilla.org/ Ship It]
* [https://aus4-admin.mozilla.org/ Balrog]
</div>
</div>

 

= About Us =
We ship Firefox to the world!

Release Engineering does this by:

* Developing world-class, scalable, open, secure, and maintainable infrastructure for application builds and tests, on a diverse set of desktop and mobile platforms
* Building fast and reliable pipelines to transform source code into shipped products

= Contacting Release Engineering =
Our team is located on 3 continents and in 5 time zones. This allows us to provide better service, especially if you contact us using one of the following methods:
* To request a new service, please [https://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering file a bug]
* To ask a general question, ping in #ci on IRC, or email release@m.c
* To request status on an existing request, set the "needinfo" flag in the bug.
Our team works normal business hours for our respective time zones. If you need to escalate an issue that is both '''urgent''' ''and'' '''important''', use the following methods:
# Use #ci in IRC or email release@m.c, starting your message with 'URGENT:'. If no response in 15 minutes, then
# Be sure the bug severity is set to "<tt>blocker</tt>"
# Call one or more of the following people (nagios page preferred if you have access):
#* [https://phonebook.mozilla.org/#search/catlee Chris AtLee] Eastern Time
#* [https://phonebook.mozilla.org/#search/jlund Jordan Lund] Pacific Time
If you get voice mail, please leave your callback number.

= Contributing =
If you are interested in getting involved with Release Engineering, please see our [[ReleaseEngineering/Contribute|contributing page]]! We'd love to hear from you.

= Team =

Mozilla's Release Engineering:
* twitter: [https://twitter.com/MozRelEng @MozRelEng]
* blogs: [http://planet.mozilla.org/releng Planet Release Engineering]
* calendar: [https://www.google.com/calendar/embed?src=mozilla.com_2d32343333353036312d393737%40resource.calendar.google.com&ctz=America/Los_Angeles Google Calendar]
* mailing list: [https://groups.google.com/forum/#!forum/mozilla.release.engineering release-engineering@lists.mozilla.org]

{| border=1
| '''Name'''
| '''Profile'''
| '''Twitter'''
| '''Blog'''
|-
| Ben Hearsum
| [https://mozillians.org/u/bhearsum bhearsum]
| [https://twitter.com/bhearsum @bhearsum]
| [http://hearsum.ca/blog Ben's blog]
|-
| Justin Wood
| [https://mozillians.org/en-US/u/Callek/ Callek]
|
| [http://blog.drapostles.org/ "Callek's Blog"]
|-
| Chris AtLee
| [https://mozillians.org/u/catlee catlee]
| [https://twitter.com/chrisatlee @chrisatlee]
| [https://atlee.ca/blog/ chris' random ramblings]
|-
| Jordan Lund
| [https://mozillians.org/u/jlund jlund]
|
| [http://jordan-lund.ghost.io/ ghost blog]
|-
| Nick Thomas
| [https://mozillians.org/u/nthomas nthomas]
| [https://twitter.com/nthomasftang @nthomasftang]
| [http://ftangftang.wordpress.com/ F'tang F'tang Release wrangling at the bottom of the world]
|-
| Rail Aliiev
| [https://mozillians.org/u/rail rail]
| [https://twitter.com/railaliiev @railaliiev]
| [http://rail.merail.ca/ Rail's blog]
|-
| Aki Sasaki
| [https://mozillians.org/en-US/u/aki/ aki]
| [https://twitter.com/escapewindow @escapewindow]
| [http://escapewindow.dreamwidth.org/ escapewindow]
|-
| Mihai Tabara
| [https://mozillians.org/en-US/u/mtabara/ mtabara]
| [https://twitter.com/mihaitabara @mihaitabara]
|
|-
| Rok Garbas
|
| [https://twitter.com/garbas @garbas]
| [https://garbas.si/ Rok's blog]
|-
| Johan Lorenzo
| [https://mozillians.org/u/jlorenzo/ jlorenzo]
|
| [https://johanlorenzo.github.io/blog/ Johan's blog]
|-
| Simon Fraser
| [https://mozillians.org/en-US/u/sfraser/ sfraser]
|
|
|-
| Tom Prince
| [https://mozillians.org/en-US/u/tom.prince tprince]
|
|
|}

== Interns ==
* [[ReleaseEngineering/Interns|Interns]]

= Releng Offerings =

As mentioned above, Releng offer pipelines to transform source code into shipped products. The following sections break down how that happens. Each section can be understood as both isolated and combined parts.

== Initiating a Release ==

To start a release, [[Release Management]] submit a form in [[ReleaseEngineering/Applications/Ship It|Ship It]] with what product they would like to release, what revision to use, and a number of other inputs.

== Configuring & Scheduling ==

[[ReleaseEngineering/Applications/Ship It|Ship It]] then creates [https://docs.taskcluster.net/manual/tasks Taskcluster tasks] that are needed to take a builds and release them to users. These tasks make up the release graph that are derived from templates within [https://firefox-source-docs.mozilla.org/taskcluster/taskcluster/index.html taskgraph] and are submitted via [https://firefox-source-docs.mozilla.org/taskcluster/taskcluster/actions.html action tasks].

== Signing ==

The signing of build artifacts and binaries. Some features of this:

* signing is done through a client/server model
* integrated with Taskcluster thanks to [https://github.com/mozilla-releng/signingscript signingscript] in conjunction with [https://github.com/mozilla-releng/scriptworker scriptworker], a securely managed [https://docs.taskcluster.net/manual/tasks/workertypes Taskcluster worker type]. signingscript tasks are part of the task graph that is generated from [[ReleaseEngineering/Applications/Ship It|Ship It]]
* keys are generated with modern standards
* keys are secured by our infra and rotated regularly
* coming soon: self served signing through exposed API

== Uploading ==

As important release artifacts are created, they are then uploaded to [https://archive.mozilla.org/ archive.m.o] where they can served from and live permanently for historical or longstanding support reasons.

Like signing, uploading is integrated with Taskcluster thanks to [https://github.com/mozilla-releng/beetmoverscript beetmoverscript] in conjunction with [https://github.com/mozilla-releng/scriptworker scriptworker], a securely managed [https://docs.taskcluster.net/manual/tasks/workertypes Taskcluster worker type]. signingscript tasks are part of the task graph that is generated from [[ReleaseEngineering/Applications/Ship It|Ship It]]

== Install & Update Tests ==

Like product tests, releases likewise need tests. This ensures that initial downloads install correctly on all supported platforms and updates work as expected, all before actually releasing. Right now this is done through our Update and Final Verify tests. Which, like all other sections are part of the release graph.

TODO: link to documentation and source of these

== Shipping & Updates ==

Finally, we release the product via both initial downloads and as new updates.

For Desktop Firefox:
* Initial installs are served via updating the download page, mozilla.com ([[Mozilla.com/Bedrock|Bedrock]]) and updating [[Bouncer]], Mozilla's mirroring and redirection server.
* Updates are handled by our own application, [http://mozilla-balrog.readthedocs.io/en/latest/index.html Balrog]. Balrog is our server side component of the update system
** this is integrated with Taskcluster thanks to [https://github.com/mozilla-releng/balrogscript balrogscript] in conjunction with [https://github.com/mozilla-releng/scriptworker scriptworker], a securely managed [https://docs.taskcluster.net/manual/tasks/workertypes Taskcluster worker type]. balrogscript tasks are part of the task graph that is generated from [[ReleaseEngineering/Applications/Ship It|Ship It]]

For Android Products, including Firefox:
* releases are submitted to the Play Store
** this is integrated with Taskcluster thanks to [https://github.com/mozilla-releng/pushapkscript pushapkscript] in conjunction with [https://github.com/mozilla-releng/scriptworker scriptworker], a securely managed [https://docs.taskcluster.net/manual/tasks/workertypes Taskcluster worker type]. pushapkscript tasks are part of the task graph that is generated from [[ReleaseEngineering/Applications/Ship It|Ship It]]

= Development =

== Best Practices ==
* [[ReleaseEngineering/Python Standards|Python Standards]] - Our Python coding standards
* [[ReleaseEngineering/Development Best Practices|Development Best Practices]] - read this when you're starting something new
* [[ReleaseEngineering/Deployment|Deployment]] - How we deploy our tools, and how do we update them.
* [[ReleaseEngineering/Wiki Guidelines|Wiki Guidelines]] - how to wiki garden for fun and profit

== Bugzilla ==
* [[ReleaseEngineering/Bugzilla/Flags|Flags]]
* [[ReleaseEngineering/Bugzilla/Whiteboard|Whiteboard]]
* [[ReleaseEngineering/Bugzilla/Triage|Triage]]

= Operations =

Release Engineering has historically been involved with the Build and Release pipeline of a product. As part of that, we have some responsibility to help ensure that the build (continuous integration) and release (deployment) pipeline infrastructure is in good running health. We have two functional groups that monitor and respond to any operational issues that come up.

== Buildduty ==

Buildduty provides support for Firefox's build and test infrastructure

The role of Buildduty has grown beyond Release Engineering's scope. There is a dedicated team of 6 people, providing 24/7 support of Mozilla's build and test infrastructure.

See the [[Buildduty|Buildduty home page]] and [[Buildduty/manifesto|manifesto]] for more information.

== Releaseduty ==

Releaseduty provides support for Firefox's release infrastructure and is currently handled by Releng

See the [[ReleaseEngineering/Releaseduty home page]] for more information.

= Releng Applications =
* [[ReleaseEngineering/Applications|Applications]] - Various applications and services that RelEng provides
* TODO update application list with: Services, Scriptworker, ChainOfTrust

= Mozilla Applications =

* [[Taskcluster]] - our task execution tool for providing continuous integration and release pipelines
* [[EngineeringProductivity/Projects/Treeherder|Treeherder]] - aggregates all the build, test, and release tasks as a reporting dashboard
* [[ReleaseEngineering/PuppetAgain|PuppetAgain]] - Release Operations tool and puppet configs for configurating build and release machines. Both for workers and hosting applications
* TODO - add Release Operations replacement for Slaveapi: Controller

= External Third Party Dependencies =

* AWS
* Heroku
* Papertrail
* Nagios
* Sentry

= Deprecated =

below pages should be deleted or be given a "warning" outdated box where appropriate.

== Buildbot ==
* Master
** [[ReleaseEngineering/Master Naming|Master Naming]]
** [[ReleaseEngineering/Master Setup|Master Setup]]
** [[ReleaseEngineering/Upgrading Buildbot|Upgrading Buildbot]]
** [[ReleaseEngineering/Preproduction|Preproduction]]
** [https://intranet.mozilla.org/RelEngWiki/index.php/Masters Masters] (authentication required)
** [[ReleaseEngineering/Queue directories|Queue directories]]
* Slave
** [[ReleaseEngineering/Buildslave Versions|Buildslave Versions]]
** [[ReleaseEngineering/Buildslave Startup Process|Buildslave Startup Process]]
** [[ReleaseEngineering/Slave_Naming|Slave naming]]
* Development
** [[ReleaseEngineering/Virtualenv|Virtualenv]] - How to set up and use python virtual environments
** [[ReleaseEngineering:TestingTechniques|Methods for testing your changes]]
** [[ReleaseEngineering/Buildbot Best Practices|Buildbot Best Practices]]
** [[ReleaseEngineering/Landing Buildbot Master Changes|Landing Buildbot Master Changes]]
* References
** [[ReleaseEngineering/Buildbot_Database_Schemas|Buildbot Database Schemas]]

== VCS Sync ==
* [[ReleaseEngineering/VCSSync/HowTo|HowTo]]
* [[ReleaseEngineering/VCSSync/History|History]]

== Releases ==
* [[Releases/RelEngChecklist | Release Checklist]] and [[Releases/BuildNotesIndex | Build Notes]]
* [[:Category:Releases:Post_Mortem_Meetings | Releng Post Mortem Meetings]]
* [[ReleaseEngineering/Releaseduty | What is releaseduty?]]
* [https://github.com/mozilla/releasewarrior Release Promotion]
* [[UpdateGeneration|Update Generation]]
** [[ReleaseEngineering/PatcherTags|Patcher tags for release updates]]

== Platform Support ==
* [[ReferencePlatforms|Reference Platforms]]
* [[ReleaseEngineering/Applications#slavealloc|Slave Allocator]]
* [[ReleaseEngineering/Official Platform Support Checklist|Official Platform Support Checklist]]

== Tryserver ==
* [[ReleaseEngineering/TryserverAsBranch|Tryserver As Branch]]
* [[ReleaseEngineering/TryServer | TryServer]]
* [[ReleaseEngineering/TryChooser | TryChooser]]

== Testing ==
* [[ReleaseEngineering:IntermittentErrors | Intermittent Errors]]
* [[ReleaseEngineering/RequestingASlave | Requesting A Slave]]
* [[ReleaseEngineering/Testing a new Signing Server | Testing a new Signing Server]]

= Subpages =
* [[ReleaseEngineering/Presentations|Presentations]]
* [[ReleaseEngineering/How Tos|How Tos]]
* [http://moz-releng-docs.readthedocs.org/en/latest/ Releng Docs] '''''outdated'''''
* [[Special:PrefixIndex/{{FULLPAGENAME}}/]] - all subpages of [[{{FULLPAGENAME}}]] in the wiki
* [[ReleaseEngineering/Buildbot Migration|Buildbot Migration]] - '''''outdated''''' What builds are produced where? Buildbot, TaskCluster, or other?
* [[ReleaseEngineering/Obsolete Pages]]
* [[ReleaseEngineering/Day_1_Checklist|day 1 checklist]]
* [[ReleaseEngineering/Day_-1_Checklist|day -1 checklist]]

ReleaseEngineering/TryChooser

2018-05-14T08:15:25Z

Dlabici: Update BuildDuty to CiDuty. Renaming project

[https://mozilla-releng.net/trychooser/ TryChooser] lets you select specific jobs you want to run on [https://wiki.mozilla.org/Build:TryServer tryserver]. You use TryChooser by putting a <tt>try:</tt> block at the end of the topmost commit message when you push to try.
==Syntax Builder==
'''To simplify using the TryChooser, you can use the [https://mozilla-releng.net/trychooser/ TryChooser Syntax Builder] website'''
or the [http://hg.mozilla.org/users/pbiggar_mozilla.com/trychooser/file/tip mercurial extension] (which also functions as a command-line application, but is not maintained by RelEng).

==Defaults==

If you don't specify an option or if you give an invalid value for an option, TryChooser will use its default for that option:

try: --build do
--platform all
--unittests all
--talos none

==Examples==
; <tt>try: -b do -u crashtest</tt>
: Debug and optimized builds, all available platforms, only crashtests, no talos:

; <tt>try: -b d -p linux -u none</tt>
: Linux debug build, no unittests, no talos:

; <tt>try: -b do -p all -u all -t none</tt>
: Run all platforms, all tests, no talos. This is a lot of tests, which will use a lot of resources! Unless absolutely necessary, narrow the list of platforms and/or the list of unit tests to those you really need.:

; <tt>try: -b do -u all -t chrome,nochrome</tt>
: Debug and optimized builds for all platforms, all unittests, select talos:

; <tt>try: -b do -u all -t chrome,nochrome mozharness: --spsProfile</tt>
: Same as above but run talos in profiling mode so that it produces zip files with profiles that can be opened in Cleopatra. See [https://wiki.mozilla.org/Buildbot/Talos/Profiling Talos Profiling].

; <tt>try: -b o -u jsreftest,crashtest,mochitest-1 -t tp4,scroll</tt>
: Opt builds on all available platforms, select unittests, select talos

; <tt>try: -b o -u all -n --post-to-bugzilla bug 456234</tt>
: Opt builds on all available platforms, all unittests, no emails, results posted to bug 456234

; <tt>try: -b d -p macosx64 -u all[10.7]</tt>
: Debug builds for 64-bit OSX, run all tests only on OSX 10.7 machines

; <tt>try: -b d -p macosx64 -u all[-10.7]</tt>
: Debug builds for 64-bit OSX, run all tests but not on all OSX test platforms except OSX 10.7

; <tt>try: -b do -p all -u all[x64]</tt>
: "T push" -- good replacement for <tt>-p all -u all</tt>. Builds on all platforms, runs all tests on linux64.

; <tt>try: -b do -p all -u all[10.8]</tt>
: Same as above, but run the tests on OSX 10.8 instead

You may precede the <tt>try:</tt> section with other text:
bug 12345 attempt 2, try: -b d -p linux64 -t jsreftest

==Workflow==

===Without MQ===

If you're not using Mercurial Queues, you can use TryChooser by including a "try:" string in the message of the commit you push to try:

$ (edit your code)
$ hg commit -m "Bug XXXXXX - Change foo to bar. try: -b d -p all"
$ hg push -f <nowiki>ssh://hg.mozilla.org/try</nowiki>

===With MQ===

'''Note:''' If you are running Mercurial 2.1 or higher, see [[Build:TryServer#hg_phases|this note]] about phases. ({{bug|725362}})

If you're using Mercurial Queues, you can use TryChooser by adding an empty patch with a "try:" string to the top of your queue:

hg qgoto your-patch
hg qnew -m "try: <Insert options here>" try
hg push -f -rtip <nowiki>ssh://hg.mozilla.org/try</nowiki>

After pushing, you can remove the empty patch:

hg qpop
hg qremove try

==FAQ==
===What if I want PGO for my build===
Until {{bug|691673}} is resolved, you'll have to do a small hack if you want PGO to work on your try push. Add
mk_add_options MOZ_PGO=1
to <tt>[https://dxr.mozilla.org/mozilla-central/source/build/mozconfig.common.override build/mozconfig.common.override]</tt>.

===What if I don't put "try:" in my commit message?===
If you don't give a "try:" string in your commit message, you'll get an error message on attempts to push to try as per {{bug|649402}} where try syntax use was made mandatory.

===What if make a syntax error in my commit message?===
If you pass an invalid parameter, TryChooser will ignore it and substitute the default in its place. For instance, if you specify <tt>try: -b f -u crashtest</tt>, you'll get both debug and optimized builds, since "f" is not a recognized argument. You'll only get crashtest run on those builds, however.

===Where's the full list of unit tests and talos tests I can request?===

Try should have the same test/talos suites available to it as mozilla-central does so you can refer to [https://treeherder.mozilla.org Treeherder] for complete listings.

To match up the Talos suite name to the tests it runs you can go [http://hg.mozilla.org/build/buildbot-configs/file/701b6ccd8e55/mozilla-tests/config.py#l19 here]. (eg: 'chrome' runs ts:tdhtml:twinopen:tsspider)

If you need help building the syntax line you can also go to the [http://trychooser.pub.build.mozilla.org/ TryChooser Syntax Builder] page.

===What do I do if I need to add a job after I push?===
Use try extender on Treeherder. To be announced in November.

===What do I do if I need to remove a job after I push?===
Check out [https://build.mozilla.org/buildapi/self-serve Self Serve!]. With Self-Serve you can cancel pending builds, re-build/run completed builds/test/talos runs.

===What do I do if I used the wrong syntax and I'm missing a job?===
With the [[ReleaseEngineering/How_To/Trigger_arbitrary_jobs|arbitrary jobs API]] you can trigger jobs that would have not been triggered according to your Try syntax.
You can even specify a different build or tests.zip.

===Where are my builds/test/talos?===
If you think Treeherder is missing results or it feels like it's been a while since you kicked the build off and you're getting antsy for results. Here are some things you can look into before pinging the person on ciduty:
* Did the build compile successfully? If it didn't, you won't get unittests or talos results.
**Look for your directory in [http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/ the builds FTP server] and see what has been uploaded. (You can also find here the email-changeset string that you'll need if you end up pinging releng.)
* Check [http://build.mozilla.org/buildapi/running/ Running] and [http://build.mozilla.org/buildapi/pending/ Pending] to view your build's status, since these are where Treeherder gets the job data from, so are closer to the source of truth.

===What if I want to run a sub-set of tests for a job?===
'''Note: This is currently broken, see [https://bugzilla.mozilla.org/show_bug.cgi?id=1336606 bug 1336606]'''

Tests can be tagged via the *.ini files, e.g.

<pre>
[default]
tags = search
</pre>

or

<pre>
[default]

[browser_426329.js]
tags = search
</pre>

You can then run just the required tag by appending `--tag search` onto your try command, e.g.

<pre>
./mach try -b d -p linux64 -u mochitest-e10s-bc -t none --artifact --tag search
</pre>

==Parsing details==

The parser for TryChooser is [http://hg.mozilla.org/build/buildbotcustom/file/tip/try_parser.py here].

The basic algorithm is:
* The commit message is split on 'try:'.
* Whatever follows after 'try:' is passed to try_parser as a string.
* The string is processed into a list of args handed to parse_known_args (thus keeping anything that is not a recognized argument from causing errors).

The email preferences are parsed out in the [http://hg.mozilla.org/build/buildbotcustom/file/tip/bin/try_mailer.py try_mailer] script, and the --post-to-bugzilla flag is currently a work in progress but will land in the RelEng [http://hg.mozilla.org/build/tools/scripts/autoland tools] repo as part of the [https://wiki.mozilla.org/BugzillaAutoLanding Bugzilla Autolanding] project.

==CiDuty Issues==
=== How to Deploy TryChooser changes ===
[[ReleaseEngineering/How_To/Update_TryChooser]]

=== Jobs not scheduled at all? ===
Recreate the comment of their change with https://mozilla-releng.net/trychooser/ and compare it to make sure is correct.

Then do a sendchange and tail the scheduler master:
<pre>
buildbot sendchange --master buildbot-master10:9301 --revision 923103d5a656 \
--branch try --username mpalmgren@mozilla.com \
--comments "try: -b d -p linux -u all" doit
</pre>

* If tryserver was just reset verify that [[ReleaseEngineering/How_To/Reset_the_Try_Server#Try_Hg_Poller_state|the scheduler has been reset]]

=== Using the TryChooser to submit build/test requests ===
CiDuty can also use the same TryChooser syntax as developers use to (re)submit build and testing requests. Here is an example:

<pre>
buildbot sendchange --master buildbot-master10:9301 --revision 923103d5a656 \
--branch try --username mpalmgren@mozilla.com \
--comments "try: -b d -p linux -u all" doit
</pre>

=== Unable to schedule builds due to missing 'requests' package ===
As with pip 8.0, the '--download-cache' option has been removed (it was already deprecated since 6.0). The fix for this issue was removing the argument in mozharness config (see https://hg.mozilla.org/try/rev/1c396281c01b).
In order to build with a base revision prior to '1c396281c01b', you will need to rebase your patch to use a newer version.

== Source code ==

The trychooser syntax builder lives in the buildtools repository, under the subdirectory "trychooser":

* Mercurial: https://hg.mozilla.org/build/tools/
* Git: https://github.com/mozilla/build-tools/

ReleaseEngineering/Slave Health

2018-05-14T06:42:56Z

Dlabici: Update BuildDuty to CiDuty. Renaming project

= What is slave health? =
Slave health is a web tool used by releng, specifically ciduty, to monitor and affect the health of the various slave pools.

https://secure.pub.build.mozilla.org/builddata/reports/slave_health/index.html

= Deployment =
{{Release Engineering Deployment|slave health}}
Slave health is deployed on cruncher under <code>/home/buildduty/slave_health</code>

There are currently two cronjobs on cruncher that correspond to slave health:

#m h d mth dow
*/15 * * * * /home/buildduty/slave_health/slave_health_cron.sh
15 0 * * * source /home/buildduty/buildduty/bin/activate && \
python /home/buildduty/slave_health/slave_health/scripts/generate_chart_objects.py

<code>/home/buildduty/slave_health/slave_health_cron.sh</code> should be copied from <code>/home/buildduty/slave_health/slave_health/scripts</code> on when the script is updated in version control.

= CiDuty Report =

We also stick the [https://secure.pub.build.mozilla.org/builddata/reports/slave_health/buildduty_report.html CiDuty report] under slave health, however its not generated from the slave health repo.

It is run from cron with:
#m h d mth dow
1 5-18 * * 1-5 /home/buildduty/buildduty/buildduty_report.sh

The related files are in braindump at:
* [http://hg.mozilla.org/build/braindump/file/default/reports/buildduty_report.sh buildduty_report.sh]
* [http://hg.mozilla.org/build/braindump/file/default/reports/buildduty_report.py buildduty_report.py]

ReleaseEngineering:Buildduty:Downtimes

2018-05-14T06:28:37Z

Dlabici: Update BuildDuty to CiDuty. Renaming project

==How do I schedule a downtime?==

Whenever RelEng/IT/WebDev wants a tree-closing downtime for work on any systems, they should contact the CiDuty person of the day in #ci, by email or preferably by ''putting exact wording they'd like in the downtime notice as it relates to their bug, and then nominating the bug using the "needs-treeclosure?" flag''.
* NOTE: '''All ServerOps components now have "needs-treeclosure" flag enabled''' (previously only some ServerOps components had the flag).
* NOTE: '''ServerOps "infra only" bugs are not visible to CiDuty by default.''' If the bug can be changed to "Moco specific", then it will be visible to all of RelEng, and show up in all the queries. If the bug needs to remain "infra only", then the bug assignee needs to file, and closely track all changes to, a new separate bug that CiDuty can use for scheduling.

When planning a downtime, CiDuty should consider:
* the urgency of the work
* what other work, if any, can be safely done in the same downtime

...and propose a time that:
* is low impact to developers
* is low impact to releases (assessed by asking a list of pre-approvers).
* is not scheduled near any other downtimes / planned outages
* fits the schedule of the person who understands the work
* has CiDuty available to handle tree closing/opening and field questions in #developers (other channels as needed)
* '''gives >1 day notice to newsgroups if at all possible'''

=== Preparing for the downtime ===
* review all bugs nominated with "needs-treeclosure?"
* for bugs approved to land in the downtime, CiDuty will:
** verify that the bug is assigned to the person who will actually be doing the work in the downtime
** set the "needs-treeclosure+" flag in the bugs
** set the whiteboard field with the proposed time/date.
** write a downtime notice that includes:
*** bug# and description for each item.
*** for security sensitive work, be vague, but still include the bug# and vague description. (this reduces confusion about whether an item of work is in/out of a downtime).
*** boilerplate text about the timing of the downtime and how it will affect developers. I've included the boilerplate below:
====Downtime Boilerplate====
<pre>
* When can I push?

We pride ourselves on having the self-serve tools[1] to make
it easier to recover from build failures caused by a
downtime. However, we understand that some developers may
not be available to re-trigger failed runs after a downtime
is done, or may not want to incur that hassle. Some would
rather push early enough to receive all their results before
the downtime starts, others would rather wait until the
downtime is complete.

If you have LDAP access to Mozilla servers, which if you're
landing code you likely do, you can check the current
end-to-end times for your chosen development branch[2].
Compare your end-to-end time with the declared start of the
downtime in order to make an informed decision about whether
you really want to push _now_.

1. https://build.mozilla.org/buildapi/self-serve
2. https://build.mozilla.org/buildapi/reports/endtoend
</pre>

==== Who do I notify, and when? ====
# etherpad/email everyone who will be working in the downtime, and ask to approve the draft downtime notice
# '''BEFORE''' posting to the newsgroups, send draft copy of the downtime notice to the list of pre-approvers. Ask for any objections/questions. As of 2012/04/25, the current list of pre-approvers is:
#* Bob Moss <bmoss@m.c>, Chris Hofmann <chofmann@m.c>, Alex Keybl <akeybl@m.c>, Damon Sicore <dsicore@m.c>, Johnathan Nightingale <johnath@m.c>, JP Rosevear <jpr@m.c>, Sheila Mooney <smooney@m.c>
#* cc release@m.c and infra-all@m.c
# post the downtime notice to the '''dev.planning''' & '''dev.tree-management''' newsgroups, and send a copy of the notice to '''all@m.c'''.
# err on the side of over-communication, i.e. play it safe: if you think a group will be impacted by a downtime and they are not included in the lists above, contact them.
# All of the above notifications should go out '''*at least 24 hours*''' before the planned downtime.

===Running the downtime===
# Be sure to check dev.planning, dev.tree-management newsgroups and planet.m.o regularly to ensure nothing comes up in response that would require changes to, or outright cancelling of, the downtime. A standout example here would be a chem-spill release.
# Before starting the downtime, CiDuty will notify sheriff in #developers, and close trees.

=== After the downtime ===
CiDuty will:
# reopen the trees
# verify with sheriff that trees open, all ok?
# update the bugs with status (landed-and-stuck", rolled-back) - it's possible this will be done first by the person who attempted the downtime. However, if not, CiDuty should update the bugs for the record.
# send "TREE OPEN" newsgroup post/email, listing what did / didn't get done

== How do I coordinate downtimes with IT? ==
Some IT maintenance requires tree closure. Maintaining or rebooting any of the following systems needs coordinated downtimes with RelEng and IT. It usually also needs advance notice of Tree Closures posted to the [usual sources:

* build.m.o (clobberer, build data, tryserver symbols)
* cruncher.build.m.o (graphs dashboard, dumping build data, dashboards, pulse)
* cvs.mozilla.org (talos)
* hg.mozilla.org (firefox source repos, build repos)
* tinderbox.mozilla.org (central reporting point for all builds)
* ftp.mozilla.org (release updates on beta channel)
* stage.mozilla.org (publishing builds, downloading builds for talos/unittest)
* graphs.mozilla.org (performance tracking)
* buildbot-rw-vip.db.scl3.mozilla.com (buildbot scheduler db, graphserver?)
* buildbot-ro-vip.db.scl3.mozilla.com (used by cruncher)
* mail.build.mozilla.org - currently dm-mail03.m.o (build mail to tinderbox)
* aus3-staging.mozilla.org (update snippets)
* nm-ops03.build.mozilla.org (releng VMs)
* nagios.mozilla.org (monitoring)
* relengweb1.dmz.scl3.mozilla.com (replacement for build.m.o)
* tbpl.mozilla.org (build status)

For any questions, or if you're not sure about a particular server, please check with '''|ciduty''' in #ci.

If possible, consolidate RelEng and IT downtimes that need tree closures to avoid the disruption of having two tree closures soon after each other. This is "nice to do", not a "requirement"; if it reduces risk by doing two separate downtimes, that's fine!

Buildduty/day 1 checklist

2018-05-14T06:23:58Z

Dlabici: Update BuildDuty to CiDuty. Renaming project

__TOC__

Welcome to CiDuty!

This page is meant to serve as a resource for new CiDuty people getting up to speed and ensuring they have access to the appropriate systems that they are expected to use.

= CiDuty and the Manifesto =

* Before you start it would be a good idea to make sure you know what CiDuty is all about. Please have a read through of [[ReleaseEngineering/Buildduty_manifesto|the manifesto]]

= Access =

== SSO ==

Generally, we rely on [https://auth0.com/ auth0] across Mozilla for authentication and [https://mana.mozilla.org/wiki/display/SYSADMIN/LDAP+Architecture LDAP] for authorization. Once given LDAP and you have created a permanent password, you can use that to login to the [https://sso.mozilla.com SSO portal]. From SSO, you should have links to various services from email, irc, calendar, slack, mana, etc. More on each of those later on this page

== login.mozilla.com ==

[https://login.mozilla.com/ login.mozilla.com] is where you can change a number of authentication/authorization access bits that you have control over. Each todo in this section assumes you have access to this page.

=== LDAP password reset ===

If you were given a temporary ldap password or you haven't created your own password yet, you should do this now.

=== SSH ===

Upload your public ssh key. It is a good idea to generate a separate ssh keypair from your personal one or any other that you have created in the past and use that explicitly for CiDuty and upload that. Follow this [[Security/Guidelines/OpenSSH#OpenSSH_client|SSH guidelines doc]] on how to generate, configure, and use your ssh key.

note: example ssh config for accessing our systems given below in Jumphost section

=== PGP ===

We use pgp keys to share private information, secrets, and verify that the source came from someone we trust. Generate a keypair for this and upload your public key so others can find it. It would be really good if you could have other people sign your key, adding more trust that this key really belongs to you.

You can use the the [https://mana.mozilla.org/wiki/display/SD/Generating+a+GPG+Public+Key pgp quickstart guide on mana] or you can use the The [https://www.gnupg.org/gph/en/manual.html GNU Privacy Handbook] for reference.

=== VPN ===

Many of our systems are behind a private network in addition to auth0. Follow the prompts to generate and download an openVPN certificate that you can use to import to your vpn client.

See the instructions on how to [https://mana.mozilla.org/wiki/display/SD/VPN install and configure your VPN client] and help choosing the right client for your platform.

note: macOS and Windows users should use [https://www.sparklabs.com/viscosity/ Viscosity]. This application comes with a free 30 day trial. During your trial, your manager can help you create a ServiceNow ticket to get a Viscosity full license.

=== MFA ===

This MFA account is specific to login.mozilla.com and is used for LDAP/auth0 based logins. Follow the instructions to download the Duo Mobile app and create a Mozilla account.

note: later on in this page we will create more MFA accounts for various systems like Github and accessing our Jumphost

== Jumphost ==

To access any of Release Engineering, Taskcluster, and Release Operations hosts directly, you will need to go through VPN -> a Jumphost machine -> Separate MFA -> your target host.

To do that, you and your manager will need to file a ticket against Release Operations and have them send you an invite to add an MFA account on your Duo App.

Then once you have your Jumphost MFA setup correctly, you will need to have your ssh config to correctly route through the jumphost before trying the target host you want.

example ssh config:
<source lang="ruby">
# Ensure KnownHosts are unreadable if leaked - it is otherwise easier to know which hosts your keys have access to.
HashKnownHosts yes
# Host keys the client accepts - order here is honored by OpenSSH
HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256

Host hg.mozilla.org git.mozilla.org
User USERNAME@mozilla.com
Compression yes
ServerAliveInterval 300

Host *.mozilla.com
User USERNAME
IdentityFile ~/.ssh/id_rsa_mozilla_2017-05-12
Compression yes
ServerAliveInterval 300

Host *.build.mozilla.org
Compression yes
User cltbld
ServerAliveInterval 300

Host rejh?.srv.releng.????.mozilla.com
ControlMaster auto
ControlPath ~/.ssh/ssh-%C
ControlPersist 10m
ForwardAgent no

Host *.releng.mdc1.mozilla.com !rejh?.srv.releng.mdc1.mozilla.com !*.private.releng.????.mozilla.com
ProxyJump rejh1.srv.releng.mdc1.mozilla.com

Host *.releng.us??.mozilla.com *.releng.scl3.mozilla.com !rejh?.srv.releng.????.mozilla.com !*.private.releng.scl3.mozilla.com
ProxyJump rejh1.srv.releng.scl3.mozilla.com
</source>
== CiDuty LDAP groups ==

You may have access to the [https://ldapadmin1.private.scl3.mozilla.com/manage/ ldap admin page] and see your own groups that you have on your record. This page is behind vpn and auth0.

Although you can read your current groups, you will not be able to modify them. To extend with CiDuty groups that you need. You and your manager will need to file a ticket for them under "MOC: Service Requests"

example ldap groups they may have by default:
cn=corp-vpn,ou=groups,dc=mozilla
cn=IntranetWiki,ou=groups,dc=mozilla
cn=irccloud,ou=groups,dc=mozilla
cn=mfa,ou=groups,dc=mozilla
cn=phonebook_access,ou=groups,dc=mozilla
cn=team_moco,ou=groups,dc=mozilla
cn=vpn_corp,ou=groups,dc=mozilla
cn=vpn_default,ou=groups,dc=mozilla

example ldap groups you may need to file for and request added (example, Bug 1434168):
cn=releng,ou=groups,dc=mozilla
cn=RelEngWiki,ou=groups,dc=mozilla
cn=vpn_releng,ou=groups,dc=mozilla
cn=vpn_releng_loan,ou=groups,dc=mozilla
cn=vpn_relengwiki,ou=groups,dc=mozilla
cn=vpn_tooltooleditor,ou=groups,dc=mozilla
cn=inventory,ou=groups,dc=mozilla
cn=inventory_build,ou=groups,dc=mozilla
cn=vpn_inventory,ou=groups,dc=mozilla
cn=nagiosadmin,ou=groups,dc=mozilla
cn=GraphsAdmin,ou=groups,dc=mozilla
cn=active_scm_level_1,ou=groups,dc=mozilla
cn=all_scm_level_1,ou=groups,dc=mozilla
cn=vpn_genericrhel6,ou=groups,dc=mozilla

= Communications =

== Mail ==
Mozilla mail is handled by [https://mail.google.com/ Gmail] now.

Have your manager subscribe you to this list if you are not already.

=== Mailing lists ===

Need permission:
* [https://groups.google.com/a/mozilla.com/forum/?hl=en#!forum/buildduty buildduty@mozilla.com] - this is our team email. Ask owner (e.g. jlund) for access
* [https://groups.google.com/a/mozilla.com/forum/?hl=en#!forum/releng-puppet-mail Puppet Mail] (warning: you will want to filter this as it can send a lot of mail)

You'll also need to manually subscribe to the following public lists:
* [https://lists.mozilla.org/listinfo/release-engineering release-engineering] public mailing list
* [https://lists.mozilla.org/listinfo/dev-planning mozilla.dev.planning]
* [https://lists.mozilla.org/listinfo/dev-tree-management mozilla.dev.tree-management]
* https://mail.mozilla.org/listinfo/taskcluster-announce - announcements of events, major changes (low volume, no discussion)
* https://lists.mozilla.org/listinfo/tools-taskcluster - general taskcluster discussion
* [https://groups.google.com/a/mozilla.com/forum/#!forum/firefox-ci firefox-ci] mailing list
* TODO public relops

These are available as [news://news.mozilla.org newsgroups], google groups, and [https://lists.mozilla.org/listinfo Mailman lists]

== Calendar ==
Like mail, we now use [https://www.google.com/calendar/ Google calendar].

You'll want to subscribe to the following public calendars:
* [https://calendar.google.com/calendar/embed?src=mozilla.com_2d32343333353036312d393737%40resource.calendar.google.com Release Engineering - Public]
* [https://www.google.com/calendar/feeds/mozilla.com_toi1svbfjd878aslutkgj32dco%40group.calendar.google.com/public/basic Releng PTO]
* TODO public taskcluster
* TODO public relops

== Bugzilla ==

Almost everything at Mozilla goes through Bugzilla. [https://bugzilla.mozilla.org/createaccount.cgi Create a Bugzilla account] if you have not already. You should setup MFA and can reuse your LDAP based MFA account.

You'll need a few tweaks to your account to get access to everything releng-related:
* Use your LDAP email if you would like. You can use a personal one or bugzilla specific one if you prefer to filter that way
* Add privileges for bugzilla group "build" (Mozilla Build Team) (Can be done by catlee or bugzilla admin.)
* Add your irc nickname & ldap username as "aliases" for your account
** log into bugzilla & follow links "Preferences" -> "Account Information"
** append the aliases, with a leading ':' and enclosed in brackets ('[]') to the "Real Name" field
** e.g.: "<tt>Chris AtLee [:catlee]</tt>"
* [https://bugzilla.mozilla.org/page.cgi?id=quicksearch.html QuickSearch help]

== Vidyo ==

Our primary two way video meeting platform is Vidyo. Basic usage instructions are [https://mana.mozilla.org/wiki/display/SD/Vidyo+Desktop+3 here]. Especially if you are running linux, it is highly recommended that you install the client and make test calls prior to any meeting.

Special [https://mana.mozilla.org/wiki/display/~gene@mozilla.com/How+to+get+Vidyo+working+on+Ubuntu+16.04+and+newer Ubuntu instructions] that may help

Add to your contact list the CiDuty room

== IRC ==

Historically, IRC is the primary place for chat based communication. Many people use the [https://mana.mozilla.org/wiki/display/SD/IRCCloud+Account+Setup%20IRCCloud%20cloud locally-hosted irccloud] instance. SSO should have a link to the irccloud instance. Servicedesk has some great [https://mana.mozilla.org/wiki/display/SD/Internet+Relay+Chat+-+IRC getting started tips] for IRC.

Useful channels
* #ci, #taskcluster, #developers, #mobile, #ateam, #moc

Protected channels
* #platform-ops-soc, #platform-ops-alerts, #access-alerts
* pw in secrets repo
* you don't need to join all of these but some may be useful: https://mana.mozilla.org/wiki/display/SYSADMIN/IRC+use+within+IT
* pw in mana

== Slack ==

Some parts of Mozilla prefer Slack to IRC, more info on [https://mana.mozilla.org/wiki/display/CCT/Slack mana].

== Wiki ==

wiki.mozilla.org (here) is the main source for public documentation

== Mana ==

Some internal Mozilla systems (IT, HR) are documented on [https://mana.mozilla.org mana]. File a [https://mozilla.service-now.com/ ServiceNow] ticket if you don't have access when you start.

== Google Drive ==

Google Drive (formerly Google docs) is a preferred way to share things these days. This includes spreadsheets and documents that will change a great deal over time.

Google Drive access should be enabled with your email account when you start. If you need access to a particular document, talk to the document owner or your manager/mentor.

= Development =

== Mercurial (hg) ==

Most development in releng (and at Mozilla writ-large) is stored in version control using [http://mercurial.selenic.com/ hg].

There is an excellent step-by-step guide for setting up and using hg: [https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html Mercurial for Mozillians]

The root webview of the Mozilla hg repositories is here: https://hg.mozilla.org/

There are 3 levels of commit access:
* Level 1 access allows you to use the [[ReleaseEngineering/TryServer|Try Server]] and setup user repos. As a new contributor, you should request this on day one.
* Level 2 access is required to land code in the build and project repos. Once you have a proven track record of successful patches, you can ask your manager/mentor to vouch for your Level 2 access. Your manager/mentor can also land patches for you until you receive Level 2 access.
* Level 3 access is required to land code in [https://hg.mozilla.org/mozilla-central mozilla-central] and its derived integration & release branches. At some point in your Mozilla contribution story, you may need Level 3 access but many contributors never do. Talk to your manager/mentor if you think you need this access. You should already have Level 2 access when you request Level 3.

You need to file an IT bug to get hg commit access. Follow the instructions for [https://www.mozilla.org/en-US/about/governance/policies/commit/ Becoming a Mozilla Committer], and for Level 2, specify you need access to (at least) hg.mozilla.org/build/* (Product/Component: mozilla.org/Repository Account Requests).
* example request: {{bug|703351}}

== Git & Github ==

There are git mirrors of many popular Mozilla repositories. One of the Mozilla github admins (jlund) can add you to the following GitHub groups:
* TODO create a Builduty team under Mozilla

There are also a handful of git repos hosted directly by Mozilla. Your manager/mentor will let you know if you need access to one of these. ([[Github|See also]])

= Secrets =

Releng and Relops have a shared repo of secrets. Please reach out to jlund or another manager for instructions on how to access and ask to be add yourself as a recipient to every secret that the rest of the CiDuty team (users/buildduty-fingerprints) has access to

= Releng AWS account =

File a release engineering "general" ticket and needinfo jlund or another releng manager to create a user account and add you to the CiDuty group

= Other Services =

For access to other services, you'll need file a couple of bugs:
* Access to [http://nagios.mozilla.org/nagios/ Nagios]
** File a bug in bugzilla under 'MOC: Service Requests'
* Access to [https://inventory.mozilla.org/en-US/#inventory inventory]
** File a bug under 'Infrastructure & Operations::WebOps: Inventory'
* Access to Papertrail
** ask jlund or another manager to be added to the encrypted file in the secrets repo

CIDuty/Reconfigs

2018-05-14T06:16:13Z

Dlabici: Update BuildDuty to CiDuty. Renaming project

__TOC__
{{Release Engineering How To|Reconfigs}}

A reconfig (short for "reconfigure the buildbot masters") is how changes to buildbot configurations make it into production. CiDuty is responsible for running reconfigs.

In a nutshell, a reconfig consists of:
* moving the ''production'' tag for the [https://hg.mozilla.org/build/buildbot-configs buildbot-configs] repository, and the ''production-0.8'' tag for the [https://hg.mozilla.org/build/buildbotcustom buildbotcustom] to the current tip (or chosen revision)
* updating the source checkout of both repositories on all the buildbot masters
* updating the [https://hg.mozilla.org/build/tools tools] checkout on each master to the current tip
* executing a buildbot reconfig command on each buildbot master

Please see the [[ReleaseEngineering/How_To/Land_Buildbot_Master_Changes|instructions for how to land buildbot changes]] for more information.

It is polite to ask in #releng if anyone has further changes to land before starting the reconfig process.

= How to reconfig =
The current state of the art is to use the [https://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/end_to_end_reconfig.sh end_to_end_reconfig.sh] script. To see all the available options the script provides:
<pre>
bash ./end_to_end_reconfig.sh -h
</pre>

The end_to_end_reconfig.sh script uses the original [https://wiki.mozilla.org/ReleaseEngineering/Managing_Buildbot_with_Fabric fabric scripts] as it's core, but also takes care of updating the wiki, updating bugs affected by the merge, and updating the tools checkouts on foopies as well. As such, the script has a few, indirect python module dependencies:
* fabric
* requests
However, the script will try to install the dependencies automatically if they are missing.

In order to update the wiki/bugzilla, the script also requires your credentials for those services in a config file:

# Needed if updating wiki - note the wiki does *not* use your LDAP credentials...
export WIKI_USERNAME=XXX
export WIKI_PASSWORD=XXXXXXXX

# Needed if updating Bugzilla bugs to mark them as in production - *no* Persona integration - must be a native Bugzilla account.. .
# Details for the 'Release Engineering SlaveAPI Service' <slaveapi@mozilla.releng.tld> Bugzilla user can be found in the RelEng
# private repo, in file passwords/slaveapi-bugzilla.txt.gpg (needs decrypting with an approved gpg key).
export BUGZILLA_USERNAME='XXX@mozilla.com'
export BUGZILLA_PASSWORD='XXXXXXXXX'

# Used for slaveapi actions
export LDAP_USERNAME='XXX@mozilla.com'
export LDAP_PASSWORD='XXXXXXXX'

The script will also:
* create a template config file (default is ''~/.reconfig/config'') if it does not exist, but you'll need to fill in your own credentials before it will work.
* attempt to update IRC with reconfig status. To do so, it uses a minimal bash IRC client called [http://tools.suckless.org/ii/ ii]. You can [http://tools.suckless.org/ii/ download ii from their website], or install it via a package manager (e.g. port install ii). Updating irc is non-fatal, but make sure ''ii'' is in your PATH if you want it to work.
* create a temporary folder (''/tmp/reconfig'') to store the reconfig files. When starting a new reconfig, you should delete the reconfig folder corresponding to an older run of the script before attempting to run it again.

= Updating the pinned version of mozharness on mozilla-central =
The revision of [https://hg.mozilla.org/build/mozharness mozharness] used by a particular branch of mozilla code is now tracked in-tree. As a courtesy to developers and sheriffs, CiDuty is expected to update the pinned revision in the mozilla-central integration branch when they move the <code>production</code> tag. The change will be merged from mozilla-central to other branches by sheriffs as part of their normal duties.

The pinned revision is tracked in this file: http://hg.mozilla.org/mozilla-central/file/920ded6a1f77/testing/mozharness/mozharness.json

Update the revision in the file to point to the revision of the new mozharness <code>production</code> tag and land normally.

NOTE: once all of mozharness moves in-tree, this step will be unnecessary.

= Updating master/master_config.json =

If you have added/removed a platform that will change the content of master/master_config.json in tools/buildfarm/maintenance/production_masters.json, you'll need to manually update the masters that this change impacts because the end_to_end_reconfig.sh script does not do this step. {{bug|1215294}} opened to enable this in the script.

Example
<pre>
cd /builds/buildbot/tests1-linux64/
export PRODUCTION_MASTERS=tools/buildfarm/maintenance/production-masters.json
python buildbot-configs/update-master-json.py $PRODUCTION_MASTERS master/master_config.json
make checkconfig
make reconfig
</pre>

= Help, my reconfig failed! =
Assuming you're using the end_to_end_reconfig.sh script, you can resume after fixing the error. Errors and exit state can be found in the manage_masters-##########.log which is created in /tmp/reconfig by default. The hourly auto reconfig logs are in the bb dir as <tt>reconfig.{log,lock}</tt>.

Running the script again will yield the following menu:

* Please select one of the following options:
1) Continue with existing reconfig (e.g. if you have resolved a merge conflict)
2) Delete saved state for existing reconfig, and start from fresh
3) Abort and exit reconfig process

Number 1 is usually the best option here, especially if the hg operations actually succeeded during the previous attempt. This way the wiki/bugzilla updates will still be properly applied.

= Help, my reconfig is stuck! =
Reconfigs can take as little as 30 minutes to run, but can take up to 2 hours depending on how busy the systems are.

In general, linux test masters are the slowest to reconfig. You can tail the manage_masters-##########.log to keep up with progress. By default, this log is created in /tmp/reconfig. The hourly auto reconfig logs are in the bb dir as <tt>reconfig.{log,lock}</tt>.

If the reconfig gets stuck, see [[ReleaseEngineering/How_To/Unstick_a_Stuck_Slave_From_A_Master|How To/Unstick a Stuck Slave From A Master]].

CiDuty/actionable

2018-05-14T06:12:30Z

Dlabici: Updated BuildDuty to CiDuty. Renaming project

== Intro ==
CiDuty is responsible for routine keep-the-lights-on (KTLO) waterline tasks. These include but are not limited to providing loaners to developers, triaging urgent break-fix issues, handling trees closures, and maintaining documentation. If any of these come up early in CiDuty morning, they take priority over anything else. CiDuty should continue performing checks on a regular basis throughout
the day to make sure things are in a good state.

== Continuously ==
'''1.''' Nagios and SNS Alerts - Monitor alerts from [https://nagios.mozilla.org/releng-scl3/cgi-bin/status.cgi?host=all&servicestatustypes=28&hoststatustypes=15&serviceprops=270346&hostprops=270346/ SCL3] and [https://nagios1.private.releng.mdc1.mozilla.com/releng-mdc1/cgi-bin/status.cgi?host=all&servicestatustypes=28/ MDC1] Nagios instances and SNS alerts from [https://papertrailapp.com/dashboard/ papertrail] in the #ci IRC channel. Triage unacknowledged alerts and file/fix bugs as necessary according to the [https://wiki.mozilla.org/ReleaseEngineering/How_To/ Release Engineering How-Tos]. Make sure that all CiDuty bugs have the correct priority set according to the [[ReleaseEngineering/Buildduty_actionable#Buildduty_Bugzilla_Priority_levels/pending_counts| priority list]] below. A few examples of such alerts include:
* High CI wait times
* CI pending job backlogs
* Buildbot misconfigurations
* Relengbot failures
* Golden AMI generation failures
* Unresponsive machines
* DIsk/RAM/CPU issues
* Failed processes
* Buildbot master process age

'''2.''' Monitor the #releng and #taskcluster irc channels for requests/questions from developers and other ops teams.

'''3.''' Bug triaging - The [https://secure.pub.build.mozilla.org/builddata/reports/slave_health/buildduty_report.html CiDuty report] (generated hourly) should be your starting point for bug triaging. Click on the bug number link in the CiDuty report to see the specified bug in bugzilla. You can also view the list of bugs with this [https://mzl.la/2wDHhDZ bugzilla search].
* At the top, it lists unassigned bugs for loan requests. Keep this queue empty and make sure developers are unblocked. The wiki has instructions for [https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave/ how to loan a machine].
* After loans are taken care of, make sure that bugs in the "No dependencies" section get dependencies filed, e.g. diagnosis bug, decomm bug, etc. The specific next steps will depend on the issue; [https://wiki.mozilla.org/ReleaseEngineering/How_To/ Release Engineering How-Tos] has details covering each case.
* Do the same for bugs in the "All dependencies resolved" section to make sure the next action is taken (re-image, decomm, return to production, etc). Again, the specific next steps will depend on the issue.
* Systemic issues (e.g. test failures that require further investigation) should not stay in the CiDuty bugzilla component. It may be OK for you to take the bug and work on it depending on how much time you have, but generally these types of bugs should be escalated to the appropriate team and moved to their component (e.g. General Automation) once CiDuty has triaged them.

== Daily ==
'''1.''' Review email from AWS looking for announced maintenance, degraded instances, etc. Resolve any issues with instances specified in the AWS email, and notify the appropriate groups of any planned maintenance.

'''2.''' Check for and terminate long-running/outdated AWS instances.

'''3.''' Check slave health for errored instances.

'''4.''' Perform buildbot reconfigs if needed.

'''5.''' Perform buildbot master restarts as needed.

== Weekly ==
'''1.''' Review AWS instances. All those AWS that have 'Unknown State/Type' or have been 'stopped for a while'.
* check the AWS sanity logs in [https://papertrailapp.com/dashboard/ papertrail]
* for each host under heading "''[https://papertrailapp.com/systems/74950614/events?q=Unknown%20State Unknown State]''", "''[https://papertrailapp.com/systems/74950614/events?q=Unknown%20Type Unknown Type]''", follow this [https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_AWS_slaves#Unknown_Type_Or_State_Instances/ guide]
* for each host under heading "''[https://papertrailapp.com/systems/74950614/events?q=Stopped%20For%20A%20While Stopped For A While]''", follow this [https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_AWS_slaves#Stopped_For_A_While_Instances/ guide]

'''2.''' Run the aws_manage_routingtables script to make sure that the AWS routing tables are up to date and error free. Fix any reported errors

'''3.''' Review the loaner email report and verify that loaners are still required. Reimage/terminate returned loaners as necessary

== CiDuty Bugzilla Priority levels ==

'''P1:''' Waterline KTLO work.
This includes developer loaners, urgent break-fix, trees closures, etc.

'''P2:''' Projects above waterline that take high priority and have hard deadlines.
This includes various fixes that we need to take care of once, not on a recurring basis. Examples of this are [https://bugzilla.mozilla.org/show_bug.cgi?id=1363897/ 1363897] (legacy extensions disabling prior to 57) and [https://bugzilla.mozilla.org/show_bug.cgi?id=1315977/ 1315977] (upgrading python to 2.7.6 on mac builders)

'''P3.''' Training/Daily Documentation.
This includes onboarding and runbook documentation. Creating/expanding this documentation will help us deliver faster on other bugs.

'''P4.''' TC migration cleanup.
This includes all tasks related to decommissioning, capacity and load reduction, etc as we transition from Buildbot to Taskcluster.

'''P5.''' Operational development.
This includes adding new scripts or improving existing scripts, e.g. to reduce running times, improve logging and notifications system, etc.

CIDuty/How To/How To Loan BD

2018-04-09T18:34:22Z

Dlabici: Slight alteration on layout.

= Introduction =
Use this page if you want to loan a machine to a developer. This guide has been updated on '''26 March 2018''' and reflects the new changes/requirements on how you should approach the process.

The format will be as follows:
* Finding a server
* Cleaning it up
* Activating and Testing the VNC connection to the server
* Sending the credentials to the requester
Everything you need to do when loaning one of the two type of servers is on this page. You can safely click on the links when instructed too as you will be automatically scrolled to the next step (eg: Server Steps -> VNC -> Email).

= OSX Yosemite r7 - Test - Build Bot =
== Find a server ==
Head to SlaveAlloc, following this link: [https://secure.pub.build.mozilla.org/slavealloc/ui/#slaves https://secure.pub.build.mozilla.org/slavealloc/ui/#slaves] and in the top-right search for '''yosemite-r7''':
Look for a server that '''DOESN'T''' have a note to it, such as the note in the example below, keep in mind servers with "'''migration to taskcluster'''" are not good for loaning either:
[[File:Slavealloc-search-example.png]]
----
Take note of the server number, then lets find out if there is a task running on the server.
Follow [https://secure.pub.build.mozilla.org/buildapi/running THIS LINK] and in the top-right search for '''10.10.5''' (version of OSX yosemite). At this point there should be 2 things:
1) Nothing shows up. Make sure your search query is correct and if it is, it's safe to assume there are no tasks running on any of the selected servers, so feel free to pick whichever you find fancy.
----
----
----
Now if you have the server '''[https://wiki.mozilla.org/Buildduty/How_To/How_To_Loan_BD#Connect_to_the_server SKIP TO Connect to the server section]'''.
----
----
----
2) There is stuff that shows up, which means we need to do some more investigation. If after the search you see anything (as shown below) click on all the '''Buildmaster''' column links:
[[File:Slavealloc-search-with-results.png]]
----
The link will lead you to the task-execution page which shows you what's happening on the slave. At this point (if there are more than 5 results) I would '''recommend''' you to open a text editor as we need to keep track of which server is running. You can extract the number from the textfield in the example below:
[[File:Slave-alloc-buildmaster-tasknumber.png]]
----
After you have a complete list of all the busy servers, see if there are servers that are free/idle. If not, come back again 30 minutes later and check again.

== Connect to the server ==
Now that we have a server number, we need to SSH into it, you can do that by typing this into the terminal. Change the DLAP to your mozilla username and XXX to the server number which you want to loan:
ssh LDAP@t-yosemite-r7-XXXX.test.releng.scl3.mozilla.com

Now run the following to commands:
passwd root
passwd cltbld
And change the passwords to the ones offered by your team leader.

== Cleaning up BuildBot and Puppet files ==
At this step we gonna run, again, 3 commands, to clean up some files from server. Like last time, run this commands one at a time.
<nowiki> rm -rf /var/lib/puppet/ssl/private_keys /Library/LaunchDaemons/com.mozilla.puppet.plist \
/Library/LaunchDaemons/org.mozilla.puppetize.plist ~cltbld/.bash_history \
~root/.bash_history ~root/.sh_history /etc/kcpassword ~cltbld/.ssh \
/opt/runner /etc/taskcluster-worker.yml /usr/local/bin/taskcluster-worker \
/usr/local/bin/generic-worker/ \
/Library/LaunchDaemons/org.mozilla.pf.plist \
/Library/LaunchDaemons/org.mozilla.pflog.plist \
/Library/LaunchAgents/net.generic.worker.plist

find /builds -maxdepth 1 -type f -print -delete

sudo reboot</nowiki>
'''IMPORTANT''': Make sure you '''waste no time with the reboot''' after running the 2 commands. Puppet might try to be sneaky and re-puppetize the machine.
----
----
----
At this point you can skip to the next step, which is: [https://wiki.mozilla.org/Buildduty/How_To/How_To_Loan_BD#Setup.2C_test_and_grant_VNC_access Setup Test and Grant VNC access]
----
----
----

= OSX Yosemite r7 - Test - TaskCluster =
== Find a server ==
TBA

== Connect to the server ==
TBA

== Terminate the TC Worker ==
TBA

== Cleaning up TC files ==
TBA

= Setup, test and grant VNC access =
== Activate VNC on the server ==
TBA

== Test the VNC connection ==
TBA

== Grant VNC access/rights to the requester ==
TBA

At this point all thats left if to send the credentials to the requester.

= Send the credentials for the server to the requester =

CIDuty/How To/How To Loan BD

2018-04-09T18:32:54Z

Dlabici: Added how to loan OSX Yosemite r7 Test Build Bot machines.

= Introduction =
Use this page if you want to loan a machine to a developer. This guide has been updated on '''26 March 2018''' and reflects the new changes/requirements on how you should approach the process.

The format will be as follows:
* Finding a server
* Cleaning it up
* Activating and Testing the VNC connection to the server
* Sending the credentials to the requester
Everything you need to do when loaning one of the two type of servers is on this page. You can safely click on the links when instructed too as you will be automatically scrolled to the next step (eg: Server Steps -> VNC -> Email).

= OSX Yosemite r7 - Test - Build Bot =
== Find a server ==
Head to SlaveAlloc, following this link: [https://secure.pub.build.mozilla.org/slavealloc/ui/#slaves https://secure.pub.build.mozilla.org/slavealloc/ui/#slaves] and in the top-right search for '''yosemite-r7''':
Look for a server that '''DOESN'T''' have a note to it, such as the note in the example below, keep in mind servers with "'''migration to taskcluster'''" are not good for loaning either:
[[File:Slavealloc-search-example.png]]
----
Take note of the server number, then lets find out if there is a task running on the server.
Follow [https://secure.pub.build.mozilla.org/buildapi/running THIS LINK] and in the top-right search for '''10.10.5''' (version of OSX yosemite). At this point there should be 2 things:
1) Nothing shows up. Make sure your search query is correct and if it is, it's safe to assume there are no tasks running on any of the selected servers, so feel free to pick whichever you find fancy.
Now if you have the server '''[https://wiki.mozilla.org/Buildduty/How_To/How_To_Loan_BD#Connect_to_the_server SKIP TO Connect to the server section]'''.
----
2) There is stuff that shows up, which means we need to do some more investigation. If after the search you see anything (as shown below) click on all the '''Buildmaster''' column links:
[[File:Slavealloc-search-with-results.png]]
----
The link will lead you to the task-execution page which shows you what's happening on the slave. At this point (if there are more than 5 results) I would '''recommend''' you to open a text editor as we need to keep track of which server is running. You can extract the number from the textfield in the example below:
[[File:Slave-alloc-buildmaster-tasknumber.png]]
----
After you have a complete list of all the busy servers, see if there are servers that are free/idle. If not, come back again 30 minutes later and check again.

== Connect to the server ==
Now that we have a server number, we need to SSH into it, you can do that by typing this into the terminal. Change the DLAP to your mozilla username and XXX to the server number which you want to loan:
ssh LDAP@t-yosemite-r7-XXXX.test.releng.scl3.mozilla.com

Now run the following to commands:
passwd root
passwd cltbld
And change the passwords to the ones offered by your team leader.

== Cleaning up BuildBot and Puppet files ==
At this step we gonna run, again, 3 commands, to clean up some files from server. Like last time, run this commands one at a time.
<nowiki> rm -rf /var/lib/puppet/ssl/private_keys /Library/LaunchDaemons/com.mozilla.puppet.plist \
/Library/LaunchDaemons/org.mozilla.puppetize.plist ~cltbld/.bash_history \
~root/.bash_history ~root/.sh_history /etc/kcpassword ~cltbld/.ssh \
/opt/runner /etc/taskcluster-worker.yml /usr/local/bin/taskcluster-worker \
/usr/local/bin/generic-worker/ \
/Library/LaunchDaemons/org.mozilla.pf.plist \
/Library/LaunchDaemons/org.mozilla.pflog.plist \
/Library/LaunchAgents/net.generic.worker.plist

find /builds -maxdepth 1 -type f -print -delete

sudo reboot</nowiki>
'''IMPORTANT''': Make sure you '''waste no time with the reboot''' after running the 2 commands. Puppet might try to be sneaky and re-puppetize the machine.

At this point you can skip to the next step, which is: [https://wiki.mozilla.org/Buildduty/How_To/How_To_Loan_BD#Setup.2C_test_and_grant_VNC_access Setup Test and Grant VNC access]

= OSX Yosemite r7 - Test - TaskCluster =
== Find a server ==
TBA

== Connect to the server ==
TBA

== Terminate the TC Worker ==
TBA

== Cleaning up TC files ==
TBA

= Setup, test and grant VNC access =
== Activate VNC on the server ==
TBA

== Test the VNC connection ==
TBA

== Grant VNC access/rights to the requester ==
TBA

At this point all thats left if to send the credentials to the requester.

= Send the credentials for the server to the requester =

CIDuty/How To/How To Loan BD

2018-04-09T18:24:31Z

Dlabici: Live preview of formating.

File:Slave-alloc-buildmaster-tasknumber.png

2018-04-09T18:19:36Z

Dlabici:

Slave-alloc-buildmaster-tasknumber

File:Slavealloc-search-with-results.png

2018-04-09T18:18:27Z

Dlabici:

Slavealloc-search-with-results

File:Slavealloc-search-example.png

2018-04-09T18:13:55Z

Dlabici:

Image that shows a slavealloc page with some example servers.

CIDuty/How To/How To Loan BD

2018-03-27T04:39:18Z

Dlabici: Initial release

CIDuty/How To

2018-03-27T04:27:04Z

Dlabici: Added new step-by-step link

===== Alerts =====
: [https://wiki.mozilla.org/ReleaseEngineering/Queue_directories Command queues]
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Dealing_with_high_pending_counts High Pending Counts]
: '''Processes'''
:: [https://wiki.mozilla.org/ReleaseEngineering/Applications/BuildbotBridge Buildbot Bridge]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Restart_Buildbot_Masters Buildbot Master - buildbot age]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_spot_AMIs Golden AMI]
: [https://mana.mozilla.org/wiki/display/NAGIOS/Pending+Scriptworker+Tasks Pending Scriptworker Tasks]
: [https://mana.mozilla.org/wiki/display/NAGIOS/Puppet+Freshness Puppet Freshness]
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Fix_Build4h_not_updating Build4h not updating]

===== Buildbot =====
: '''Manage Buildbot Masters'''
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Setup_buildbot_masters_in_AWS Add a New Buildbot Master in AWS]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Bring_Buildbot_Masters_Up_After_an_Outage Bring Buildbot Masters Up After an Outage]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Decommission_buildbot_masters Decommission a Master]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Land_Buildbot_Master_Changes Land Buildbot Master Changes]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Restart_Buildbot_Masters Restart Buildbot Masters]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Setup_Personal_Development_Master Setup a Personal Development Master]
: '''Manage Buildbot Slaves'''
:: Hardware Machines
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Connect_To_IPMI Connect to IPMI/PDU]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Decommission_Slave Decommission a Slave]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Handle_an_Idle_Slave Handle an Idle Slave ]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Reboot_Multiple_Slaves_Using_Slaveapi Reboot Multiple Machines]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Request_That_a_Machine_Be_Reimaged Re-image machines]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Set_Up_a_Freshly_Imaged_Slave Return a re-imaged machine to production]
::: [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Slave_Management Slave_Management]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Move_slaves_between_pools Move slaves between pools]
:: Virtual Machines
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Add_Disk_to_an_EC2_Instance Add a new disk to an EC2 instance]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_AWS_slaves Manage AWS instances]
:: Loaners
::: [https://wiki.mozilla.org/Buildduty/How_To/How_To_Loan_BD How to loan servers - New step-by-step guide ]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave Loan a Machine]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Update_VPN_ACL Grant VPN Access]
: [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Slave_Management#Slave_Tracking Machine Tracking]
: [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Reconfigs Perform a Reconfig]
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Work_with_Golden_AMIs Work with Golden AMIs]

===== Puppet =====
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Build_DEBs Build DEBs]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Build_DMGs Build DMGs]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Build_RPMs Build RPMs]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Hack_on_PuppetAgain Hack on PuppetAgain]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Push_changes_to_Production Push Changes to Production]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Modules/users Rotate Passwords across Releng Infra]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Set_up_a_user_environment Setup a User Environment]
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Update_Mercurial Update Mercurial]

===== TaskCluster =====
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave#t-yosemite-r7_taskcluster Loan an OSX Worker]
: [https://wiki.mozilla.org/ReleaseEngineering:TestingTechniques Test a Patch]
: [https://wiki.mozilla.org/BuildDuty:QuarantineInstances Quarantine instances]

===== Tree Management =====
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Close_or_Open_the_Tree Close or Open the Tree]
: [https://wiki.mozilla.org/ReleaseEngineering/TryServer Push to Try Server]

===== Other Tasks =====
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Upload_to_internal_Pypi Upload a Python Package to the Internal Pypi]
: [https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool Upload a Package/Binary to ToolTool]
: [https://wiki.mozilla.org/Buildduty/How_To/Add_new_secrets_to_Puppet_Hiera Add new secrets to Puppet Hiera]

===== See Also =====
: Relevant [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Training_-_Relevant_Buildduty_Bugs buildduty bugs] that would be suitable for training

CIDuty/How To

2018-03-27T04:22:11Z

Dlabici: Fixed link for Hiera after change in structure

===== Alerts =====
: [https://wiki.mozilla.org/ReleaseEngineering/Queue_directories Command queues]
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Dealing_with_high_pending_counts High Pending Counts]
: '''Processes'''
:: [https://wiki.mozilla.org/ReleaseEngineering/Applications/BuildbotBridge Buildbot Bridge]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Restart_Buildbot_Masters Buildbot Master - buildbot age]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_spot_AMIs Golden AMI]
: [https://mana.mozilla.org/wiki/display/NAGIOS/Pending+Scriptworker+Tasks Pending Scriptworker Tasks]
: [https://mana.mozilla.org/wiki/display/NAGIOS/Puppet+Freshness Puppet Freshness]
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Fix_Build4h_not_updating Build4h not updating]

===== Buildbot =====
: '''Manage Buildbot Masters'''
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Setup_buildbot_masters_in_AWS Add a New Buildbot Master in AWS]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Bring_Buildbot_Masters_Up_After_an_Outage Bring Buildbot Masters Up After an Outage]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Decommission_buildbot_masters Decommission a Master]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Land_Buildbot_Master_Changes Land Buildbot Master Changes]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Restart_Buildbot_Masters Restart Buildbot Masters]
:: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Setup_Personal_Development_Master Setup a Personal Development Master]
: '''Manage Buildbot Slaves'''
:: Hardware Machines
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Connect_To_IPMI Connect to IPMI/PDU]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Decommission_Slave Decommission a Slave]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Handle_an_Idle_Slave Handle an Idle Slave ]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Reboot_Multiple_Slaves_Using_Slaveapi Reboot Multiple Machines]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Request_That_a_Machine_Be_Reimaged Re-image machines]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Set_Up_a_Freshly_Imaged_Slave Return a re-imaged machine to production]
::: [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Slave_Management Slave_Management]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Move_slaves_between_pools Move slaves between pools]
:: Virtual Machines
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Add_Disk_to_an_EC2_Instance Add a new disk to an EC2 instance]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_AWS_slaves Manage AWS instances]
:: Loaners
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave Loan a Machine]
::: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Update_VPN_ACL Grant VPN Access]
: [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Slave_Management#Slave_Tracking Machine Tracking]
: [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Reconfigs Perform a Reconfig]
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Work_with_Golden_AMIs Work with Golden AMIs]

===== Puppet =====
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Build_DEBs Build DEBs]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Build_DMGs Build DMGs]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Build_RPMs Build RPMs]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Hack_on_PuppetAgain Hack on PuppetAgain]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Push_changes_to_Production Push Changes to Production]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Modules/users Rotate Passwords across Releng Infra]
: [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Set_up_a_user_environment Setup a User Environment]
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Update_Mercurial Update Mercurial]

===== TaskCluster =====
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave#t-yosemite-r7_taskcluster Loan an OSX Worker]
: [https://wiki.mozilla.org/ReleaseEngineering:TestingTechniques Test a Patch]
: [https://wiki.mozilla.org/BuildDuty:QuarantineInstances Quarantine instances]

===== Tree Management =====
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Close_or_Open_the_Tree Close or Open the Tree]
: [https://wiki.mozilla.org/ReleaseEngineering/TryServer Push to Try Server]

===== Other Tasks =====
: [https://wiki.mozilla.org/ReleaseEngineering/How_To/Upload_to_internal_Pypi Upload a Python Package to the Internal Pypi]
: [https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool Upload a Package/Binary to ToolTool]
: [https://wiki.mozilla.org/Buildduty/How_To/Add_new_secrets_to_Puppet_Hiera Add new secrets to Puppet Hiera]

===== See Also =====
: Relevant [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Training_-_Relevant_Buildduty_Bugs buildduty bugs] that would be suitable for training

CIDuty

2018-03-27T04:18:39Z

Dlabici: /* Useful Links */ - Minor changes

__TOC__

= What is Buildduty? =
Buildduty is a team dedicated to helping out developers with Firefox continuous integration infra issues. We currently have six people based in Romania that provide 24/7 support. Buildduty complement the [[Sheriff|sheriffing team]] where sheriffs respond to Firefox code regressions, Buildduty respond to the infrastructure that builds and tests Firefox code.

= Manifesto =
The [[ReleaseEngineering/Buildduty_manifesto| buildduty manifesto]] describes the team responsibilities in a nutshell.

= Team =

{| border=1
| '''Name'''
| '''Profile'''
| '''Social'''
| '''Blog'''
|-
| Jordan Lund
| [https://mozillians.org/u/jlund jlund]
| [https://github.com/lundjordan github]
| [http://jordan-lund.ghost.io/ blog]
|-
| Zsolt Fay
| [https://mozillians.org/en-US/u/zfay/ zfay]
| [https://github.com/Rivulu5 github]
| N/A
|-
| Radu Iman
| [https://mozillians.org/en-US/u/riman/ riman]
| [https://github.com/raduiman github]
| N/A
|-
| Bogdan Crisan
| [https://mozillians.org/en-US/u/bcrisan/ bcrisan]
| [https://github.com/bccrisan github]
| N/A
|-
| Danut Labici
| [https://mozillians.org/en-US/u/dlabici/ dlabici]
| [https://github.com/akhliskun github]
| N/A
|-
| Roland Mutter
| [https://mozillians.org/en-US/u/rmutter/ rmutter]
| [https://github.com/mutterroland github]
| N/A
|}

= Buildduty priorities =
The [[ReleaseEngineering/Buildduty_actionable| buildduty actionable]] enumerates their daily/weekly sanity job.

== Others ==
There is a long list of '''[[ReleaseEngineering/Buildduty/Other_Duties|other, less-frequent duties]]''' that buildduty can assist with.

= Documentation =
There's a [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/How_To wiki page] that aggregates useful info related to the tasks Buildduty is taking care of (as of December 2017).

= Useful Links =
* [[ReleaseEngineering/Buildduty/day_1_checklist|Day 1 checklist]]
* [https://secure.pub.build.mozilla.org/builddata/reports/slave_health/ Slave Health]
* [https://secure.pub.build.mozilla.org/slavealloc/ui/#slaves Slave Alloc]
* [https://tools.taskcluster.net/provisioners Provision Explorer]
* [https://secure.pub.build.mozilla.org/builddata/reports/slave_health/buildduty_report.html Buildduty Report]
* [https://secure.pub.build.mozilla.org/buildapi/ Build Dashboard Main Page]
** You can get JSON dumps for people to analyze by adding <code>&format=json</code>
* [https://wiki.mozilla.org/Buildduty/How_To Public "How To" documents]('''new''')
* [[ReleaseEngineering/How_To|Old/Deprecated Public "How To" documents]]
* [https://mana.mozilla.org/wiki/dosearchsite.action?queryString=title%3A%22How%20To%22&where=RelEng Private "How To" documents]

= Meeting Notes =
* [https://etherpad.mozilla.org/buildduty-notes Daily buildduty stand-up notes]
* [[ReleaseEngineering/Buildduty/Meetings|Old buildduty weekly meetings notes]]
* [[ReleaseEngineering/Buildduty/SVMeetings| SoftVision buildduty stand-up notes]]

CIDuty

2018-03-27T04:17:23Z

Dlabici: Added link to Buildduty new How to page

__TOC__

= What is Buildduty? =
Buildduty is a team dedicated to helping out developers with Firefox continuous integration infra issues. We currently have six people based in Romania that provide 24/7 support. Buildduty complement the [[Sheriff|sheriffing team]] where sheriffs respond to Firefox code regressions, Buildduty respond to the infrastructure that builds and tests Firefox code.

= Manifesto =
The [[ReleaseEngineering/Buildduty_manifesto| buildduty manifesto]] describes the team responsibilities in a nutshell.

= Team =

{| border=1
| '''Name'''
| '''Profile'''
| '''Social'''
| '''Blog'''
|-
| Jordan Lund
| [https://mozillians.org/u/jlund jlund]
| [https://github.com/lundjordan github]
| [http://jordan-lund.ghost.io/ blog]
|-
| Zsolt Fay
| [https://mozillians.org/en-US/u/zfay/ zfay]
| [https://github.com/Rivulu5 github]
| N/A
|-
| Radu Iman
| [https://mozillians.org/en-US/u/riman/ riman]
| [https://github.com/raduiman github]
| N/A
|-
| Bogdan Crisan
| [https://mozillians.org/en-US/u/bcrisan/ bcrisan]
| [https://github.com/bccrisan github]
| N/A
|-
| Danut Labici
| [https://mozillians.org/en-US/u/dlabici/ dlabici]
| [https://github.com/akhliskun github]
| N/A
|-
| Roland Mutter
| [https://mozillians.org/en-US/u/rmutter/ rmutter]
| [https://github.com/mutterroland github]
| N/A
|}

= Buildduty priorities =
The [[ReleaseEngineering/Buildduty_actionable| buildduty actionable]] enumerates their daily/weekly sanity job.

== Others ==
There is a long list of '''[[ReleaseEngineering/Buildduty/Other_Duties|other, less-frequent duties]]''' that buildduty can assist with.

= Documentation =
There's a [https://wiki.mozilla.org/ReleaseEngineering/Buildduty/How_To wiki page] that aggregates useful info related to the tasks Buildduty is taking care of (as of December 2017).

= Useful Links =
* [[ReleaseEngineering/Buildduty/day_1_checklist|Day 1 checklist]]
* [https://secure.pub.build.mozilla.org/builddata/reports/slave_health/ Slave Health]
* [https://secure.pub.build.mozilla.org/slavealloc/ui/#slaves Slave Alloc]
* [https://tools.taskcluster.net/provisioners Provision Explorer]
* [https://secure.pub.build.mozilla.org/builddata/reports/slave_health/buildduty_report.html Buildduty Report]
* [https://secure.pub.build.mozilla.org/buildapi/ Build Dashboard Main Page]
** You can get JSON dumps for people to analyze by adding <code>&format=json</code>
* [https://wiki.mozilla.org/Buildduty/How_To Public "How To" documents]('''new''') / [[ReleaseEngineering/How_To|Public "How To" documents]] ('''old''')
* [https://mana.mozilla.org/wiki/dosearchsite.action?queryString=title%3A%22How%20To%22&where=RelEng Private "How To" documents]

= Meeting Notes =
* [https://etherpad.mozilla.org/buildduty-notes Daily buildduty stand-up notes]
* [[ReleaseEngineering/Buildduty/Meetings|Old buildduty weekly meetings notes]]
* [[ReleaseEngineering/Buildduty/SVMeetings| SoftVision buildduty stand-up notes]]

BuildDuty/How To/Add new secrets to Puppet Hiera

2018-03-27T04:14:42Z

Dlabici: Dlabici moved page BuildDuty/How To/Add new secrets to Puppet Hiera to Buildduty/How To/Add new secrets to Puppet Hiera

#REDIRECT [[Buildduty/How To/Add new secrets to Puppet Hiera]]

CIDuty/How To/Add new secrets to Puppet Hiera

2018-03-27T04:14:42Z

Dlabici: Dlabici moved page BuildDuty/How To/Add new secrets to Puppet Hiera to Buildduty/How To/Add new secrets to Puppet Hiera

= Task Overview =
If a developer ask for some credentials to be added to Puppet, PuppetAgain and/or Hiera, such as in this [https://bugzilla.mozilla.org/show_bug.cgi?id=1402371 bug 1402371] it requires you to access a puppet master, open a file which (at the date of writing this guide) has over 4300 lines, encrypt passwords, backups files, create some string names, verify everything then share the name of the string(s).

= Connect to puppet master =
After you got the credentials which you need to add to Puppet, please connect as yourself to:

<pre>
~$ ssh LDAP-USERNAME@releng-puppet2.srv.releng.scl3.mozilla.com
</pre>
After you are connected to the master, make yourself root:
<pre>
~$ sudo su -
</pre>
Now changes directory to:
<pre>
~$ cd /etc/hiera
</pre>

= Backup existing secrets file =
This is very important! if anything goes wrong with the next steps, you will have a "safe spot" to come back too.
<pre>
~$ cp /etc/hiera/secrets.eyaml /etc/hiera/screts.eyaml.LDAP-YYYYMMDD
</pre>
Example:
<pre>
~$ cp /etc/hiera/secrets.eyaml /etc/hiera/screts.eyaml.dlabici-20180322
</pre>

= Encrypt and add new credentials =
== Check if name is available ==
Before we do anything we need to check the "secrets vault" if the name we want to use for our string is not used already so to do that we can check it with the following command. As an added example, I have to add 2 passwords, so I will be listing the commands that I used to make sure the string name is free to be taken. '''Keep in mind''': Names should be all lowercase with underlines '''like_this_for_example'''.
<pre>
~$ hiera NAME_OF_STRING
</pre>
Example for task which needs us to add AWS IAM credentials:
<pre>
~$ hiera cloud_watcher_access_key_id
~$ hiera cloud_watcher_secret_access_key
</pre>
If the command above returns "'''nil'''" then we can use the name!

== Encrypt the password/credential ==
Now that we verified if the string name is usable we can encrypt everything and prepare to add it to secrets.eyaml.
Simple run this command in the terminal:
<pre>
~$: eyaml encrypt --pkcs7-private-key /etc/hiera/keys/private_key.pem --pkcs7-public-key /etc/hiera/keys/public_key.pem \
--output examples --password -l 'NAME_OF_STRING_THAT_YOU_VERIFIED_LAST_STEP'
</pre>

When you execute the command, you will be asked to "Enter Password:" this is where you put the credential that you want to securely store!
After you enter the password, you will receive 2 outputs. Copy the second one as we will need it later. Output should look something like this:
<pre>
cloudwatcher_access_key_id: >
XXXXXXXXX,XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
</pre>

== Add the encrypted credentials to secrets.eyaml ==
Now with everything ready, we have to add the credentials at the very end of the file. Open the file with your favorite editor and paste it there with 1 empty line above it!
Save the file then quickly run the next step.
'''TIP''': If you use '''nano''' to edit the file press '''CTRL+SHIFT+_''' (underline) and when asked for which Line Number you want to jump to, press '''CTRL+V'''. Those 2 commands together will jump you to the very end of the file.

= Validate the new secrets =
Okay, now that we added and saved everything that we need to, please run the following command:
<pre>
~$: python -c "import yaml; f=open('secrets.eyaml','r'); yaml.load(f)" && echo "pass" || echo "FAIL"
</pre>
We are looking for the output to be "'''pass'''" and nothing else! If it says anything else, please backup the file you made at step 3!

''Okay so the output is pass, is there a way to verify everything worked?''

Yeah! Simple run '''hiera name_of_your_string''' and if you get an output that is not "nil", you are done!
You have added new secrets to Puppet!