MozillaWiki - User contributions [en]

ReleaseEngineering/Day 1 Checklist

2018-11-28T23:50:33Z

NThomas: /* Jumphost */ Update host specs

__TOC__

Welcome to Release Engineering!

This page is meant to get new hires, interns, or interested community members up to speed with the right software, configurations, and communication channels to contribute effectively to the release engineering pipeline.

= Overview =

* [[ReleaseEngineering|Release Engineering]] home page
* [[ReleaseEngineering/Video_Resources|Video Resources]] - this page has a mix of introductions, tutorials, and presentations.

= Development Best Practices =

* Read and keep up to date with: [[ReleaseEngineering/Development_Best_Practices|Development Best Practices]]
*
= Access =

== SSO ==

Generally, we rely on [https://auth0.com/ auth0] across Mozilla for authentication and [https://mana.mozilla.org/wiki/display/SYSADMIN/LDAP+Architecture LDAP] for authorization. Once given LDAP and you have created a permanent password, you can use that to login to the [https://sso.mozilla.com SSO portal]. From SSO, you should have links to various services from email, irc, calendar, slack, mana, etc. More on each of those later on this page

== login.mozilla.com ==

[https://login.mozilla.com/ login.mozilla.com] is where you can change a number of authentication/authorization access bits that you have control over. Each todo in this section assumes you have access to this page.

=== LDAP password reset ===

If you were given a temporary ldap password or you haven't created your own password yet, you should do this now.

'''''Warning for people who already have an LDAP account:''''' '''Change your password.''' Otherwise, adding you to the releng group may lock your account without further notice.

'''''Note:There is a stronger password policy in releng: users must change their password every 3 months.''''' If you don't change your password, the only symptom will be that one day or another (already observed after 8 days), your regular password won't work anymore. If this happens to you, contact people in #servicedesk, they will be able to reset your password.

This is mostly applicable only to employees and interns, although it *is* possible for other contributors to acquire some limited LDAP access. Speak to someone you work with on the releng team if you would like to investigate this.

=== SSH ===

Upload your public ssh key. It is a good idea to generate a separate ssh keypair from your personal one or any other that you have created in the past and use that explicitly for Releng and upload that. Follow this [[Security/Guidelines/OpenSSH#OpenSSH_client|SSH guidelines doc]] on how to generate, configure, and use your ssh key.

note: example ssh config for accessing our systems given below in Jumphost section

=== PGP ===

We use pgp keys to share private information, secrets, and verify that the source came from someone we trust. Generate a keypair for this and upload your public key so others can find it. It would be really good if you could have other people sign your key, adding more trust that this key really belongs to you.

You can use the the [https://mana.mozilla.org/wiki/display/SD/Generating+a+GPG+Public+Key pgp quickstart guide on mana] or you can use the The [https://www.gnupg.org/gph/en/manual.html GNU Privacy Handbook] for reference.

=== VPN ===

Many of our systems are behind a private network in addition to auth0. Follow the prompts to generate and download an openVPN certificate that you can use to import to your vpn client.

See the instructions on how to [https://mana.mozilla.org/wiki/display/SD/VPN install and configure your VPN client] and help choosing the right client for your platform.

note: macOS and Windows users should use [https://www.sparklabs.com/viscosity/ Viscosity]. This application comes with a free 30 day trial. During your trial, your manager can help you create a ServiceNow ticket to get a Viscosity full license.

=== MFA ===

This MFA account is specific to login.mozilla.com and is used for LDAP/auth0 based logins. Follow the instructions to download the Duo Mobile app and create a Mozilla account.

note: later on in this page we will create more MFA accounts for various systems like Github and accessing our Jumphost

== Mercurial (hg) ==
Most development in releng (and at Mozilla writ-large) is stored in version control using [http://mercurial.selenic.com/ hg].

There is an excellent step-by-step guide for setting up and using hg: [https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html Mercurial for Mozillians]

The root webview of the Mozilla hg repositories is here: https://hg.mozilla.org/

Most releng code lives in repos under https://hg.mozilla.org/build

There are 3 levels of commit access:
* Level 1 access allows you to use the [[ReleaseEngineering/TryServer|Try Server]] and setup user repos. As a new contributor, you should request this on day one.
* Level 2 access is required to land code in the build and project repos. Once you have a proven track record of successful patches, you can ask your manager/mentor to vouch for your Level 2 access. Your manager/mentor can also land patches for you until you receive Level 2 access.
* Level 3 access is required to land code in [https://hg.mozilla.org/mozilla-central mozilla-central] and its derived integration & release branches. At some point in your Mozilla contribution story, you may need Level 3 access but many contributors never do. Talk to your manager/mentor if you think you need this access. You should already have Level 2 access when you request Level 3.

You need to file an IT bug to get hg commit access. Follow the instructions for [https://www.mozilla.org/en-US/about/governance/policies/commit/ Becoming a Mozilla Committer], and for Level 2, specify you need access to (at least) hg.mozilla.org/build/* (Product/Component: mozilla.org/Repository Account Requests).
* example request: {{bug|703351}}

== Git & Github ==
There are git mirrors of many popular Mozilla repositories. One of the Mozilla github admins (catlee, kmoir, jlund) can add you to the following GitHub groups:
* [https://github.com/mozilla-releng Mozilla-Releng organization]
* [https://github.com/orgs/mozilla/teams/releng Releng team within the Mozilla organization]

There are also a handful of git repos hosted directly by Mozilla. Your manager/mentor will let you know if you need access to one of these. ([[Github|See also]])

= Communication =

== Mail ==
Mozilla mail is handled by [https://mail.google.com/ Gmail] now.

You should be added to the release@mozilla.com google group as a new hire/intern. This mailing list is managed by Google groups. Owners of this group will be able to add you. Send a test message to release@m.c to verify that your address has been added/subscribed. Talk to your manager if it is not working.

'''WARNING''': release@m.c can contain security-sensitive information. Do not automatically forward your email to a system that is not under Mozilla's control.

=== Mailing lists ===
You'll need to manually subscribe to:
* [https://lists.mozilla.org/listinfo/release-engineering release-engineering] public mailing list
* [https://lists.mozilla.org/listinfo/dev-planning mozilla.dev.planning]
* [https://lists.mozilla.org/listinfo/tools-taskcluster]
* [https://mail.mozilla.org/listinfo/release-drivers] private list

These are available as [news://news.mozilla.org newsgroups], google groups, and [https://lists.mozilla.org/listinfo Mailman lists]

=== Mail Filtering ===

With all that new email, you will want to set up some filters in Gmail (https://mail.google.com/mail/u/0/#settings/filters) to filter some of the higher-volume automated mail into a folder. You may eventually want to handle this information, but on day one hundreds of nagios notifications are not going to be educational.

Here is [http://people.mozilla.org/~coop/mozillaMailFilters.xml an imperfect set of Gmail filters] that you can import to get you started.

A list of new (and some older) automated emails are indexed by subject, along with relevant actions, [https://wiki.mozilla.org/ReleaseEngineering/How_To/Process_release_email here].

If you are going to working on puppet, you should also look at this page on [https://intranet.mozilla.org/RelEngWiki/index.php/How_To/Read_Releng-Shared_Emails how to read releng shared emails].

== Calendar ==
Like mail, we now use [https://www.google.com/calendar/ Google calendar].

You'll want to subscribe to the following public calendars:
* [https://calendar.google.com/calendar/embed?src=mozilla.com_2d32343333353036312d393737%40resource.calendar.google.com Release Engineering - Public]
* [https://www.google.com/calendar/feeds/mozilla.com_toi1svbfjd878aslutkgj32dco%40group.calendar.google.com/public/basic Releng PTO]

Talk to your manager/mentor to get added to the various other private calendars as appropriate.

== Bugzilla ==
Almost everything at Mozilla goes through Bugzilla. [https://bugzilla.mozilla.org/createaccount.cgi Create a Bugzilla account] if you have not already.

You'll need a few tweaks to your account to get access to everything releng-related:
* Add privileges for bugzilla group "build" (Mozilla Build Team) (Can be done by catlee or bugzilla admin.)
* Add your irc nickname & ldap username as "aliases" for your account
** log into bugzilla & follow links "Preferences" -> "Account Information"
** append the aliases, with a leading ':' and enclosed in brackets ('[]') to the "Real Name" field
** e.g.: "<tt>Chris AtLee [:catlee]</tt>"
* [https://bugzilla.mozilla.org/page.cgi?id=quicksearch.html QuickSearch help]

== Filing bugs against Release Engineering ==
The product to use is, unsurprisingly, "Release Engineering." There are multiple possible components under that product, so take your best guess or ask for guidance in IRC.

== IRC ==
The majority of day-to-day communication in releng happens on IRC, and many people use the locally-hosted irccloud instance. Servicedesk has some great [https://mana.mozilla.org/wiki/display/SD/Internet+Relay+Chat+-+IRC getting started tips] for IRC.

If you run a traditional IRC client:
* https://wiki.mozilla.org/IRC irc://irc.mozilla.org
* irc.mozilla.org:6697 #mozbuild (use SSL, ask for the channel key; your nick needs to be registered)
** private team channel
<pre class="_fck_mw_lspace"> * /attach ircs://irc.mozilla.org:6697/?pass=[your_irc_pass]
* /join #mozbuild [access_key]</pre>

* irc.mozilla.org #releng
** public channel for discussion with sheriffs, and continuous integration bot
* keywords to hilite on:
** "!squirrel" is "I need eyes on this now" keyword in #mozbuild & #releng- please set your client to alert on it.
** "r?" - team member looking for a review, perhaps not in but (irc/pastebin)

* also useful to join
** #developers, #airmozilla, #sf, #mobile, #planning, #release-drivers, #ateam,
** #moco access_key is [https://mana.mozilla.org/wiki/display/AVSE/MoCo+Vidyo+Room+and+%23moco+IRC+Channel+Security on mana]
** #firebot for hiliting when you're mentioned in a bug, review request, etc.

IT now also provides a hosted [https://mana.mozilla.org/wiki/display/SD/IRCCloud+Account+Setup IRCCloud cloud] instance you can partake of.
* NOTE: if you're using IRCCloud, and are invited to a password protected channel, IRCCloud will not have the password available to re-connect you. Do the following as soon as you enter the channel:
*# Click the "Options" button
*# Copy the password
*# Click the "Leave" button
*# Click the "Rejoin" button
*# Paste the password into the text box.

== Slack ==
Some parts of Mozilla prefer Slack to IRC, more info on [https://mana.mozilla.org/wiki/display/CCT/Slack mana].

== Vidyo Services ==
Our primary two way video meeting platform is Vidyo. Basic usage instructions are [https://mana.mozilla.org/wiki/display/SD/Vidyo+Desktop+3 here]. Especially if you are running linux, it is highly recommended that you install the client and make test calls prior to any meeting. Many of our team meetings are held in the '''ReleaseEngineering''' room.
* ''Pro tip: many folks have found the mobile client useful to have preinstalled as a backup device.''
* If you're going to record a meeting, practice first. (Instructions are linked from [https://mana.mozilla.org/wiki/display/SD/Vidyo+Desktop+3 mana page].)
* Ask team members for details on recording in the '''ReleaseEngineering''' room.

== Wiki ==
If you're reading this now, you found the wiki! ;)

Please add yourself and details to the [[ReleaseEngineering#Team | list of team members]].

Don't be shy about making improvements to releng pages based on your experiences. Getting someone in releng to review your changes first is good practice. Just ask in #releng.
* Useful [https://wiki.mozilla.org/Special:Templates templates] (aka "macros")
** for releng, use <tt>{{bug|999}}</tt> for bugzilla references; <tt>{{rev|releases/mozilla-release|6e453b4f7056}}</tt> for hg revisions is also quite nice.

== Pastebin ==
There's a local pastebin instance at http://pastebin.mozilla.org/, so you don't need to paste multi-lines into a channel. You'll find a helper script in the [http://hg.mozilla.org/build/braindump/ braindump] repo at <tt>utils/pastebin</tt>.

* it creates a new page from stdin, and outputs the url to stdout
* an argument will be interpretted as the syntax highlighting format to use
* sample usage: <tt>hg diff | pastebin python | pbcopy</tt> (last part mac specific)
* set & export <tt>PASTEBIN_NAME</tt> if you don't want to post as anonymous.

(Many folks now use the pastebin integrated into IRCCloud.)

== Google Drive ==
Google Drive (formerly Google docs) is a preferred way to share things these days. This includes spreadsheets and documents that will change a great deal over time.

Google Drive access should be enabled with your email account when you start. If you need access to a particular document, talk to the document owner or your manager/mentor.

== Mana ==
Some internal Mozilla systems (IT, HR) are documented on [https://mana.mozilla.org mana]. File a [https://mozilla.service-now.com/ ServiceNow] ticket if you don't have access when you start.

== Other Resources ==
* [https://docs.google.com/document/d/1VcEjW82jBxr77aYi3TVaha9S4uJwAkwEapXnhhMwcgg RelEng crowd-sourced Glossary of Terms]
* Join https://mozillians.org/ (public, and yet-another-set-of-credentials)
** join the "release engineering" group
* [https://intranet.mozilla.org/ Mozilla intranet]
** https://intranet.mozilla.org/StaffMeeting
* https://mana.mozilla.org/wiki/display/RelEng/Release+Engineering+Home (for sensitive information only) (LDAP Note: Req attribute RelEng)
* https://trello.com/b/KwwYSXE1/release-engineering-status-board (Projects and deliverables status)
* Sign up for [https://wiki.mozilla.org/Safari_Books O'Reilly's online library] of their boooks - a great resource
* Overview of dev cycle: http://k0s.org/mozilla/workflow.svg (slightly dated). And browse http://k0s.org/mozilla
* [[ReleaseEngineering/Tips_And_Tricks|Releng Tips and Tricks]]Firefox Desktop + Firefox Mobile release process docs:
* [[Release_Management/Release_Process]]
* [[Release_Management/Release_Process/FAQ]]
* [https://intranet.mozilla.org/TravelPolicies#Corporate_Travel_Accounts Egencia account]: this should be accessible via [https://mana.mozilla.org/wiki/display/SD/SSO+Quick+Links SSO]

== Future Access as you need it ==

Talk to your mentor/manager to see which of these make sense. For each section below, request for these as you need them.

=== Nagios ===

https://nagios.mozilla.org/nagios/

File a bug in bugzilla under 'MOC: Service Requests'

=== AWS ===

We have a Releng AWS account. To get access file a Release Engineering: General ticket and request for a user account with a policy that grants you access to what you need in each service. Also enable MFA

=== Private Secrets ===

We have a secrets vault that holds access to various passwords and keys. As you need access to various parts of infra, you will need to get access to the vault and then ask for your gpg key be added to the encrypted secret. Talk to your manager as this comes up.

=== LDAP Groups ===

You may have access to the [https://ldapadmin1.private.scl3.mozilla.com/manage/ ldap admin page] and see your own groups that you have on your record. This page is behind vpn and auth0.

Although you can read your current groups, you will not be able to modify them. To extend with Releng groups that you need. You and your manager will need to file a ticket for them under "MOC: Service Requests"

example ldap groups they may have by default:
cn=corp-vpn,ou=groups,dc=mozilla
cn=IntranetWiki,ou=groups,dc=mozilla
cn=irccloud,ou=groups,dc=mozilla
cn=mfa,ou=groups,dc=mozilla
cn=phonebook_access,ou=groups,dc=mozilla
cn=team_moco,ou=groups,dc=mozilla
cn=vpn_corp,ou=groups,dc=mozilla
cn=vpn_default,ou=groups,dc=mozilla

TODO audit these and break them up by security level (Bug 1465535)

Example ldap groups you may need to file for and request added (example, Bug 1434168):
cn=releng,ou=groups,dc=mozilla
cn=RelEngWiki,ou=groups,dc=mozilla
cn=vpn_releng,ou=groups,dc=mozilla
cn=vpn_releng_loan,ou=groups,dc=mozilla
cn=vpn_relengwiki,ou=groups,dc=mozilla
cn=vpn_tooltooleditor,ou=groups,dc=mozilla
cn=nagiosadmin,ou=groups,dc=mozilla
cn=GraphsAdmin,ou=groups,dc=mozilla
cn=vpn_genericrhel6,ou=groups,dc=mozilla

=== Jumphost ===

To access any of Release Engineering, Taskcluster, and Release Operations hosts directly, you will need to go through VPN -> a Jumphost machine -> Separate MFA -> your target host.

To do that, you and your manager will need to file a ticket against Release Operations and have them send you an invite to add an MFA account on your Duo App.

Then once you have your Jumphost MFA setup correctly, you will need to have your ssh config to correctly route through the jumphost before trying the target host you want.

example ssh config:
<source lang="ruby">
# Ensure KnownHosts are unreadable if leaked - it is otherwise easier to know which hosts your keys have access to.
HashKnownHosts yes
# Host keys the client accepts - order here is honored by OpenSSH
HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256

Host hg.mozilla.org git.mozilla.org
User USERNAME@mozilla.com
Compression yes
ServerAliveInterval 300

Host *.mozilla.com
User USERNAME
IdentityFile ~/.ssh/id_rsa_mozilla_2017-05-12
Compression yes
ServerAliveInterval 300

Host *.build.mozilla.org
Compression yes
User cltbld
ServerAliveInterval 300

Host rejh?.srv.releng.????.mozilla.com
ControlMaster auto
ControlPath ~/.ssh/ssh-%C
ControlPersist 10m
ForwardAgent no

Host *.releng.mdc1.mozilla.com !rejh?.srv.releng.mdc1.mozilla.com
ProxyJump rejh1.srv.releng.mdc1.mozilla.com

Host *.releng.us??.mozilla.com *.releng.mdc2.mozilla.com !rejh?.srv.releng.mdc2.mozilla.com !*.private.releng.????.mozilla.com
ProxyJump rejh1.srv.releng.mdc2.mozilla.com
</source>

ReleaseEngineering/Tupperware

2018-10-08T07:31:37Z

NThomas: Deprecation notice

<font color=red>'''This page contains obsolete content. Buildbot and BuildAPI are no longer in use.'''</font>

= Overview =
_._
_.-``__ ''-._
_.-``.-``` ```-. ''-._
.-`` .-`` ``._ ''-._
( ' `, `.
|`-._ `-._ _.' _.--'
`-._`-._ `-._ _.-' _.-'_.
| `-._`-._ `-._.-'_.-'_.-' ,
\ `-._`-.__.-'_.-' /
`-. `-.__.-' .-'
``-. .-''
``-. .-''
``''
_______
|__ __|
| |_ _ _ __ _ __ ___ _ ____ ____ _ _ __ ___
| | | | | '_ \| '_ \ / _ \ '__\ \ /\ / / _` | '__/ _ \
| | |_| | |_) | |_) | __/ | \ V V / (_| | | | __/
|_|\__,_| .__/| .__/ \___|_| \_/\_/ \__,_|_| \___|
| | | |
|_| |_|
Mozilla apps in Docker.
Portable, Reusable, and Containerized.

Great for keeping an unpolluted development environment, testing patches and enabling new contributors to join the effort by lowering the barriers to entry (being one of the biggest deterrents to potential new contributors).

Vagrant is used as a quick and easy way to provision the docker apps and make the setup truly plug n' play. The current setup only has a single Vagrantfile which launches BuildAPI and BuildBot, with their dependency apps RabbitMQ and MySQL.

= Project Requirements =
There is only a single project dependency that you have to worry about
* [https://www.vagrantup.com/download-archive/v1.6.3.html Vagrant 1.6.3]

= Currently Included Apps =
Currently Tupperware is used for only Release Engineering apps, but that is not meant as a preclusion to avoid other apps, so by all means add some!

Currently included Mozilla apps:
* [http://hg.mozilla.org/build/buildapi/ BuildAPI]
* [http://hg.mozilla.org/build/buildbot/ BuildBot]

Dependency apps currently included:
* [https://www.rabbitmq.com/ RabbitMQ]
* [https://www.mysql.com/ MySQL] - Forked from [https://github.com/orchardup/docker-mysql/tree/76f9fa1566cfb5dad8b1ff8fe4606d5d3a2b6d72 orchardup/mysql]

= Getting Started =
How to run:
* Install [https://www.vagrantup.com/download-archive/v1.6.3.html Vagrant 1.6.3]
* <pre>hg clone https://hg.mozilla.org/build/tupperware/ && cd tupperware && vagrant up</pre>
** (takes >10 minutes the first time)

Where to see apps:
* BuildAPI: http://127.0.0.1:8888/
* BuildBot: http://127.0.0.1:8000/
* RabbitMQ Management: http://127.0.0.1:15672/

= Troubleshooting Tips =
== App not visible? ==
ex. Buildbot is not visible at http://127.0.0.1:8000/
* If the others are up and Buildbot is not, then run "vagrant ssh", then "docker restart buildbot"

== Vagrant Error: The guest machine entered an invalid state... ==
When running "vagrant up", I am getting a error which states, "The guest machine entered an invalid state while waiting for it to boot. Valid states are 'starting, running'. The machine is in the 'poweroff' state. Please verify everything is configured properly and try again."

There are a few possibilities:
* OS X Mavericks may be causing problems with the 64-bit vagrant boxes (https://github.com/mitchellh/vagrant/issues/2157)
** could be a missing kernel driver, there is a hidden virtualbox GUI window that will give more information on errors
* Make sure you are running
** [https://www.vagrantup.com/download-archive/v1.6.3.html Vagrant 1.6.3+]
** [https://www.virtualbox.org/ VirtualBox 4.3.6+]
* Switching to a 32-bit vagrant box seems to be a temporary fix

==ProTips==
Nothing yet :)

ReleaseEngineering/Contribute

2018-10-08T07:13:55Z

NThomas: Remove obsolete content

Mozilla's Release Engineering team is responsible for the project's build, test, and release pipeline. 'Releng' deals with problems at great scale, and there's always room for new contributors!

== What is Release Engineering? ==

''"Release engineering deals with all activities in between regular development and delivery of a software product to the end user, i.e., integration, build, test execution, packaging and delivery of software"''
<small>From the [http://releng.polymtl.ca/RELENG2014/html/index.html RELENG 2014 Workshop]</small>

[http://youtu.be/7j0NDGJVROI "Release Engineering As A Force Multiplier"] is a talk given by Mozilla's former Director of Release Engineering and is a highly recommended introduction to the field.

== Release Engineering at Mozilla ==

Release Engineering at Mozilla is concerned with building Mozilla's release pipeline - the infrastructure that takes the work developers do on our products and delivers it to our customers. This involves being responsible for Mozilla's build and test infrastructure, ie. continuous integration. Release Engineering at Mozilla works to make everybody else to be more efficient and more effective, thus allowing a relatively small organisation like Mozilla to compete with corporate giants and advance Mozilla's mission.

On a day-to-day basis, Mozilla's release engineers write Python applications, maintain the pipeline, and configure & wrestle thousands of servers.

== Contributing to Releng ==

We're always looking for new contributors to help us advance the open web! If you have experience in Python, system administration, or configuration management - or you're looking to learn - we'd love to have you onboard.

Mozilla's Release Engineering team is highly remote and distributed, and we live online. Our timezone spread means that there will always be someone online to help you contribute to the open web.

=== Getting Started ===
If you're interested in contributing to Release Engineering, find a project or bug you like to help with. Feel free to leave a comment in the bug asking how to get started.

=== First Projects ===

==== Mentored Bugs ====

The Mozilla Project has the idea of a 'Good First Bugs' and 'Mentored Bugs'. A 'Good First Bug' is a bug which has been marked as a good first step into the Project and a 'Mentored Bug' is a bug which an experienced contributors has marked as one they will help a new contributor to fix.

Here you can see [https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=RelEng%20good%20first%20bugs&sharer_id=464696 RelEng good first bugs], and [https://codetribute.mozilla.org/projects/releng mentored bugs]. Don't be discouraged if you don't understand much (or anything!) about the bug, that's completely normal. Just ask for help and you'll be a contributor in no time.

==== Mozilla Release Services ====

[https://github.com/mozilla/release-services/ Release Services] is our interface to Release Engineering services.

We're building new services on top of it, and porting old ones to the new structure. Porting an existing application could be a great way to gain familiarity with a big chunk of the existing architecture at your own pace.

Plenty of [https://github.com/mozilla/release-services/ documentation] exists, and there's an irc channel at #release-services on [https://developer.mozilla.org/en-US/docs/Mozilla/QA/Getting_Started_with_IRC irc.mozilla.org].

Ask the team how you can help, there's always a bunch to do!

== Further Information ==

This page is only intended as a starting guide, more information can be found starting from the [[ReleaseEngineering|main Release Engineering page]] or from the sources below.

=== IRC ===

Come find us in irc.mozilla.org channel: '''#ci''' with your questions, we want you to be able to successfully work on code and submit useful patches so don't be afraid to ask about whatever is blocking you.

=== Planet Releng ===

To help the team keep in touch with each other and contributors, many members of Mozilla Releng write regular blog posts on what they're doing. You can find an RSS feed of those posts [http://planet.mozilla.org/releng/ here].

=== Glossary of terms ===

If you ever feel lost with all the jargon around you, here's a [https://docs.google.com/document/d/1VcEjW82jBxr77aYi3TVaha9S4uJwAkwEapXnhhMwcgg/edit# handy reference (Google doc)] .

Balrog/Client Domains

2018-09-16T23:59:01Z

NThomas: /* Pinning Requirements */ Started using aus4.m.o at 4.0

This page documents all of domains that Balrog serves, when various applications switched to them, their SSL pinning requirements, and active certificates.

== SSL Certificates ==
{| class="wikitable sortable"
|-
! Domain
! Issuer
! Serial Number
! Primary/Backup
! Expiration
! Links
! Comments
|-
| rowspan="4" | aus5.mozilla.org
| rowspan="2" | DigiCert
| 07:10:8B:20:9E:D3:45:6C:EE:88:94:91:44:C4:56:0C
| Primary
| August 13, 2019
| {{bug|1369143}}
|
|-
| 07:D5:0D:C7:F3:68:98:2F:AB:5E:19:B9:C5:FB:A1:5C
| Retired on July 20, 2017
| July 28, 2017
| {{bug|1179339}}
|
|-
| rowspan="2" | Thawte
| 0c:96:80:24:b6:b2:72:81:42:8b:53:a5:24:94:52:fb
| Backup
| August 14, 2020
| {{bug|1369143}}
|
|-
| ???
| Retired Backup
| August 10, 2017
| {{bug|1179339}}
|
|-
| rowspan="2" | aus4.mozilla.org
| DigiCert
| 05:5A:F0:03:C4:5E:01:11:4A:D0:5E:24:D7:74:3B:1E
| Primary
| December 7, 2018
| {{bug|832461}}
|
|-
| Thawte
| 25:a8:fd:b6:7a:1f:6c:b8:95:99:e0:91:5c:69:71:05
| Retired Backup
| September 24, 2017
| {{bug|919746}}
| Explicitly not renewing this cert, per https://bugzilla.mozilla.org/show_bug.cgi?id=1340880#c60
|-
| rowspan="2" | aus3.mozilla.org
| Thawte
| 5b:44:41:c9:34:ed:c8:9c:81:b9:32:0d:09:43:45:a9
| Primary
| February 7, 2020
| {{bug|1340880}}
| rowspan="2" | Not possible to have a backup cert because Thawte is the only Issuer compatible with all clients using this domain.
|-
| Thawte
| 14:6A:AB:C3:52:09:8C:4D:51:7B:FA:1B:AA:21:2C:6A
| Retired Primary (retired on September 5, 2017)
| September 8, 2017
| ???
|}

== Pinning Requirements ==
{| class="wikitable sortable"
|-
! Domain
! Application
! Versions
! Issuer Pinned To
! HPKP(inning)
! Links
! Renewable?
|-
| rowspan="7" | aus5.mozilla.org
| Firefox
| rowspan="3" | 42.0 and up
| Nothing
| None
| {{bug|1116409}}
| rowspan="7" | YES - No pinning requirements for some apps, and we can get certs for those that do pin.
|-
| Fennec
| Nothing
| None
| {{bug|1116409}}
|-
| GMP
| "CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US"
"CN=thawte SSL CA - G2,O=thawte, Inc.,C=US"
| None
| {{bug|1116409}}
|-
| rowspan="2" | Thunderbird
| 51.0 and up
| Nothing
| rowspan="2" | None
| {{bug|1182352}}
|-
| 42.0 - 50.0
| "CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US"
"CN=thawte SSL CA - G2,O=thawte, Inc.,C=US"
| {{bug|1116409}}
|-
| B2G
| ???
| Nothing
| None
| {{bug|1116409}}
|-
| SystemAddons
| 44.0 and up
| Any CA included in Firefox's root store.
| None
| {{bug|1213348}}
|-
| rowspan="5" | aus4.mozilla.org
| Firefox
| rowspan="2" | 36.0 - 41.0
| "CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US"
"CN=Thawte SSL CA,O=\"Thawte, Inc.\",C=US"
| None
| {{bug|885477}}
| rowspan="5" | NO - All apps do pinning, and we cannot get certs that are compatible.
|-
| Thunderbird
| "CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US"
"CN=Thawte SSL CA,O=\"Thawte, Inc.\",C=US"
| None
| {{bug|922264}}
|-
| Fennec
| 27.0 - 42.0
| "CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US"
"CN=Thawte SSL CA,O=\"Thawte, Inc.\",C=US"
| None
| {{bug|885477}}
|-
| B2G
| ???
| Nothing
| None
| {{bug|918068}}
|-
| GMP
| 37.0 - 41.0
| "CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US"
"CN=Thawte SSL CA,O=\"Thawte, Inc.\",C=US"
| None
|
|-
| rowspan="4" | aus3.mozilla.org
| rowspan="2" | Firefox
| 26.0 - 35.0
| "CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US"
"CN=Thawte SSL CA,O=\"Thawte, Inc.\",C=US"
| None
| {{bug|921045}}
| rowspan="4" | NO - All apps do pinning, and we cannot get certs that are compatible.
|-
| 4.0 - 25.0
| "OU=Equifax Secure Certificate Authority,O=Equifax,C=US"
"CN=Thawte SSL CA,O=\"Thawte, Inc.\",C=US"
| None
| {{bug|586213}}
|-
| rowspan="2" | Thunderbird
| 27.0 - 35.0
| "CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US"
"CN=Thawte SSL CA,O=\"Thawte, Inc.\",C=US"
| None
| {{bug|942748}}
|-
| 14.0 - 26.0
| "OU=Equifax Secure Certificate Authority,O=Equifax,C=US"
"CN=Thawte SSL CA,O=\"Thawte, Inc.\",C=US"
| None
| {{bug|751679}}
|-
| rowspan="2" | aus2.mozilla.org
| Firefox
| 2.0 - 3.6
| Nothing
| None
| {{bug|302721}}
| rowspan="2" | YES - No pinning requirements. We just 302 to another domain at this point, though.
|-
| Fennec
| 26.0 and earlier
| Nothing
| None
| {{bug|302721}}
|}

NB: Beginning with 24.0, Thunderbird started shipping release channel builds of ESR repos. This means that they have not shipped any release builds from Gecko versions other than 24.0, 31.0, 38.0, 45.0, 52.0, etc. The version numbers in the table still apply for Betas shipped from the major versions listed.

ReleaseEngineering/Releaseduty/FAQ

2018-07-12T22:38:50Z

NThomas: Update link to releasewarrior-2.0

This page has been moved to [https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/day1/ReleaseDuty-Day-1-Checklist-and-FAQ.md releasewarrior]

ReleaseEngineering/Day -1 Checklist

2018-07-10T23:21:09Z

NThomas: /* If the person is staying at Mozilla */ Balrog ldap group

__TOC__

Cleaning up state, handing off work, and removing access

This page is meant to track all the work that should be done when someone from Buildduty transitions off the team

= Person Leaving Actions =

== LDAP, SSH, and VPN ==

* This is handled automatically by IT. Nothing to do.

== Bugzilla & Github ==

* For bugs attached to you: resolve, un-assign yourself, or summarize current state and escalate where appropriate

== IRC & Slack ==

* Because of the potential sensitive information occasionally shared in certain rooms, please drop fromf from all private (pw locked) channels.

== Loaners & Host personal state ==

* If you have any machines on loan or artifacts living on remote machines, please clean those up where appropriate

== Email & Calendar ==

* Both are controlled by LDAP and IT. Make sure you reach out to the target audience you want to connect with before access is revoked. This is a good time to pass on your permanent email and way to reach you, should you want to. It's also good to clear your calendar where it makes sense.

= Manager Actions =

== LDAP, SSH, and VPN ==

=== If the person is leaving Mozilla ===

* SSH & VPN are managed by LDAP and login.mozilla.com. IT should handle this automatically when they revoke LDAP. This largely means you don't need to manually revoke machine and network access. Some caveats and code cleaning in sections later in this wiki

=== If the person is staying at Mozilla ===

* They will continue to have LDAP and therefore it would be good to sanity check that the releng/buildduty/balrog LDAP groups have been purged from their record.

== Jumphost MFA ==

* file a ticket under "Infrastructure & Operations : RelOps" and request to revoke MFA jumphost access. Attach a patch that removes the person from the [https://dxr.mozilla.org/build-central/rev/0f9c36bceb7655c3bc411aaa280d3ff62c00508b/puppet/manifests/moco-config.pp#232 jumphost list].

== Puppet and Signing Shortlist ==

* Some access privileges are defined within the puppet repo. Particularly who has authorized access to our signing servers. Go through [https://dxr.mozilla.org/build-central/rev/0f9c36bceb7655c3bc411aaa280d3ff62c00508b/puppet/manifests/moco-config.pp#202 this list] and make sure it's up to date.

== Taskcluster & Scopes ==

While Taskcluster accounts should largely be controlled by LDAP, scopes should be added or removed where appropriate. Particularly when people stay at Mozilla but leave Releng/Buildduty

== Bugzilla & Github ==

* Find owners for dropped, unresolved bugs
* Find component owners where openings are made
* Remove person from github team and org where appropriate

== IRC & Slack ==

* Rotate password protected channels

== Email ==

* While this is automatically handled it is a good idea, for cleanliness sake, to sanity check list of recipients in private groups. e.g. buildduty@mozilla.com, release@mozilla.com

== AWS ==

* Identity and Access Management (IAM) should be removed where appropriate

== Secrets ==

This is the big one. In our private repo:

1. determine which encrypted files the person has access to
2. under COOKBOOK.txt, follow the steps under "4. How do I remove a user from a file" and remove access for the person leaving to every file they have access to.
3. remove the person from the users/*-fingerprints files.
4. To err on side of caution, it is best to rotate the passwords and keys the person previously had access to. You can use your discretion case by case. To rotate: file a bug, rotate, and update the private repo with new secret

== Heroku ==

TODO

== Dockerhub ==

TODO

Balrog

2018-07-04T03:58:17Z

NThomas: /* Environments */ DB reset info

If you are looking for the general documentation that used to live here, it has been moved into [https://github.com/mozilla/balrog the Balrog repository], and a built version of it [http://mozilla-balrog.readthedocs.io/en/latest/index.html is available on Read The Docs].

This page will continue to host information about Balrog that doesn't make sense to put into the repository, such as meeting notes and things related to our hosted versions of Balrog.

Balrog is the software that runs the server side component of the update system used by Firefox and other Mozilla products. It is the successor to AUS (Application Update Service), which did not scale to our current needs nor allow us to adapt to more recent business requirements. Balrog helps us ship updates faster and with much more flexibility than we’ve had in the past.

= Infrastructure =
== Environments ==
We have a number of different Balrog environments with different purposes:
{| class="wikitable"
|-
! Environment
! App
! URL
! Deploys
! Purpose
|-
| rowspan="2" | Production
| Admin
| https://aus4-admin.mozilla.org (VPN Required)
| rowspan="2" | Manually by CloudOps
| rowspan="2" | Manage and serve production updates
|-
| Public
| https://aus5.mozilla.org and others (see the [[Balrog/Client_Domains | Client Domains page]] for details)
|-
| rowspan="2" | Stage
| Admin
| https://balrog-admin.stage.mozaws.net/ (VPN Required)
| rowspan="2" | When version tags (eg: v2.40) are created
| rowspan="2" | A place to submit staging Releases and verify new Balrog code with automation
|-
| Public
| https://aus4.stage.mozaws.net/
https://aus5.stage.mozaws.net/
|-
| rowspan="2" | Dev
| Admin
| https://balrog-admin.dev.mozaws.net (VPN Required)
| rowspan="2" | Whenever new code is pushed to Balrog's master branch
| rowspan="2" | Manual verification of Balrog code changes in a deployed environment
|-
| Public
| https://aus5.dev.mozaws.net/
|}

The database for the stage or dev environments can be rebuilt (using the latest production data dump) via the <tt>/__rebuilddb__</tt> endpoint on the respective admin host.

== Support & Escalation ==
''If the issue may be visible to users, please make sure [irc://irc.mozilla.org/#moc #moc] is also notified. They can also assist with the notifications below.''

RelEng is the first point of contact for issues. To contact them, follow [[ReleaseEngineering#Contacting_Release_Engineering|the standard RelEng escalation path]].

If RelEng is unable to correct the issue, they may [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations escalate to CloudOps].

== Monitoring & Metrics ==
Metrics from RDS, EC2, and Nginx are available in the [https://app.datadoghq.com/dash/156924/balrog-web-aus5mozillaorg?live=true&page=0&is_auto=false&tile_size=m&fullscreen=false Datadog Dashboard].

We aggregate exceptions from both the [https://sentry.prod.mozaws.net/operations/prod-public/ public apps] and [https://sentry.prod.mozaws.net/operations/prod-admin/ admin app] to [https://sentry.prod.mozaws.net/operations/ CloudOps' Sentry instance].

== ELB Logs ==
Balrog publishes logs to S3 buckets which are [https://sql.telemetry.mozilla.org available for querying in Redash]. The relevant tables are:
* balrog_elb_logs_aus{3,4,5} - These tables contain update request records sourced from the ELB logs of the named domain (eg: aus5). If you're looking to do ad-hoc queries of update request (eg: estimate how many users are on a particular version or channel), the balrog_elb_logs_aus5 is probably what you want to query.
* balrog_elb_logs_aus_api - This table contains request logs for the aus-api.mozilla.org domain
* log_balrog_admin_nginx_access - This table contains access logs for the admin app sourced from nginx access logs.
* log_balrog_admin_nginx_error - This table contains error logs for the admin app sourced from nginx error logs.
* log_balrog_admin_syslog_admin_fixed - This table contains syslog output from the admin app's Docker container.
* log_balrog_admin_syslog_agent - This table contains syslog output from the agent's Docker container.
* log_balrog_web_syslog_web_fixed - This table contains syslog output from the public app's Docker containers.

Redash should show you the table schemas in the pane on the left. If not, you can inspect with them with "describe $table".

== Backups ==
Balrog uses the built-in RDS backups. The database in snapshotted nightly, and incremental backups are done throughout the day. If necessary, we have the ability to recover to within a 5 minute window. Database restoration is done by CloudOps, and they should be contacted immediately if needed.

== Deploying Changes ==
Balrog's [https://mana.mozilla.org/wiki/display/SVCOPS/Firefox+Update+Service stage and production infrastructure] is managed by [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations the Cloud Operations team]. This section describes how to go from a reviewed patch to deploying it in production. You should generally begin this process '''at least 24 hours''' before you want the new code live in production. This gives the new code a chance to bake in stage.

At a high level, the deployment process looks like this:
* Verify the new code in dev
* Bake the new code in stage
* Deploy to prod

Each part of this process is described in more detail below.

=== Is now a good time? ===
Before you deploy, consider whether or not it's an appropriate time to. Some factors to consider:
* Are we in the middle of an important release such as a chemspill? If so, it's probably not a good time to deploy.
* Is it Friday? You probably don't want to deploy on a Friday except in extreme circumstances.
* Do you have enough time to safely do a push? Most pushes take at most 60 minutes to complete once the production push has begun.

=== Schema Upgrades ===
If you need to do a schema change you '''must''' ensure that either the current production code can run with your schema change applied, or that your new code can run with the old schema. Code and schema changes '''cannot''' be done at the same instant, so you must be able to support one of these scenarios. Generally, additive changes (column or table additions) should do the schema change first, while destructive changes (column or table deletions) should do the schema change second. You can simulate the upgrade with your local Docker containers to verify which is right for you.

A quick way to find out if you have a schema change is to diff the current tip of the master branch against the currently deployed tag, eg:
tag=REPLACEME
git diff $tag

When you file the deployment bug (see below), include a note about the schema change in it. Something like:
This push requires a schema change that needs to be done _prior_ to the new code going out. That can be performed by running the Docker image with the "upgrade-db" command, with DBURI set.

{{bug|1295678}} is an example of a push with a schema change.

=== Verification in dev ===
The dev environment automatically deploys new code from the master branch of [https://github.com/mozilla/balrog the Balrog repository] (including any necessary schema changes). Before beginning the deployment procedure, you should do some functional testing there. At the very least, you should do explicit testing of all the new code that would be included in the push. Eg: if you're changing the format of a blob, make sure that you can add a new blob of that type, and that the XML response looks correct.

'''If you have schema changes''' you must also ensure that the existing deployed code will work with the new schema. To do this, CloudOps will downgrade the dev apps. You should do some routine testing (make some changes to some objects, try some update requests) to ensure that everything works. If you have any issues you CANNOT proceed to production.

=== Baking in stage ===
To get the new code in stage you must create a new Release in Github as follows:
# Tag the repository with a "vX.Y" tag. Eg: "git tag -s vX.Y && git push --tags"
# Diff against the previous release tag. Eg: "git diff v2.24 v2.25", to double whether or not there's schema changes.
#* Look for anything unexpected, or any '''schema changes'''. If schema changes are present, see the above section for instructions on handling them.
# [https://github.com/mozilla/balrog/releases Create a new Release on Github]. This create new Docker images tagged with your version, and deploys them to stage. It may take upwards of 30 minutes for the deployment to happen.

Once the changes are deployed to stage, let them bake for at least 24 hours. You can do additional targeted testing here if you wish, or simply wait for nightlies/releases to prod things along. It's a good idea to watch Sentry for new exceptions that may show up, and Datadog for any notable changes in the shape of the traffic.

=== Pushing to production ===
Pushing live requires CloudOps. For non-urgent pushes, you should begin this procedure a few hours in advance to give CloudOps time to notice and respond. For urgent pushes, file the bug immediately and [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations escalate if no action is taken quickly]. Either way, you must follow this procedure to push:
# [https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=oremj%40mozilla.com&bug_file_loc=http%3A%2F%2F&bug_ignored=0&bug_severity=normal&bug_status=NEW&cc=oremj%40mozilla.com&cc=jbuckley%40mozilla.com&cc=bhearsum%40mozilla.com&cf_blocking_fennec=---&cf_fx_iteration=---&cf_fx_points=---&cf_status_firefox56=---&cf_status_firefox57=---&cf_status_firefox58=---&cf_status_firefox_esr52=---&cf_tracking_firefox56=---&cf_tracking_firefox57=---&cf_tracking_firefox58=---&cf_tracking_firefox_esr52=---&cf_tracking_firefox_relnote=---&comment=Balrog%20version%20X.Y%20is%20ready%20to%20be%20pushed%20to%20prod.%20Please%20deploy%20the%20new%20Docker%20images%20%28vX.Y%29%20for%20admin%2C%20public%2C%20and%20the%20agent.%0D%0A%0D%0AWe%27d%20like%20the%20production%20push%20for%20this%20to%20happen%20around%2011am%20pacific%20on%20%28DATE%20GOES%20HERE%29.%0D%0A%0D%0AONE%20OF%3A%0D%0A%28NO%20SCHEMA%20CHANGE%29%20This%20release%20does%20not%20contain%20a%20schema%20change.%0D%0A%28ADDITIVE%20SCHEMA%20CHANGE%29%20This%20release%20requires%20a%20schema%20change%20that%20needs%20to%20be%20done%20_prior_%20to%20the%20new%20code%20going%20out.%20It%20can%20be%20performed%20by%20running%20the%20Docker%20image%20with%20the%20%22upgrade-db%22%20command%2C%20with%20DBURI%20set.%0D%0A%28DESTRUCTIVE%20SCHEMA%20CHANGE%29%20This%20release%20requires%20a%20schema%20change%20that%20needs%20to%20be%20done%20_after_%20to%20the%20new%20code%20is%20fully%20deployed.%20It%20can%20be%20performed%20by%20running%20the%20Docker%20image%20with%20the%20%22upgrade-db%22%20command%2C%20with%20DBURI%20set.%0D%0A%0D%0AONE%20OF%3A%0D%0A%28NO%20SCHEMA%20CHANGE%20OR%20ADDITIVE%20SCHEMA%20CHANGE%29%20If%20anything%20goes%20wrong%20with%20the%20new%20version%20in%20production%2C%20we%20can%20safely%20rollback.%0D%0A%28DESTRUCTIVE%20SCHEMA%20CHANGE%29%20If%20anything%20goes%20wrong%20with%20the%20new%20version%20in%20production%2C%20we%20must%20revert%20the%20schema%20change%20prior%20to%20rolling%20any%20code%20back%20by%20running%20the%20following%20command%20in%20the%20admin%20container%3A%0D%0Apython%20%2Fapp%2Fscripts%2Fmanage-db.py%20-d%20%24DBURI%20--version%20X%0D%0A%28UNABLE%20TO%20COME%20UP%20WITH%20SAFE%20ROLLBACK%20PROCEDURE%20-%20THIS%20SHOULD%20NEVER%20HAPPEN%29%20This%20push%20contains%20changes%20that%20cannot%20be%20safely%20rolled%20back.%20If%20anything%20goes%20wrong%20with%20the%20new%20version%20in%20production%2C%20please%20escalate%20to%20me.%0D%0A%0D%0AONE%20OF%3A%0D%0A%28NO%20NEW%20INTERDEPENDENT%20CODE%20BETWEEN%20admin%2Fpublic%2Fagent%29%20If%20anything%20goes%20wrong%20with%20one%20of%20the%20apps%2C%20it%20may%20be%20rolled%20back%20independently%20of%20the%20others.%0D%0A%28NEW%20CODE%20REQUIRES%20admin%2Fpublic%2Fagent%20ON%20THE%20MOST%20RECENT%20VERSION%29%20If%20anything%20goes%20wrong%20with%20just%20one%20of%20the%20apps%2C%20all%20of%20them%20must%20be%20rolled%20back.&component=Operations%3A%20Deployment%20Requests&contenttypemethod=autodetect&contenttypeselection=text%2Fplain&defined_groups=1&flag_type-37=X&flag_type-4=X&flag_type-5=X&flag_type-607=X&flag_type-708=X&flag_type-721=X&flag_type-737=X&flag_type-787=X&flag_type-800=X&flag_type-803=X&flag_type-846=X&flag_type-864=X&flag_type-914=X&flag_type-916=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=--&product=Cloud%20Services&rep_platform=Unspecified&short_desc=please%20deploy%20balrog%20X.Y%20to%20prod&target_milestone=---&version=unspecified File a bug] to have the new version pushed to production.
#* Wednesdays around 11am Pacific are usually the best day to push to production, because they are generally free of release events, nightlies, and cronjobs. Unless you have a specific need to deploy on a different day, you should request the prod push for a Wednesday between those hours
#* You should link any bugs being deployed is the "Blocks" field.
#* Make sure you substitute the version number and choose the correct options from the bug template.
# Once the push has happened, verify that the code was pushed to production by checking the __version__ endpoints on [https://aus4-admin.mozilla.org/__version__ the Admin] and [https://aus5.mozilla.org/__version__ Public] apps.
# Bump the [https://github.com/mozilla/balrog/blob/master/version.txt in-repo version] to the next available one to ensure the next push gets a new version.

= Meeting Notes =
* [[Balrog/Meetings/CloudOps - June 22, 2016 | CloudOps Migration Meeting - June 22, 2016]]
* [[Balrog/Meetings/Balrog+absearch Information Exchange | Balrog/absearch Information Exchange - June 28, 2016]]
* [[Balrog/Meetings/CloudOps - June 29, 2016 | CloudOps Migration Meeting - June 29, 2016]]
* [[Balrog/Meetings/CloudOps - July 6, 2016 | CloudOps Migration Meeting - July 6, 2016]]
* [[Balrog/Meetings/CloudOps - July 12, 2016 | CloudOps Final Cut Over Planning - July 12, 2016]]
* [[Balrog/Meetings/Balrog Worker Brainingstorming - July 19, 2016 | Balrog Worker Brainstorming - July 19, 2016]]
* [[Balrog/Meetings/CloudOps - July 27, 2016 | CloudOps Meeting - July 27, 2016]]
* [[Balrog/Meetings/CloudOps - August 10, 2016 | CloudOps Meeting - August 10, 2016]]
* [[Balrog/Meetings/CloudOps - September 7, 2016 | CloudOps Meeting - September 7, 2016]]
* [[Balrog/Meetings/CloudOps - September 14, 2016 | CloudOps Meeting - September 14, 2016]]
* [[Balrog/Meetings/CloudOps - September 21, 2016 | CloudOps Meeting - September 21, 2016]]
* [[Balrog/Meetings/CloudOps - September 28, 2016 | CloudOps Meeting - September 28, 2016]]
* [[Balrog/Meetings/CloudOps - October 19, 2016 | CloudOps Meeting - October 19, 2016]]
* [[Balrog/Meetings/CloudOps - November 2, 2016 | CloudOps Meeting - November 2, 2016]]
* [[Balrog/Meetings/CloudOps - November 9, 2016 | CloudOps Meeting - November 9, 2016]]
* [[Balrog/Meetings/CloudOps - January 18, 2017 | CloudOps Meeting - January 17, 2017]]
* [[Balrog/Meetings/CloudOps - February 22, 2017 | CloudOps Meeting - February 22, 2017]]

Template:NextReleaseDate

2017-11-14T02:28:20Z

NThomas: merge day

2018-01-23

Template:Version/Gecko/release/current

2017-11-14T02:27:33Z

NThomas: merge day

Template:Version/Gecko/beta/current

2017-11-14T02:25:21Z

NThomas: merge day

Template:Version/Gecko/release/next

2017-11-14T02:24:46Z

NThomas: merge day

Template:Version/Gecko/central/current

2017-11-13T22:22:02Z

NThomas: merge day

ReleaseEngineering/Applications/BuildbotBridge

2017-10-30T21:34:45Z

NThomas: /* How to restart the services */ Update service names for buildbot_bridge2_reflector

== Buildbot Bridge (BBB) ==
The Buildbot Bridge is a set of services that allow us to schedule jobs in Taskcluster but run them in Buildbot. The Taskcluster Listener listens for pending Taskcluster Tasks and creates BuildRequests for them. The Buildbot Listener listens for Buildbot events for those jobs, and updates the Taskcluster Tasks accordingly. The Reflector reclaims Taskcluster Tasks and polls for changes that can't be detected by listening for Taskcluster on Buildbot events.

The source code is located at https://github.com/mozilla/buildbot-bridge

== Interactions with other systems ==
The Bridge interacts with and has credentials for many different systems:
* [https://tools.taskcluster.net/ Taskcluster] - To claim and resolve Tasks
* Buildbot Scheduler DB - To create BuildRequests
* Buildbot Bridge Database - To track ongoing jobs
* [https://pulse.mozilla.org Pulse] - To subscribe to Taskcluster and Buildbot exchanges
* [https://secure.pub.build.mozilla.org/buildapi/self-serve Self Serve] - To cancel Builds and BuildRequests

== Development ==
When working on low or medium risk patches to the Buildbot Bridge it's easiest to write unit tests, get review, and then use the supported development environment for testing. For more major changes, it's better to set up your own instances of each component, pointed at your own Buildbot Master. Details on each of these is below:

=== Official Development Instance ===
Working with the official development evironment is just like working with production. It is managed by Puppet, and must be deployed through it. Once your new code is deployed to it (see below for how to do that), you can create Tasks for builders on the "alder" project branch and dev will pay attention to them.

=== DIY ===
You will need a few things to have a fully functioning Buildbot Bridge set-up:
* [https://tools.taskcluster.net/auth/ Taskcluster credentials]
* [[ReleaseEngineering/How_To/Setup_Personal_Development_Master | A Build and/or Test Buildbot master]]
** You must be running command_runner.py and pulse_publisher.py as well.
** If you need both, they must be pointed at the same database
* [[https://pulse.mozilla.org/profile A Pulse account]]

Once you have those, adjust the config to use them as well as your own provisioner id, worker group, and worker id. For example, here is the config that bhearsum uses:
<pre>
{
"taskcluster_queue_config": {
"credentials": {
"clientId": "<redacted>",
"accessToken": "<redacted>"
}
},
"buildbot_scheduler_db": "sqlite:////builds/buildbot/bhearsum/build1/master/state.sqlite",
"bbb_db": "sqlite:///bbb.db",
"selfserve_url": "http://foo.com/junk",

"pulse_user": "bhearsum-publisher",
"pulse_password": "<redacted>",
"pulse_queue_basename": "queue/bhearsum-publisher",

"restricted_builders": [],
"ignored_builders": [],

"tclistener": {
"pulse_exchange_basename": "exchange/taskcluster-queue/v1",
"worker_type": "buildbot-bridge-bhearsum",
"provisioner_id": "buildbot-bridge-bhearsum",
"logfile": "tclistener.log"
},
"bblistener": {
"pulse_exchange": "exchange/bhearsum-publisher/buildbot",
"tc_worker_group": "buildbot-bridge-bhearsum",
"tc_worker_id": "buildbot-bridge-bhearsum",
"logfile": "bblistener.log"
},
"reflector": {
"interval": 60,
"logfile": "reflector.log"
}
}
</pre>

Note that the selfserve_url is junk. This is because there is no staging vesrion of BuildAPI (like most of the rest of our CI infrastructure), so you'll be unable to test changes that depend on it in dev unless you set-up your own instance.

== Deployment ==
The Buildbot Bridge services run on multiple machines for redundancy and increased throughput. The installations are fully deployed and managed by Puppet. The running services are managed by supervisord. You can find them in "/builds/bbb" on the following Buildbot masters.
* Production:
** buildbot-master86.bb.releng.scl3.mozilla.com
** buildbot-master72.bb.releng.usw2.mozilla.com
** buildbot-master82.bb.releng.scl3.mozilla.com
*** buildbot-master70.bb.releng.use1.mozilla.com [ no longer used for bbb, was replaced with bm86]
* Dev:
** buildbot-master84.bb.releng.scl3.mozilla.com

=== How to update ===
To deploy new Buildbot Bridge code you must generate a new Python package and have Puppet deploy it. Once your code has been reviewed and landed, do the following to deploy it:
# Bump the version in setup.py
# Run "python setup.py sdist" to generate a new tarball.
# [[ReleaseEngineering/PuppetAgain/Data#.._add_a_data_file | Copy the tarball to the puppet server]]
# Update the dev or prod [https://github.com/mozilla/build-puppet/blob/master/manifests/moco-config.pp version in Puppet].
# Wait for Puppet to update the installations and restart the instances

=== How to restart the services ===
'''with ansible (much easier!)'''

pip install ansible and checkout releng ansible repo as described [[ReleaseEngineering/How_To/Use_Ansible_for_AdHoc_Updates#Set_it_up_locally|here]]:

then run:
ansible-playbook -v -i bbb-inventory.ini supervisord-action.yml -e desired_state=restarted -l all

'''manually'''

running the following as root (don't forget it runs on multiple different machines):
for i in buildbot_bridge2_reflector buildbot_bridge_bblistener buildbot_bridge_tclistener; do supervisorctl restart $i; done

=== What to expect in /builds/bbb ===
<pre>
root@buildbot-master70.bb.releng.use1.mozilla.com bbb]# ls -tlr
total 2756
drwxr-xr-x 3 cltbld cltbld 4096 May 5 08:18 lib
lrwxrwxrwx 1 cltbld cltbld 15 May 5 08:18 lib64 -> /builds/bbb/lib
drwxr-xr-x 2 cltbld cltbld 4096 May 5 08:18 include
-rw------- 1 cltbld cltbld 1241 May 21 05:41 config.json
drwxr-xr-x 2 cltbld cltbld 4096 May 25 07:07 bin
-rw-r--r-- 1 cltbld cltbld 2372446 May 25 10:55 reflector.log
-rw-r--r-- 1 cltbld cltbld 359635 May 25 11:48 bblistener.log
-rw-r--r-- 1 cltbld cltbld 56047 May 26 07:12 tclistener.log
</pre>

As you can see, each service has its own log file. The supervisord logs sometimes have additional information in error cases, and can be found in /var/log/supervisord.

ReleaseEngineering/Repositories/balrog-ui

2017-10-25T02:28:28Z

NThomas: moved into the main repo

Deprecated

Release:Release Automation on Mercurial:Updates through Shipping

2017-08-23T09:37:33Z

NThomas: /* Set-up whatsnew page */ We have a Download button now

{{Release:DocMainHeader}}

= Update/snippet Generation =
Deliverables:
* Updated patcher config file
* Updated update verify configs
* Release blobs submitted to production Balrog
* Updated test channel rules in Balrog, pointing at the new release Blob

After checking out the necessary modules and tools this builder goes to work doing the following:
* Bumping & checking in a patcher config file
* Creating & uploading update snippets

= Set-up whatsnew page =
At the time of writing, manual work is required if we're shipping a release that has a mix of old versions that need/don't need a whatsnew page. If Release Management hasn't specified that a whatsnew page is needed, there's nothing to do. This must be done after initial update submission is done, and you'll need to know which where the cut off line for the whatsnew is. For example, when 35.0 shipped everything earlier than 34.0 required a whatsnew page, but 34.0 and 34.0.5 did not. If you don't grok how Balrog rules work it would be a good idea [[Balrog#Rules | to read up on them]] before getting started here.

Note: do not modify blobs until both "updates" builders have finished. Your changes may be overwritten otherwise.

To create a blob with the whatsnew page enabled, do the following:
* Load the [https://aus4-admin.mozilla.org/ui/ Balrog Admin UI].
* Go to the "Releases" section.
* Find the release you're working on by filtering.
* Click "Download".
* Edit the local file and add the following at the top level of the blob (replace the VERSION variable with the version of the release, do NOT replace %LOCALE% or %OLD_VERSION% - those are substituted by Balrog/Firefox):
"openURL": "https://www.mozilla.org/%LOCALE%/firefox/$MANUALLY_REPLACE_THIS_VARIABLE_CALLED_VERSION/whatsnew/?oldversion=%OLD_VERSION%",
"actions": "showURL",
* Change the "name" in the blob to something unique (usually by appending "-whatsnew" to it).
* Back in the Admin UI, click "Add a new release".
* The name should match the one you just edited in the blob, product should be Firefox.
* Upload the new blob.

Now that you've got the the new blob you need to adjust rules to point at it. This is usually done in two stage: once for the test channels, and once for the live channel. Test channel updates are typically done as soon as the automated channel updates become available. Live channel updates typically happen at the same time we push the release live. Take care when modifying rules - any changes you make to live channels take effect immediately.

Because the changes that need to be made are highly dependent on the state of the rules at the time and the specific versions requiring/not requiring a whatsnew page, specific step-by-step instructions are explicitly not provided. You may need to manipulate existing rules, add new ones, or even delete some. As an example, here's what the release live and test channels rules looked like after 35.0 was shipped:<br/>
[[File:Balrog_whatsnew_rules_example.png]]

The rules above are listed from highest priority to lowest. The first is ignorable for our purposes (it's a long standing watershed update rule for users on versions earlier than 10.0). The next 3 are are rules that point to the whatsnew version of the blob created above - one rule each for the cdntest, localtest, and live channels. The last 3 point users at the non-whatsnew blob that was fully created by automation. Note that the whatsnew rules specify a version ("<34.0") and have a higher priority.

= Update Verify =
Code used:
* [http://hg.mozilla.org/build/tools/file/tip/release/updates/verify.sh release/updates/verify.sh]

Deliverables:
* Update verify logs per platform. Details on them are below.

Update verify, in its current form, verifies a few different things:
* That the "updater" binary works
* That all MAR files can be applied, and result in a functionally equivalent build to the installer.
* That all previous versions (across all locales), have snippets available

It does this in two different types of tests:
* Full update verify unpacks a previous version's installer, pings AUS, downloads the MAR it finds, applies it, and diffs the result against the current version's installer. Differences between the two are reported in the log.
** Run for all locales in the oldVersion
** Run for select locales ("de", "en-US", "ru" at the time of writing) for all other previous versions
** Completes and partials (if present) are both tested
* Quick update verify does a subset of the above: it pings AUS, downloads the MAR it finds, and ensures that it is non-zero in size
** Run against previous versions for all locales not tested as part of the full update verify.

Update verify is split into multiple chunks per platform (10, by default). The full and quick checks are spaced evenly throughout the chunks giving each one roughly equivalent run time.

== Log Analysis ==
This step is pretty good at reporting errors. Most errors will turn the step red, and be pretty clearly visible in the log. Nonetheless it's a good idea to have a scan through the log for anything abnormal (searching for FAIL and WARN are helpful here).

Here's some common failures:
* "Binary files source/bin/freebl3.chk and target/bin/freebl3.chk differ" (or softokn3.chk) on win32 in the complete MAR
** This is normal in the win32 logs and is caused by {{bug|404340}}. Once this bug is fixed it should go away.
* "CRC check failed" and/or "LoadSourceFile failed" on a partial MAR
** This typically happens when the previous release's complete MAR differs from the complete package. Because partial MARs are generated by diffing the previous complete MAR with the current complete MAR the partial will fail the CRC check when this happens. There's nothing that can be done about this, because the problem is with an already-release-build. The correct thing to do here is inform release-drivers that there will not be partials available on this platform, trim them out of the candidates directory, and remove the partial snippets.
* "FAIL: partial from xxxxx wrong size", "FAIL: actual size: 0"
** This is usually caused by missing MARs. You should first check to see if the MAR in question exists in the candidates directory. If it doesn't, go back and look at the logs to see if there was a problem generating or uploading it. If it does exist in the candidates directory make sure the permissions are set properly (ffxbld:firefox, 664).
* "FAIL: Could not download source xxx"
** Usually indicates missing packages or bad permissions in the candidates directory. Take the same action as above.
* "FAIL: no partial update found for xxx" or "FAIL: no complete update found for xxx"
** Check the logs from the previous step to make sure 'pushsnip' was run successfully. This error usually indicates that snippets weren't pushed.

= Check Permissions =
**only applicable for non release promotion releases**
This builder:
# Checks if the FTP directories and files have proper permissions and modes
# Runs rsync in dry-run mode

This builder is triggered automatically after the updates builder.

No reconfig needed for this builder in manual mode. Automatic mode requires reconfig only on version bump.

= Antivirus check =
**only applicable for non release promotion releases. release promotion bundles antivirus within beetmover tasks**
This builder:
# Runs antivirus locally ''← report these results in build notes''
# Runs rsync in dry-run mode

This builder is triggered automatically after the updates builder.

If a long time has passed since the first run, this builder should be run a second time before pushing files to mirrors by hitting "Rebuild" button or forcing the builder with 2 properties set:
* '''Property 1 name''': script_repo_revision use FIREFOX_*_RELEASE tag here
* '''Property 2 name''' -> release_config: mozilla/release-firefox-mozilla-2.0.py

No reconfig needed for this builder in manual mode. Automatic mode requires reconfig only on version bump.

= Push to mirrors =

**only applicable for non release promotion releases**

For GA release builds, this step is not run automatically, and must be manually triggered (below) after receiving the go from RelMgmt. For all other releases, this step is run automatically.

Code: [http://hg.mozilla.org/build/tools/file/tip/scripts/release/stage-tasks.py scripts/release/stage-tasks.py]

* Release
** [http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-mozilla-release-fennec_push_to_mirrors Fennec Release]
*** prerequisites: wait for driver email subj like '(Android|fennec|mobile firefox) $version $buildnum (uploaded|pushed|published) to (Google Play|play store)'
*** Note: push to mirrors step can happen after we push to play store, since we serve updates through google play and not balrog (so cdn/mirrors don't need apks).
** [http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-comm-esr38-thunderbird_push_to_mirrors Thunderbird 38 Release]
** [http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-comm-esr45-thunderbird_push_to_mirrors Thunderbird 45 Release]
** [http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-comm-esr52-thunderbird_push_to_mirrors Thunderbird 52 Release]
*** prerequisites: wait for driver email subj like 'push Thunderbird $version $buildnum to (release-cdntest channel|cdn|mirrors)'
*** Note: antivirus and update_verify automation emails should be completed

This builder:
# Runs rsync and pushes files to the release directory
# Triggers mirror uptake monitoring poller.
# If partner builds are enabled (ie Firefox on release), copies some builds into place

When enableAutomaticPushToMirrors is True, this is triggered automatically after the antivirus scan is done. When False, it must be manually triggered with Force Build. Step will automatically send email when complete.

= Final Verification =

**release promotion doesn't use ReleaseFinalVerification factory. it uses taskcluster and calls final-verification.sh directly**

Code:
* 'ReleaseFinalVerification' from [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py process/factory.py]
* [http://hg.mozilla.org/build/tools/file/tip/release/final-verification.sh final-verification.sh]

Once mirror uptake has reached 50 or 60% this builder should be automatically started by the uptake monitoring process. If these don't get fired automatically, manually verify uptake on bouncer, then start the builder for each platform with 'Force Build' (<em>no parameters needed</em>) from the Buildbot waterfall. If the buildbot scheduler does decide to auto-start the builds after they've been manually started, fear not, as they are pretty quick builds and won't hurt anything.

This is a simple test which is run after mirrors are propagated to ensure there's no (major) problems with them. It does the same thing as the second part of Update Verify: for every locale/platform/version combination it checks for an update snippet, and then performs an HTTP HEAD request on the MAR the snippet points to.

The step will turn red if there are http codes other than 200 or 302, or if FAIL is found in the log. If it does this, you should have a look in the log, and find out which mirror is failing. It's a good idea to keep an eye on that mirror for awhile to make sure it finishes getting all of the files.

Searching the log for errors is made easier by downloading it and running the following locally:
grep HTTP/1 text | grep -v 302 | grep -v 200
grep FAIL text
grep "HTTP/1.1 4" text
grep "HTTP/1.1 5" text

= Bouncer Submitter =
This builder is triggered automatically after tagging completes, and creates and/or updates bouncer entries used on -cdntest and live update channels, and bouncer entries that Bedrock points at.

This step adds adds bouncer entries for installers, complete and partial updates for all platforms listed in enUSPlatforms and l10nPlatforms and locales specified in shipped-locales (en-US is being added in any case).

The step will turn red if tuxedo-add.py exits non zero. There are 2 major error sources:
* tuxedoUsername has not enough privileges to add entries
* Tuxedo is not aware of new locales added in the current version. Usually the log contains "HTTPError: HTTP Error 400: BAD REQUEST". See [[Releases/Firefox_4.0b7/BuildNotes#Update_Bouncer|this link]] for an example workflow on how to resolve this problem.

= Publish in Balrog =

**only applicable for non release promotion releases**

== Betas/Final Releases ==
'''These instructions cover _typical_ cases. If any special requirements have been requested (ie, whatsnew page) you may need to do different or additional things at the same time.'''<br/>

Before publishing, you should verify that the throttle rate is set as expected - automation will not update this for you. Compare the rate in the push e-mail (eg: 25% or 100%) to the rate on [https://aus4-admin.mozilla.org/rules the Balrog admin interface]. More on throttle rates can be found in the [[#Throttling | Throttling section of this page]]

We have a builder that updates the right parts of Balrog to push a release live. When instructed, you can use "force build" to start it.

'''NOTE:''' this step requires a "double authorization" - appropriate sign off by QE or RelMan, ''and'' verifying that automation is ready (e.g. 'ready to ship' notification).

Example buildbot URLs for the builder:
* [http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-comm-beta-update_shipping_beta Thunderbird Beta]
* [http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-comm-esr45-update_shipping_release Thunderbird 45 Release]
* [http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-comm-esr52-update_shipping_release Thunderbird 52 Release]

''Note that the master you force this on, is not necessarily the master that takes the job, watch your e-mail for completion status. Subject will be "[https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/release_templates/update_shipping_success %(releaseName)s: Updates available on %(releaseChannel)s]".''

'''In order to guard against accidents, this builder aborts unless a special property is set. You must enter a "shipit" property on the form, whose value is also "shipit".'''

= Throttling =
The throttle rate is number from 0 to 100, used in the sense of a percentage a gas pedal is pushed down. So
* 0 means stopped, no requests will be served an update
* 100 means full speed, all requests will be served an update
* 25 means quarter speed, 1 in 4 requests will be served an update (on average!)

Update queries which carry the parameter force=1, eg:
https://aus3.mozilla.org/Firefox/.../update.xml?force=1
will not be throttled. Checks which are initiated by the user have this parameter, while background checks (initiated by the app itself) do not.

== Do I need to throttle? ==
Generally, there's two different ways we do throttling for releases. Most of the time we ship a release partly throttled (10% served), throttle it completely 48 hours later (at the request of RelMan through release-drivers and/or a bug), once some amount of users have it in their hands. In certain cases (for example, when our release day coincides with a Microsoft patch Tuesday), we ship a release fully throttled (0% of requests are served an update).

'''In both cases you must make throttling changes before making the updates live.'''

We're usually told explicity "please push release xxxx fully throttled" or "please push release xxxx partly throttled", but if you're unclear about whether a release should be throttled or not, ask on release-drivers prior making the updates live. If you're confused about anything else, ask another RelEng person.

== How to throttle ==
Throttling is an attribute on each rule in Balrog. To change the throttle rate, simply adjust the "Rate" on the appropriate rule and save the changes:
[[File:Throttling in Balrog.png|upright]]

== Verifying Throttling ==
See http://people.mozilla.com/~nthomas/update-watch/release/ for example AUS links

Some links to check:
* Automatic (idle time check) update links:
** 10.0.2 en-US [https://aus3.mozilla.org/update/1/Firefox/10.0.2/20120215223356/WINNT_x86-msvc/en-US/release/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86-gcc3/en-US/release/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml mac] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86_64-gcc3/en-US/release/update.xml linux64]
** 9.0 en-US [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/WINNT_x86-msvc/en-US/release/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Linux_x86-gcc3/en-US/release/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml mac] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Linux_x86_64-gcc3/en-US/release/update.xml linux64]
** 10.0.2esr en-US [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/WINNT_x86-msvc/en-US/esr/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Linux_x86-gcc3/en-US/esr/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/esr/update.xml mac] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Linux_x86_64-gcc3/en-US/esr/update.xml linux64]
** 10.0esr en-US [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/WINNT_x86-msvc/en-US/esr/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Linux_x86-gcc3/en-US/esr/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/esr/update.xml mac] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Linux_x86_64-gcc3/en-US/esr/update.xml linux64]
** 3.6.27 en-US [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/WINNT_x86-msvc/en-US/release/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/Linux_x86-gcc3/en-US/release/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/Darwin_x86-gcc3-u-ppc-i386/en-US/release/update.xml mac] ← ''only updates to 12.0''
* Manual update links:
** 10.0.2 en-US [https://aus3.mozilla.org/update/1/Firefox/10.0.2/20120215223356/WINNT_x86-msvc/en-US/release/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86-gcc3/en-US/release/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml?force=1 mac] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86_64-gcc3/en-US/release/update.xml?force=1 linux64]
** 9.0 en-US [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/WINNT_x86-msvc/en-US/release/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Linux_x86-gcc3/en-US/release/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml?force=1 mac] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Linux_x86_64-gcc3/en-US/release/update.xml?force=1 linux64]
** 10.0.2esr en-US [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/WINNT_x86-msvc/en-US/esr/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Linux_x86-gcc3/en-US/esr/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/esr/update.xml?force=1 mac] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Linux_x86_64-gcc3/en-US/esr/update.xml?force=1 linux64]
** 10.0esr en-US [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/WINNT_x86-msvc/en-US/esr/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Linux_x86-gcc3/en-US/esr/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/esr/update.xml?force=1 mac] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Linux_x86_64-gcc3/en-US/esr/update.xml?force=1 linux64]
** 3.6.27 en-US [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/WINNT_x86-msvc/en-US/release/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/Linux_x86-gcc3/en-US/release/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/Darwin_x86-gcc3-u-ppc-i386/en-US/release/update.xml?force=1 mac] ← ''only updates to 12.0''

= Post-release tasks =

**only applicable for non release promotion releases**

Releases are considered 'shipped' when
* android - the apks have been uploaded to the Play Store
The post-release tasks should be run immediately afterwards.

== Mark as shipped on Ship It ==
Releases need to be marked as shipped on Ship It. To do so, visit https://ship-it.mozilla.org/releases.html, find the release in question, and click the "Shipped!" button.

== postrelease builder ==
The "post release" builder needs to be started with "force build". This builder will update bouncer aliases.

Example buildbot URLs for the builder:
* Beta
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-mozilla-beta-fennec_postrelease Fennec Beta]
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-comm-beta-thunderbird_postrelease Thunderbird Beta]

* Release
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-mozilla-release-fennec_postrelease Fennec Release]
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-comm-esr45-thunderbird_postrelease Thunderbird 45 Release]
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-comm-esr52-thunderbird_postrelease Thunderbird 52 Release]
''Note that the master you force this on, is not necessarily the master that takes the job, watch your e-mail for completion status. Subject will be "[https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/release_templates/default_success %(releaseName)s: completed %(stage)s]".''

ReleaseEngineering/Applications/Ship It

2017-07-27T22:17:25Z

NThomas: /* Release runner */ add releasetasks info

= What is it? =
Ship it is a system that accepts requests for releases from humans and translates them into information and requests that our [https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Documentation Buildbot-based release automation] can process. It consists of a web application/api and a backend script ("release runner") to do much of the actual work.

Source code for the web application lives on [https://github.com/mozilla-releng/ship-it github]. Release runner is part of [https://hg.mozilla.org/build/tools/file/default/buildfarm/release/release-runner.py the build/tools repository]. Release promotion uses the default branch, while legacy automation (eg Thunderbird) uses the <tt>old-release-runner</tt> branch.

= Where is it? =
The web application is deployed on IT's generic web cluster. There are [https://ship-it-dev.allizom.org/ dev], [https://ship-it.allizom.org/ stage], and [https://ship-it.mozilla.org/ production] instances of it. More in-depth, IT-oriented, information is available on [https://mana.mozilla.org/wiki/display/websites/ship-it.mozilla.org mana].

Release runner is deployed through Puppet on buildbot-master85 for release promotion, and on buildbot-master81 for legacy automation. Supervisord manages the running process.

= Requirements/dependencies =
The web application depends on many Python packages. Most are included in a vendor library in the repository. Required binary packages are noted in [https://github.com/mozilla-releng/ship-it/blob/master/requirements/compiled.txt its requirements list].

Release runner's dependencies are described [https://hg.mozilla.org/build/puppet/file/default/modules/releaserunner/manifests/init.pp in its puppet manifests].

= Deployment =
== Web app ==
The dev instance of Ship It automatically pulls the tip of the master branch of the git repository - no human intervention is needed to deploy to it. Stage and production deployment are handled through Chief, which let us push explicit revisions at will:
* [http://genericadm.private.phx1.mozilla.com/chief/shipit.stage Stage's Chief]
* [http://genericadm.private.phx1.mozilla.com/chief/shipit.prod Production's Chief]

When deploying, you should enter your own name in the first field, the deployment password in the second, and the revision you want to push in the third. If you don't have the deployment passwords, ask someone for them.

== Release runner ==
Release runner's repository needs to be updated by hand to deploy a newly checked in version. Run the following on buildbot-master85 (release promotion) or buildbot-master81 (legacy) to do so:
cd /builds/releaserunner/tools
hg pull && hg up
Switch to root and run
supervisorctl restart releaserunner

=== releasetasks ===
Release runner uses [https://github.com/mozilla/releasetasks releasetasks] to generate the taskcluster graph, and this is also updated by hand. Run the following as cltbld@buildbot-master85:
cd ~/releasetasks
git pull

= Using it =
Standard usage of Ship It is documented as part of the full [https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Starting_a_Release#Submit_to_Ship_It release automation documentation]. [https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Troubleshooting Troubleshooting information] is also available there.

ReleaseEngineering/PuppetAgain/HowTo/Set up a user environment

2017-06-28T03:16:16Z

NThomas: /* Secrets (back on master node) */ Heira envs work fine

Approved people have SSH logins on all puppet masters and a puppet environment at /etc/puppet/environments/$username.

You can use this environment to do your development.

= Common =

== On the master machine ==

In general, you'll want to clone http://hg.mozilla.org/build/puppet at /etc/puppet/environments/$USER. You can use git if you'd like, doesn't matter. Then link in the config and nodes for the org you're working with; for moco, that's:

cd /etc/puppet/environments/$USER/manifests
ln -s moco-config.pp config.pp
ln -s moco-nodes.pp nodes.pp

== On the worker node(s) ==

Next, you can run puppet agent on a worker and tell it to use your environment on the server you've selected:

$ su -
# puppet agent --test --environment=dmitchell --server=releng-puppet2.srv.releng.scl3.mozilla.com

On a mac worker, on the first run you will have to specify the ssl dir. For example:

$ su -
# puppet agent --test --server=releng-puppet2.srv.releng.scl3.mozilla.com --environment=kmoir --pluginsync --ssldir=/var/lib/puppet/ssl

== Secrets (back on master node) ==

Note that your secrets will come from the same Hiera datasource as everything else. You can override secrets for your env only in <tt>/etc/hiera/environments/<yourname>_secrets.eyaml</tt>.

= Problems =

The most common problem that you'll see is that your version control system will helpfully make the files in your repository not world-readable, and in particular preclude puppet from reading them.

This will result in this error message when try to run puppet on your worker
"Could not parse for environment $yourid: Permission denied"
in /etc/puppet/environments/$yourid

find . -type f | xargs chmod o+r
find . -type d | xargs chmod o+rx

This ''should'' not be a problem anymore - puppet is now a member of each user's group, so it should be able to read the repository with the 'g' permissions. Please file bugs or contact dustin for any permissions problems.

= Pinning =

If you want to make a node use your environment on every run (e.g., for workers that run puppet at boot), you can "pin" the host to your environment. Edit the node definition like this:
node "hostname" {
# the pins must come *before* the toplevel include
$pin_puppet_server = "releng-puppet2.srv.releng.scl3.mozilla.com"
$pin_puppet_env = "dmitchell"
include toplevel::slave::releng::build
}

This will result in a puppet.conf on the client that specifies the server and environment.

= Git =

If you're using git, set things up as follows:

cd /etc/puppet/environments/$USER
git init

edit .git/config, and add

sharedRepository = 0644

to the [core] section. Then clone a copy of the git repository (e.g., from http://github.com/mozilla/build-puppet), and start hacking.

git remote add mozilla git@github.com:mozilla/build-puppet.git
git fetch mozilla
git reset --hard mozilla/master

== Using git-remote hg ==

{{warning|git-remote-hg is real, released software, but it is buggy as all get-out. Expect errors}}

If you want to be able to push changes to hg, you can use git-remote-hg (see {{bug|1058165}}). Again, you can use whatever repo you would like here. Make sure your SSH config is set up properly (with your username, not mine, of course):

cat ~/.ssh/config
Host hg.mozilla.org
User dmitchell@mozilla.com
Compression yes
ServerAliveInterval 300
chmod 600 ~/.ssh/config

then add a new remote named 'moco'

git remote add moco hg::ssh://hg.mozilla.org/build/puppet
git fetch moco

Note that it creates `moco/branches/default` and `moco/branches/production`. Don't use `moco/master` -- it doesn't work the way you want.

You can generally hack away on your own branch, based on default:

git fetch moco
git checkout -b bug1234 moco/branches/default
.. hack hack hack ..

to rebase your branch onto the latest from hg:

git pull --rebase moco branches/default

Sometimes you'll need to do this twice, or things get confused. If in doubt, try re-pulling.

Pushing to default is a little more complicated, as the push operation requires the source branch name to be 'branches/default'.

git branch -D branches/default
git checkout -b branches/default
git push moco branches/default
git checkout @{-1}

This has the effect of blowing away and re-creating the `branches/default` git branch, based on the current HEAD, pushing that, and then jumping back to the original branch.

In fact, I have an alias for these; in my .git/config:

[alias]
hgpush = "!/bin/sh -c '[ -z \"$0\" ] && exit 1; [ -z \"$1\" ] && exit 1; git branch -D branches/$1; git checkout -b branches/$1 && { git push $0 branches/$1; rc=$?; git checkout @{-1}; exit $rc; }'"
hgpull = "!/bin/sh -c '[ -z \"$0\" ] && exit 1; [ -z \"$1\" ] && exit 1; git pull --rebase $0 branches/$1'"

So I can run

git hgpull moco default

and

git hgpush moco default

respectively. Note that you often need to run `hgpull` twice to get the full effect. I haven't quite figured out what the bug is here.

=== Merging to Production ===

Here's my script for merging to production. I cherry-pick my changes to a local branch named "moco" before pushing, and I use "moco-prod" to track the upstream production branch.

git checkout moco
git hgpull moco default
git hgpull moco default
git hgpush moco default
git checkout moco-prod
git hgpull moco production
git hgpull moco production
git hgmerge
git hgpush moco production
git push github moco-prod
git checkout moco

=== Fixing Problems ===

I have this script (with a name I won't repeat) which can fix things when they get out of whack.

git remote rm moco
rm -rf .git/refs/remotes/moco/ .git/refs/hg .git/hg
git remote add moco hg::ssh://hg.mozilla.org/build/puppet
git fetch moco

= Hg =

If you're using hg, first make sure you're using /usr/local/bin/hg:

which hg

Then set things up as follows (using whatever repo you'd like):

cd /etc/puppet/environment/$USER
mv environment.conf ~
hg --config format.dotencode=0 clone --pull http://hg.mozilla.org/build/puppet .
mv ~/environment.conf .

ReleaseEngineering/Day 1 Checklist

2017-06-05T09:27:26Z

NThomas: Assorted updating (moar ircloud links, slack, move vidyo up, ssh key generation, remove people ahead of EOL

__TOC__

Welcome to Release Engineering!

This page is meant to get new hires, interns, or interested community members up to speed with the right software, configurations, and communication channels to contribute effectively to the release engineering pipeline.

= Video Introductions =

* [[ReleaseEngineering/Video_Resources|Video Resources]] - this page has a mix of introductions, tutorials, and presentations.

= Development Best Practices =

* Read and keep up to date with: [[ReleaseEngineering/Development_Best_Practices|Development Best Practices]]
* Read and keep up to date with: http://moz-releng-docs.readthedocs.org/en/latest/
** Please fix any issues you find: https://github.com/mozilla/build-relengdocs As someone new to the project, your fresh perspective helps shine a light on missing information or assumptions we take for granted.

= Single Sign-On (SSO) =
Many Mozilla services use SSO now, including Gmail, Drive, and ServiceNow. The [https://mozilla.okta.com/app/UserHome?fromLogin=true Single Sign-On homepage] has a list of the available services.

'''NOTE''': SSO requires that [[#LDAP.2C_SSH.2C_VPN|LDAP be setup first]].

= Mail =
Mozilla mail is handled by [https://mail.google.com/ Gmail] now.

You should be added to the release@mozilla.com google group as a new hire/intern. This mailing list is managed by Google groups. Owners of this group will be able to add you. Send a test message to release@m.c to verify that your address has been added/subscribed. Talk to your manager if it is not working.

'''WARNING''': release@m.c can contain security-sensitive information. Do not automatically forward your email to a system that is not under Mozilla's control.

== Mailing lists ==
You'll need to manually subscribe to:
* [https://lists.mozilla.org/listinfo/release-engineering release-engineering] public mailing list
* [https://lists.mozilla.org/listinfo/dev-planning mozilla.dev.planning]
* [https://lists.mozilla.org/listinfo/dev-tree-management mozilla.dev.tree-management]

These are available as [news://news.mozilla.org newsgroups], google groups, and [https://lists.mozilla.org/listinfo Mailman lists]

== Mail Filtering ==

With all that new email, you will want to set up some filters in Gmail (https://mail.google.com/mail/u/0/#settings/filters) to filter some of the higher-volume automated mail into a folder. You may eventually want to handle this information, but on day one hundreds of nagios notifications are not going to be educational.

Here is [http://people.mozilla.org/~coop/mozillaMailFilters.xml an imperfect set of Gmail filters] that you can import to get you started.

A list of new (and some older) automated emails are indexed by subject, along with relevant actions, [https://wiki.mozilla.org/ReleaseEngineering/How_To/Process_release_email here].

If you are going to working on puppet, you should also look at this page on [https://intranet.mozilla.org/RelEngWiki/index.php/How_To/Read_Releng-Shared_Emails how to read releng shared emails].

== Calendar ==
Like mail, we now use [https://www.google.com/calendar/ Google calendar].

You'll want to subscribe to the following public calendars:
* [https://www.google.com/calendar/feeds/mozilla.com_2d32343333353036312d393737%40resource.calendar.google.com/public/basic Release Engineering - Public]
* [https://www.google.com/calendar/feeds/mozilla.com_toi1svbfjd878aslutkgj32dco%40group.calendar.google.com/public/basic Releng PTO]

Talk to your manager/mentor to get added to the various other private calendars as appropriate.

= Access =

== Bugzilla ==
Almost everything at Mozilla goes through Bugzilla. [https://bugzilla.mozilla.org/createaccount.cgi Create a Bugzilla account] if you have not already.

You'll need a few tweaks to your account to get access to everything releng-related:
* Add privileges for bugzilla group "build" (Mozilla Build Team) (Can be done by catlee or bugzilla admin.)
* Add your irc nickname & ldap username as "aliases" for your account
** log into bugzilla & follow links "Preferences" -> "Account Information"
** append the aliases, with a leading ':' and enclosed in brackets ('[]') to the "Real Name" field
** e.g.: "<tt>Chris AtLee [:catlee]</tt>"
* Excellent video showing how to do power searches: [http://people.mozilla.org/~johnath/bugzilla/BeltznerDoesQuicksearch.ogv Bugzilla Quick Search] (5 minutes you NEED to give yourself)
** [https://bugzilla.mozilla.org/page.cgi?id=quicksearch.html QuickSearch help]

== Filing bugs against Release Engineering ==
The product to use is, unsurprisingly, "Release Engineering." There are multiple possible components under that product, so take your best guess or ask for guidance in IRC.

== Vidyo Services ==
Our primary two way video meeting platform is Vidyo. Basic usage instructions are [https://mana.mozilla.org/wiki/display/SD/Vidyo+Desktop+3 here]. Especially if you are running linux, it is highly recommended that you install the client and make test calls prior to any meeting. Many of our team meetings are held in the '''ReleaseEngineering''' room.
* ''Pro tip: many folks have found the mobile client useful to have preinstalled as a backup device.''
* If you're going to record a meeting, practice first. (Instructions are linked from [https://mana.mozilla.org/wiki/display/SD/Vidyo+Desktop+3 mana page].)
* Ask team members for details on recording in the '''ReleaseEngineering''' room.

== LDAP, SSH, VPN ==

'''''Warning for people who already have an LDAP account:''''' '''Change your password.''' Otherwise, adding you to the releng group may lock your account without further notice.

'''''Note:There is a stronger password policy in releng: users must change their password every 3 months.''''' If you don't change your password, the only symptom will be that one day or another (already observed after 8 days), your regular password won't work anymore. If this happens to you, contact people in #servicedesk, they will be able to reset your password.

This is mostly applicable only to employees and interns, although it *is* possible for other contributors to acquire some limited LDAP access. Speak to someone you work with on the releng team if you would like to investigate this.

SSH keys are used for access to most machines, and some of them are stored in your LDAP record:
* Generate a key for Mozilla based on the [https://wiki.mozilla.org/Security/Guidelines/OpenSSH#OpenSSH_client Security's recommendations]. If you are connecting to buildbot masters and slaves you will need to use the Intermediate ssh config.
* Add your SSH public key to your LDAP account, at https://login.mozilla.com/

You'll need a number of other bits set in your LDAP object to access releng networks and systems via the VPN.
* Full instructions for [https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=30769829#MozillaVPN%28DatacenterVPN%29 accessing the VPN] are on [https://mana.mozilla.org mana].
** this usually starts by generating a certificate at https://login.mozilla.com/
* file a bug against Product/Component: Infrastructure & Operations/MOC: Service Requests asking to be added to the '''releng''' and '''vpn_releng''' groups. Something similar to {{bug|1223449}}
* If you're on MacOSX or Windows, you might want to request a license for Viscosity (A VPN client). To do so from [https://mozilla.service-now.com/ service-now].
** 'Order Stuff' > 'Software Applications' > 'Viscosity VPN' > 'Buisness Support && follow onscreen instructions
* Your manager is responsible for making sure you belong to the following LDAP groups, ideally before your first day:
cn=balrog,ou=groups,dc=mozilla
cn=corp-vpn,ou=groups,dc=mozilla
cn=CrashReportsAdmin,ou=groups,dc=mozilla
cn=GraphsAdmin,ou=groups,dc=mozilla
cn=IntranetWiki,ou=groups,dc=mozilla
cn=inventory,ou=groups,dc=mozilla
cn=irccloud,ou=groups,dc=mozilla
cn=mfa,ou=groups,dc=mozilla
cn=nagiosadmin,ou=groups,dc=mozilla
cn=phonebook_access,ou=groups,dc=mozilla
cn=releng,ou=groups,dc=mozilla
cn=RelEngWiki,ou=groups,dc=mozilla
cn=StatsDashboard,ou=groups,dc=mozilla
cn=tableau_users,ou=groups,dc=mozilla
cn=team_moco,ou=groups,dc=mozilla
cn=vpn_corp,ou=groups,dc=mozilla
cn=vpn_default,ou=groups,dc=mozilla
cn=vpn_releng,ou=groups,dc=mozilla
cn=vpn_releng_loan,ou=groups,dc=mozilla
cn=vpn_releng_self_serve,ou=groups,dc=mozilla
cn=vpn_releng_slaveapi_dev,ou=groups,dc=mozilla
cn=vpn_relengwiki,ou=groups,dc=mozilla
cn=vpn_tooltooleditor,ou=groups,dc=mozilla

=== Build Machines ===
Releng runs many, many machines, including:
* staging and production buildbot masters
* staging and production slaves
* servers hosting a variety of services
* a handful of shared releng machines for development

We access these machines via SSH (all), VNC (all), or remote desktop (Windows), depending on the context of the activity we're performing.

To add your SSH key to the buildslaves, you'll need to do something like {{bug|899315}}.
# File a bug in "Release Engineering: Platform Support"
# Checkout the puppet repo and create a patch
# Ask for a reviewer. Your mentor/manager are good options here.

If you need an account on cruncher-aws (shared releng machine) ask catlee, and send him your SSH public key. (If you aren't sure you need the account, you probably do not for now.)

=== SSH Config ===
You'll want to update your ~/.ssh/config accordingly:

<pre class="_fck_mw_lspace">Host hg.mozilla.org
User <short-ldap-name-here>@mozilla.com
Compression yes
ServerAliveInterval 300

Host *.mozilla.com
User <short-ldap-name-here>
Compression yes
ServerAliveInterval 300

Host *.build.mozilla.org
Compression yes
User cltbld
ServerAliveInterval 300</pre>

You can add an <tt>IdentityFile ~/.ssh/<filename></tt> line to those blocks to specify separate keys for different systems (eg Github).

== Mercurial (hg) ==
Most development in releng (and at Mozilla writ-large) is stored in version control using [http://mercurial.selenic.com/ hg].

There is an excellent step-by-step guide for setting up and using hg: [https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html Mercurial for Mozillians]

The root webview of the Mozilla hg repositories is here: https://hg.mozilla.org/

Most releng code lives in repos under https://hg.mozilla.org/build

There are 3 levels of commit access:
* Level 1 access allows you to use the [[ReleaseEngineering/TryServer|Try Server]] and setup user repos. As a new contributor, you should request this on day one.
* Level 2 access is required to land code in the build and project repos. Once you have a proven track record of successful patches, you can ask your manager/mentor to vouch for your Level 2 access. Your manager/mentor can also land patches for you until you receive Level 2 access.
* Level 3 access is required to land code in [https://hg.mozilla.org/mozilla-central mozilla-central] and its derived integration & release branches. At some point in your Mozilla contribution story, you may need Level 3 access but many contributors never do. Talk to your manager/mentor if you think you need this access. You should already have Level 2 access when you request Level 3.

You need to file an IT bug to get hg commit access. Follow the instructions for [https://www.mozilla.org/en-US/about/governance/policies/commit/ Becoming a Mozilla Committer], and for Level 2, specify you need access to (at least) hg.mozilla.org/build/* (Product/Component: mozilla.org/Repository Account Requests).
* example request: {{bug|703351}}

== Git & Github ==
There are git mirrors of many popular Mozilla repositories. One of the Mozilla github admins (catlee, kmoir, jlund) can add you to the following GitHub groups:
* [https://github.com/mozilla-releng Mozilla-Releng organization]
* [https://github.com/orgs/mozilla/teams/releng Releng team within the Mozilla organization]

There are also a handful of git repos hosted directly by Mozilla. Your manager/mentor will let you know if you need access to one of these. ([[Github|See also]])

== PGP ==
PGP keys are used for encrypting sensitive data - you are responsible for generating and managing you own keys. A [https://intranet.mozilla.org/AndroidMarket#How_To:_GPG_Encryption.2FDecryption quick, Mac-focused primer is available on the intranet] or you can use the [https://www.gnupg.org/gph/en/manual.html The GNU Privacy Handbook] for reference. After you have created your keys, do try to get your key signed by other team members, and uploaded to [https://gpg.mozilla.org gpg.mozilla.org].
* It is okay to reuse an existing gpg key, provided you add an additional ID for your canonical LDAP name (e.g. 'hwine [at] mozilla.com', not an alias like 'Hal_9001 [at] mozilla.com').

== Other Services ==
For access to other services, you'll need file a couple of bugs:
* Access to [http://nagios.mozilla.org/nagios/ Nagios]
** File a bug in bugzilla under 'MOC: Service Requests'
* Access to [https://inventory.mozilla.org/en-US/#inventory inventory]
** File a bug under 'Infrastructure & Operations::WebOps: Inventory'

= Communication =

== IRC ==
The majority of day-to-day communication in releng happens on IRC, and many people use the locally-hosted irccloud instance. Servicedesk has some great [https://mana.mozilla.org/wiki/display/SD/Internet+Relay+Chat+-+IRC getting started tips] for IRC.

If you run a traditional IRC client:
* https://wiki.mozilla.org/IRC irc://irc.mozilla.org
* irc.mozilla.org:6697 #mozbuild (use SSL, ask for the channel key; your nick needs to be registered)
** private team channel
<pre class="_fck_mw_lspace"> * /attach ircs://irc.mozilla.org:6697/?pass=[your_irc_pass]
* /join #mozbuild [access_key]</pre>

* irc.mozilla.org #releng
** public channel for discussion with sheriffs, and continuous integration bot
* keywords to hilite on:
** "!squirrel" is "I need eyes on this now" keyword in #mozbuild & #releng- please set your client to alert on it.
** "r?" - team member looking for a review, perhaps not in but (irc/pastebin)

* also useful to join
** #developers, #airmozilla, #sf, #mobile, #planning, #release-drivers, #ateam,
** #moco access_key is [https://mana.mozilla.org/wiki/display/AVSE/MoCo+Vidyo+Room+and+%23moco+IRC+Channel+Security on mana]
** #firebot for hiliting when you're mentioned in a bug, review request, etc.

* http://chatzilla.hacksrus.com/faq/#connect

Some folks run a BNC (an IRC bouncer) (e.g. znc) and/or <tt>irssi</tt> under screen to get continuous view of traffic. Ask around. (some [https://mana.mozilla.org/wiki/display/SYSADMIN/IRC+use+within+IT#IRCusewithinIT-irssi good irssi notes]).

See [[#pastebin|below]] for a local pastebin install, so you don't paste huge amounts in channel (irccloud handles this internally).

IT now also provides a hosted [https://mana.mozilla.org/wiki/display/SD/IRCCloud+Account+Setup IRCCloud cloud] instance you can partake of.
* NOTE: if you're using IRCCloud, and are invited to a password protected channel, IRCCloud will not have the password available to re-connect you. Do the following as soon as you enter the channel:
*# Click the "Options" button
*# Copy the password
*# Click the "Leave" button
*# Click the "Rejoin" button
*# Paste the password into the text box.

== Slack ==
Some parts of Mozilla prefer Slack to IRC, more info on [https://mana.mozilla.org/wiki/display/CCT/Slack mana].

== Wiki ==
If you're reading this now, you found the wiki! ;)

Please add yourself and details to the [[ReleaseEngineering#Team | list of team members]].

Don't be shy about making improvements to releng pages based on your experiences. Getting someone in releng to review your changes first is good practice. Just ask in #releng.
* Useful [https://wiki.mozilla.org/Special:Templates templates] (aka "macros")
** for releng, use <tt>{{bug|999}}</tt> for bugzilla references; <tt>{{rev|releases/mozilla-release|6e453b4f7056}}</tt> for hg revisions is also quite nice.

== Pastebin ==
There's a local pastebin instance at http://pastebin.mozilla.org/, so you don't need to paste multi-lines into a channel. You'll find a helper script in the [http://hg.mozilla.org/build/braindump/ braindump] repo at <tt>utils/pastebin</tt>.

* it creates a new page from stdin, and outputs the url to stdout
* an argument will be interpretted as the syntax highlighting format to use
* sample usage: <tt>hg diff | pastebin python | pbcopy</tt> (last part mac specific)
* set & export <tt>PASTEBIN_NAME</tt> if you don't want to post as anonymous.

(Many folks now use the pastebin integrated into IRCCloud.)

== Google Drive ==
Google Drive (formerly Google docs) is a preferred way to share things these days. This includes spreadsheets and documents that will change a great deal over time.

Google Drive access should be enabled with your email account when you start. If you need access to a particular document, talk to the document owner or your manager/mentor.

== Mana ==
Some internal Mozilla systems (IT, HR) are documented on [https://mana.mozilla.org mana]. File a [https://mozilla.service-now.com/ ServiceNow] ticket if you don't have access when you start.

== Other Resources ==
=== Short-term ===
* [https://docs.google.com/document/d/1VcEjW82jBxr77aYi3TVaha9S4uJwAkwEapXnhhMwcgg RelEng crowd-sourced Glossary of Terms]
* Join https://mozillians.org/ (public, and yet-another-set-of-credentials)
** join the "release engineering" group
* [https://intranet.mozilla.org/ Mozilla intranet]
** https://intranet.mozilla.org/StaffMeeting
* https://mana.mozilla.org/wiki/display/RelEng/Release+Engineering+Home (for sensitive information only) (LDAP Note: Req attribute RelEng)
* https://trello.com/b/KwwYSXE1/release-engineering-status-board (Projects and deliverables status)
* Sign up for [https://wiki.mozilla.org/Safari_Books O'Reilly's online library] of their boooks - a great resource
* Overview of dev cycle: http://k0s.org/mozilla/workflow.svg (slightly dated). And browse http://k0s.org/mozilla
* [[ReleaseEngineering/Tips_And_Tricks|Releng Tips and Tricks]]

=== Longer-term ===
Talk to your mentor/manager to see which of these make sense.
* https://build.inventory.mozilla.org/build
* https://nagios.mozilla.org/nagios/
* https://metrics.mozilla.com/
* background for starting to understand buildbot releases (stale, but s/b roughly accurate): [http://people.mozilla.com/~bhearsum/mozilla-beta-schedulers.png Beta schedulers]
* How teams coordinate during a release: [http://moz-releng-docs.readthedocs.org/en/latest/release_workflows release work flows]

Firefox Desktop + Firefox Mobile release process docs:
* [[Release_Management/Release_Process]]
* [[Release_Management/Release_Process/FAQ]]
* [https://intranet.mozilla.org/TravelPolicies#Corporate_Travel_Accounts Egencia account]: this should be accessible via [https://mana.mozilla.org/wiki/display/SD/SSO+Quick+Links SSO]

ReleaseEngineering/Day 1 Checklist

2017-06-05T08:42:46Z

NThomas: /* Bugzilla */ update admin users

__TOC__

Welcome to Release Engineering!

This page is meant to get new hires, interns, or interested community members up to speed with the right software, configurations, and communication channels to contribute effectively to the release engineering pipeline.

= Video Introductions =

* [[ReleaseEngineering/Video_Resources|Video Resources]] - this page has a mix of introductions, tutorials, and presentations.

= Development Best Practices =

* Read and keep up to date with: [[ReleaseEngineering/Development_Best_Practices|Development Best Practices]]
* Read and keep up to date with: http://moz-releng-docs.readthedocs.org/en/latest/
** Please fix any issues you find: https://github.com/mozilla/build-relengdocs As someone new to the project, your fresh perspective helps shine a light on missing information or assumptions we take for granted.

= Single Sign-On (SSO) =
Many Mozilla services use SSO now, including Gmail, Drive, and ServiceNow. The [https://mozilla.okta.com/app/UserHome?fromLogin=true Single Sign-On homepage] has a list of the available services.

'''NOTE''': SSO requires that [[#LDAP.2C_SSH.2C_VPN|LDAP be setup first]].

= Mail =
Mozilla mail is handled by [https://mail.google.com/ Gmail] now.

You should be added to the release@mozilla.com google group as a new hire/intern. This mailing list is managed by Google groups. Owners of this group will be able to add you. Send a test message to release@m.c to verify that your address has been added/subscribed. Talk to your manager if it is not working.

'''WARNING''': release@m.c can contain security-sensitive information. Do not automatically forward your email to a system that is not under Mozilla's control.

== Mailing lists ==
You'll need to manually subscribe to:
* [https://lists.mozilla.org/listinfo/release-engineering release-engineering] public mailing list
* [https://lists.mozilla.org/listinfo/dev-planning mozilla.dev.planning]
* [https://lists.mozilla.org/listinfo/dev-tree-management mozilla.dev.tree-management]

These are available as [news://news.mozilla.org newsgroups], google groups, and [https://lists.mozilla.org/listinfo Mailman lists]

== Mail Filtering ==

With all that new email, you will want to set up some filters in Gmail (https://mail.google.com/mail/u/0/#settings/filters) to filter some of the higher-volume automated mail into a folder. You may eventually want to handle this information, but on day one hundreds of nagios notifications are not going to be educational.

Here is [http://people.mozilla.org/~coop/mozillaMailFilters.xml an imperfect set of Gmail filters] that you can import to get you started.

A list of new (and some older) automated emails are indexed by subject, along with relevant actions, [https://wiki.mozilla.org/ReleaseEngineering/How_To/Process_release_email here].

If you are going to working on puppet, you should also look at this page on [https://intranet.mozilla.org/RelEngWiki/index.php/How_To/Read_Releng-Shared_Emails how to read releng shared emails].

== Calendar ==
Like mail, we now use [https://www.google.com/calendar/ Google calendar].

You'll want to subscribe to the following public calendars:
* [https://www.google.com/calendar/feeds/mozilla.com_2d32343333353036312d393737%40resource.calendar.google.com/public/basic Release Engineering - Public]
* [https://www.google.com/calendar/feeds/mozilla.com_toi1svbfjd878aslutkgj32dco%40group.calendar.google.com/public/basic Releng PTO]

Talk to your manager/mentor to get added to the various other private calendars as appropriate.

= Access =

== Bugzilla ==
Almost everything at Mozilla goes through Bugzilla. [https://bugzilla.mozilla.org/createaccount.cgi Create a Bugzilla account] if you have not already.

You'll need a few tweaks to your account to get access to everything releng-related:
* Add privileges for bugzilla group "build" (Mozilla Build Team) (Can be done by catlee or bugzilla admin.)
* Add your irc nickname & ldap username as "aliases" for your account
** log into bugzilla & follow links "Preferences" -> "Account Information"
** append the aliases, with a leading ':' and enclosed in brackets ('[]') to the "Real Name" field
** e.g.: "<tt>Chris AtLee [:catlee]</tt>"
* Excellent video showing how to do power searches: [http://people.mozilla.org/~johnath/bugzilla/BeltznerDoesQuicksearch.ogv Bugzilla Quick Search] (5 minutes you NEED to give yourself)
** [https://bugzilla.mozilla.org/page.cgi?id=quicksearch.html QuickSearch help]

== Filing bugs against Release Engineering ==
The product to use is, unsurprisingly, "Release Engineering." There are multiple possible components under that product, so take your best guess or ask for guidance in IRC.

== LDAP, SSH, VPN ==

'''''Warning for people who already have an LDAP account:''''' '''Change your password.''' Otherwise, adding you to the releng group may lock your account without further notice. There is a stronger password policy in releng: users must change their password every 3 months. If you don't change your password, the only symptom will be that one day or another (already observed after 8 days), your regular password won't work anymore. If this happens to you, contact people in #servicedesk, they will be able to reset your password.

This is mostly applicable only to employees and interns, although it *is* possible for other contributors to acquire some limited LDAP access. Speak to someone you work with on the releng team if you would like to investigate this.

SSH keys are used for access to most machines, and some of them are stored in your LDAP record:
* Add your SSH public key to your LDAP account, at https://login.mozilla.com/

You'll need a number of other bits set in your LDAP object to access releng networks and systems via the VPN.
* Full instructions for [https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=30769829#MozillaVPN%28DatacenterVPN%29 accessing the VPN] are on [https://mana.mozilla.org mana].
** this usually starts by generating a certificate at https://login.mozilla.com/
* file a bug against Product/Component: Infrastructure & Operations/MOC: Service Requests asking to be added to the '''releng''' and '''vpn_releng''' groups. Something similar to {{bug|1223449}}
* If you're on MacOSX or Windows, you might want to request a license for Viscosity (A VPN client). To do so from [https://mozilla.service-now.com/ service-now].
** 'Order Stuff' > 'Software Applications' > 'Viscosity VPN' > 'Buisness Support && follow onscreen instructions
* Your manager is responsible for making sure you belong to the following LDAP groups, ideally before your first day:
cn=balrog,ou=groups,dc=mozilla
cn=corp-vpn,ou=groups,dc=mozilla
cn=CrashReportsAdmin,ou=groups,dc=mozilla
cn=GraphsAdmin,ou=groups,dc=mozilla
cn=IntranetWiki,ou=groups,dc=mozilla
cn=inventory,ou=groups,dc=mozilla
cn=irccloud,ou=groups,dc=mozilla
cn=mfa,ou=groups,dc=mozilla
cn=nagiosadmin,ou=groups,dc=mozilla
cn=phonebook_access,ou=groups,dc=mozilla
cn=releng,ou=groups,dc=mozilla
cn=RelEngWiki,ou=groups,dc=mozilla
cn=StatsDashboard,ou=groups,dc=mozilla
cn=tableau_users,ou=groups,dc=mozilla
cn=team_moco,ou=groups,dc=mozilla
cn=vpn_corp,ou=groups,dc=mozilla
cn=vpn_default,ou=groups,dc=mozilla
cn=vpn_releng,ou=groups,dc=mozilla
cn=vpn_releng_loan,ou=groups,dc=mozilla
cn=vpn_releng_self_serve,ou=groups,dc=mozilla
cn=vpn_releng_slaveapi_dev,ou=groups,dc=mozilla
cn=vpn_relengwiki,ou=groups,dc=mozilla
cn=vpn_tooltooleditor,ou=groups,dc=mozilla

=== Common Machines ===
Once your SSH public key is associated with your LDAP record, you should have access a number of shared Mozilla hosts:
* people.mozilla.org (account is not automatic, use [https://mozilla.service-now.com/ service-now] to request).
* hg.mozilla.org (use full email address as username) (commit access is not automatic, see below for access)

=== Build Machines ===
Releng runs many, many machines, including:
* staging and production buildbot masters
* staging and production slaves
* servers hosting a variety of services
* a handful of shared releng machines for development

We access these machines via SSH (all), VNC (all), or remote desktop (Windows), depending on the context of the activity we're performing.

To add your SSH key to the buildslaves, you'll need to do something like {{bug|899315}}.
# File a bug in "Release Engineering: Platform Support"
# Checkout the puppet repo and create a patch
# Ask for a reviewer. Your mentor/manager are good options here.

If you need an account on cruncher (shared releng machine), ask catlee, and send him your SSH public key. (If you aren't sure you need the account, you probably do not for now.)

=== SSH Config ===
You'll want to update your ~/.ssh/config accordingly:

<pre class="_fck_mw_lspace">Host hg.mozilla.org git.mozilla.org
User <short-ldap-name-here>@mozilla.com
Compression yes
ServerAliveInterval 300

Host *.mozilla.com
User <short-ldap-name-here>
Compression yes
ServerAliveInterval 300

Host *.build.mozilla.org
Compression yes
User cltbld
ServerAliveInterval 300</pre>

== Mercurial (hg) ==
Most development in releng (and at Mozilla writ-large) is stored in version control using [http://mercurial.selenic.com/ hg].

There is an excellent step-by-step guide for setting up and using hg: [https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html Mercurial for Mozillians]

The root webview of the Mozilla hg repositories is here: https://hg.mozilla.org/

Most releng code lives in repos under https://hg.mozilla.org/build

There are 3 levels of commit access:
* Level 1 access allows you to use the [[ReleaseEngineering/TryServer|Try Server]] and setup user repos. As a new contributor, you should request this on day one.
* Level 2 access is required to land code in the build and project repos. Once you have a proven track record of successful patches, you can ask your manager/mentor to vouch for your Level 2 access. Your manager/mentor can also land patches for you until you receive Level 2 access.
* Level 3 access is required to land code directly on [https://hg.mozilla.org/mozilla-central mozilla-central] and its derived release branches. At some point in your Mozilla contribution story, you may need Level 3 access but many contributors never do. Talk to your manager/mentor if you think you need this access. You should already have Level 2 access when you request Level 3.

You need to file an IT bug to get hg commit access. Follow the instructions for [https://www.mozilla.org/en-US/about/governance/policies/commit/ Becoming a Mozilla Committer], and for Level 2, specify you need access to (at least) hg.mozilla.org/build/* (Product/Component: mozilla.org/Repository Account Requests).
* example request: {{bug|703351}}

== Git & Github ==
There are git mirrors of many popular Mozilla repositories. One of the Mozilla github admins (catlee, coop) can add you to the following GitHub groups:
* [https://github.com/mozilla-releng Mozilla-Releng organization]
* [https://github.com/orgs/mozilla/teams/releng Releng team within the Mozilla organization]

There are also a handful of git repos hosted directly by Mozilla. Your manager/mentor will let you know if you need access to one of these. ([[Github|See also]])

== PGP ==
PGP keys are used for encrypting sensitive data - you are responsible for generating and managing you own keys. A [https://intranet.mozilla.org/AndroidMarket#How_To:_GPG_Encryption.2FDecryption quick, Mac-focused primer is available on the intranet] or you can use the [https://www.gnupg.org/gph/en/manual.html The GNU Privacy Handbook] for reference. After you have created your keys, do try to get your key signed by other team members, and uploaded to [https://gpg.mozilla.org gpg.mozilla.org].
* It is okay to reuse an existing gpg key, provided you add an additional ID for your canonical LDAP name (e.g. 'hwine [at] mozilla.com', not an alias like 'Hal_9001 [at] mozilla.com').

== Other Services ==
For access to other services, you'll need file a couple of bugs:
* Access to [http://nagios.mozilla.org/nagios/ Nagios]
** File a bug in bugzilla under 'MOC: Service Requests'
* Access to [https://inventory.mozilla.org/en-US/#inventory inventory]
** File a bug under 'Infrastructure & Operations::WebOps: Inventory'

= Communication =

== IRC ==
The majority of day-to-day communication in releng happens on IRC:
* https://wiki.mozilla.org/IRC irc://irc.mozilla.org
* irc.mozilla.org:6697 #mozbuild (use SSL, ask for the channel key; your nick needs to be registered)
** private team channel
<pre class="_fck_mw_lspace"> * /attach ircs://irc.mozilla.org:6697/?pass=[your_irc_pass]
* /join #mozbuild [access_key]</pre>

* irc.mozilla.org #releng
** public channel for discussion with sherrifs, and continuous integration bot
* keywords to hilite on:
** "!squirrel" is "I need eyes on this now" keyword in #mozbuild & #releng- please set your client to alert on it.
** "r?" - team member looking for a review, perhaps not in but (irc/pastebin)

* also useful to join
** #developers, #airmozilla, #sf, #mobile, #planning, #release-drivers, #ateam,
** #infra #sysadmins access_key is here https://mana.mozilla.org/wiki/display/SYSADMIN/IRC+use+within+IT
** #moco [access_key is here [https://intranet.mozilla.org/StaffMeeting [n]]]
** #firebot for hiliting when you're mentioned in a bug, review request, etc.

* http://chatzilla.hacksrus.com/faq/#connect

Many folks run a BNC (an IRC bouncer) (e.g. znc) and/or <tt>irssi</tt> under screen to get continuous view of traffic. Ask around. (some [https://mana.mozilla.org/wiki/display/SYSADMIN/IRC+use+within+IT#IRCusewithinIT-irssi good irssi notes])

See [[#pastebin|below]] for a local pastebin install, so you don't paste huge amounts in channel.

IT now also provides a hosted [https://mana.mozilla.org/wiki/display/SD/IRCCloud+Account+Setup IRCCloud cloud] instance you can partake of.
* NOTE: if you're using IRCCloud, and are invited to a password protected channel, IRCCloud will not have the password available to re-connect you. Do the following as soon as you enter the channel:
*# Click the "Options" button
*# Copy the password
*# Click the "Leave" button
*# Click the "Rejoin" button
*# Paste the password into the text box.

== Wiki ==
If you're reading this now, you found the wiki! ;)

Please add yourself and details to the [[ReleaseEngineering#Team | list of team members]].

Don't be shy about making improvements to releng pages based on your experiences. Getting someone in releng to review your changes first is good practice. Just ask in #releng.
* Useful [https://wiki.mozilla.org/Special:Templates templates] (aka "macros")
** for releng, use <tt>{{bug|999}}</tt> for bugzilla references; <tt>{{rev|releases/mozilla-release|6e453b4f7056}}</tt> for hg revisions is also quite nice.

== Pastebin ==
There's a local pastebin instance at http://pastebin.mozilla.org/, so you don't need to paste multi-lines into a channel. You'll find a helper script in the [http://hg.mozilla.org/build/braindump/ braindump] repo at <tt>utils/pastebin</tt>.

* it creates a new page from stdin, and outputs the url to stdout
* an argument will be interpretted as the syntax highlighting format to use
* sample usage: <tt>hg diff | pastebin python | pbcopy</tt> (last part mac specific)
* set & export <tt>PASTEBIN_NAME</tt> if you don't want to post as anonymous.

(Many folks now use the pastebin integrated into IRCCloud.)

== Google Drive ==
Google Drive (formerly Google docs) is a preferred way to share things these days. This includes spreadsheets and documents that will change a great deal over time.

Google Drive access should be enabled with your email account when you start. If you need access to a particular document, talk to the document owner or your manager/mentor.

== Mana ==
Some internal Mozilla systems (IT, HR) are documented on [https://mana.mozilla.org mana]. File a [https://mozilla.service-now.com/ ServiceNow] ticket if you don't have access when you start.

== Vidyo Services ==
Our primary two way video meeting platform is Vidyo. Basic usage instructions are [https://mana.mozilla.org/wiki/display/SD/Vidyo+Desktop+3 here]. Especially if you are running linux, it is highly recommended that you install the client and make test calls prior to any meeting. Many of our team meetings are held in the '''ReleaseEngineering''' room.
* ''Pro tip: many folks have found the mobile client useful to have preinstalled as a backup device.''
* If you're going to record a meeting, practice first. (Instructions are linked from [https://mana.mozilla.org/wiki/display/SD/Vidyo+Desktop+3 mana page].)
* Ask team members for details on recording in the '''ReleaseEngineering''' room.

== Other Resources ==
=== Short-term ===
* [https://docs.google.com/document/d/1VcEjW82jBxr77aYi3TVaha9S4uJwAkwEapXnhhMwcgg RelEng crowd-sourced Glossary of Terms]
* Join https://mozillians.org/ (public, and yet-another-set-of-credentials)
** join the "release engineering" group
* [https://intranet.mozilla.org/ Mozilla intranet]
** https://intranet.mozilla.org/StaffMeeting
* https://mana.mozilla.org/wiki/display/RelEng/Release+Engineering+Home (for sensitive information only) (LDAP Note: Req attribute RelEng)
* https://trello.com/b/KwwYSXE1/release-engineering-status-board (Projects and deliverables status)
* Sign up for [https://wiki.mozilla.org/Safari_Books O'Reilly's online library] of their boooks - a great resource
* Overview of dev cycle: http://k0s.org/mozilla/workflow.svg (slightly dated). And browse http://k0s.org/mozilla
* [[ReleaseEngineering/Tips_And_Tricks|Releng Tips and Tricks]]

=== Longer-term ===
Talk to your mentor/manager to see which of these make sense.
* https://build.inventory.mozilla.org/build
* https://nagios.mozilla.org/nagios/
* https://metrics.mozilla.com/
* background for starting to understand buildbot releases (stale, but s/b roughly accurate): [http://people.mozilla.com/~bhearsum/mozilla-beta-schedulers.png Beta schedulers]
* How teams coordinate during a release: [http://moz-releng-docs.readthedocs.org/en/latest/release_workflows release work flows]

Firefox Desktop + Firefox Mobile release process docs:
* [[Release_Management/Release_Process]]
* [[Release_Management/Release_Process/FAQ]]
* [https://intranet.mozilla.org/TravelPolicies#Corporate_Travel_Accounts Egencia account]: this should be accessible via [https://mana.mozilla.org/wiki/display/SD/SSO+Quick+Links SSO]

ReleaseEngineering/How To/Restart Buildbot Masters

2017-03-28T23:44:20Z

NThomas: /* By script */ Update for bug 1340925

__TOC__
We occasionally need to restart buildbot masters for various reasons:
* upgrades to the underlying OS
* gradual increase in memory usage over time, leading to reduced master performance

= Manually =
If you need to restart a single master by hand, here's the sequence you should follow:
* disable the master in [https://secure.pub.build.mozilla.org/slavealloc/ui/#masters slavealloc]. This prevents the master from taking more slave connections while you're waiting for it to shutdown.
* click the "Clean Shutdown" button on the web interface for the given master, e.g. http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/
* wait for the jobs currently running on that master to complete. You can track progress by searching in-page for "Running" on the master's buildslaves page, e.g. http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/buildslaves?no_builders=1
* once the master is shutdown, perform whatever upgrades are required, etc.
* restart the master. '''NOTE:''' buildbot masters are configured to restart buildbot automatically on boot, so if you reboot the master, buildbot will restart itself. To restart manually:
xebec:buildduty ccooper$ ssh cltbld@buildbot-master82
Unauthorized access prohibited
[cltbld@buildbot-master82.bb.releng.scl3.mozilla.com ~]$ cd /builds/buildbot/build1/
[cltbld@buildbot-master82.bb.releng.scl3.mozilla.com build1]$ make start
* re-enable the master in [https://secure.pub.build.mozilla.org/slavealloc/ui/#masters slavealloc].

= By script =
The above actions have been encapsulated into a script: https://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/restart_masters.py. The script is setup to run on <tt>buildduty-tools.srv.releng.usw2.mozilla.com</tt>, located at <tt>/home/buildduty/restart_masters/</tt> under buildduty's account.

The wrapper script <tt>/home/buildduty/restart_masters.sh</tt> is used to update repositories and call restart_masters.py. The latter will prompt for the usernames and passwords it needs (ldap for slavealloc, cltbld and root for ssh).

Here is an example invocation:

# ssh -A buildduty@buildduty-tools
$ screen -R restart_masters
$ /home/buildduty/restart_masters.sh

You can enter nonsense for the cltbld and root passwords because key auth is used. Forwarding your ssh agent is required for ssh access to the masters. If you use a timeout make sure it's sufficiently long for the script to repeat, but don't leave it running indefinitely. '''NB: rebooting masters by adding the -r arg to restart_masters.py is non-functional because root logins are disabled.'''

The logs go to [https://papertrailapp.com/systems/255918104/events?q=restart_masters papertrail], outputting a progress report every 60 minutes. For really minimal view (which may hide errors) see this [https://papertrailapp.com/systems/255918104/events?q=restart_masters+-%22Connection+as+cltbld%22+-%22Attempting+to+connect+to%22+-%22Checking+shutdown+status%22+-lockfile+-Disabling+-Initiating+-%22Re-enabling%22+-%22Attempting+to+restart+master%22 filtered view].

You can send a SIGUSR1 to restart_masters.py to prompt an extra progress report; allow time for the current status check to complete first.

= Automated =
The above script requires sensitive credentials that shouldn't be stored on disk. For now, we're still running this script by hand.

ReleaseEngineering/Merge Duty/Steps

2017-02-26T23:10:50Z

NThomas: /* Bump bouncer entry versions */ add win64 locations for stub installer, bug 1342814

= Bump the buildbot-configs gecko version for mozilla-release =
* Bump the mozilla-release version [https://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/gecko_versions.json gecko_versions.json]
** Get review, land, and merge to production before the RC build is kicked off in Ship It.

= Bump the buildbot-configs gecko version for other branches =
* bump mozilla-central, mozilla-aurora, mozilla-beta, comm-beta, comm-aurora in [https://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/gecko_versions.json gecko_versions.json]
** get review, land when you're ready for the next reconfig to push this live.
** these need a reconfig to go live.

= Throttle Aurora nightly updates =
* https://wiki.mozilla.org/index.php?title=ReleaseEngineering/How_To/Enable_or_Disable_Updates_on_Aurora#Disabling_Updates

= Unthrottle Aurora nightly updates =
* Requires access to bounceradmin (ask rail/bhearsum/nthomas to add you)
* Get "go" email from Relman around the Friday of merge week. (There will be 2 QA emails, one each for desktop and mobile)
* https://wiki.mozilla.org/index.php?title=ReleaseEngineering/How_To/Enable_or_Disable_Updates_on_Aurora#Enabling_Updates
** After bouncer update for nightly and aurora, check if disk space needs cleanup per {{bug|703559}}

= Perform mozilla-beta -> mozilla-release migration =
* Get the list of l10n locales to remove (if required) from release management.
* be available in #r-d as there is often some chatter about merge day in there.

== Set up HG Extensions ==
Edit ~/.hgrc to include

[extensions]
transplant=
mq=
rebase=
graphlog=

== Close mozilla-beta ==
* Use https://mozilla-releng.net/treestatus to close mozilla-beta.
** Please use the "Remember previous state" option, so you can easily restore the previous status messages when you are finished (there will be a restore button at the bottom of the page).
** In order to change tree statuses, your browserID email needs to be added with sheriff permissions (ping edmorley/philor/catlee or ask in #developers).
** mozilla-beta will remain closed until after the Aurora->Beta uplift the following Monday (or a beta re-spin is necessary).

== Sync mozilla-beta to mozilla-release ==
Overview of the process:
* Mercurial repositories used for final releases will be synced from mozilla-beta.
* Tip of default on mozilla-release will be tagged and closed.
* mozilla-beta will be tagged and pulled. It creates new tip of default on mozilla-release.
* L10N repositories use noop (dummy) merge if needed to simplify localizers life.

=== Beta to Release Merge day helper Script ===
To handle this process use the following scripts:
Gecko migration script: [https://hg.mozilla.org/build/mozharness/file/production/scripts/merge_day/gecko_migration.py gecko_migration.py]

==== Running gecko_migration.py for mozilla-release ====
Example invocation:
# checkout mozharness from the gecko tree using archiver
wget https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/archiver_client.py
python archiver_client.py mozharness --destination mozharness-beta --repo releases/mozilla-beta --rev default --debug
# run the script
python mozharness-beta/scripts/merge_day/gecko_migration.py -c merge_day/beta_to_release.py
* the script will tag mozilla-beta
* it will continue by pulling mozilla-beta to mozilla-release, merging the heads in a fast-forward manner (using <tt>hg debugsetparents</tt>), adjusting shipped-locales and changing some configs.
* Once the script finishes, run an <tt>hg diff</tt> to see the uncommitted changes that the script generated:
hg -R build/mozilla-release diff
* It's best to get a visual review by someone else as well.
* Commit+push the final bits by
# Make sure you already pushed your release version bump and reconfiged before this part!
python mozharness-beta/scripts/merge_day/gecko_migration.py \
-c selfserve/production.py -c merge_day/beta_to_release.py \
--create-virtualenv --commit-changes --push
* The push should be available at https://treeherder.mozilla.org/#/jobs?repo=mozilla-release

In case of failure, you can start again from the top; no need to clobber (the on-by-default clean-repos action will be sufficient). It should be faster the second time, since you won't be pulling in as many changesets from hg.m.o.

==== Bump ESR version ====
This will bump the second digit in ESR's version. Repeat for all ESR that require the bump.

ESR_VERSION=45
# checkout mozharness from the gecko tree using archiver
wget https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/archiver_client.py
python archiver_client.py mozharness --destination mozharness-esr$ESR_VERSION --repo releases/mozilla-esr$ESR_VERSION --rev default --debug
# run the script
python mozharness-esr$ESR_VERSION/scripts/merge_day/gecko_migration.py -c mozharness-esr$ESR_VERSION/configs/merge_day/bump_esr.py
hg -R build/mozilla-esr$ESR_VERSION diff
python mozharness-esr$ESR_VERSION/scripts/merge_day/gecko_migration.py -c mozharness-esr$ESR_VERSION/configs/merge_day/bump_esr.py \
--commit-changes --push

==== Uplift locales ====
Helper script: [https://hg.mozilla.org/build/braindump/file/default/releases-related/beta2release_l10n.sh beta2release_l10n.sh]
* The script uses <tt>ffxbld_rsa</tt> key in the current directory (make sure to copy it and chmod 600) to push l10n repos on behalf on ffxbld user. It can be run from cruncher.
* The script is based on [https://hg.mozilla.org/users/axel_mozilla.com/branch-l10n/file/default/merge_l10n.py Axel's merge script]
Example invocation:
mkdir l10n && cd l10n
wget https://hg.mozilla.org/build/braindump/raw-file/default/releases-related/beta2release_l10n.sh
chmod 755 beta2release_l10n.sh
# copy ffxbld_rsa
chmod 600 ffxbld_rsa
./beta2release_l10n.sh
* It takes 30-45 minutes
* In case of failure it can be rerun and it will skip already cloned locales. If a locale failed while pushing, you can delete it to make the script process it again.

==== Notify r-d that the migration has completed! ====
By responding to the migration request.

= Perform mozilla-aurora -> mozilla-beta and mozilla-central -> mozilla-aurora migrations =

Gecko migration script : https://hg.mozilla.org/build/mozharness/file/production/scripts/merge_day/gecko_migration.py

== Wait for Go ==

Wait for an email to r-d with the subject: "Please bump & merge m-c, m-a, m-b", with the following contents:
* all locales confirmed (or a list of changes needed with bug numbers)
* no config changes needed (or a list of features to enable/disable with bug numbers)
* confirm that all IDL checks have been done, and that RelMan will ensure that any remaining IDL issues are addressed prior to go-to-build for m-a.

Ensure that you understand all of the requested changes.

== Set up HG Extensions ==

Edit ~/.hgrc to include

[extensions]
transplant=
mq=
rebase=
graphlog=

== Close mozilla-aurora ==
* Use https://mozilla-releng.net/treestatus to close the tree.
* Please use the "Remember previous state" option, so you can easily restore the previous status messages when you are finished.
* mozilla-beta should already be closed from the previous Monday.

== Running gecko_migration.py for mozilla-beta ==
You can run this in the exact same directory as you ran the beta_to_release migration.

Example invocation:
# checkout mozharness from the gecko tree using archiver
wget https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/archiver_client.py
python archiver_client.py mozharness --destination mozharness-aurora --repo releases/mozilla-aurora --rev default --debug
# run the script
python mozharness-aurora/scripts/merge_day/gecko_migration.py -c merge_day/aurora_to_beta.py
* the script will tag mozilla-aurora
* it will continue by pulling mozilla-aurora to mozilla-beta, merging the heads in a fast-forward manner (using <tt>hg debugsetparents</tt>), adjusting shipped-locales and changing some configs.
* run an <tt>hg -R build/mozilla-beta diff</tt> to view the uncommitted changes.
* Push the final bits by
python mozharness-aurora/scripts/merge_day/gecko_migration.py \
-c selfserve/production.py -c merge_day/aurora_to_beta.py \
--create-virtualenv --commit-changes --push --trigger-builders
* The push should be available at https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta

In case of failure, you can start again from the top; no need to clobber (the on-by-default clean-repos action will be sufficient). It should be faster the second time, since you won't be pulling in as many changesets from hg.m.o. If --trigger-builders fails, you can rerun the same script with only that action specified (no --commit-changes --push).

=== Leave mozilla-beta closed, hand it off to #sheriffs ===
* Best case, until [https://bugzilla.mozilla.org/show_bug.cgi?id=1303904 we can run gecko-decision on merges] we need another no-op push after the merge to see whether TaskCluster jobs are working; worst case it will take a day of fixing up to get the tree green.
* If there isn't an obvious sheriff on duty hanging around in #releng, shouting in #sheriffs should work.

== Running gecko_migration.py for mozilla-aurora ==
You can run this in the exact same directory as you ran the aurora_to_beta migration.

Example invocation:
# checkout mozharness from the gecko tree using archiver
wget https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/archiver_client.py
python archiver_client.py mozharness --destination mozharness-central --repo mozilla-central --rev default --debug
# run the script
python mozharness-central/scripts/merge_day/gecko_migration.py -c merge_day/central_to_aurora.py
* the script will tag mozilla-central
* it will continue by pulling mozilla-central to mozilla-aurora, merging the heads in a fast-forward manner (using <tt>hg debugsetparents</tt>), adjusting shipped-locales and changing some configs.
* Once the diff is shown and reviewed on screen, the script will commit the changes.
* Push the final bits by
scp cltbld@buildbot-master81.bb.releng.scl3.mozilla.com:/builds/buildbot/build_scheduler/master/BuildSlaves.py oauth.txt
python mozharness-central/scripts/merge_day/gecko_migration.py \
-c balrog/production.py -c selfserve/production.py \
-c merge_day/central_to_aurora.py \
--create-virtualenv --lock-update-paths --commit-changes \
--push --trigger-builders
* The push should be available at https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora
In case of failure, you can start again from the top; no need to clobber (the on-by-default clean-repos action will be sufficient). It should be faster the second time, since you won't be pulling in as many changesets from hg.m.o. If --trigger-builders fails, you can rerun the same script with only that action specified (no --commit-changes --push).
* Freeze aurora updates using [https://wiki.mozilla.org/index.php?title=ReleaseEngineering/How_To/Enable_or_Disable_Updates_on_Aurora#Disabling_Updates|this page] until <tt>--lock-update-paths</tt> is fixed ({{bug|1163998}})

== Change Template Version Numbers ==
Now that everything has been bumped edit the following so the Channel Notes and other wiki pages have the correct info and queries now (increment by 1). Also they are very important for daily cronjobs that read the values and nag many many engineers based on triage queries that use them.

The script is at https://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/update_merge_day_wiki.sh

If the ESR version is NOT changing,

export WIKI_USERNAME=asasaki
export WIKI_PASSWORD=*******
update_merge_day_wiki.sh

If the ESR version IS changing,

export WIKI_USERNAME=asasaki
export WIKI_PASSWORD=*******
NEW_ESR_VERSION=52
./update_merge_day_wiki.sh -e $NEW_ESR_VERSION

=== Double check that the wiki changes happened properly! ===

* [https://wiki.mozilla.org/Template:Version/Gecko/release/next NEXT_VERSION]
* [https://wiki.mozilla.org/Template:Version/Gecko/central/current CENTRAL_VERSION]
* [https://wiki.mozilla.org/Template:Version/Gecko/aurora/current AURORA_VERSION]
* [https://wiki.mozilla.org/Template:Version/Gecko/beta/current BETA_VERSION]
* [https://wiki.mozilla.org/Template:Version/Gecko/release/current RELEASE_VERSION]
* [https://wiki.mozilla.org/Template:Version/Gecko/esr/current ESR_VERSION] (every 7 releases)
* [https://wiki.mozilla.org/index.php?title=Template:NextReleaseDate NextReleaseDate]
** Which causes these to auto-update (with the magic of math!)
** [https://wiki.mozilla.org/index.php?title=Template:FIREFOX_SHIP_DATE] (the next ship date)
** [https://wiki.mozilla.org/index.php?title=Template:FIREFOX_MERGE_DATE] (the next merge date)
** [https://wiki.mozilla.org/index.php?title=Template:CURRENT_CYCLE] based upon [https://wiki.mozilla.org/RapidRelease/Calendar the calendar]

== Send Merge Completion Email ==
Send an email to r-d with a subject of "Bump & Merge m-c,m-a,m-b complete". Make sure to CC <tt>thunderbird-drivers@mozilla.org</tt>.

The contents should confirm that:
* wikimo version templates have been updated
* Notify qa-drivers that nighlty builds are in progress and aurora updates are frozen.

== Leave mozilla-aurora closed, hand it off to #sheriffs ==
* Best case, until [https://bugzilla.mozilla.org/show_bug.cgi?id=1303904 we can run gecko-decision on merges] we need another no-op push after the merge to see whether TaskCluster jobs are working; worst case it will take a day of fixing up to get the tree green.
* If there isn't an obvious sheriff on duty hanging around in #releng, shouting in #sheriffs should work.

== Notify interested people (Optional) ==
* Ping Callek and Pike in #r-d to warn about push

== Bump bouncer entry versions ==
* When we have nightlies from mozilla-central with the new version, update the bouncer locations for:
** [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=2005 firefox-nightly-latest]
** [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=6508 firefox-nightly-latest-ssl]
** [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=6506 firefox-nightly-latest-l10n]
** [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=6507 firefox-nightly-latest-l10n-ssl]
** [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=6509 firefox-nightly-stub]
** [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=6512 firefox-nightly-stub-l10n]

NB: it is expected that the two stub products have a win and win64 location which both point to the same location. We don't have a win64 stub installer, instead the mislabeled 32bit stub selects the correct full installer at runtime.

== Trim bouncer's Check Now list ==
* Once per release cycle we should stop checking old releases to see if they're present
* Visit the [https://bounceradmin.mozilla.com/admin/mirror/product/?all=&checknow__exact=1 list of check now enabled products]
* you should leave all current and upcoming releases enabled, as well as the updates for watershed releases. Typically this is about 100 products
* select all the old releases not covered above, and use the 'Remove Check Now on selected products' option in the Action dropdown

ReleaseEngineering/Archive/Enable or Disable Updates on Aurora

2017-02-26T23:05:37Z

NThomas: /* Update bouncer */ add win64 stub locations, bug 1342814

{{Release Engineering How To|Enable or Disable Updates on Aurora}}
When Nightly merges into Aurora every 6 weeks we lock Aurora updates at a specific old version for a period of time (days to a week) while QA runs some tests. Once their tests have passed, we re-enable updates. We're also occaisonly asked to disable updates temporarily for other reasons. This document describes how to do both of these things.

Note the magic rule ids below. These correspond to ids of rows in the "rules" table of Balrog. If we fix {{bug|1067402}} we can pass more intelligable values here.

= Disabling Updates =
NOTE: Disabling no longer has to be done by hand during uplifts. The uplift script does it.

= Enabling Updates =
Unlike disabling, Firefox and Fennec aren't always re-enabled at the same time. Make sure you are connected to releng VPN.

'''NOTE''': You may not be able to run these scripts unless you have a linux laptop. See {{bug|1151633}} for details and workarounds.

== Fennec ==
<pre>
cd /tmp
rm -rf tools
hg clone https://hg.mozilla.org/build/tools
cd tools/scripts/updates
scp cltbld@buildbot-master81.bb.releng.scl3.mozilla.com:/builds/buildbot/build_scheduler/master/BuildSlaves.py oauth.txt
# Run with dry run first, sanity check the "Would've unlocked" lines.
python2.7 balrog-nightly-locker.py -a https://aus4-admin.mozilla.org/api -c oauth.txt -u ffxbld -r 8 -n unlock
# Run without dry run to unlock the updates
python2.7 balrog-nightly-locker.py -a https://aus4-admin.mozilla.org/api -c oauth.txt -u ffxbld -r 8 unlock
</pre>

== Firefox ==
<pre>
cd /tmp
rm -rf tools
hg clone https://hg.mozilla.org/build/tools
cd tools/scripts/updates
scp cltbld@buildbot-master81.bb.releng.scl3.mozilla.com:/builds/buildbot/build_scheduler/master/BuildSlaves.py oauth.txt
# Run with dry run first, sanity check the "Would've unlocked" lines.
python2.7 balrog-nightly-locker.py -a https://aus4-admin.mozilla.org/api -c oauth.txt -u ffxbld -r 10 -n unlock
# Run without dry run to unlock the updates
python2.7 balrog-nightly-locker.py -a https://aus4-admin.mozilla.org/api -c oauth.txt -u ffxbld -r 10 unlock
</pre>

== Cleanup ==
<pre>
# preferably with srm or equivalent
srm oauth.txt
</pre>

== Update bouncer ==
When re-enable updates for Aurora, update the following bouncer locations to use the new version for aurora:
* [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=2006 firefox-aurora-latest]
* [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=3686 firefox-aurora-latest-ssl]
* [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=3685 firefox-aurora-latest-l10n]
* [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=6510 firefox-aurora-latest-l10n-ssl]
* [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=2061 firefox-aurora-stub]
* [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=6511 firefox-aurora-stub-l10n]

NB: it is expected that the two stub products have a win and win64 location which both point to the same location. We don't have a win64 stub installer, instead the mislabeled 32bit stub selects the correct full installer at runtime.

ReleaseEngineering/BuildAPI

2017-02-09T03:17:49Z

NThomas: /* BuildAPI */ formatting

{{Release Engineering How To|BuildAPI}}
= Overview =
The BuildAPI is a Pylons project used by RelEng to surface information collected from two databases updated through our buildbot masters as they run jobs.

= Project Requirements =
To run an instance of the buildAPI locally, you will need the following:
* Python with easy_install
* MySQL
* [http://pypi.python.org/pypi/MySQL-python/ MySQL-Python]
* [https://code.google.com/p/google-visualization-python/ Google-visualization-python]
* [http://memcached.org/ Memcached]
* Snapshots of our two databases 'schedulerdb' and 'statusdb'. They are available on Cruncher

It is highly recommended to set up and work within a python [[ReleaseEngineering/Virtualenv|virtualenv]].

If you are looking to setup a local virtual environment to run BuildAPI, then follow [[ReleaseEngineering/BuildAPI/Setup_Local_Virtualenv_for_BuildAPI|this wiki]].

= Setup =
Mana documentation:
* [https://mana.mozilla.org/wiki/display/IT/Redis Redis]
* [https://mana.mozilla.org/wiki/display/IT/BuildAPI BuildAPI]
* [https://mana.mozilla.org/wiki/display/IT/RelEng+RabbitMQ]

Puppet setup:
* [http://hg.mozilla.org/build/puppet/file/default/modules/selfserve_agent selfserve_agent]
* [http://hg.mozilla.org/build/puppet-manifests/file/default/modules/buildapi buildapi]
* [http://hg.mozilla.org/build/puppet-manifests/file/default/modules/redis redis]

= Getting Started =
The source for the buildapi is available here: [http://hg.mozilla.org/build/buildapi buildapi source]

Before you get started with that, you should setup your MySQL database instances.
To do that all you need to do is download, extract and load the sql dumps provided. So, create the databases first
<pre>
$ mysql -u <user> -p
<at prompt>
mysql>create database schedulerdb;
mysql>create database statusdb;
mysql>exit
</pre>

Download the sql dumps, then load them into the db with:
<pre>
mysql schedulerdb -u <user> -p < schedulerdb.sql
mysql statusdb -u <user> -p < statusdb.sql
</pre>

'''NOTE''': The files unzipped can account for more than 10GB. Watch out! :)
'''NOTE2''': The import can take a lot of time.

Now, to get started with the pylons project, simply run:
<pre>
python setup.py install
</pre>
In the buildapi/ directory. (And inside your virtualenv, if you're using one).
which should handle grabbing pylon project dependencies. You will also need to grab and install the google python visualization library from [http://code.google.com/p/google-visualization-python/ here].

Next you will need to generate a config file for your project, do this by running:
<pre>
paster make-config buildapi config.ini
</pre>

Now you will need to edit that config.ini to use the correct host(localhost should be fine), and database URLs.
Note: MySQL databases take the format `mysql://[username][:password][@hostname]/database_name`. For example:
sqlalchemy.scheduler_db.url = mysql://username:password@localhost/schedulerdb
sqlalchemy.status_db.url = mysql://username:password@localhost/statusdb

You should now be all set up now! Running the project locally can be done through:
<pre>
paster serve --reload --daemon config.ini
</pre>
Which starts running the buildapi on your local machine. To view it, open up http://localhost:5000.

= Building a simple controller =
The buildAPI is built on top of the pylons framework, which enforces a strict [http://en.wikipedia.org/wiki/Model-view-controller MVC] stack.
To add a new feature to the buildapi you will typically want to create a new controller and associate it with one or more models and views.

To create a simple 'hello world' controller, first use paster to create a template to work with.
<pre>
# in buildapi/
paster controller hello_world
</pre>
This automatically creates a workable template and a functional test case for your new controller.
You can now hack the newly created file under buildapi/controllers to your heart's content.
Models associated with your controller can be created under buildapi/model.
Views are created using Mako python templates. Create a template file under buildapi/templates for your new controller.

To associate with a model, simply add an import to your controller; for example:
<pre>
from buildapi.model.<model_name> import <functions to import>
</pre>

Lastly, once you're done editing your controller and have a resultset to publish, you probably want to render a page. To do this you can call render('/<template-name>.mako') from your controller body to render a view with your results. To access results from your controller, it is best to store the results in a pylons.tmpl_context object, which will make them available to your mako template.

===Simple code sample===
Now, to write a simple controller that prints "Hello World" using an M-V-C stack, we need three files:
* controllers/hello_world.py
* model/hello_world.py
* templates/hello_world.mako

Our controller lives in: controllers/hello_world.py
<pre>
import logging

from pylons import request, response, session, tmpl_context as c, url
from pylons.controllers.util import abort, redirect

from buildapi.lib.base import BaseController, render
from buildapi.model.hello_world import GetMessage

log = logging.getLogger(__name__)

class HelloController(BaseController):

def index(self):
# Return a rendered template
c.message = GetMessage()
return render('/hello_world.mako')
</pre>

Our model lives in: model/hello_world.py
<pre>
def GetMessage():
return "Hello world!"
</pre>

Our view lives in: templates/hello_world.mako
Note that the tmpl_context object is available in the template.
<pre>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<title>Hello World!</title>
</head>
<body>
<h1>${c.message}</h1>
</body>
</html>
</pre>

To access this page, we need to set up a route to it. To do this we add to config/routing.py:
<pre>
map.connect('/hello_world', controller='hello_world', action='index')
</pre>
Now restart buildapi, and point your browser to [http://localhost:5000/hello_world http://localhost:5000/hello_world].

= Maintenance =
== Updating code ==

BuildAPI has two components: BuildAPI and Selfserve Agent. They're deployed differently

=== BuildAPI ===

BuildAPI updates are handled like any other webapp, through pushes. See also [[ReleaseEngineering/How_To/Update_BuildAPI]]

The steps are roughly as follows, from the base directory of buildapi:
* Push new version of BuildAPI to HG
** Edit setup.py to show the updated version (eg version='0.3.0' becomes version='0.3.1')
* Build a new sdist tarball of BuildAPI, using normal python techniques, and upload it to the our pypi repository on relengwebadm.private.scl3.mozilla.com.
** Tarballs can be seen here: http://pypi.pub.build.mozilla.org/pub/
python setup.py sdist
VERSION=0.3.1
scp dist/buildapi-${VERSION}.tar.gz relengwebadm.private.scl3.mozilla.com:

ssh relengwebadm.private.scl3.mozilla.com
VERSION=0.3.1
sudo mv buildapi-${VERSION}.tar.gz /mnt/netapp/relengweb/pypi/pub/
sudo chmod 664 /mnt/netapp/relengweb/pypi/pub/buildapi-${VERSION}.tar.gz

* Deploy to the webheads, reusing the relengwebadm ssh session from above
sudo su -
cd /data/releng/src/buildapi # or for staging /data/releng-stage/src/buildapi
./update 0.3.1

=== Selfserve Agent ===

Deploying new functionality may require deploying a new version of the <code>scripts/selfserve_agent.py</code> file on build masters. This process is handled by puppet, but orthogonal to buildapi itself. The steps are roughly:

* bump the version of the buildapi repository in "<code>setup.py</code>" and commit (following semantic versioning guidelines, please)
* add a new tag to the buildapi repository for that version, commit & push
* create a python sdist tarball of that version:
** make sure you have a clean checkout
** locally modify (but do not commit) <code>setup.cfg</code> and comment out "<code>tag_build</code>"
** run "python setup.py "<code>sdist</code>"
** updload the file from "<code>dist/</code>"
* [[ReleaseEngineering/How_To/Add_a_Python_Package_to_PuppetAgain|upload to the internal puppetagain python]] repo.
* update the manifests for the new version in [http://hg.mozilla.org/build/puppet/file/default/modules/selfserve_agent/manifests/install.pp puppet]
* wait for puppet to deploy the new agent.

== Adding branches ==
buildapi and self-serve pull from [http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json] to determine which branches are active and caches the result for a few minutes.

== Kicking ==
If things are broken, figure out if it's
* self-serve requests not completing (check the self-serve agents and rabbit connections)
* buildapi HTTP requests not working (check buildapi - [[ReleaseEngineering/How_To/Restart_BuildAPI]])
* builds-4hr.js not being updated (check the [[ReleaseEngineering/BuildAPI#report_crontask|report crontask]] report crontask)

=== report crontask ===

There's a crontask on [https://mana.mozilla.org/wiki/display/websites/Releng+Cluster relengwebadm] that runs every minute to generate the builds-4hr.js file. If this crontask gets "hung", it will prevent updates from occuring. This can happen due to DB connection issues, or bad data in the DB. In these cases, killing the hung crontask is the appropriate fix. There may be a stale lockfile that need removing under /var/lock/buildapi too.

However, the crontask takes about 15 minutes to run on a cold cache. So if its cache (memcached) has gone cold, then killing it before 15 minutes have elapsed is only delaying the failure.

From [https://bugzilla.mozilla.org/show_bug.cgi?id=1005342#c6 bug 1005342]:

{| class="wikitable"
|-
| ''Dustin J. Mitchell [:dustin] 2014-05-02 16:18:14 PDT''

I looked at 4:01 pacific, and saw that builds-4hr.js.gz had last been generated at 2:47. This is generated by a crontab that runs every minute, and makes a lot of heavy queries against the DB. On a good day with a cold cache, it's about 15 minutes. With a hot cache it can finish comfortably in one minute. It would seem that Matt's heavy query caused the builds-4hr.js.gz queries to go slowly, and thus caused the file's generation to fall behind.

I found the crontask, appropriately dated 2:47, and killed it. Cron started a new one up on the minute, and that completed in something like 45s.

I don't know why the original job didn't complete soon after mpressman cancelled his query.
|}

= Staging =
You can visit https://secure-pub-build.allizom.org/buildapi

From Dustin:
Buildapi itself doesn't write anything but jobrequests. For the staging instance, those go to a different DB than for production. Like the production version, it reads from the production status and scheduler db's.

The bit that it's missing is a running selfserve-agent. If you trigger some action on the staging instance, it pushes a message to the staging rabbitmq virtualhost and waits for a response, but since there's no agent that response never comes.

= Development environment =
Running your own instance of BuildAPI is relatively simple. You should:
* Create a virtualenv ([[ReleaseEngineering/BuildAPI/Setup_Local_Virtualenv_for_BuildAPI|details]])
* Install buildapi with its setup.py script
* Create a config. You may base your config on the production one, with at least the following changes:
** Change DEFAULT.email_to to your own email address
** Change server:main.port to 0.0.0.0
** Change server:main.port to something unused
** Comment out app:main.carrot*
** Set app:main.buildapi.cache to nothing

= Troubleshooting Tips =
== Host ==
NOTE: buildapi01 is largely unused now and the hosts are managed by webops

==No MySQLdb==
If after installing you run:
<pre>
paster serve --reload --daemon config.ini
</pre>
and it does not start the server, check paster.log and if you see "ImportError: No module named MySQLdb" then you need to easy_install MySQL-python into the site-packages of the python it's looking for MySQLdb in.

ReleaseEngineering/BuildAPI

2017-02-09T03:17:04Z

NThomas: /* BuildAPI */ Update for changes in bug 1318969

{{Release Engineering How To|BuildAPI}}
= Overview =
The BuildAPI is a Pylons project used by RelEng to surface information collected from two databases updated through our buildbot masters as they run jobs.

= Project Requirements =
To run an instance of the buildAPI locally, you will need the following:
* Python with easy_install
* MySQL
* [http://pypi.python.org/pypi/MySQL-python/ MySQL-Python]
* [https://code.google.com/p/google-visualization-python/ Google-visualization-python]
* [http://memcached.org/ Memcached]
* Snapshots of our two databases 'schedulerdb' and 'statusdb'. They are available on Cruncher

It is highly recommended to set up and work within a python [[ReleaseEngineering/Virtualenv|virtualenv]].

If you are looking to setup a local virtual environment to run BuildAPI, then follow [[ReleaseEngineering/BuildAPI/Setup_Local_Virtualenv_for_BuildAPI|this wiki]].

= Setup =
Mana documentation:
* [https://mana.mozilla.org/wiki/display/IT/Redis Redis]
* [https://mana.mozilla.org/wiki/display/IT/BuildAPI BuildAPI]
* [https://mana.mozilla.org/wiki/display/IT/RelEng+RabbitMQ]

Puppet setup:
* [http://hg.mozilla.org/build/puppet/file/default/modules/selfserve_agent selfserve_agent]
* [http://hg.mozilla.org/build/puppet-manifests/file/default/modules/buildapi buildapi]
* [http://hg.mozilla.org/build/puppet-manifests/file/default/modules/redis redis]

= Getting Started =
The source for the buildapi is available here: [http://hg.mozilla.org/build/buildapi buildapi source]

Before you get started with that, you should setup your MySQL database instances.
To do that all you need to do is download, extract and load the sql dumps provided. So, create the databases first
<pre>
$ mysql -u <user> -p
<at prompt>
mysql>create database schedulerdb;
mysql>create database statusdb;
mysql>exit
</pre>

Download the sql dumps, then load them into the db with:
<pre>
mysql schedulerdb -u <user> -p < schedulerdb.sql
mysql statusdb -u <user> -p < statusdb.sql
</pre>

'''NOTE''': The files unzipped can account for more than 10GB. Watch out! :)
'''NOTE2''': The import can take a lot of time.

Now, to get started with the pylons project, simply run:
<pre>
python setup.py install
</pre>
In the buildapi/ directory. (And inside your virtualenv, if you're using one).
which should handle grabbing pylon project dependencies. You will also need to grab and install the google python visualization library from [http://code.google.com/p/google-visualization-python/ here].

Next you will need to generate a config file for your project, do this by running:
<pre>
paster make-config buildapi config.ini
</pre>

Now you will need to edit that config.ini to use the correct host(localhost should be fine), and database URLs.
Note: MySQL databases take the format `mysql://[username][:password][@hostname]/database_name`. For example:
sqlalchemy.scheduler_db.url = mysql://username:password@localhost/schedulerdb
sqlalchemy.status_db.url = mysql://username:password@localhost/statusdb

You should now be all set up now! Running the project locally can be done through:
<pre>
paster serve --reload --daemon config.ini
</pre>
Which starts running the buildapi on your local machine. To view it, open up http://localhost:5000.

= Building a simple controller =
The buildAPI is built on top of the pylons framework, which enforces a strict [http://en.wikipedia.org/wiki/Model-view-controller MVC] stack.
To add a new feature to the buildapi you will typically want to create a new controller and associate it with one or more models and views.

To create a simple 'hello world' controller, first use paster to create a template to work with.
<pre>
# in buildapi/
paster controller hello_world
</pre>
This automatically creates a workable template and a functional test case for your new controller.
You can now hack the newly created file under buildapi/controllers to your heart's content.
Models associated with your controller can be created under buildapi/model.
Views are created using Mako python templates. Create a template file under buildapi/templates for your new controller.

To associate with a model, simply add an import to your controller; for example:
<pre>
from buildapi.model.<model_name> import <functions to import>
</pre>

Lastly, once you're done editing your controller and have a resultset to publish, you probably want to render a page. To do this you can call render('/<template-name>.mako') from your controller body to render a view with your results. To access results from your controller, it is best to store the results in a pylons.tmpl_context object, which will make them available to your mako template.

===Simple code sample===
Now, to write a simple controller that prints "Hello World" using an M-V-C stack, we need three files:
* controllers/hello_world.py
* model/hello_world.py
* templates/hello_world.mako

Our controller lives in: controllers/hello_world.py
<pre>
import logging

from pylons import request, response, session, tmpl_context as c, url
from pylons.controllers.util import abort, redirect

from buildapi.lib.base import BaseController, render
from buildapi.model.hello_world import GetMessage

log = logging.getLogger(__name__)

class HelloController(BaseController):

def index(self):
# Return a rendered template
c.message = GetMessage()
return render('/hello_world.mako')
</pre>

Our model lives in: model/hello_world.py
<pre>
def GetMessage():
return "Hello world!"
</pre>

Our view lives in: templates/hello_world.mako
Note that the tmpl_context object is available in the template.
<pre>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<title>Hello World!</title>
</head>
<body>
<h1>${c.message}</h1>
</body>
</html>
</pre>

To access this page, we need to set up a route to it. To do this we add to config/routing.py:
<pre>
map.connect('/hello_world', controller='hello_world', action='index')
</pre>
Now restart buildapi, and point your browser to [http://localhost:5000/hello_world http://localhost:5000/hello_world].

= Maintenance =
== Updating code ==

BuildAPI has two components: BuildAPI and Selfserve Agent. They're deployed differently

=== BuildAPI ===

BuildAPI updates are handled like any other webapp, through pushes. See also [[ReleaseEngineering/How_To/Update_BuildAPI]]

The steps are roughly as follows, from the base directory of buildapi:
* Push new version of BuildAPI to HG
** Edit setup.py to show the updated version (eg version='0.3.0' becomes version='0.3.1')
* Build a new sdist tarball of BuildAPI, using normal python techniques, and upload it to the our pypi repository on relengwebadm.private.scl3.mozilla.com.
** Tarballs can be seen here: http://pypi.pub.build.mozilla.org/pub/
python setup.py sdist
VERSION=0.3.1
scp dist/buildapi-${VERSION}.tar.gz relengwebadm.private.scl3.mozilla.com:
ssh relengwebadm.private.scl3.mozilla.com

VERSION=0.3.1
sudo mv buildapi-${VERSION}.tar.gz /mnt/netapp/relengweb/pypi/pub/
sudo chmod 664 /mnt/netapp/relengweb/pypi/pub/buildapi-${VERSION}.tar.gz

* Deploy to the webheads, reusing the relengwebadm ssh session from above
sudo su -
cd /data/releng/src/buildapi # or for staging /data/releng-stage/src/buildapi
./update 0.3.1

=== Selfserve Agent ===

Deploying new functionality may require deploying a new version of the <code>scripts/selfserve_agent.py</code> file on build masters. This process is handled by puppet, but orthogonal to buildapi itself. The steps are roughly:

* bump the version of the buildapi repository in "<code>setup.py</code>" and commit (following semantic versioning guidelines, please)
* add a new tag to the buildapi repository for that version, commit & push
* create a python sdist tarball of that version:
** make sure you have a clean checkout
** locally modify (but do not commit) <code>setup.cfg</code> and comment out "<code>tag_build</code>"
** run "python setup.py "<code>sdist</code>"
** updload the file from "<code>dist/</code>"
* [[ReleaseEngineering/How_To/Add_a_Python_Package_to_PuppetAgain|upload to the internal puppetagain python]] repo.
* update the manifests for the new version in [http://hg.mozilla.org/build/puppet/file/default/modules/selfserve_agent/manifests/install.pp puppet]
* wait for puppet to deploy the new agent.

== Adding branches ==
buildapi and self-serve pull from [http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json] to determine which branches are active and caches the result for a few minutes.

== Kicking ==
If things are broken, figure out if it's
* self-serve requests not completing (check the self-serve agents and rabbit connections)
* buildapi HTTP requests not working (check buildapi - [[ReleaseEngineering/How_To/Restart_BuildAPI]])
* builds-4hr.js not being updated (check the [[ReleaseEngineering/BuildAPI#report_crontask|report crontask]] report crontask)

=== report crontask ===

There's a crontask on [https://mana.mozilla.org/wiki/display/websites/Releng+Cluster relengwebadm] that runs every minute to generate the builds-4hr.js file. If this crontask gets "hung", it will prevent updates from occuring. This can happen due to DB connection issues, or bad data in the DB. In these cases, killing the hung crontask is the appropriate fix. There may be a stale lockfile that need removing under /var/lock/buildapi too.

However, the crontask takes about 15 minutes to run on a cold cache. So if its cache (memcached) has gone cold, then killing it before 15 minutes have elapsed is only delaying the failure.

From [https://bugzilla.mozilla.org/show_bug.cgi?id=1005342#c6 bug 1005342]:

{| class="wikitable"
|-
| ''Dustin J. Mitchell [:dustin] 2014-05-02 16:18:14 PDT''

I looked at 4:01 pacific, and saw that builds-4hr.js.gz had last been generated at 2:47. This is generated by a crontab that runs every minute, and makes a lot of heavy queries against the DB. On a good day with a cold cache, it's about 15 minutes. With a hot cache it can finish comfortably in one minute. It would seem that Matt's heavy query caused the builds-4hr.js.gz queries to go slowly, and thus caused the file's generation to fall behind.

I found the crontask, appropriately dated 2:47, and killed it. Cron started a new one up on the minute, and that completed in something like 45s.

I don't know why the original job didn't complete soon after mpressman cancelled his query.
|}

= Staging =
You can visit https://secure-pub-build.allizom.org/buildapi

From Dustin:
Buildapi itself doesn't write anything but jobrequests. For the staging instance, those go to a different DB than for production. Like the production version, it reads from the production status and scheduler db's.

The bit that it's missing is a running selfserve-agent. If you trigger some action on the staging instance, it pushes a message to the staging rabbitmq virtualhost and waits for a response, but since there's no agent that response never comes.

= Development environment =
Running your own instance of BuildAPI is relatively simple. You should:
* Create a virtualenv ([[ReleaseEngineering/BuildAPI/Setup_Local_Virtualenv_for_BuildAPI|details]])
* Install buildapi with its setup.py script
* Create a config. You may base your config on the production one, with at least the following changes:
** Change DEFAULT.email_to to your own email address
** Change server:main.port to 0.0.0.0
** Change server:main.port to something unused
** Comment out app:main.carrot*
** Set app:main.buildapi.cache to nothing

= Troubleshooting Tips =
== Host ==
NOTE: buildapi01 is largely unused now and the hosts are managed by webops

==No MySQLdb==
If after installing you run:
<pre>
paster serve --reload --daemon config.ini
</pre>
and it does not start the server, check paster.log and if you see "ImportError: No module named MySQLdb" then you need to easy_install MySQL-python into the site-packages of the python it's looking for MySQLdb in.

ReleaseEngineering/How To/Add New AWS Subnets

2017-01-19T22:32:41Z

NThomas: typo fix

{{Release Engineering How To|Add_New_AWS_Subnets}}

Below you can find the steps to add new aws subnets

== Create new subnets ==
* Add subnets to https://github.com/mozilla/build-cloud-tools/blob/master/configs/subnets.yml
** run `scripts/aws_manage_subnets.py configs/subnets.yml` from aws-manager server to create the subnets in VPC

Please check in AWS the new VPC subnets

More details {{bug|1165432}} or {{bug|1239263}} and notes here https://github.com/mozilla/build-cloud-tools/pull/72

== Update firewall rules ==
* Open bugs with netops to include new flows, eg {{bug|1254144}}
* Add the subnets to https://github.com/mozilla/build-cloud-tools/blob/master/configs/securitygroups.yml
** run `scripts/aws_manage_securitygroups.py configs/securitygroups.yml` from aws-manager server to create the security groups

'''NOTE:''' After the firewall rules has been added by netops team and the new subnets in securitygroups.yml, the changes will be applied overnight. Or you can ask dustin to run the command manually. (We don't have permissions to sudo to fwunit)

<pre>
dmitchell@fwunit1 ~/firewall-tests [master] $ sudo su - fwunit
-sh-4.1$ umask 022
-sh-4.1$ cd /opt/fwunit/releng/src/tests/
-sh-4.1$ source /opt/fwunit/releng/bin/activate
(releng)-sh-4.1$ fwunit aws_releng
[2016-03-09 14:19:09,178] running aws_releng
...
(releng)-sh-4.1$ fwunit releng
...
</pre>

* Update firewall tests to include new subnets (private repo releng/firewall-tests) and ensure tests pass after netops has made the changes ( more details https://mana.mozilla.org/wiki/display/IT/How+to+Run+Flow+Tests )

== Increase size of the pool ==
* Updated the size limit https://github.com/ciobancai/build-cloud-tools/blob/master/configs/watch_pending.cfg

== Add the new platforms in build-cloud-tools==
=== tst-linux64 ===
* Add the new subnets for tst-linux64: https://github.com/mozilla/build-cloud-tools/blob/master/configs/tst-linux64

ReleaseEngineering/Merge Duty/Steps

2017-01-19T04:04:09Z

NThomas: /* Trim bouncer's Check Now list */ mention watershed updates too

= Bump the buildbot-configs gecko version for mozilla-release =
* Bump the mozilla-release version [https://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/gecko_versions.json gecko_versions.json]
** Get review, land, and merge to production before the RC build is kicked off in Ship It.

= Bump the buildbot-configs gecko version for other branches =
* bump mozilla-central, mozilla-aurora, mozilla-beta, comm-beta, comm-aurora in [https://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/gecko_versions.json gecko_versions.json]
** get review, land when you're ready for the next reconfig to push this live.
** these need a reconfig to go live.

= Throttle Aurora nightly updates =
* https://wiki.mozilla.org/index.php?title=ReleaseEngineering/How_To/Enable_or_Disable_Updates_on_Aurora#Disabling_Updates

= Unthrottle Aurora nightly updates =
* Requires access to bounceradmin (ask rail/bhearsum/nthomas to add you)
* Get "go" email from Relman around the Friday of merge week. (There will be 2 QA emails, one each for desktop and mobile)
* https://wiki.mozilla.org/index.php?title=ReleaseEngineering/How_To/Enable_or_Disable_Updates_on_Aurora#Enabling_Updates
** After bouncer update for nightly and aurora, check if disk space needs cleanup per {{bug|703559}}

= Perform mozilla-beta -> mozilla-release migration =
* Get the list of l10n locales to remove (if required) from release management.
* be available in #r-d as there is often some chatter about merge day in there.

== Set up HG Extensions ==
Edit ~/.hgrc to include

[extensions]
transplant=
mq=
rebase=
graphlog=

== Close mozilla-beta ==
* Use https://treestatus.mozilla.org/ to close mozilla-beta.
** Please use the "Remember previous state" option, so you can easily restore the previous status messages when you are finished (there will be a restore button at the bottom of the page).
** In order to change tree statuses, your browserID email needs to be added with sheriff permissions (ping edmorley/philor/catlee or ask in #developers).
** mozilla-beta will remain closed until after the Aurora->Beta uplift the following Monday (or a beta re-spin is necessary).

== Sync mozilla-beta to mozilla-release ==
Overview of the process:
* Mercurial repositories used for final releases will be synced from mozilla-beta.
* Tip of default on mozilla-release will be tagged and closed.
* mozilla-beta will be tagged and pulled. It creates new tip of default on mozilla-release.
* L10N repositories use noop (dummy) merge if needed to simplify localizers life.

=== Beta to Release Merge day helper Script ===
To handle this process use the following scripts:
Gecko migration script: [http://hg.mozilla.org/build/mozharness/file/production/scripts/merge_day/gecko_migration.py gecko_migration.py]

==== Running gecko_migration.py for mozilla-release ====
Example invocation:
# checkout mozharness from the gecko tree using archiver
wget http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/archiver_client.py
python archiver_client.py mozharness --repo mozilla-central --rev default --debug
# run the script
python mozharness/scripts/merge_day/gecko_migration.py -c mozharness/configs/merge_day/beta_to_release.py
* the script will tag mozilla-beta
* it will continue by pulling mozilla-beta to mozilla-release, merging the heads in a fast-forward manner (using <tt>hg debugsetparents</tt>), adjusting shipped-locales and changing some configs.
* Once the script finishes, run an <tt>hg diff</tt> to see the uncommitted changes that the script generated:
hg -R build/mozilla-release diff
* It's best to get a visual review by someone else as well.
* Commit+push the final bits by
# Make sure you already pushed your release version bump and reconfiged before this part!
python mozharness/scripts/merge_day/gecko_migration.py \
-c mozharness/configs/selfserve/production.py \
-c mozharness/configs/merge_day/beta_to_release.py \
--create-virtualenv --commit-changes --push
* The push should be available at https://treeherder.mozilla.org/#/jobs?repo=mozilla-release

In case of failure, you can start again from the top; no need to clobber (the on-by-default clean-repos action will be sufficient). It should be faster the second time, since you won't be pulling in as many changesets from hg.m.o.

==== Bump ESR version ====
This will bump the second digit in ESR's version.
python mozharness/scripts/merge_day/gecko_migration.py -c mozharness/configs/merge_day/bump_esr.py
hg -R build/mozilla-esr45 diff
python mozharness/scripts/merge_day/gecko_migration.py -c mozharness/configs/merge_day/bump_esr.py \
--commit-changes --push

==== Uplift locales ====
Helper script: [http://hg.mozilla.org/build/braindump/file/default/releases-related/beta2release_l10n.sh beta2release_l10n.sh]
* The script uses <tt>ffxbld_rsa</tt> key in the current directory (make sure to copy it and chmod 600) to push l10n repos on behalf on ffxbld user. It can be run from cruncher.
* The script is based on [http://hg.mozilla.org/users/axel_mozilla.com/branch-l10n/file/default/merge_l10n.py Axel's merge script]
Example invocation:
mkdir l10n && cd l10n
wget http://hg.mozilla.org/build/braindump/raw-file/default/releases-related/beta2release_l10n.sh
chmod 755 beta2release_l10n.sh
# copy ffxbld_rsa
chmod 600 ffxbld_rsa
./beta2release_l10n.sh
* It takes 30-45 minutes
* In case of failure it can be rerun and it will skip already cloned locales. If a locale failed while pushing, you can delete it to make the script process it again.

==== Notify r-d that the migration has completed! ====
By responding to the migration request.

= Perform mozilla-aurora -> mozilla-beta and mozilla-central -> mozilla-aurora migrations =

Gecko migration script : http://hg.mozilla.org/build/mozharness/file/production/scripts/merge_day/gecko_migration.py

== Wait for Go ==

Wait for an email to r-d with the subject: "Please bump & merge m-c, m-a, m-b", with the following contents:
* all locales confirmed (or a list of changes needed with bug numbers)
* no config changes needed (or a list of features to enable/disable with bug numbers)
* confirm that all IDL checks have been done, and that RelMan will ensure that any remaining IDL issues are addressed prior to go-to-build for m-a.

Ensure that you understand all of the requested changes.

== Set up HG Extensions ==

Edit ~/.hgrc to include

[extensions]
transplant=
mq=
rebase=
graphlog=

== Close mozilla-aurora ==
* Use https://treestatus.mozilla.org/ to close the tree.
* Please use the "Remember previous state" option, so you can easily restore the previous status messages when you are finished.
* mozilla-beta should already be closed from the previous Monday.

== Running gecko_migration.py for mozilla-beta ==
You can run this in the exact same directory as you ran the beta_to_release migration.

Example invocation:
# checkout mozharness from the gecko tree using archiver
wget http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/archiver_client.py
python archiver_client.py mozharness --repo mozilla-central --rev default --debug
# run the script
python mozharness/scripts/merge_day/gecko_migration.py -c mozharness/configs/merge_day/aurora_to_beta.py
* the script will tag mozilla-aurora
* it will continue by pulling mozilla-aurora to mozilla-beta, merging the heads in a fast-forward manner (using <tt>hg debugsetparents</tt>), adjusting shipped-locales and changing some configs.
* run an <tt>hg -R build/mozilla-beta diff</tt> to view the uncommitted changes.
* Push the final bits by
python mozharness/scripts/merge_day/gecko_migration.py \
-c mozharness/configs/selfserve/production.py \
-c mozharness/configs/merge_day/aurora_to_beta.py \
--create-virtualenv --commit-changes --push --trigger-builders
* The push should be available at https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta

In case of failure, you can start again from the top; no need to clobber (the on-by-default clean-repos action will be sufficient). It should be faster the second time, since you won't be pulling in as many changesets from hg.m.o. If --trigger-builders fails, you can rerun the same script with only that action specified (no --commit-changes --push).

=== Leave mozilla-beta closed, hand it off to #sheriffs ===
* Best case, until [https://bugzilla.mozilla.org/show_bug.cgi?id=1303904 we can run gecko-decision on merges] we need another no-op push after the merge to see whether TaskCluster jobs are working; worst case it will take a day of fixing up to get the tree green.
* If there isn't an obvious sheriff on duty hanging around in #releng, shouting in #sheriffs should work.

== Running gecko_migration.py for mozilla-aurora ==
You can run this in the exact same directory as you ran the aurora_to_beta migration.

Example invocation:
# checkout mozharness from the gecko tree using archiver
wget http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/archiver_client.py
python archiver_client.py mozharness --repo mozilla-central --rev default --debug
# run the script
python mozharness/scripts/merge_day/gecko_migration.py -c mozharness/configs/merge_day/central_to_aurora.py
* the script will tag mozilla-central
* it will continue by pulling mozilla-central to mozilla-aurora, merging the heads in a fast-forward manner (using <tt>hg debugsetparents</tt>), adjusting shipped-locales and changing some configs.
* Once the diff is shown and reviewed on screen, the script will commit the changes.
* Push the final bits by
scp cltbld@buildbot-master81.bb.releng.scl3.mozilla.com:/builds/buildbot/build_scheduler/master/BuildSlaves.py oauth.txt
python mozharness/scripts/merge_day/gecko_migration.py \
-c mozharness/configs/balrog/production.py \
-c mozharness/configs/selfserve/production.py \
-c mozharness/configs/merge_day/central_to_aurora.py \
--create-virtualenv --lock-update-paths --commit-changes \
--push --trigger-builders
* The push should be available at https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora
In case of failure, you can start again from the top; no need to clobber (the on-by-default clean-repos action will be sufficient). It should be faster the second time, since you won't be pulling in as many changesets from hg.m.o. If --trigger-builders fails, you can rerun the same script with only that action specified (no --commit-changes --push).
* Freeze aurora updates using [https://wiki.mozilla.org/index.php?title=ReleaseEngineering/How_To/Enable_or_Disable_Updates_on_Aurora#Disabling_Updates|this page] until <tt>--lock-update-paths</tt> is fixed ({{bug|1163998}})

== Change Template Version Numbers ==
Now that everything has been bumped edit the following so the Channel Notes and other wiki pages have the correct info and queries now (increment by 1). Also they are very important for daily cronjobs that read the values and nag many many engineers based on triage queries that use them.

The script is at http://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/update_merge_day_wiki.sh

If the ESR version is NOT changing,

export WIKI_USERNAME=asasaki
export WIKI_PASSWORD=*******
update_merge_day_wiki.sh

If the ESR version IS changing,

export WIKI_USERNAME=asasaki
export WIKI_PASSWORD=*******
NEW_ESR_VERSION=52
./update_merge_day_wiki.sh -e $NEW_ESR_VERSION

=== Double check that the wiki changes happened properly! ===

* [https://wiki.mozilla.org/Template:Version/Gecko/release/next NEXT_VERSION]
* [https://wiki.mozilla.org/Template:Version/Gecko/central/current CENTRAL_VERSION]
* [https://wiki.mozilla.org/Template:Version/Gecko/aurora/current AURORA_VERSION]
* [https://wiki.mozilla.org/Template:Version/Gecko/beta/current BETA_VERSION]
* [https://wiki.mozilla.org/Template:Version/Gecko/release/current RELEASE_VERSION]
* [https://wiki.mozilla.org/Template:Version/Gecko/esr/current ESR_VERSION] (every 7 releases)
* [https://wiki.mozilla.org/index.php?title=Template:NextReleaseDate NextReleaseDate]
** Which causes these to auto-update (with the magic of math!)
** [https://wiki.mozilla.org/index.php?title=Template:FIREFOX_SHIP_DATE] (the next ship date)
** [https://wiki.mozilla.org/index.php?title=Template:FIREFOX_MERGE_DATE] (the next merge date)
** [https://wiki.mozilla.org/index.php?title=Template:CURRENT_CYCLE] based upon [https://wiki.mozilla.org/RapidRelease/Calendar the calendar]

== Send Merge Completion Email ==
Send an email to r-d with a subject of "Bump & Merge m-c,m-a,m-b complete". Make sure to CC <tt>thunderbird-drivers@mozilla.org</tt>.

The contents should confirm that:
* wikimo version templates have been updated
* Notify qa-drivers that nighlty builds are in progress and aurora updates are frozen.

== Leave mozilla-aurora closed, hand it off to #sheriffs ==
* Best case, until [https://bugzilla.mozilla.org/show_bug.cgi?id=1303904 we can run gecko-decision on merges] we need another no-op push after the merge to see whether TaskCluster jobs are working; worst case it will take a day of fixing up to get the tree green.
* If there isn't an obvious sheriff on duty hanging around in #releng, shouting in #sheriffs should work.

== Notify interested people (Optional) ==
* Ping Callek and Pike in #r-d to warn about push

== Bump bouncer entry versions ==
* When we have nightlies from mozilla-central with the new version, update the bouncer locations for:
** [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=2005 firefox-nightly-latest]
** [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=6508 firefox-nightly-latest-ssl]
** [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=6506 firefox-nightly-latest-l10n]
** [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=6507 firefox-nightly-latest-l10n-ssl]
** [https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=6509 firefox-nightly-stub]
** [https://bounceradmin.mozilla.com/admin/mirror/location/33419/ firefox-nightly-stub-l10n]

== Trim bouncer's Check Now list ==
* Once per release cycle we should stop checking old releases to see if they're present
* Visit the [https://bounceradmin.mozilla.com/admin/mirror/product/?all=&checknow__exact=1 list of check now enabled products]
* you should leave all current and upcoming releases enabled, as well as the updates for watershed releases. Typically this is about 100 products
* select all the old releases not covered above, and use the 'Remove Check Now on selected products' option in the Action dropdown

ReleaseEngineering/How To/Add New AWS Subnets

2017-01-09T21:08:10Z

NThomas: /* Update firewall rules */ typos

{{Release Engineering How To|Add_New_AWS_Subnets}}

Below you can find the steps to add new aws subnests

== Create new subnets ==
* Add subnets to https://github.com/mozilla/build-cloud-tools/blob/master/configs/subnets.yml
** run `scripts/aws_manage_subnets.py configs/subnets.yml` from aws-manager server to create the subnets in VPC

Please check in AWS the new VPC subnets

More details {{bug|1165432}} or {{bug|1239263}} and notes here https://github.com/mozilla/build-cloud-tools/pull/72

== Update firewall rules ==
* Open bugs with netops to include new flows, eg {{bug|1254144}}
* Add the subnets to https://github.com/mozilla/build-cloud-tools/blob/master/configs/securitygroups.yml
** run `scripts/aws_manage_securitygroups.py configs/securitygroups.yml` from aws-manager server to create the security groups

'''NOTE:''' After the firewall rules has been added by netops team and the new subnets in securitygroups.yml, the changes will be applied overnight. Or you can ask dustin to run the command manually. (We don't have permissions to sudo to fwunit)

<pre>
dmitchell@fwunit1 ~/firewall-tests [master] $ sudo su - fwunit
-sh-4.1$ umask 022
-sh-4.1$ cd /opt/fwunit/releng/src/tests/
-sh-4.1$ source /opt/fwunit/releng/bin/activate
(releng)-sh-4.1$ fwunit aws_releng
[2016-03-09 14:19:09,178] running aws_releng
...
(releng)-sh-4.1$ fwunit releng
...
</pre>

* Update firewall tests to include new subnets (private repo releng/firewall-tests) and ensure tests pass after netops has made the changes ( more details https://mana.mozilla.org/wiki/display/IT/How+to+Run+Flow+Tests )

== Increase size of the pool ==
* Updated the size limit https://github.com/ciobancai/build-cloud-tools/blob/master/configs/watch_pending.cfg

== Add the new platforms in build-cloud-tools==
=== tst-linux64 ===
* Add the new subnets for tst-linux64: https://github.com/mozilla/build-cloud-tools/blob/master/configs/tst-linux64

ReleaseEngineering/BuildAPI

2016-11-20T23:04:25Z

NThomas: /* BuildAPI */ Update for the passage of time

{{Release Engineering How To|BuildAPI}}
= Overview =
The BuildAPI is a Pylons project used by RelEng to surface information collected from two databases updated through our buildbot masters as they run jobs.

= Project Requirements =
To run an instance of the buildAPI locally, you will need the following:
* Python with easy_install
* MySQL
* [http://pypi.python.org/pypi/MySQL-python/ MySQL-Python]
* [https://code.google.com/p/google-visualization-python/ Google-visualization-python]
* [http://memcached.org/ Memcached]
* Snapshots of our two databases 'schedulerdb' and 'statusdb'. They are available on Cruncher

It is highly recommended to set up and work within a python [[ReleaseEngineering/Virtualenv|virtualenv]].

If you are looking to setup a local virtual environment to run BuildAPI, then follow [[ReleaseEngineering/BuildAPI/Setup_Local_Virtualenv_for_BuildAPI|this wiki]].

= Setup =
Mana documentation:
* [https://mana.mozilla.org/wiki/display/IT/Redis Redis]
* [https://mana.mozilla.org/wiki/display/IT/BuildAPI BuildAPI]
* [https://mana.mozilla.org/wiki/display/IT/RelEng+RabbitMQ]

Puppet setup:
* [http://hg.mozilla.org/build/puppet/file/default/modules/selfserve_agent selfserve_agent]
* [http://hg.mozilla.org/build/puppet-manifests/file/default/modules/buildapi buildapi]
* [http://hg.mozilla.org/build/puppet-manifests/file/default/modules/redis redis]

= Getting Started =
The source for the buildapi is available here: [http://hg.mozilla.org/build/buildapi buildapi source]

Before you get started with that, you should setup your MySQL database instances.
To do that all you need to do is download, extract and load the sql dumps provided. So, create the databases first
<pre>
$ mysql -u <user> -p
<at prompt>
mysql>create database schedulerdb;
mysql>create database statusdb;
mysql>exit
</pre>

Download the sql dumps, then load them into the db with:
<pre>
mysql schedulerdb -u <user> -p < schedulerdb.sql
mysql statusdb -u <user> -p < statusdb.sql
</pre>

'''NOTE''': The files unzipped can account for more than 10GB. Watch out! :)
'''NOTE2''': The import can take a lot of time.

Now, to get started with the pylons project, simply run:
<pre>
python setup.py install
</pre>
In the buildapi/ directory. (And inside your virtualenv, if you're using one).
which should handle grabbing pylon project dependencies. You will also need to grab and install the google python visualization library from [http://code.google.com/p/google-visualization-python/ here].

Next you will need to generate a config file for your project, do this by running:
<pre>
paster make-config buildapi config.ini
</pre>

Now you will need to edit that config.ini to use the correct host(localhost should be fine), and database URLs.
Note: MySQL databases take the format `mysql://[username][:password][@hostname]/database_name`. For example:
sqlalchemy.scheduler_db.url = mysql://username:password@localhost/schedulerdb
sqlalchemy.status_db.url = mysql://username:password@localhost/statusdb

You should now be all set up now! Running the project locally can be done through:
<pre>
paster serve --reload --daemon config.ini
</pre>
Which starts running the buildapi on your local machine. To view it, open up http://localhost:5000.

= Building a simple controller =
The buildAPI is built on top of the pylons framework, which enforces a strict [http://en.wikipedia.org/wiki/Model-view-controller MVC] stack.
To add a new feature to the buildapi you will typically want to create a new controller and associate it with one or more models and views.

To create a simple 'hello world' controller, first use paster to create a template to work with.
<pre>
# in buildapi/
paster controller hello_world
</pre>
This automatically creates a workable template and a functional test case for your new controller.
You can now hack the newly created file under buildapi/controllers to your heart's content.
Models associated with your controller can be created under buildapi/model.
Views are created using Mako python templates. Create a template file under buildapi/templates for your new controller.

To associate with a model, simply add an import to your controller; for example:
<pre>
from buildapi.model.<model_name> import <functions to import>
</pre>

Lastly, once you're done editing your controller and have a resultset to publish, you probably want to render a page. To do this you can call render('/<template-name>.mako') from your controller body to render a view with your results. To access results from your controller, it is best to store the results in a pylons.tmpl_context object, which will make them available to your mako template.

===Simple code sample===
Now, to write a simple controller that prints "Hello World" using an M-V-C stack, we need three files:
* controllers/hello_world.py
* model/hello_world.py
* templates/hello_world.mako

Our controller lives in: controllers/hello_world.py
<pre>
import logging

from pylons import request, response, session, tmpl_context as c, url
from pylons.controllers.util import abort, redirect

from buildapi.lib.base import BaseController, render
from buildapi.model.hello_world import GetMessage

log = logging.getLogger(__name__)

class HelloController(BaseController):

def index(self):
# Return a rendered template
c.message = GetMessage()
return render('/hello_world.mako')
</pre>

Our model lives in: model/hello_world.py
<pre>
def GetMessage():
return "Hello world!"
</pre>

Our view lives in: templates/hello_world.mako
Note that the tmpl_context object is available in the template.
<pre>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<title>Hello World!</title>
</head>
<body>
<h1>${c.message}</h1>
</body>
</html>
</pre>

To access this page, we need to set up a route to it. To do this we add to config/routing.py:
<pre>
map.connect('/hello_world', controller='hello_world', action='index')
</pre>
Now restart buildapi, and point your browser to [http://localhost:5000/hello_world http://localhost:5000/hello_world].

= Maintenance =
== Updating code ==

BuildAPI has two components: BuildAPI and Selfserve Agent. They're deployed differently

=== BuildAPI ===

BuildAPI updates are handled like any other webapp, through pushes. See also [[ReleaseEngineering/How_To/Update_BuildAPI]]

The steps are roughly as follows, from the base directory of buildapi:
* Push new version of BuildAPI to HG
** Edit setup.py to show the updated version (ie version='0.3.0' becomes version='0.3.1')
* Build a new sdist tarball of BuildAPI, using normal python techniques, and upload it to the puppetagain python repository at releng-puppet2.srv.releng.scl3.mozilla.com (/data/python/packages on the distinguished puppetmaster)
** Tarballs can be seen here: http://releng-puppet2.srv.releng.scl3.mozilla.com/python/packages/
python setup.py sdist
VERSION=0.3.1
scp dist/buildapi-${VERSION}.tar.gz releng-puppet2.srv.releng.scl3.mozilla.com:
ssh releng-puppet2.srv.releng.scl3.mozilla.com

VERSION=0.3.1
sudo mv buildapi-${VERSION}.tar.gz /data/python/packages
cd /data/python/packages
sudo chown puppetsync:puppetsync buildapi-${VERSION}.tar.gz
sudo chmod 664 buildapi-${VERSION}.tar.gz
* Login to relengwebadm.private.scl3.mozilla.com and become root. Go to /data/releng-stage/src/buildapi (or /releng/ for production) and have a look at README.txt. Where 0.3.1 is the version you want to install, it boils down to:
./update 0.3.1

=== Selfserve Agent ===

Deploying new functionality may require deploying a new version of the <code>scripts/selfserve_agent.py</code> file on build masters. This process is handled by puppet, but orthogonal to buildapi itself. The steps are roughly:

* bump the version of the buildapi repository in "<code>setup.py</code>" and commit (following semantic versioning guidelines, please)
* add a new tag to the buildapi repository for that version, commit & push
* create a python sdist tarball of that version:
** make sure you have a clean checkout
** locally modify (but do not commit) <code>setup.cfg</code> and comment out "<code>tag_build</code>"
** run "python setup.py "<code>sdist</code>"
** updload the file from "<code>dist/</code>"
* [[ReleaseEngineering/How_To/Add_a_Python_Package_to_PuppetAgain|upload to the internal puppetagain python]] repo.
* update the manifests for the new version in [http://hg.mozilla.org/build/puppet/file/default/modules/selfserve_agent/manifests/install.pp puppet]
* wait for puppet to deploy the new agent.

== Adding branches ==
buildapi and self-serve pull from [http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json] to determine which branches are active and caches the result for a few minutes.

== Kicking ==
If things are broken, figure out if it's
* self-serve requests not completing (check the self-serve agents and rabbit connections)
* buildapi HTTP requests not working (check buildapi - [[ReleaseEngineering/How_To/Restart_BuildAPI]])
* builds-4hr.js not being updated (check the [[ReleaseEngineering/BuildAPI#report_crontask|report crontask]] report crontask)

=== report crontask ===

There's a crontask on [https://mana.mozilla.org/wiki/display/websites/Releng+Cluster relengwebadm] that runs every minute to generate the builds-4hr.js file. If this crontask gets "hung", it will prevent updates from occuring. This can happen due to DB connection issues, or bad data in the DB. In these cases, killing the hung crontask is the appropriate fix. There may be a stale lockfile that need removing under /var/lock/buildapi too.

However, the crontask takes about 15 minutes to run on a cold cache. So if its cache (memcached) has gone cold, then killing it before 15 minutes have elapsed is only delaying the failure.

From [https://bugzilla.mozilla.org/show_bug.cgi?id=1005342#c6 bug 1005342]:

{| class="wikitable"
|-
| ''Dustin J. Mitchell [:dustin] 2014-05-02 16:18:14 PDT''

I looked at 4:01 pacific, and saw that builds-4hr.js.gz had last been generated at 2:47. This is generated by a crontab that runs every minute, and makes a lot of heavy queries against the DB. On a good day with a cold cache, it's about 15 minutes. With a hot cache it can finish comfortably in one minute. It would seem that Matt's heavy query caused the builds-4hr.js.gz queries to go slowly, and thus caused the file's generation to fall behind.

I found the crontask, appropriately dated 2:47, and killed it. Cron started a new one up on the minute, and that completed in something like 45s.

I don't know why the original job didn't complete soon after mpressman cancelled his query.
|}

= Staging =
You can visit https://secure-pub-build.allizom.org/buildapi

From Dustin:
Buildapi itself doesn't write anything but jobrequests. For the staging instance, those go to a different DB than for production. Like the production version, it reads from the production status and scheduler db's.

The bit that it's missing is a running selfserve-agent. If you trigger some action on the staging instance, it pushes a message to the staging rabbitmq virtualhost and waits for a response, but since there's no agent that response never comes.

= Development environment =
Running your own instance of BuildAPI is relatively simple. You should:
* Create a virtualenv ([[ReleaseEngineering/BuildAPI/Setup_Local_Virtualenv_for_BuildAPI|details]])
* Install buildapi with its setup.py script
* Create a config. You may base your config on the production one, with at least the following changes:
** Change DEFAULT.email_to to your own email address
** Change server:main.port to 0.0.0.0
** Change server:main.port to something unused
** Comment out app:main.carrot*
** Set app:main.buildapi.cache to nothing

= Troubleshooting Tips =
== Host ==
NOTE: buildapi01 is largely unused now and the hosts are managed by webops

==No MySQLdb==
If after installing you run:
<pre>
paster serve --reload --daemon config.ini
</pre>
and it does not start the server, check paster.log and if you see "ImportError: No module named MySQLdb" then you need to easy_install MySQL-python into the site-packages of the python it's looking for MySQLdb in.

ReleaseEngineering/How To/Restart Buildbot Masters

2016-11-20T22:21:35Z

NThomas: Add filtered view for logs

__TOC__
We occasionally need to restart buildbot masters for various reasons:
* upgrades to the underlying OS
* gradual increase in memory usage over time, leading to reduced master performance

= Manually =
If you need to restart a single master by hand, here's the sequence you should follow:
* disable the master in [https://secure.pub.build.mozilla.org/slavealloc/ui/#masters slavealloc]. This prevents the master from taking more slave connections while you're waiting for it to shutdown.
* click the "Clean Shutdown" button on the web interface for the given master, e.g. http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/
* wait for the jobs currently running on that master to complete. You can track progress by searching in-page for "Running" on the master's buildslaves page, e.g. http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/buildslaves?no_builders=1
* once the master is shutdown, perform whatever upgrades are required, etc.
* restart the master. '''NOTE:''' buildbot masters are configured to restart buildbot automatically on boot, so if you reboot the master, buildbot will restart itself. To restart manually:
xebec:buildduty ccooper$ ssh cltbld@buildbot-master82
Unauthorized access prohibited
[cltbld@buildbot-master82.bb.releng.scl3.mozilla.com ~]$ cd /builds/buildbot/build1/
[cltbld@buildbot-master82.bb.releng.scl3.mozilla.com build1]$ make start
* re-enable the master in [https://secure.pub.build.mozilla.org/slavealloc/ui/#masters slavealloc].

= By script =
The above actions have been encapsulated into a script: https://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/restart_masters.py. The script is setup to run on <tt>buildduty-tools.srv.releng.usw2.mozilla.com</tt>, located at <tt>/home/buildduty/restart_masters/</tt> under buildduty's account.

The wrapper script <tt>/home/buildduty/restart_masters.sh</tt> is used to update repositories and call restart_masters.py. The latter will prompt for the usernames and passwords it needs (ldap for slavealloc, cltbld and root for ssh).

Here is an example invocation:

# buildduty@buildduty-tools
$ screen -R restart_masters
$ /home/buildduty/restart_masters.sh

The logs go to [https://papertrailapp.com/systems/255918104/events?q=restart_masters papertrail], outputting a progress report every 60 minutes. For really minimal view (which may hide errors) see this [https://papertrailapp.com/systems/255918104/events?q=restart_masters+-%22Connection+as+cltbld%22+-%22Attempting+to+connect+to%22+-%22Checking+shutdown+status%22+-lockfile+-Disabling+-Initiating+-%22Re-enabling%22+-%22Attempting+to+restart+master%22 filtered view].

You can send a SIGUSR1 to restart_masters.py to prompt an extra progress report; allow time for the current status check to complete first.

= Automated =
The above script requires sensitive credentials that shouldn't be stored on disk. For now, we're still running this script by hand.

ReleaseEngineering/How To/Restart Buildbot Masters

2016-11-20T22:02:50Z

NThomas: /* By script */ Streamline docs for running on buildduty-tools

__TOC__
We occasionally need to restart buildbot masters for various reasons:
* upgrades to the underlying OS
* gradual increase in memory usage over time, leading to reduced master performance

= Manually =
If you need to restart a single master by hand, here's the sequence you should follow:
* disable the master in [https://secure.pub.build.mozilla.org/slavealloc/ui/#masters slavealloc]. This prevents the master from taking more slave connections while you're waiting for it to shutdown.
* click the "Clean Shutdown" button on the web interface for the given master, e.g. http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/
* wait for the jobs currently running on that master to complete. You can track progress by searching in-page for "Running" on the master's buildslaves page, e.g. http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/buildslaves?no_builders=1
* once the master is shutdown, perform whatever upgrades are required, etc.
* restart the master. """NOTE:""" buildbot masters are configured to restart buildbot automatically on boot, so if you reboot the master, buildbot will restart itself. To restart manually:
xebec:buildduty ccooper$ ssh cltbld@buildbot-master82
Unauthorized access prohibited
[cltbld@buildbot-master82.bb.releng.scl3.mozilla.com ~]$ cd /builds/buildbot/build1/
[cltbld@buildbot-master82.bb.releng.scl3.mozilla.com build1]$ source bin/activate
(build1)[cltbld@buildbot-master82.bb.releng.scl3.mozilla.com build1]$ make start
* re-enable the master in [https://secure.pub.build.mozilla.org/slavealloc/ui/#masters slavealloc].

= By script =
The above actions have been encapsulated into a script: https://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/restart_masters.py. The script is setup to run on <tt>buildduty-tools.srv.releng.usw2.mozilla.com</tt>, located at <tt>/home/buildduty/restart_masters/</tt> under buildduty's account.

The wrapper script <tt>/home/buildduty/restart_masters.sh</tt> is used to update repositories and call restart_masters.py. The latter will prompt for the usernames and passwords it needs (ldap for slavealloc, cltbld and root for ssh).

Here is an example invocation:

# buildduty@buildduty-tools
$ screen -R restart_masters
$ /home/buildduty/restart_masters.sh

The logs go to [https://papertrailapp.com/systems/255918104/events?q=restart_masters papertrail], outputting a progress report every 60 minutes. You can send a SIGUSR1 to restart_masters.py to prompt an extra progress report; allow time for the current status check to complete first.

= Automated =
The above script requires sensitive credentials that shouldn't be stored on disk. For now, we're still running this script by hand.

ReleaseEngineering/How To/Setup Personal Development Master

2016-11-08T23:45:26Z

NThomas: Use python 2.7.3 when creating a master

__TOC__
{{Release Engineering How To|Setup a Personal Development Master}}
= Setup =
== Prerequisites ==
* python 2.7
* virtualenv
** If you need to create a virtualenv manually, you must call it directly with: <pre>python /tools/misc-python/virtualenv.py </path/to/destination></pre>
** This is not necessary for a buildbot master setup if you use the methods described below

'''NOTE''': You will need root access to create a user account for yourself on dev-master1 and a buildbot area under /builds/buildbot/your_name.<br />
Ask someone with root access to set it up following instructions in [https://intranet.mozilla.org/Build/Mentor_Checklist here].<br />
If you are setting dev-master1 you probably want to use root to set up an account in /builds/buildbot/your_name since you might get this error (I think it has to do with which group your account belongs to):
from twisted.python.zippath import ZipArchive
ImportError: No module named zippath

== General ==
* Instructions specific to dev-master1 (dev-master1.build.mozilla.org)
<pre>
# as root
WHO=coop
mkdir /builds/buildbot/${WHO}
chown ${WHO} /builds/buildbot/${WHO}
</pre>

* Notes:
** BASEDIR refers to where you want the new master to be created
** MASTER_NAME is the name of one of the predefined masters. Search for a name prefixed by "staging".
*** /usr/bin/python2.6 setup-master.py -l -- display a list of enabled masters
*** /usr/bin/python2.6 setup-master.py -l -8 -- display a hardcoded list of 0.8.0 masters
** PYTHON, HG, VIRTUALENV should be adjusted to your local environment

== Create a build master ==

Use [https://hg.mozilla.org/build/braindump/file/tip/buildbot-related/create-staging-master.pl this automation script] to create a build master, or follow the manual steps below.

First, clone the <code>buildbot-configs</code> repository.
<pre>
$ hg clone https://hg.mozilla.org/build/buildbot-configs
</pre>

Add <code>'try'</code> to the list of <code>ACTIVE_BRANCHES</code> in <code>master/master_localconfig.py</code>.

<code>create_master.sh</code>
<pre>
USERNAME=`whoami`
make -f Makefile.setup \
USE_DEV_MASTER=1 \
MASTER_NAME=$USERNAME \
BASEDIR=/builds/buildbot/$USERNAME/build1 \
PYTHON=/tools/python27/bin/python \
VIRTUALENV=virtualenv \
BUILDBOTCUSTOM_BRANCH=default \
BUILDBOTCONFIGS_BRANCH=default \
USER=$USERNAME \
HTTP_PORT=8888 PB_PORT=9999 SSH_PORT=7777 ROLE=build \
virtualenv deps install-buildbot master master-makefile
</pre>

=== Universal Master ===

To create a universal master:
$ cd /builds/buildbot/$USERNAME/build1/master
$ ln -sf ../buildbot-configs/mozilla/universal_master_sqlite.cfg master.cfg

You will probably want to avoid running scheduled builds (nightlies, etc.) from starting on your personal master. Clear the ACTIVE_* lists in <code>mozilla/build_localconfig.py</code> to achieve this. You can also set <code>c['schedulers']=[]</code> but take care to do that '''before''' the release schedulers are added to the list or sendchange will fail to kick off a release build.

=== build-master path values ===
* Configs: buildbot-configs/mozilla/
* master_localconfig: ???
* host: preprod-builder

=== Signing server setup ===
Ever since we started signing all builds, a proper signing server setup in your dev master is necessary to do proper tests. The signing servers are configured in passwords.py and require real secrets (meaning, do not distribute them publicly). The [https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/passwords.py.template templatized version of passwords.py] will show you the format of the file, but you'll need to get the real values elsewhere. bhearsum's version is generally up-to-date and available at /builds/buildbot/bhearsum/build1/master/passwords.py on dev-master1. Note that in staging you must be careful not to use the Release or Nightly signing server instances, which will sign builds with real certificates (instead of self signed ones) -- so do NOT copy a passwords.py file from a production build master. If you have trouble with this, ask for help.

=== Disable periodic builds ===

If you are testing changes on your build master, it can be annoying for pgo builds to be invoked and run on your master consuming your build machine resources. To avoid this, comment out lines that define pgo_strategy like this BRANCHES['mozilla-central']['pgo_strategy'] = 'periodic' or BRANCHES['mozilla-esr31']['pgo_strategy'] = 'per-checkin'. Conversely, if you are testing pgo builds, you can change the start_hour and start_minute on a per branch basis.

== Create a tests master ==

Use [https://hg.mozilla.org/build/braindump/file/tip/buildbot-related/create-staging-master.pl this automation script] (recommended) to create a build master, or use the same approach as for build master.

perl create-staging-master.pl -u coop --master-kind=test --master-dir=tests-master --http-port=8045

= Customize your configs =
* In <code>master/master_config.json</code>, adjust ports for <code>pb_port</code>, <code>http_port</code> & <code>ssh_port</code>
** Check [https://secure.pub.build.mozilla.org/slavealloc/ui/#masters slavealloc] to see what ports are in use by others. Search for 'staging-pers' and be sure to Show all Entries.
** Try to use similar ports, i.e. if you choose 9099 for pb_port, use 8099 for your http_port and 7099 for your ssh_port.
* Consider [[#Optimizations|optimizing the number of branches]] you're running.

== Create your own queue dir ==
To avoid this error:
OSError: [Errno 1] Operation not permitted: '/dev/shm/queue/commands/tmp'

Do this:
<pre>
WHO=coop
mkdir -p /dev/shm/${WHO}/queue
chown ${WHO} /dev/shm/${WHO}
</pre>
* modify your master/master_localconfig.py so that QUEUEDIR points to your new dir:
QUEUEDIR = "/dev/shm/${WHO}/queue"
* or (newer) <tt>master/master_config.json</tt> to add the line:
"queuedir": "/dev/shm/${WHO}/queue"

= Does it work? =
* If you want production schedulers to run, then "<tt>cd $BASEDIR/master; ln -sf ../buildbot-configs/mozilla/universal_master_sqlite.cfg master.cfg; cd -</tt>" (build master) or "<tt>cd $BASEDIR/master; ln -sf ../buildbot-configs/mozilla-tests/universal_master_sqlite.cfg master.cfg; cd -</tt>" (tests master). <small>Note: the default scheduler <code>builder_master.cfg</code> is usually fine.</small>
** Read the above as "you '''always''' want <tt>universal_master_sqlite.cfg</tt> unless you have a good and solid reason for not wanting it <small>(or want to test your antacid options)</small>.
* Run "make checkconfig" to make sure the default configs are in place and good.
* Run "make start" to make sure you master starts.
** You should have a Makefile at $BASEDIR/Makefile to start/stop your master
** Visit http://dev-master1.build.mozilla.org:${YOUR_WEBSTATUS_PORT}/waterfall
* If it works, congrats! Now run "make stop" to shut it down until you're ready to actually start testing.

== Sendchanges to your tests master from your build master ==
* If you're running *both* a build master and a tests master and want to test the sendchanges between them, you should have a similar local change on your build master (replace 9041 with the relevant port on your tests master):

<pre>
diff --git a/mozilla/staging_config.py b/mozilla/staging_config.py
--- a/mozilla/staging_config.py
+++ b/mozilla/staging_config.py
@@ -49,3 +49,3 @@ GLOBAL_VARS = {
'talos_masters': [
- ('dev-master1.build.mozilla.org:9009', True, 1),
+ ('staging-master:9041', True, 1),
],
@@ -55,4 +55,3 @@ GLOBAL_VARS = {
'unittest_masters': [
- ('staging-master.build.mozilla.org:9009', True, 1),
- ('talos-staging-master02.build.mozilla.org:9012', True, 1),
+ ('dev-master1:9041', True, 1),
],
</pre>

= Adding your master to slavealloc =
To lock slaves onto your master you need to add it to slavealloc, see [[ReleaseEngineering:Buildduty:Slave_Management#Adding_a_master | adding a master]]

= Modifying your old master in slavealloc =
If you have an old master you can modify it:
* Visit http://slavealloc.build.mozilla.org/ui/#masters
* Modify the FQDN of the master and the port (e.g. dev-master1.build.mozilla.org)

= Setup/Running local master scheduler on laptop - not dev-master =
<ol><li> Download [http://hg.mozilla.org/build/braindump/raw-file/default/buildbot-related/fresh_bb_setup fresh_bb_setup] to ~/</li>
<li>bash ./fresh_bb_setup Buildbot</li>
<li>cd Buildbot; source bin/activate</li>
<li>cd buildbot-configs; ./test-masters.sh</li>
<ul><li><strong>NOTE:</strong> you can stop here if your only goal is to test config changes. Make them in this directory, then run <code>./test-masters.sh</code>.</li></ul>
<li>cd ~</li>
<li>cd BASEDIR/build-master/</li>
<li>Modify line 2 of Makefile from BUILDBOT=$(PWD)/bin/buildbot to be BUILDBOT=$(PWD)/../bin/buildbot</li>
<li>Modify line 11 of master_config.json so that 'release_branches' contains an empty list. It did contain ["mozilla-1.9.2", "mozilla-beta"] and they are unnecessary for a local setup.</li>
<li>make checkconfig
* Should say: Config file is good!</li>
<li>make start
<ul><li> It will appear as if start has failed, but fear not! You'll see the message below, but this is simply because the buildbot process is launched as a dameon and buildmaster waits to give the all configuration is complete signal. However, Mozilla's configuration files are way to complicated to be dealt with in under 10 seconds, so it times out and you see the message below:
<pre>(Buildbot)localhost:build-master jzeller$ make start
cd master && /Users/jzeller/Buildbot/build-master/../bin/buildbot start $PWD
Following twistd.log until startup finished..
2014-01-22 14:47:30-0800 [-] Log opened.
2014-01-22 14:47:30-0800 [-] twistd 12.0.0 (/Users/jzeller/Buildbot/bin/python 2.6.7) starting up.
2014-01-22 14:47:30-0800 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2014-01-22 14:47:30-0800 [-] monkeypatch_twisted_cbLogin applied
2014-01-22 14:47:30-0800 [-] Creating BuildMaster — buildbot.version: 0.8.2
2014-01-22 14:47:30-0800 [-] loading configuration from /Users/jzeller/Buildbot/build-master/master.cfg
2014-01-22 14:47:30-0800 [-] unable to import dnotify, so Maildir will use polling instead
2014-01-22 14:47:30-0800 [-] nextAWSSlave: start
2014-01-22 14:47:30-0800 [-] nextAWSSlave: start
2014-01-22 14:47:37-0800 [-] nextAWSSlave: start
2014-01-22 14:47:37-0800 [-] nextAWSSlave: start

The buildmaster took more than 10 seconds to start, so we were unable to
confirm that it started correctly. Please 'tail twistd.log' and look for a
line that says 'configuration update complete' to verify correct startup.

make: *** [start] Error 1</pre></li></ul>
<li>To doublecheck that the configuration was successful, just follow the directions and type: less twistd.log | grep "configuration update complete". Make sure that the most recent timestamp matches the time with which you tried a make start, but as long as you see it pop up then you're good!</li>
<li>Go to http://localhost:8501/</li>
<li>If you see a page with "Welcome to the Buildbot for the Firefox project!" then you have been successful!</li>
<li>You can now check out what buildbot has pending by going to http://localhost:8501/waterfall</li></ol>

= Setting up a local slave =
<pre>
cd /path/to/master
source ./bin/activate
buildslave --help
</pre>

The buildslave command should be available at this point, if not:
<pre>pip install buildbot-slave</pre>

Now we create a slave using the directory my-slave in the master directory.
<ul>
<li>To find the relevant ports, check master/master_localconfig.py</li>
<li>The port to access your master via the browser is under WebStatus. </li>
<li>The port for slave access (used when creating a slave) is under slavePortnum.</li>
<li>Choose a slave name that the master is already looking for. To find one go to 127.0.0.1:{browserport}/builders, select a builder (OS X 10.6.2 mozilla-central nightly for example) and choose one of the names under "Buildslaves"
</li>

<pre>
# From the master's directory
buildslave create-slave my-slave 127.0.0.1:<port> <name> pass
</pre>

If you haven't already, start the master, then start the slave
<pre>
cd /path/to/master
make start
# Start the slave
cd my-slave
buildslave start
</pre>

Your slave should be picked up momentarily by the master.

= Pointing dev slaves to your master =

Using the slaves interface of slavealloc (http://slavealloc.build.mozilla.org/ui/#slaves), search for slaves that are currently in the preprod (or preprod-tests) pool that are not currently locked to someone else's dev master. Edit the slave and lock that slave to your master.

'''NOTE''': the slave will need to be rebooted to pick up this change.

If no slaves are available for your required platform, find out who has them all and ask to borrow one.

= Setup a user source repository =
Depending on how invasive your changes are going to be, you can choose to setup your own user repo for any of the releng repos or even the Firefox code repos.

Follow the instructions for [https://developer.mozilla.org/en/Publishing_Mercurial_Clones Publishing Mercurial Clones] and then replace references to releng repos in the configs with your own repo.

As of 2012-12-06, you need a mozharness user repo just run b2g compile jobs in staging. It is the only place to change the upload hosts to the staging host.

= Optimizations =

== master_localconfig.py ==
* Comment out/change ACTIVE_{BRANCHES,PLATFORMS,PROJECTS} in master_localconfig.py as needed to reduce your workspace.
** e.g. I usually run with only one branch: ACTIVE_BRANCHES = ['mozilla-central']

==master_config.json==

* Limit the number of platforms your dev-master will run tests on. For instance, if you only want to run tests for Android devices, add this to master_config.json

<pre>
"limit_fx_platforms": [],
"limit_mobile_platforms": [
"android",
"android-armv6",
"android-noion"
],
"limit_tb_platforms": [],
"limit_b2g_platforms": [],
</pre>

==BuildSlaves.py==

* Update with the appropriate staging passwords so your slaves can connect

==staging_config.py==

* Update with the STAGING_SLAVES dictionary with the appropriate type and range of devices that you want to connect to you connect to your master

== Resetting db state ==
This is generally most useful when you want to get rid of all pending builds. First, add this to your master's Makefile:
reset-db:
rm master/state.sqlite
cd master && $(BUILDBOT) upgrade-master $$PWD

Then run:
make stop
make reset-db
make start

== Getting build notifications ==
It might be useful to know when a build finishes or the status of certain builds. This way you don't have to keep an eye on your dev master and move on to other awesome things. There are a few ways to accomplish this:

* Using Buildbot's IRC bot
**first add the following to your list of status obj in your master.cfg:
from buildbot.status import words
...
irc = words.IRC("irc.mozilla.org", "<your-bot-nick>",
channels=["<irc-channel>"], useSSL=True,
port=6697, allowForce=False,
notify_events={
'exception': 1,
'successToFailure': 1,
'failureToSuccess': 1,
})
c['status'].append(irc)

now reconfig and your should see your bot join <irc-channel>. From there you can do a number of things: http://docs.buildbot.net/0.8.0/IRC-Bot.html

I like to start a private chat with my bot and request the build status when a job finishes. This gives me actual pings!

you can do this like so from irc:
> /query <your-bot-nick>
> notify on finished

* Using Check4Change Firefox extension:
** this extension will check if a page changes without having to actually refresh! You can hook this up to your dev master buildbot web UI
** for more details: http://www.check4change.com/

= See Also =
* [[ReleaseEngineering/How_To/Staging_Release_Build]]
* http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla
* http://coop.deadsquid.com/2011/10/thinnerbox-updated/
* [https://hg.mozilla.org/build/braindump/file/d84e1ece72c6/community/create_community_slaves_and_masters.sh Script to setup staging local buildbot masters and slaves]

ReleaseEngineering/Maintenance

2016-11-07T20:43:02Z

NThomas: /* Reconfigs / Deployments */ deploy bug 1306723

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

[[ReleaseEngineering:BuildbotBestPractices]] describes how we manage changes to our master buildbots.

= Relevant repositories =
* [https://hg.mozilla.org/build/buildbot/ buildbot]
* [https://hg.mozilla.org/build/buildbot-configs/ buildbot-configs]
* [https://hg.mozilla.org/build/buildbotcustom/ buildbotcustom]
* [https://hg.mozilla.org/build/tools/ tools]
* [https://hg.mozilla.org/build/mozharness/ mozharness]
* [https://hg.mozilla.org/build/puppet/ puppet]

= Non-releng repositories involved in the CI =
== [[Auto-tools/Projects/Treeherder|Treeherder]] ==
* [http://treeherderadm.private.scl3.mozilla.com/chief/treeherder.prod/history Chief self-deploy recent history] (needs VPN; more readable than raw logs, but is cleared if the Chief service is restarted)
* [treeherderadm.private.scl3.mozilla.com/chief/treeherder.prod/logs/ Chief self-deploy logs archive] (needs VPN; use if the history page is empty)

= Mozharness deployments =
Merges to the <em>production</em> branch of [http://hg.mozilla.org/build/mozharness/graph the mozharness repository] are live immediately. Please see that repo for push history.

= Puppet deployments =

Merges to the <em>production</em> branch of [http://hg.mozilla.org/build/puppet/graph the puppet repository] are automatically deployed by Puppet within a 1/2 hour. Please see that repo for push history.

Please note if you are applying a puppet change, '''unless you have good reason not to do so''' please merge it to production branch immediately. The reason for this is that we do not want puppet changes to "back up" on default, and then land ''en masse'' in production. It can be very difficult to untangle multiple puppet changes that arrive together.

The reason we have a production branch at all, is to allow contributions on default branch from a larger group of people, than those responsible for maintaining the infrastructure controlled by the puppet repo - in other words, a smaller group of people can merge to production, than the group of people that can land on default.

= Reconfigs / Deployments =
This page is updated by the person who does a [[ReleaseEngineering:Buildduty:Reconfigs|reconfig]] on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as <i>mozharness</i> changes.)

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Older changes are archived in: [[ReleaseEngineering/Changes/2015|2015]] | [[ReleaseEngineering/Changes/2014|2014]] | [[ReleaseEngineering/Changes/2013|2013]] | [[ReleaseEngineering/Changes/2012|2012]] | [[ReleaseEngineering/Changes/2011|2011]] | [[ReleaseEngineering/Changes/2010|2010]] | [[ReleaseEngineering/Changes/2009|2009]]

{| class="fullwidth-table wikitable sortable"
| style="background:#cccccc" | '''Outcome'''
| style="background:#cccccc" | '''When'''
| style="background:#cccccc" | '''Bug #(s)''' - '''Description(s)'''
|-
| in production
| 2016-11-07 13:00 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1306723&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1306723}} - Make non-mozharness jobs in builds-4hr use 40 character SHA revision, r=nthomas ([https://hg.mozilla.org/build/buildbotcustom/rev/46bf43f4514d 46bf43f4514d])
|-
| in production
| 2016-10-31 12:07 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1305491&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1305491}} - Tracking bug for Nov-07-2016 migration work, bump m-r - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/3239e7157676 3239e7157676])
|-
| in production
| 2016-10-27 11:38 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1308544,1313434&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1308544}} - Enable automation jobs on Cedar twig - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/188eea293ee4 188eea293ee4])
* {{bug|1308544}} - Enable automation jobs on Cedar twig - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/8455db7716ed 8455db7716ed])
* {{bug|1313434}} - tracking bug for build and release of Firefox 47.0.2 - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/bcb07cf862bf bcb07cf862bf])
* No bug - revert changeset 188eea293ee4 ([https://hg.mozilla.org/build/buildbot-configs/rev/8e85d4585108 8e85d4585108])
|-
| in production
| 2016-10-25 19:00 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1312650&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1312650}} - Windows buildbot factory jobs should know about TOOLTOOL_CACHE, r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/22368ebeb3c1 22368ebeb3c1])
|-
| in production
| 2016-10-25 09:24 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1309894&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1309894}} - Use *-dev update channels on jamun - r=mtabara ([https://hg.mozilla.org/build/buildbot-configs/rev/c3ba8bc05df6 c3ba8bc05df6])
|-
| in production
| 2016-10-24 17:58 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1309894&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1309894}} - Use dev-* update channels on jamun - r=nthomas ([https://hg.mozilla.org/build/buildbot-configs/rev/d65e81712d2c d65e81712d2c])
|-
| in production
| 2016-10-21 06:56 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1309894&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1309894}} - Use "poll_all_branches" in configs to explicitly set polling behaviour - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/f3d8389cb9bf f3d8389cb9bf])
* No bug - Update release config for Fennec-49.0.2-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/08de42085241 08de42085241])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1309894}} - Use "poll_all_branches" to explicitly set polling behaviour - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/422f21b8f5fc 422f21b8f5fc])
|-
| in production
| 2016-10-18 08:54 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1308789&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1308789}} - Add test case to reproduce failure - r=nthomas ([https://hg.mozilla.org/build/buildbotcustom/rev/5a283b5466d6 5a283b5466d6])
* {{bug|1308789}} - Fix setting of default platforms for tests - r=nthomas ([https://hg.mozilla.org/build/buildbotcustom/rev/bfe42c243d4a bfe42c243d4a])
|-
| in production
| 2016-10-13 10:14 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1308789,1309894&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1308789}} - Set the Win64 builds to be the default try_by_default: True, because we want them to build by default, it's only the Win8 tests that are supposed to be try_by_default: False - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/a9bbcd872ad9 a9bbcd872ad9])
* {{bug|1309894}} - Investigate shipping old release version off of jamun - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/6f25921852c3 6f25921852c3])
* No bug - Update release config for Fennec-50.0b6-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/f1f2fc3fa06e f1f2fc3fa06e])
|-
| in production
| 2016-10-10 07:54 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1292656,1300413&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1292656}} - decommission another 10 t-snow-r4 machines - r=coop ([https://hg.mozilla.org/build/buildbot-configs/rev/02b3c211cc48 02b3c211cc48])
* {{bug|1300413}} - Use unbuffered python in case we're missing output - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/1d22c76665b8 1d22c76665b8])
* No bug - Update release config for Thunderbird-45.4.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/a4d817817321 a4d817817321])
* No bug - Update release config for Thunderbird-50.0b1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/ab02ce360da9 ab02ce360da9])
* No bug - Update release config for Fennec-50.0b4-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/f854aac09b67 f854aac09b67])
* No bug - Update release config for Thunderbird-50.0b1-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/acc59373912b acc59373912b])
|-
| in production
| 2016-09-21 10:45 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* No bug - Update release config for Fennec-50.0b1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/d17d80076abd d17d80076abd])
|-
| in production
| 2016-09-19 22:09 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1303914&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1303914}} - buildbot - turn on fennec debug builds and tests for beta 50 - r=aki ([https://hg.mozilla.org/build/buildbot-configs/rev/2726220a2e49 2726220a2e49])
|-
| in production
| 2016-09-19 06:37 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1275601,1297519&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1275601}} - Tracking bug for Sep-13-2016 migration work - r=mtabara ([https://hg.mozilla.org/build/buildbot-configs/rev/bb6a28e7b06d bb6a28e7b06d])
* {{bug|1297519}} - Generate multilocale Firefox snaps - r=jlorenzo ([https://hg.mozilla.org/build/buildbot-configs/rev/4354ce6042f5 4354ce6042f5])
* No bug - Update release config for Fennec-49.0-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/4b6d645e8c31 4b6d645e8c31])
|-
| in production
| 2016-09-14 13:31 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1302256,1302530&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1302256}} - Run Win7 GFX jobs on try by default, now that they are production jobs - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/1911d93964d0 1911d93964d0])
* {{bug|1302256}} - Run Win7 GFX jobs on try by default, now that they are production jobs - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/1911d93964d0 1911d93964d0])
* {{bug|1302530}} - Add ondemand g-w732 instances - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/ba50299bcd9f ba50299bcd9f])
* {{bug|1302530}} - Add ondemand g-w732 instances - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/ba50299bcd9f ba50299bcd9f])
* No bug - Update release config for Fennec-49.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/0788239cf1d8 0788239cf1d8])
* No bug - Update release config for Fennec-49.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/0788239cf1d8 0788239cf1d8])
* No bug - Backed out changeset 1911d93964d0 ([https://hg.mozilla.org/build/buildbot-configs/rev/46e667b9878b 46e667b9878b])
|-
| in production
| 2016-09-06 10:59 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1300368&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1300368}} - Make autoland trigger periodic builds more frequently ([https://hg.mozilla.org/build/buildbot-configs/rev/8d3743998c94 8d3743998c94])
* No bug - Update release config for Fennec-49.0b11-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/6fd24999974a 6fd24999974a])
|-
| in production
| 2016-09-05 07:59 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1275601&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1275601}} - Bump mozilla-release version - r=mtabara ([https://hg.mozilla.org/build/buildbot-configs/rev/0f7da3a3e475 0f7da3a3e475])
|-
| in production
| 2016-08-26 11:33 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1288434&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1288434}} - Add bb-configs for mark release as shipped task - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/3dc06ea07b8d 3dc06ea07b8d])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1288434}} - Add bb-custom builder for mark release as shipped task - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/91327dcd59ef 91327dcd59ef])
|-
| in production
| 2016-08-25 10:56 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1296449,1298013&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1298013}} - Migrate remaining win 7 mochitest bc jobs to AWS - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/7c1e748101f6 7c1e748101f6])
* No bug - Update release config for Thunderbird-49.0b1-build4 ([https://hg.mozilla.org/build/buildbot-configs/rev/26a6a5f90516 26a6a5f90516])
* No bug - Update release config for Thunderbird-45.3.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/4b44a217b2ef 4b44a217b2ef])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1296449}} - Backed out changeset 68ec5535f7af because it was unnecessary ([https://hg.mozilla.org/build/buildbotcustom/rev/e03bbd8376e3 e03bbd8376e3])
|-
| in production
| 2016-08-23 12:23 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1297421&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1297421}} - migrate win 7 mochitest-e10s browser-chrome debug from windows HW to AWS - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/e24afa83727c e24afa83727c])
|-
| in production
| 2016-08-23 10:14 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1291015,1297173&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1291015}} - Migrate mochitest to AWS - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/6b5ab7e5fa62 6b5ab7e5fa62])
* {{bug|1297173}} - Move 20 machines from Windows 7 pool to Windows XP pool - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/0eb5c3edca2a 0eb5c3edca2a])
* No bug - Update release config for Fennec-49.0b6-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/3c0a11a61920 3c0a11a61920])
|-
| in production
| 2016-08-19 05:55 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1296026,1296449&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1296026}} - Modify --mochitest-suite for desktop mochitest-gl - r=aselagea ([https://hg.mozilla.org/build/buildbot-configs/rev/13db21d7a4f8 13db21d7a4f8])
* No bug - Update release config for Thunderbird-49.0b1-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/93a033b4676c 93a033b4676c])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1296449}} - Timeouts generating partial updates for Thunderbird on Windows - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/68ec5535f7af 68ec5535f7af])
|-
| in production
| 2016-08-11 13:16 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1290319&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1290319}} - require hostname or releng_CA in signing ssl cert - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/307646eb0970 307646eb0970])
|-
| in production
| 2016-08-10 13:50 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1268648,1290319&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1268648}} - Increase the script_timeout for mochitest-browser-screenshots to 1800s - r=armenzg ([https://hg.mozilla.org/build/buildbot-configs/rev/e538f7f45a14 e538f7f45a14])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1290319}} - fix URLPath call - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/61a1b16ba83d 61a1b16ba83d])
|-
| in production
| 2016-08-08 13:15 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* No bug - empty commit to force reconfig ([https://hg.mozilla.org/build/buildbot-configs/rev/54cc6717f498 54cc6717f498])
|-
| in production
| 2016-08-04 09:11 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1288043,1291044&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1288043}} - migrate win7 gpu jobs to gfx instances - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/199eca90a0a9 199eca90a0a9])
* {{bug|1291044}} - Use 10 chunks for wpt on win 7 vm debug to prevent timeouts - r=jmaher ([https://hg.mozilla.org/build/buildbot-configs/rev/c23e2a8603f5 c23e2a8603f5])
* No bug - [PATCH] Bug 1271153 - Adjust automation for the version_display.txt new file on tb - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/06d19bac0eff 06d19bac0eff])
|-
| in production
| 2016-08-01 06:14 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1278260&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1278260}} - Tracking bug for Aug-1-2016 migration work - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/559b89156599 559b89156599])
|-
| in production
| 2016-07-28 04:56 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1289514&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1289514}} - Remove unused import - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/e939e50055c5 e939e50055c5])
* {{bug|1289514}} - Remove bin/hgpoller.py - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/97fe1b0d7fcb 97fe1b0d7fcb])
* {{bug|1289514}} - Consume pushlog version 2 - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/f543c1e5f30a f543c1e5f30a])
* {{bug|1289514}} - Change API and semantics of pushlog parsing - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/528c39fe4a5a 528c39fe4a5a])
* {{bug|1289514}} - Factor common test code into base class - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/93f15cf0f724 93f15cf0f724])
* {{bug|1289514}} - Detect empty repos from empty lastpushid - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/d76e0a6a3900 d76e0a6a3900])
* {{bug|1289514}} - React intelligently to reset pushlogs - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/3041d0f3079f 3041d0f3079f])
* {{bug|1289514}} - Remove empty dataFailed method - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/604839f1803e 604839f1803e])
* {{bug|1289514}} - Poll pushlog from last consumed push ID - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/2b20e69710ab 2b20e69710ab])
* {{bug|1289514}} - Handle empty changesets in push entry - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/86c0a3b7c99c 86c0a3b7c99c])
|-
| in production
| 2016-07-27 09:54 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* No bug - Update release config for Fennec-48.0b11-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/9c1804c74761 9c1804c74761])
* No bug - Update release config for Fennec-48.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/49e72329db05 49e72329db05])
|-
| in production
| 2016-07-25 07:30 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1278260,1278261&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1278260}} - "Tracking bug for Aug-1-2016 migration work" - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/21ae1b8c2405 21ae1b8c2405])
* {{bug|1278261}} - drop api-9 mozilla-release release support, remove all api-9 and api-11 references - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/707b1871d7dc 707b1871d7dc])
|-
| in production
| 2016-07-22 09:39 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1233953,1282849&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1233953}} - Intermittent Linux debug command timed out: 7200 seconds elapsed running - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/2f53b99fd268 2f53b99fd268])
* {{bug|1282849}} - disable fennec debug builds and tests in buildbot - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/19f3d53951ae 19f3d53951ae])
* No bug - Update release config for Fennec-48.0b8-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/da62abc85e9d da62abc85e9d])
* No bug - Update release config for Fennec-48.0b9-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/2b19aa948928 2b19aa948928])
* No bug - Update release config for Fennec-48.0b10-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/ad5f8c375bcd ad5f8c375bcd])
|-
| in production
| 2016-07-12 16:12 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1055707,1277435&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* No bug - Update release config for Thunderbird-48.0b1-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/205cec6585a1 205cec6585a1])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1055707}} - set MOZ_NODE_PATH for Mac - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/64450127bc26 64450127bc26])
* {{bug|1277435}} - Remove funfuzz private repo reference ([https://hg.mozilla.org/build/buildbotcustom/rev/2d29e0ad789e 2d29e0ad789e])
|-
| in production
| 2016-07-07 11:34 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1267262,1278611,1285303&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1267262}} - Kill unused release config files - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/fc34658eac8e fc34658eac8e])
* {{bug|1278611}} - move valgrind tc builds to tier1 and stop building buildbot binaries - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/e582005ba6bd e582005ba6bd])
* {{bug|1285303}} - Reset projects/date configs - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/5db89ab28e80 5db89ab28e80])
* No bug - Update release config for Fennec-48.0b6-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/42a5ebf91840 42a5ebf91840])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1278611}} - move valgrind tc builds to tier1 and stop building buildbot binaries - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/ee757cc22d46 ee757cc22d46])
|-
| in production
| 2016-06-27 18:57 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1281869&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1281869}} - Do pgo builds on-checkin for larch - r=nthomas ([https://hg.mozilla.org/build/buildbot-configs/rev/4887a9a7894d 4887a9a7894d])
* No bug - Update release config for Fennec-48.0b4-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/8c97d6107105 8c97d6107105])
|-
| in production
| 2016-06-07 13:55 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1262760&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1262760}} - Use the objdir's virtualenv's python executable. -- fixes typo - r=rail (hopefully) post-landing ([https://hg.mozilla.org/build/buildbotcustom/rev/6f6a9824ca4d 6f6a9824ca4d])
|-
| in production
| 2016-06-07 11:56 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1262760&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1262760}} - Use the objdir's virtualenv's python executable ([https://hg.mozilla.org/build/buildbotcustom/rev/370d289d34a0 370d289d34a0])
* {{bug|1262760}} - Use the objdir's virtualenv's python executable ([https://hg.mozilla.org/build/buildbotcustom/rev/646b380589aa 646b380589aa])
|-
| in production
| 2016-06-02 08:45 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1276474,1277200,1277389&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1276474}} - Enable all e10s test suites for Windows 7 on Gecko 48+ - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/6b5e33436b61 6b5e33436b61])
* {{bug|1277200}} - Bump update_verify_chunks - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/63ab033a8991 63ab033a8991])
* {{bug|1277389}} - Schedule PGO builds on Ash at 9am and 9pm if the repo has changed - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/51c2e105e9de 51c2e105e9de])
* No bug - Update release config for Fennec-47.0b10-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/b641f8dd3c7f b641f8dd3c7f])
|-
| in production
| 2016-05-30 06:58 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1276389&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1276389}} - Remove esr38 from the buildbot configs - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/1831040dc4bf 1831040dc4bf])
|-
| in production
| 2016-05-29 14:55 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1253757&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1253757}} - Switch mozilla-esr45 to release build promotion - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/d3363d0677d3 d3363d0677d3])
|-
| in production
| 2016-05-27 12:05 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1271355&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1271355}} - Disable web-platform-e10s tests on IX instances - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/4422dacf9643 4422dacf9643])
|-
| in production
| 2016-05-27 09:48 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1271355&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1271355}} - Disable more suites on IX instances - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/9c52931d383c 9c52931d383c])
|-
| in production
| 2016-05-27 07:55 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1274552&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1274552}} - Disable extra jobs on IX / GFX instances - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/c1e529438581 c1e529438581])
|-
| in production
| 2016-05-27 05:59 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* No bug - Backed out changeset 899b41a5e8e5, for xpcshellcommandline bustage ([https://hg.mozilla.org/build/buildbotcustom/rev/c543cd26f4ad c543cd26f4ad])
|-
| in production
| 2016-05-27 05:51 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1274552,1275573,1275775&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1274552}} - Move devtools, jetpack and marionette jobs to Win7 VMs ([https://hg.mozilla.org/build/buildbot-configs/rev/d15d0f3bf6aa d15d0f3bf6aa])
* {{bug|1275573}} - Adjust buildbot configs to make jamun track esr45 - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/b8bcdf1032fe b8bcdf1032fe])
* {{bug|1275775}} - Remove most spidermonkey builds, add osx builds for greening - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/e760aa2e4016 e760aa2e4016])
* {{bug|1275775}} - followup, syntax error fix - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/8ffe3a22fc9c 8ffe3a22fc9c])
* {{bug|1275775}} - Remove most spidermonkey builds, add osx builds for greening - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/be79d77a7c33 be79d77a7c33])
* No bug - Backed out changeset 8ffe3a22fc9c (bug 1275775) ([https://hg.mozilla.org/build/buildbot-configs/rev/70d9c5da9f95 70d9c5da9f95])
* No bug - Backed out changeset e760aa2e4016 (bug 1275775) ([https://hg.mozilla.org/build/buildbot-configs/rev/06ca6db17cf4 06ca6db17cf4])
* No bug - Update release config for Thunderbird-45.1.1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/7bf4cbff80e8 7bf4cbff80e8])
|-
| in production
| 2016-05-26 01:20 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1262760,1275738&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1275738}} - Stop running e10s tests on OSX 10.6 - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/f5fa52ba7df1 f5fa52ba7df1])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1262760}} - Use the objdir's virtualenv's python executable - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/899b41a5e8e5 899b41a5e8e5])
|-
| in production
| 2016-05-24 10:56 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1267712,1269872&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1267712}} - Enable build automation on the autoland repo - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/2c50bb75bc0a 2c50bb75bc0a])
* {{bug|1269872}} - enable gpu/clipboard on 49+ - r=armenzg ([https://hg.mozilla.org/build/buildbot-configs/rev/7d5af12079a6 7d5af12079a6])
* No bug - Update release config for Fennec-47.0b8-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/e765101f1d77 e765101f1d77])
|-
| in production
| 2016-05-23 09:45 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1269872&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1269872}} - add config lines for gpu/clipboard asan jobs - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/c2db2be3fd78 c2db2be3fd78])
|-
| in production
| 2016-05-20 08:37 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1255533,1269872,1274300&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1255533}} - Disable win8 by default on try - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/5f7d99a176f1 5f7d99a176f1])
* {{bug|1269872}} - Create GPU/clipboard jobs for ASAN - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/1ad733905a5b 1ad733905a5b])
* {{bug|1274300}} - Remove b-2008-ix, talos-linux32 from buildbot-configs and slavealloc - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/1995d0e79cab 1995d0e79cab])
* No bug - Update release config for Thunderbird-45.2b1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/b76200a19cd1 b76200a19cd1])
* No bug - Update release config for Thunderbird-45.2b1-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/c4f4ea55bd30 c4f4ea55bd30])
|-
| in production
| 2016-05-19 10:47 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1269872,1271394&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1269872}} - android jobs should be for 4.3, not 2.3 - r=gbrown ([https://hg.mozilla.org/build/buildbot-configs/rev/8aa5b6e8e7c1 8aa5b6e8e7c1])
* {{bug|1269872}} - remove winxp e10 jobs and win8 debug e10s jobs - r=ryanvm ([https://hg.mozilla.org/build/buildbot-configs/rev/800f339e74f1 800f339e74f1])
* {{bug|1271394}} - Enable all standard tests on projects/larch branch - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/151b97c26b65 151b97c26b65])
* No bug - No bug, fix api-11 references in staging release configs, which break staging build masters ([https://hg.mozilla.org/build/buildbot-configs/rev/f00e825ae12e f00e825ae12e])
|-
| in production
| 2016-05-17 09:46 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1223509,1269872,1273135&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1223509}} - disable w732-spot-101 for issues - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/e2b015fe1d6f e2b015fe1d6f])
* {{bug|1269872}} - remove --disable-e10s and use --e10s - r=armenzg ([https://hg.mozilla.org/build/buildbot-configs/rev/10b8e6c3c984 10b8e6c3c984])
* {{bug|1273135}} - Migrate more tests to AWS instances - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/c6b4172b5b96 c6b4172b5b96])
* No bug - Update release config for Fennec-47.0b6-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/a9a4d80011e1 a9a4d80011e1])
|-
| in production
| 2016-05-16 11:22 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1272723&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1272723}} - Remove b2g - r=nthomas ([https://hg.mozilla.org/build/buildbot-configs/rev/f373116efc7d f373116efc7d])
* {{bug|1272723}} - Remove b2g - r=nthomas ([https://hg.mozilla.org/build/buildbot-configs/rev/f373116efc7d f373116efc7d])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1272723}} - Remove b2g - r=nthomas ([https://hg.mozilla.org/build/buildbotcustom/rev/14de2833fc34 14de2833fc34])
|-
| in production
| 2016-05-16 08:15 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1194533,1236954,1269872,1270962,1271355,1272005&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1194533}} - Enable all e10s tests for OSX 10.10 on Gecko 48+ - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/343ac36a1864 343ac36a1864])
* {{bug|1269872}} - Remove mochitest-chrome from Android gpu jobs - r=jmaher ([https://hg.mozilla.org/build/buildbot-configs/rev/d99a2e77873e d99a2e77873e])
* {{bug|1270962}} - create a new clipboard job, or fix clipboard to work on windows 7 vm - r=armenzg ([https://hg.mozilla.org/build/buildbot-configs/rev/ffb96494328b ffb96494328b])
* {{bug|1271355}} - Enable win7_vm by default on try - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/af3b8066baf8 af3b8066baf8])
* {{bug|1272005}} - Disable cedar branch - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/cfbd735fa485 cfbd735fa485])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1236954}} - Add standalone partner repack factory - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/546df4a98723 546df4a98723])
|-
| in production
| 2016-05-12 08:43 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1252003,1271355,1271454&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1271355}} - Migrate some win7 test suites to AWS - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/f3e3592dffd1 f3e3592dffd1])
* {{bug|1271454}} - Create entries for 600 t-w732-spot machines in AWS - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/87d2d8f7fcd9 87d2d8f7fcd9])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1252003}} - Add publish to balrog builder - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/efdb41734f8b efdb41734f8b])
|-
| in production
| 2016-05-12 02:03 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1253341,1262760,1268542,1271550&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1253341}} - Run duplicate Talos jobs in AWS for Linux (reverted) - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/2302dc47cb21 2302dc47cb21])
* {{bug|1268542}} - Disable Linux64 Debug builds and tests in buildbot - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/cfe4c07f8161 cfe4c07f8161])
* {{bug|1271550}} - Add new 10.10.5 machines to releng configs ([https://hg.mozilla.org/build/buildbot-configs/rev/750ab5df5a06 750ab5df5a06])
* No bug - Update release config for Thunderbird-45.1.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/48e0db6d4d22 48e0db6d4d22])
* No bug - Update release config for Thunderbird-45.1.0-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/c199b13b84a3 c199b13b84a3])
* No bug - Backed out changeset c199b13b84a3 to let TB 45.1.0 build3 to restart ([https://hg.mozilla.org/build/buildbot-configs/rev/a55a7e24bba1 a55a7e24bba1])
* No bug - Update release config for Thunderbird-45.1.0-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/3a7c1d3dbace 3a7c1d3dbace])
* No bug - Update release config for Fennec-47.0b4-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/f7b8a43eed4f f7b8a43eed4f])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1262760}} - Use the objdir's virtualenv's python executable - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/a15cf826c0cb a15cf826c0cb])
|-
| in production
| 2016-05-06 09:58 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1262760,1268542,1269543,1270881&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1268542}} - Disable Linux64 Debug builds and tests in buildbot - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/eefa2e45cd9e eefa2e45cd9e])
* {{bug|1269543}} - Stop running tests on OS X 10.6 on Firefox 49+ - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/f88f7f57f714 f88f7f57f714])
* {{bug|1270881}} - Make the mochitest-gl timeout 5400s for both variants - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/29f4c35c876b 29f4c35c876b])
* No bug - Update release config for Thunderbird-45.1.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/ffcc165163a8 ffcc165163a8])
* No bug - backout bug 1268542 - Disable Linux64 Debug builds and tests in buildbot ([https://hg.mozilla.org/build/buildbot-configs/rev/9f885780ab86 9f885780ab86])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1262760}} - Use the virtualenv's python with printconfigsetting.py - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/a6b085c766b5 a6b085c766b5])
|-
| in production
| 2016-05-02 13:55 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1214487&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1214487}} - Add mapping in common for uptake monitoring - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/7d70b0149b14 7d70b0149b14])
|-
| in production
| 2016-05-02 13:00 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1214487,1268895&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1214487}} - Add uptake monitoring builder config - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/07af46009466 07af46009466])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1214487}} - Add uptake monitoring builder - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/6a21eadc4deb 6a21eadc4deb])
* {{bug|1268895}} - Remove unused l10n dep logic - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/a0def3e44f74 a0def3e44f74])
|-
| in production
| 2016-04-27 10:34 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=787208&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|787208}} - Fix cypress repo_path by using common defaults - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/6403d9027a45 6403d9027a45])
* {{bug|787208}} - fix busted patch again - r=bustage ([https://hg.mozilla.org/build/buildbot-configs/rev/af5f7251deeb af5f7251deeb])
* No bug - Update release config for Fennec-47.0b1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/b47a0268849d b47a0268849d])
|-
| in production
| 2016-04-19 06:35 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1264407,1265425&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1264407}} - Switch Thunderbird ESR45 testing to r7 minis - r=aselagea ([https://hg.mozilla.org/build/buildbot-configs/rev/7fe1919120db 7fe1919120db])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1265425}} - Bump partner repacks script_maxtime - r=coop ([https://hg.mozilla.org/build/buildbotcustom/rev/700093ba063c 700093ba063c])
* {{bug|1265425}} - Use slave platform instead of target platform - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/901cbfd439c0 901cbfd439c0])
|-
| in production
| 2016-04-19 05:32 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1260415&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1260415}} - Explicitly set basedir for all platforms - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/5557a28843c5 5557a28843c5])
|-
| in production
| 2016-04-13 06:07 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1212993,1253341,1260415,1262533,1263279,1263385,1263402,1263918&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1253341}} - Run duplicate Talos jobs in AWS for Linux (large instances). r?rail ([https://hg.mozilla.org/build/buildbot-configs/rev/609fef319188 609fef319188])
* {{bug|1263279}} - add balrog rules for TB 45.0ESR builds in release configs - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/4a4c57503630 4a4c57503630])
* {{bug|1263385}} - Increase the number of wpt[-e10s] opt chunks to 5 and debug to 10 - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/ee9a45746a10 ee9a45746a10])
* {{bug|1263402}} - Remove symbol server config for TB builds - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/312fc115e5ab 312fc115e5ab])
* {{bug|1263918}} - Enable release-signing on jamun to run full tests - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/f35a9ae54e4d f35a9ae54e4d])
* No bug - Update release config for Fennec-46.0b10-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/f22ead008630 f22ead008630])
* No bug - Update release config for Firefox-45.0.2esr-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/f44ca27a5aa3 f44ca27a5aa3])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1212993}} - remove get basedir step - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/493648951b51 493648951b51])
* {{bug|1260415}} - Schedule "partner repacks" builder - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/02f5c29c5b11 02f5c29c5b11])
* {{bug|1262533}} - bbot-custom - add .idea dir to list of hgignore files for buildbot and tools repos, r?rail ([https://hg.mozilla.org/build/buildbotcustom/rev/d810ea7559de d810ea7559de])
|-
| in production
| 2016-04-04 13:40 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1253341,1259524,1260415,1260879,1261045&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1253341}} - Disable duplicate talos jobs in AWS - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/c06fa02837e9 c06fa02837e9])
* {{bug|1259524}} - release promotion - add staging mozilla-release configs to buildbot - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/fe28dc4e3993 fe28dc4e3993])
* {{bug|1260415}} - buildboti-config: add "partner repacks" builder - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/3e0f1271d6b1 3e0f1271d6b1])
* {{bug|1260879}} - removes bouncer_branch and add postrelease_bouncer_aliases items to jamun, r?kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/ce584e727954 ce584e727954])
* {{bug|1261045}} - remove symobl-server configuration - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/fa3ae48d4723 fa3ae48d4723])
* No bug - Update release config for Fennec-46.0b7-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/3fb326e95b27 3fb326e95b27])
* No bug - Update release config for Thunderbird-38.7.2-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/78ef71c34627 78ef71c34627])
* No bug - Update release config for Thunderbird-45.0b4-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/653739ef781d 653739ef781d])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1260415}} - buildbotcustom: add "partner repacks" builder - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/430fd7c357f7 430fd7c357f7])
* {{bug|1260879}} - bbotcustom removes bouncer_branch from bouncer submitter factory, r?kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/4461354684f3 4461354684f3])
|-
| in production
| 2016-03-29 13:32 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1256063&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1256063}} - Add update bouncer aliases builder - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/3d7b8b58d31e 3d7b8b58d31e])
* No bug - Update release config for Thunderbird-45.0b3-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/f1695648974e f1695648974e])
* No bug - Update release config for Fennec-46.0b6-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/07fe251fbd29 07fe251fbd29])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1256063}} - Add update bouncer aliases builder - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/fbdc1775117a fbdc1775117a])
|-
| in production
| 2016-03-21 17:41 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1249528&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1249528}} - releaserunner should handle RC channels. Fix product - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/40258e329c78 40258e329c78])
|-
| in production
| 2016-03-21 14:15 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1249528&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1249528}} - releaserunner should handle RC channels - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/42e571aa53d8 42e571aa53d8])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1249528}} - Unify update verify builders - r=Callek ([https://hg.mozilla.org/build/buildbotcustom/rev/139b15174c5d 139b15174c5d])
|-
| in production
| 2016-03-18 12:37 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1253341&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1253341}} - Allow running talos on AWS machines - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/ddb55938d842 ddb55938d842])
|-
| in production
| 2016-03-16 10:35 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1252292,1253341,1257230&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1252292}} - Don't run Ru-e10s on debug builds - r=me ([https://hg.mozilla.org/build/buildbot-configs/rev/04c8fe62d3a4 04c8fe62d3a4])
* {{bug|1253341}} - Enable talos on linux64 AWS machines on try - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/2d6158d681e0 2d6158d681e0])
* {{bug|1257230}} - Incorrect branding for 45.0.1esr build - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/b5163b4a7814 b5163b4a7814])
* No bug - Update release config for Fennec-46.0b2-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/216ac6505867 216ac6505867])
* No bug - Update release config for Firefox-38.7.1esr-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/2b0b8272dda2 2b0b8272dda2])
* No bug - backouts release change for 38.7.1esr - rev: 2b0b8272dda2 ([https://hg.mozilla.org/build/buildbot-configs/rev/ea85259cf436 ea85259cf436])
* No bug - merges backout 2b0b8272dda2 -> default ([https://hg.mozilla.org/build/buildbot-configs/rev/7b0276c224b4 7b0276c224b4])
* No bug - Update release config for Firefox-38.7.1esr-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/b735b52c0d5a b735b52c0d5a])
* No bug - Update release config for Firefox-45.0.1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/b3267390eb7d b3267390eb7d])
* No bug - Update release config for Firefox-45.0.1esr-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/1f7801999788 1f7801999788])
* No bug - Update release config for Fennec-45.0.1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/e759c66d311c e759c66d311c])
* No bug - Backed out 2d6158d681e0 for suspicion of causing intermittent checkconfig ([https://hg.mozilla.org/build/buildbot-configs/rev/7c463958098d 7c463958098d])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1253341}} - Allow running talos on AWS machines - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/42184dbdf8f3 42184dbdf8f3])
* No bug - Backed out changeset 42184dbdf8f3 for suspicion of causing intermittent checkconfig failureis ([https://hg.mozilla.org/build/buildbotcustom/rev/046c46eb099d 046c46eb099d])
|-
| in production
| 2016-03-15 07:43 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1249528,1252292,1255812,1256184&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1249528}} - releaserunner should handle RC channels - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/5235b4ceb15e 5235b4ceb15e])
* {{bug|1249528}} - Bustage fix - r=me ([https://hg.mozilla.org/build/buildbot-configs/rev/806fd2074e58 806fd2074e58])
* {{bug|1249528}} - backout - r=me ([https://hg.mozilla.org/build/buildbot-configs/rev/fc1e57afc013 fc1e57afc013])
* {{bug|1252292}} - Set Ash to the same branch priority as mozilla-central - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/50c1fdd4ff3e 50c1fdd4ff3e])
* {{bug|1255812}} - Move 10 XP test machines to the Windows 8 pool - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/36eb260ec81c 36eb260ec81c])
* {{bug|1256184}} - Remove code that unconditionally enables Talos g2 irrespective of the branch's enable_talos setting - r=jmaher ([https://hg.mozilla.org/build/buildbot-configs/rev/ec3937ad8bd7 ec3937ad8bd7])
|-
| in production
| 2016-03-11 09:50 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1186617,1242682,1245585,1250999,1255078&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1242682}} - Separate dom/media into its own subsuite - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/cc9e268a0036 cc9e268a0036])
* {{bug|1245585}} - turn off android talos tests which run on the panda boards - r=aselagea ([https://hg.mozilla.org/build/buildbot-configs/rev/771999c6f4ef 771999c6f4ef])
* {{bug|1250999}} - releng - remove api-9-10 from trunk - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/1bbe091937b4 1bbe091937b4])
* No bug - Update release config for Thunderbird-38.7.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/cf9b859c9d8c cf9b859c9d8c])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1186617}} - releng changes to decommission pandas - r=aselagea ([https://hg.mozilla.org/build/buildbotcustom/rev/695299094ba0 695299094ba0])
* {{bug|1255078}} - Android Beta tests broken: Found installer url ~/bouncer.apk - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/b4c6ed50c14c b4c6ed50c14c])
|-
| in production
| 2016-03-10 06:17 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1245585,1253858,1255088&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1245585}} - turn off android talos tests which run on the panda boards - r=aselagea ([https://hg.mozilla.org/build/buildbot-configs/rev/c58e5674bae2 c58e5674bae2])
* {{bug|1245585}} - turn off android talos tests which run on the panda boards - r=aselagea ([https://hg.mozilla.org/build/buildbot-configs/rev/87168a6da9f1 87168a6da9f1])
* {{bug|1253858}} - Turn off mochitest-push on esr45 - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/95e902ea4772 95e902ea4772])
* {{bug|1255088}} - turn on talos e10s for mozilla-beta, remove some other misconfigurations - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/8a6d6e31623f 8a6d6e31623f])
|-
| in production
| 2016-03-09 06:14 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1166420,1242682,1251351,1251477,1254902&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1166420}} - Add relengapi.tok to TB bb configs - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/d7776ec49bb4 d7776ec49bb4])
* {{bug|1242682}} - Separate dom/media into its own subsuite - r=armenzg ([https://hg.mozilla.org/build/buildbot-configs/rev/dd518a784a46 dd518a784a46])
* {{bug|1251351}} - Add config file options for marionette-e10s on Windows - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/1ce5d30d8e7c 1ce5d30d8e7c])
* {{bug|1251477}} - rename update verify configs - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/aed2ba7655a0 aed2ba7655a0])
* {{bug|1254902}} - Add relengapi.tok to buildbot configs - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/ddf4c824ca46 ddf4c824ca46])
|-
| in production
| 2016-03-07 19:31 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1254337&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1254337}} - fix tuple typo for beta releasetasks items, bbotcfgs, r?rail ([https://hg.mozilla.org/build/buildbot-configs/rev/b9a2659ad8fe b9a2659ad8fe])
|-
| in production
| 2016-03-07 16:12 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1251351,1251936,1254337&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1251351}} - Schedule Mn-e10s tests on win7 - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/8a111c694d59 8a111c694d59])
* {{bug|1251936}} - Part 3: Add buildbot configs for Windows x86 static analysis buillds - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/ae056e1b3ad7 ae056e1b3ad7])
* {{bug|1254337}} - use release-signing on mozilla-beta and disable push2releases automatic, bbotcfgs, r?rail ([https://hg.mozilla.org/build/buildbot-configs/rev/75be1891c7f8 75be1891c7f8])
* No bug - Update release config for Fennec-46.0b1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/236b949b7359 236b949b7359])
|-
| in production
| 2016-03-07 05:36 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1243796&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1243796}} - Tracking bug for Mar-7-2016 migration work - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/f03990b8f382 f03990b8f382])
|-
| in production
| 2016-03-04 13:23 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1253492,1253508,1253710,1253714&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1253492}} - Add missing e10s defines to un-break Mn-e10s and M(gl) jobs - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/7b8c9c7698ed 7b8c9c7698ed])
* {{bug|1253508}} - Let e10s jobs ride the trains - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/1e47404949d2 1e47404949d2])
* {{bug|1253710}} - Enable OSX 10.10 opt e10s tests on Gecko 47 - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/15dc4b9def85 15dc4b9def85])
* {{bug|1253714}} - Enable OSX 10.10 debug e10s browser-chrome and Marionette tests on Gecko 47 - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/e5cc6f88dcf5 e5cc6f88dcf5])
|-
| in production
| 2016-03-04 12:26 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1251477&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1251477}} - prep buildbot configs for release promotion in production on beta - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/57c166d8109b 57c166d8109b])
|-
| in production
| 2016-03-04 09:03 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1242682,1253410,1253437,1253492&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1242682}} - Separate dom/media into its own subsuite [fix e10s config-file] - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/f0acc9241544 f0acc9241544])
* {{bug|1253410}} - Enable ASAN builds/tests on ESR45 - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/e254fe53e290 e254fe53e290])
* {{bug|1253437}} - Split Linux debug/ASAN xpcshell into 4 chunks - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/9a94815dc367 9a94815dc367])
* {{bug|1253492}} - Enable more e10s test suites on Linux - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/8fb9139950ba 8fb9139950ba])
* {{bug|1253492}} - Fix up duplicate builder bustage - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/6535597727b5 6535597727b5])
|-
| in production
| 2016-03-04 07:06 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1237094,1242682&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1237094}} - tracking bug for build and release of Firefox 45.0 ESR - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/f65f8e566a68 f65f8e566a68])
* {{bug|1242682}} - Separate dom/media into its own subsuite - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/ad171d57880d ad171d57880d])
|-
| in production
| 2016-03-03 16:54 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1192994,1252248,1252809,1253364,1253429&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1192994}} - Handle Linux64 data from SETA - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/8382286ea52c 8382286ea52c])
* {{bug|1252248}} - Add more tst-linux64, tst-emulator64 capacity in AWS - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/bc958535ce5a bc958535ce5a])
* {{bug|1252809}} - Only test pretty names on comm train branches, not comm-central - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/18c0c4c99064 18c0c4c99064])
* {{bug|1253364}} - Clean up list of machines that aren't in slavealloc - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/b86333d2e140 b86333d2e140])
* {{bug|1253429}} - production-ify release promotion configs, use existing bucket - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/858343665d94 858343665d94])
* No bug - Update release config for Firefox-38.7.0esr-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/39239fcf0362 39239fcf0362])
* No bug - Update release config for Fennec-45.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/20518d5e75df 20518d5e75df])
* No bug - Update release config for Fennec-45.0-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/92ab47cbc53a 92ab47cbc53a])
* No bug - Update release config for Firefox-45.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/7f197028d1d9 7f197028d1d9])
* No bug - Update release config for Firefox-45.0esr-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/08355f5f053b 08355f5f053b])
* No bug - Backed out 3 changesets on default to restart releases ([https://hg.mozilla.org/build/buildbot-configs/rev/436d072f5952 436d072f5952])
* No bug - Update release config for Fennec-45.0-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/fc5ddedd1c1d fc5ddedd1c1d])
* No bug - Update release config for Firefox-45.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/b51aad4f9e4b b51aad4f9e4b])
* No bug - Update release config for Firefox-45.0esr-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/13e413a55b5b 13e413a55b5b])
|-
| in production
| 2016-03-02 13:51 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1237094,1251381,1252756&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1237094}} - tracking bug for build and release of Firefox 45.0 ESR - r=nthomas ([https://hg.mozilla.org/build/buildbot-configs/rev/91b85a6e4c1e 91b85a6e4c1e])
* {{bug|1251381}} - Switch gtests to riding 46 - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/2a06d32c80b9 2a06d32c80b9])
* {{bug|1252756}} - Shut off periodic PGO on b2g-inbound - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/6c131bed7fae 6c131bed7fae])
* No bug - No bug - shorten travis results to be less spammy - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/fb1af69edbec fb1af69edbec])
* No bug - Update release config for Fennec-45.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/38e4592e3d57 38e4592e3d57])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* No bug - No Bug - shorten travis results notification to be less spammy - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/4b96f5cb0d0f 4b96f5cb0d0f])
|-
| in production
| 2016-03-02 06:25 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1237094,1252258,1252328&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1237094}} - tracking bug for Firefox 45.0 ESR, channel rules - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/4b855f6b6eae 4b855f6b6eae])
* {{bug|1252258}} - Move 8 w10 test machines to the Windows 7 pool - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/25b7652be68b 25b7652be68b])
* {{bug|1252328}} - Only run e10s tests on Gecko 46+ - r= ([https://hg.mozilla.org/build/buildbot-configs/rev/e9147a125ebc e9147a125ebc])
|-
| in production
| 2016-03-01 07:30 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1242682,1249376&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1242682}} - Separate dom/media into its own subsuite - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/bf7f47ae7e0f bf7f47ae7e0f])
* {{bug|1249376}} - Move 20 XP test machines to the Windows 7 pool - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/dee25cbffdcd dee25cbffdcd])
* No bug - Update release config for Fennec-45.0b11-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/b42fec100a37 b42fec100a37])
* No bug - Update release config for Firefox-45.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/e172a9923d43 e172a9923d43])
|-
| in production
| 2016-02-29 08:10 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1249682&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1249682}} - Remove addon-sdk buildbot configs - r=Callek ([https://hg.mozilla.org/build/buildbotcustom/rev/01d5bda06fc9 01d5bda06fc9])
|-
| in production
| 2016-02-25 09:50 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1237094,1251144,1251285&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1237094}} - tracking bug for build and release of Firefox 45.0 ESR - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/8712d19cff79 8712d19cff79])
* {{bug|1251144}} - Don't merge release promotion builders - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/1668779e0c88 1668779e0c88])
* {{bug|1251285}} - Disable alder project branch ([https://hg.mozilla.org/build/buildbot-configs/rev/81db79611afa 81db79611afa])
* {{bug|1251285}} - Disable alder project branch - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/0441563764d3 0441563764d3])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1251144}} - Don't merge release promotion builders - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/0cbca385a7e2 0cbca385a7e2])
|-
| in production
| 2016-02-24 13:45 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1250585&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1250585}} - adds push to mirrors automatic date, buildbot-configs - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/8ff751a38618 8ff751a38618])
* No bug - Update release config for Fennec-45.0b9-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/fc63a5db2c0c fc63a5db2c0c])
* No bug - Update release config for Firefox-45.0b9-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/884497cab040 884497cab040])
|-
| in production
| 2016-02-23 13:53 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1245559,1250306&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1245559}} - Enable wpt e10s on Windows 7 - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/17cc02d08629 17cc02d08629])
* {{bug|1245559}} - Enable wpt e10s on Windows 7 - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/646145351982 646145351982])
* {{bug|1250306}} - Point date tools repo at rail's fork - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/9bae3c5f95f2 9bae3c5f95f2])
* No bug - Backed out changeset 17cc02d08629 ([https://hg.mozilla.org/build/buildbot-configs/rev/a06e78e32bcb a06e78e32bcb])
* No bug - [mq]: tb.cypress.tests ([https://hg.mozilla.org/build/buildbot-configs/rev/4218b4cabec6 4218b4cabec6])
|-
| in production
| 2016-02-22 13:17 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1248474&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* No bug - Update release config for Firefox-45.0b8-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/5f9a37958eaf 5f9a37958eaf])
* No bug - Update release config for Firefox-45.0b8-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/5f9a37958eaf 5f9a37958eaf])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1248474}} - Don't use a slash in buildername - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/c87b90ed718d c87b90ed718d])
* {{bug|1248474}} - Don't use a slash in buildername - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/c87b90ed718d c87b90ed718d])
|-
| in production
| 2016-02-19 13:00 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1248474&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1248474}} - [RelPromo] Run Mac/Windows update verification via BBB - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/6035a0b57bfb 6035a0b57bfb])
|-
| in production
| 2016-02-19 07:03 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1241955,1246538,1248474,1248729,1249383&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1241955}} - Enable tests on date - r=callek ([https://hg.mozilla.org/build/buildbot-configs/rev/cc561d788755 cc561d788755])
* {{bug|1248474}} - [RelPromo] Run Mac/Windows update verification via BBB - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/437d703fad2b 437d703fad2b])
* {{bug|1248729}} - Schedule e10s mochitest-push on Windows 7 patch by aselagea - r=jgriffin ([https://hg.mozilla.org/build/buildbot-configs/rev/2999a899e438 2999a899e438])
* {{bug|1249383}} - disable talos on b2g-inbound - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/279753fc211c 279753fc211c])
* No bug - Update release config for Firefox-45.0b7-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/87671b77f024 87671b77f024])
* No bug - Update release config for Thunderbird-45.0b2-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/2706c3b66654 2706c3b66654])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1246538}} - mochitest-o should trigger mochitest-a11y and mochitest-chrome-* patch by aselagea - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/926a0351a0d7 926a0351a0d7])
* {{bug|1248474}} - [RelPromo] Run Mac/Windows update verification via BBB - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/eb8290943fcf eb8290943fcf])
|-
| in production
| 2016-02-17 08:46 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1236954,1246627,1247078&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1246627}} - Schedule e10s jsreftests on win7 patch by aselagea - r=jgriffin ([https://hg.mozilla.org/build/buildbot-configs/rev/cc37bdb15492 cc37bdb15492])
* No bug - revert Bug 1246627 - Schedule e10s jsreftests on win7 ([https://hg.mozilla.org/build/buildbot-configs/rev/71b9ccb8abf1 71b9ccb8abf1])
* No bug - Update release config for Fennec-45.0b6-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/69f84fb1f35b 69f84fb1f35b])
* No bug - Update release config for Firefox-45.0b6-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/df4ed20df968 df4ed20df968])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1236954}} - add standalone partner repack factory - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/a7bd1793a7ab a7bd1793a7ab])
* {{bug|1247078}} - [tryparser] Don't schedule any jobs if no try syntax is detected - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/46fb74d54d79 46fb74d54d79])
|-
| in production
| 2016-02-14 08:43 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1244510,1245605,787208&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1245605}} - Use 'dumb' source mozconfig for TB beta builds following bug 123448 - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/22f1e00d700c 22f1e00d700c])
* {{bug|787208}} - Followup, don't set mozilla_dir - r=no-reviewer ([https://hg.mozilla.org/build/buildbot-configs/rev/8c7677a7cdb6 8c7677a7cdb6])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1244510}} - Add a try_parser alias for mochitest-e10s-(bc|browser-chrome) patch by aselagea - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/c994ec64ba03 c994ec64ba03])
|-
| in production
| 2016-02-12 07:43 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1218589,1229532,1246627,787208&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1218589}} - Nightly mozilla-central builds of graphene patch by aselagea - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/f88d45c3fbe6 f88d45c3fbe6])
* {{bug|1229532}} - remove hg bundle builder config - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/f0c9c5b5a7b8 f0c9c5b5a7b8])
* {{bug|1246627}} - Schedule e10s jsreftests on win7 patch by aselagea - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/66bc120b896c 66bc120b896c])
* {{bug|787208}} - In support of merging comm-central and mozilla-central together, configure cypress as a Thunderbird branch. r=rail - r=Fallen f=jcranmer ([https://hg.mozilla.org/build/buildbot-configs/rev/b92aaf935173 b92aaf935173])
* No bug - Update release config for Firefox-38.6.1esr-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/33fe33d65478 33fe33d65478])
* No bug - Update release config for Firefox-44.0.2-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/09cd8415adb9 09cd8415adb9])
* No bug - Update release config for Thunderbird-38.6.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/91cf6aa50268 91cf6aa50268])
* No bug - Update release config for Firefox-45.0b5-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/6376fd9567c4 6376fd9567c4])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1229532}} - remove hg bundle builders - r=bhearsum ([https://hg.mozilla.org/build/buildbotcustom/rev/1b880a68c991 1b880a68c991])
|-
| in production
| 2016-02-02 14:53 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1242103&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1242103}} - Add browser-chrome-screenshots to m-c and other try - r=armenzg ([https://hg.mozilla.org/build/buildbot-configs/rev/63614b6564ee 63614b6564ee])
* No bug - Update release config for Fennec-45.0b2-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/7a121e3abe60 7a121e3abe60])
* No bug - Update release config for Firefox-45.0b2-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/a70a9918a489 a70a9918a489])
* No bug - Update release config for Thunderbird-45.0b1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/39642e4c3ba4 39642e4c3ba4])
|-
| in production
| 2016-01-26 20:05 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1190860&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1190860}} - fix valgrind by setting up mock before running tooltool, r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/28864afc6409 28864afc6409])
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1190860}} - Copy in tooltool.py, and run it from in-mock location (with TB too), r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/10af9c3220aa 10af9c3220aa])
|-
| in production
| 2016-01-26 18:05 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1190860&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1190860}} - followup fix r=Callek over irc ([https://hg.mozilla.org/build/buildbotcustom/rev/a4bc25f76f2d a4bc25f76f2d])
* {{bug|1166464}} - followup to fix reconfig, r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/de981e55b123 de981e55b123])
|-
| in production
| 2016-01-26 16:00 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1242374,1166464,1242641,1190860,1211889&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1242374}} - set an sdkUrl to avoid interfering with packageUrl; r=bhearsum ([https://hg.mozilla.org/build/buildbotcustom/rev/34f988f27f10 34f988f27f10])
* {{bug|1166464}} - Bump version after shipping release builds r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/97febf14fe8d 97febf14fe8d])
* {{bug|1242641}} - backout 3b001e0b8438 ([https://hg.mozilla.org/build/buildbotcustom/rev/360f6f0cf623 360f6f0cf623])
* {{bug|1242641}} - backout 31ff0be74400 ([https://hg.mozilla.org/build/buildbotcustom/rev/9b9b645d00a1 9b9b645d00a1])
* {{bug|1190860}} - Run tooltool inside mock, r=Callek ([https://hg.mozilla.org/build/buildbotcustom/rev/889eb503d966 889eb503d966])
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1211889}} - split mochitest-oth into- mochitest-a11y (a), mochitest-chrome (c1, c2, c3) for linux32 debug. r=armenzg ([https://hg.mozilla.org/build/buildbot-configs/rev/d9e027f12297 d9e027f12297])
* {{bug|1166464}} - Bump version after shipping release builds r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/bb2664d7bf15 bb2664d7bf15])

|-
| in production
| 2016-01-25 20:47 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1242641&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1242641}} - followup to fix cwd for mock call, and use python from the virtualenv ([https://hg.mozilla.org/build/buildbotcustom/rev/3b001e0b8438 3b001e0b8438])
|-
| in production
| 2016-01-25 14:54 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1242641&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1242641}} - fix printconfigsetting.py calls for Firefox on m-b and higher, r=Callek ([https://hg.mozilla.org/build/buildbotcustom/rev/31ff0be74400 31ff0be74400])
|-
| in production
| 2016-01-25 05:50 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1232329&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1232329}} - Tracking bug for Jan-25-2016 migration work - r=trivial ([https://hg.mozilla.org/build/buildbot-configs/rev/c38c7f7b76e9 c38c7f7b76e9])
* No bug - Update release config for Firefox-44.0-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/3941c0a6daf6 3941c0a6daf6])
|-
| in production
| 2016-01-21 12:12 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1235961&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1235961}} - set tools_repo_cache for mac platforms - r=bustage ([https://hg.mozilla.org/build/buildbot-configs/rev/fa16bb751b50 fa16bb751b50])
|-
| in production
| 2016-01-21 10:56 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1231408,1233005,1236951,1241159,1241481&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1231408}} - Add browser-chrome-screenshots to yosemite_r7 on Try - r=armenzg ([https://hg.mozilla.org/build/buildbot-configs/rev/b74388666f8e b74388666f8e])
* {{bug|1233005}} - Enable packageSDK in release configs - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/9f0795fa7184 9f0795fa7184])
* No bug - Update release config for Fennec-44.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/2b71401d73e8 2b71401d73e8])
* No bug - Update release config for Firefox-44.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/0154167166dd 0154167166dd])
* No bug - Update release config for Firefox-38.6.0esr-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/fa10d55edf12 fa10d55edf12])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1233005}} - Set packageSDK if enabled in the config - r=bhearsum ([https://hg.mozilla.org/build/buildbotcustom/rev/5d15ab1c5ddd 5d15ab1c5ddd])
* {{bug|1236951}} - set tools_repo_cache for partner repacks - r=bustage ([https://hg.mozilla.org/build/buildbotcustom/rev/c06e740a09e4 c06e740a09e4])
* {{bug|1241159}} - handle mochites-e10s-devtools-chrome as an alias for the e10s devtools chunks - r=Callek ([https://hg.mozilla.org/build/buildbotcustom/rev/25315f9d0e2e 25315f9d0e2e])
* {{bug|1241481}} - Get rid of script_revlink - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/dc9b1cf7b9e1 dc9b1cf7b9e1])
|-
| in production
| 2016-01-20 06:16 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1236951&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* No bug - Update release config for Fennec-44.0b11-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/370703910927 370703910927])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1236951}} - use tools_repo_cache for toolsdir when set - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/6368247d8c7a 6368247d8c7a])
* {{bug|1236951}} - use WithProperties in case we actually have a revision - r=bustage ([https://hg.mozilla.org/build/buildbotcustom/rev/b049ab7ef9a7 b049ab7ef9a7])
|-
| in production
| 2016-01-19 07:36 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1225899,1236835,1236951,1239785&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1225899}} - add beetmover candidates flag to date config - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/fca43032761e fca43032761e])
* {{bug|1236951}} - Update partner repack stanza in release templates - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/2f6ad8dd5b7e 2f6ad8dd5b7e])
* {{bug|1239785}} - Move some XP test machines to the Windows 7 pool - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/31b0f9f1195c 31b0f9f1195c])
* No bug - Update release config for Fennec-44.0b10-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/cfff00d1a9f5 cfff00d1a9f5])
* No bug - Update release config for Firefox-44.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/8a87f5fe2b16 8a87f5fe2b16])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1236835}} - Disable all B2G Desktop Builds & Tests on All trees. Cleanup - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/e0ab5efe7082 e0ab5efe7082])
|-
| in production
| 2016-01-18 06:19 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1151954,1232329,1239025&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1232329}} - Tracking bug for Jan-25-2016 migration work - r=mtabara ([https://hg.mozilla.org/build/buildbot-configs/rev/a0007d5cd72b a0007d5cd72b])
* {{bug|1239025}} - Turn on reftest for e10s on win7 - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/c257ac5cf785 c257ac5cf785])
[https://hg.mozilla.org/build/mozharness mozharness]
* {{bug|1151954}} - move get_apk.sh to a mozharness file - r=sylvestre ([https://hg.mozilla.org/build/mozharness/rev/15845edae9a1 15845edae9a1])
|-
| in production
| 2016-01-15 08:55 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1236951&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* No bug - Update release config for Fennec-44.0b9-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/15dc357fc5bd 15dc357fc5bd])
* No bug - Update release config for Firefox-44.0b9-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/df3ee52930d6 df3ee52930d6])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1236951}} - Set script_repo_revision for ScriptFactory when archiver is used - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/aa885f5cdbc3 aa885f5cdbc3])
|-
| in production
| 2016-01-13 15:03 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1226180,1236835,1236951,1239074&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1226180}} - decommission t-yosemite-r5 machines - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/b05f8de5ec9d b05f8de5ec9d])
* {{bug|1236835}} - Disable all B2G Desktop Builds & Tests on All trees - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/66b14e07db81 66b14e07db81])
* {{bug|1236951}} - Re-enable partner repacks on beta and release - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/5bdbf1ac3f5f 5bdbf1ac3f5f])
* {{bug|1239074}} - Add trailing slash to ${MOZILLABUILD} so that PATH gets properly created - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/388b403428e2 388b403428e2])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1236951}} - Replace PartnerRepackFactory with a SigningScriptFactory instance - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/089b94bab05c 089b94bab05c])
|-
| in production
| 2016-01-12 09:04 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1238947&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1238947}} - Disable b2g37_v2.2 automation - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/4989f1200474 4989f1200474])
* No bug - Update release config for Fennec-44.0b8-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/e8d645fd93cd e8d645fd93cd])
* No bug - Update release config for Firefox-44.0b8-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/6196d2a86352 6196d2a86352])
* No bug - Update release config for Thunderbird-44.0b1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/6580fa6c368b 6580fa6c368b])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1238947}} - Disable b2g37_v2.2 automation - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/2135d176b8e8 2135d176b8e8])
[https://hg.mozilla.org/build/mozharness mozharness]
* {{bug|1238947}} - Disable b2g37_v2.2 automation - r=rail ([https://hg.mozilla.org/build/mozharness/rev/bcdac5ca4d07 bcdac5ca4d07])
|-
| in production
| 2016-01-11 08:10 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1238083&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1238083}} - Clean up stage tasks - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/0d695ad18d3f 0d695ad18d3f])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1238083}} - Clean up stage tasks - r=bhearsum ([https://hg.mozilla.org/build/buildbotcustom/rev/65c073287d56 65c073287d56])
|-
| in production
| 2016-01-11 07:41 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1073442,1237985&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1073442}} - adjust chunks from 5 to 8 for debug mochitests - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/5ae05a86c3ab 5ae05a86c3ab])
* {{bug|1237985}} - Disable b2g37_v2.2R automation - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/00c4b51cb1fb 00c4b51cb1fb])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1237985}} - Disable b2g37_v2.2R automation - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/7eb568e273d8 7eb568e273d8])
[https://hg.mozilla.org/build/mozharness mozharness]
* {{bug|1237985}} - Disable b2g37_v2.2R automation - r=rail ([https://hg.mozilla.org/build/mozharness/rev/9effeabb1c18 9effeabb1c18])
* {{bug|1237985}} - Disable b2g37_v2.2R automation - r=rail ([https://hg.mozilla.org/build/mozharness/rev/8da7cb911d70 8da7cb911d70])
* No bug - Revert Bug 1237985 - Disable b2g37_v2.2R automation - r=rail ([https://hg.mozilla.org/build/mozharness/rev/cdc0ebfcef83 cdc0ebfcef83])
|-
| in production
| 2016-01-07 06:39 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1211770,1215204,1221963,1229790,1232780&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1211770}} - Disable HSTS/HPKP updates on b2g 2.2 branches - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/65d3648d9b27 65d3648d9b27])
* {{bug|1221963}} - Schedule media-youtube-tests on trunk branches - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/f7d9ee4eab35 f7d9ee4eab35])
* {{bug|1229790}} - add ec2 windows slaves to configs, restricted to try - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/4a61c10820eb 4a61c10820eb])
* {{bug|1232780}} - Schedule e10s mochitest-plain on win7-opt - r=jgriffin ([https://hg.mozilla.org/build/buildbot-configs/rev/ea429699060a ea429699060a])
* No bug - Update release config for Firefox-43.0.4-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/1bc16691f520 1bc16691f520])
* No bug - Update release config for Firefox-43.0.4-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/4b67594abc24 4b67594abc24])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1215204}} - Add bouncer submitter to release promotion graph - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/93196910e3df 93196910e3df])
* {{bug|1215204}} - Add bouncer submitter to release promotion graph - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/4cda855c1404 4cda855c1404])
* No bug - revert Bug 1215204 - Add bouncer submitter to release promotion graph - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/8dda9c1cce64 8dda9c1cce64])
|}

ReleaseEngineering/Maintenance

2016-10-26T01:21:37Z

NThomas: /* Reconfigs / Deployments */ bug 1312650

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

[[ReleaseEngineering:BuildbotBestPractices]] describes how we manage changes to our master buildbots.

= Relevant repositories =
* [https://hg.mozilla.org/build/buildbot/ buildbot]
* [https://hg.mozilla.org/build/buildbot-configs/ buildbot-configs]
* [https://hg.mozilla.org/build/buildbotcustom/ buildbotcustom]
* [https://hg.mozilla.org/build/tools/ tools]
* [https://hg.mozilla.org/build/mozharness/ mozharness]
* [https://hg.mozilla.org/build/puppet/ puppet]

= Non-releng repositories involved in the CI =
== [[Auto-tools/Projects/Treeherder|Treeherder]] ==
* [http://treeherderadm.private.scl3.mozilla.com/chief/treeherder.prod/history Chief self-deploy recent history] (needs VPN; more readable than raw logs, but is cleared if the Chief service is restarted)
* [treeherderadm.private.scl3.mozilla.com/chief/treeherder.prod/logs/ Chief self-deploy logs archive] (needs VPN; use if the history page is empty)

= Mozharness deployments =
Merges to the <em>production</em> branch of [http://hg.mozilla.org/build/mozharness/graph the mozharness repository] are live immediately. Please see that repo for push history.

= Puppet deployments =

Merges to the <em>production</em> branch of [http://hg.mozilla.org/build/puppet/graph the puppet repository] are automatically deployed by Puppet within a 1/2 hour. Please see that repo for push history.

Please note if you are applying a puppet change, '''unless you have good reason not to do so''' please merge it to production branch immediately. The reason for this is that we do not want puppet changes to "back up" on default, and then land ''en masse'' in production. It can be very difficult to untangle multiple puppet changes that arrive together.

The reason we have a production branch at all, is to allow contributions on default branch from a larger group of people, than those responsible for maintaining the infrastructure controlled by the puppet repo - in other words, a smaller group of people can merge to production, than the group of people that can land on default.

= Reconfigs / Deployments =
This page is updated by the person who does a [[ReleaseEngineering:Buildduty:Reconfigs|reconfig]] on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as <i>mozharness</i> changes.)

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Older changes are archived in: [[ReleaseEngineering/Changes/2015|2015]] | [[ReleaseEngineering/Changes/2014|2014]] | [[ReleaseEngineering/Changes/2013|2013]] | [[ReleaseEngineering/Changes/2012|2012]] | [[ReleaseEngineering/Changes/2011|2011]] | [[ReleaseEngineering/Changes/2010|2010]] | [[ReleaseEngineering/Changes/2009|2009]]

{| class="fullwidth-table wikitable sortable"
| style="background:#cccccc" | '''Outcome'''
| style="background:#cccccc" | '''When'''
| style="background:#cccccc" | '''Bug #(s)''' - '''Description(s)'''
|-
| in production
| 2016-10-25 19:00 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1312650&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1312650}} - Windows buildbot factory jobs should know about TOOLTOOL_CACHE, r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/22368ebeb3c1 22368ebeb3c1])
|-
| in production
| 2016-10-25 09:24 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1309894&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1309894}} - Use *-dev update channels on jamun - r=mtabara ([https://hg.mozilla.org/build/buildbot-configs/rev/c3ba8bc05df6 c3ba8bc05df6])
|-
| in production
| 2016-10-24 17:58 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1309894&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1309894}} - Use dev-* update channels on jamun - r=nthomas ([https://hg.mozilla.org/build/buildbot-configs/rev/d65e81712d2c d65e81712d2c])
|-
| in production
| 2016-10-21 06:56 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1309894&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1309894}} - Use "poll_all_branches" in configs to explicitly set polling behaviour - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/f3d8389cb9bf f3d8389cb9bf])
* No bug - Update release config for Fennec-49.0.2-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/08de42085241 08de42085241])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1309894}} - Use "poll_all_branches" to explicitly set polling behaviour - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/422f21b8f5fc 422f21b8f5fc])
|-
| in production
| 2016-10-18 08:54 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1308789&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1308789}} - Add test case to reproduce failure - r=nthomas ([https://hg.mozilla.org/build/buildbotcustom/rev/5a283b5466d6 5a283b5466d6])
* {{bug|1308789}} - Fix setting of default platforms for tests - r=nthomas ([https://hg.mozilla.org/build/buildbotcustom/rev/bfe42c243d4a bfe42c243d4a])
|-
| in production
| 2016-10-13 10:14 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1308789,1309894&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1308789}} - Set the Win64 builds to be the default try_by_default: True, because we want them to build by default, it's only the Win8 tests that are supposed to be try_by_default: False - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/a9bbcd872ad9 a9bbcd872ad9])
* {{bug|1309894}} - Investigate shipping old release version off of jamun - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/6f25921852c3 6f25921852c3])
* No bug - Update release config for Fennec-50.0b6-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/f1f2fc3fa06e f1f2fc3fa06e])
|-
| in production
| 2016-10-10 07:54 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1292656,1300413&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1292656}} - decommission another 10 t-snow-r4 machines - r=coop ([https://hg.mozilla.org/build/buildbot-configs/rev/02b3c211cc48 02b3c211cc48])
* {{bug|1300413}} - Use unbuffered python in case we're missing output - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/1d22c76665b8 1d22c76665b8])
* No bug - Update release config for Thunderbird-45.4.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/a4d817817321 a4d817817321])
* No bug - Update release config for Thunderbird-50.0b1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/ab02ce360da9 ab02ce360da9])
* No bug - Update release config for Fennec-50.0b4-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/f854aac09b67 f854aac09b67])
* No bug - Update release config for Thunderbird-50.0b1-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/acc59373912b acc59373912b])
|-
| in production
| 2016-09-21 10:45 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* No bug - Update release config for Fennec-50.0b1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/d17d80076abd d17d80076abd])
|-
| in production
| 2016-09-19 22:09 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1303914&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1303914}} - buildbot - turn on fennec debug builds and tests for beta 50 - r=aki ([https://hg.mozilla.org/build/buildbot-configs/rev/2726220a2e49 2726220a2e49])
|-
| in production
| 2016-09-19 06:37 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1275601,1297519&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1275601}} - Tracking bug for Sep-13-2016 migration work - r=mtabara ([https://hg.mozilla.org/build/buildbot-configs/rev/bb6a28e7b06d bb6a28e7b06d])
* {{bug|1297519}} - Generate multilocale Firefox snaps - r=jlorenzo ([https://hg.mozilla.org/build/buildbot-configs/rev/4354ce6042f5 4354ce6042f5])
* No bug - Update release config for Fennec-49.0-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/4b6d645e8c31 4b6d645e8c31])
|-
| in production
| 2016-09-14 13:31 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1302256,1302530&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1302256}} - Run Win7 GFX jobs on try by default, now that they are production jobs - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/1911d93964d0 1911d93964d0])
* {{bug|1302256}} - Run Win7 GFX jobs on try by default, now that they are production jobs - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/1911d93964d0 1911d93964d0])
* {{bug|1302530}} - Add ondemand g-w732 instances - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/ba50299bcd9f ba50299bcd9f])
* {{bug|1302530}} - Add ondemand g-w732 instances - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/ba50299bcd9f ba50299bcd9f])
* No bug - Update release config for Fennec-49.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/0788239cf1d8 0788239cf1d8])
* No bug - Update release config for Fennec-49.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/0788239cf1d8 0788239cf1d8])
* No bug - Backed out changeset 1911d93964d0 ([https://hg.mozilla.org/build/buildbot-configs/rev/46e667b9878b 46e667b9878b])
|-
| in production
| 2016-09-06 10:59 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1300368&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1300368}} - Make autoland trigger periodic builds more frequently ([https://hg.mozilla.org/build/buildbot-configs/rev/8d3743998c94 8d3743998c94])
* No bug - Update release config for Fennec-49.0b11-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/6fd24999974a 6fd24999974a])
|-
| in production
| 2016-09-05 07:59 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1275601&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1275601}} - Bump mozilla-release version - r=mtabara ([https://hg.mozilla.org/build/buildbot-configs/rev/0f7da3a3e475 0f7da3a3e475])
|-
| in production
| 2016-08-26 11:33 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1288434&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1288434}} - Add bb-configs for mark release as shipped task - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/3dc06ea07b8d 3dc06ea07b8d])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1288434}} - Add bb-custom builder for mark release as shipped task - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/91327dcd59ef 91327dcd59ef])
|-
| in production
| 2016-08-25 10:56 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1296449,1298013&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1298013}} - Migrate remaining win 7 mochitest bc jobs to AWS - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/7c1e748101f6 7c1e748101f6])
* No bug - Update release config for Thunderbird-49.0b1-build4 ([https://hg.mozilla.org/build/buildbot-configs/rev/26a6a5f90516 26a6a5f90516])
* No bug - Update release config for Thunderbird-45.3.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/4b44a217b2ef 4b44a217b2ef])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1296449}} - Backed out changeset 68ec5535f7af because it was unnecessary ([https://hg.mozilla.org/build/buildbotcustom/rev/e03bbd8376e3 e03bbd8376e3])
|-
| in production
| 2016-08-23 12:23 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1297421&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1297421}} - migrate win 7 mochitest-e10s browser-chrome debug from windows HW to AWS - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/e24afa83727c e24afa83727c])
|-
| in production
| 2016-08-23 10:14 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1291015,1297173&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1291015}} - Migrate mochitest to AWS - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/6b5ab7e5fa62 6b5ab7e5fa62])
* {{bug|1297173}} - Move 20 machines from Windows 7 pool to Windows XP pool - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/0eb5c3edca2a 0eb5c3edca2a])
* No bug - Update release config for Fennec-49.0b6-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/3c0a11a61920 3c0a11a61920])
|-
| in production
| 2016-08-19 05:55 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1296026,1296449&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1296026}} - Modify --mochitest-suite for desktop mochitest-gl - r=aselagea ([https://hg.mozilla.org/build/buildbot-configs/rev/13db21d7a4f8 13db21d7a4f8])
* No bug - Update release config for Thunderbird-49.0b1-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/93a033b4676c 93a033b4676c])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1296449}} - Timeouts generating partial updates for Thunderbird on Windows - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/68ec5535f7af 68ec5535f7af])
|-
| in production
| 2016-08-11 13:16 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1290319&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1290319}} - require hostname or releng_CA in signing ssl cert - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/307646eb0970 307646eb0970])
|-
| in production
| 2016-08-10 13:50 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1268648,1290319&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1268648}} - Increase the script_timeout for mochitest-browser-screenshots to 1800s - r=armenzg ([https://hg.mozilla.org/build/buildbot-configs/rev/e538f7f45a14 e538f7f45a14])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1290319}} - fix URLPath call - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/61a1b16ba83d 61a1b16ba83d])
|-
| in production
| 2016-08-08 13:15 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* No bug - empty commit to force reconfig ([https://hg.mozilla.org/build/buildbot-configs/rev/54cc6717f498 54cc6717f498])
|-
| in production
| 2016-08-04 09:11 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1288043,1291044&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1288043}} - migrate win7 gpu jobs to gfx instances - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/199eca90a0a9 199eca90a0a9])
* {{bug|1291044}} - Use 10 chunks for wpt on win 7 vm debug to prevent timeouts - r=jmaher ([https://hg.mozilla.org/build/buildbot-configs/rev/c23e2a8603f5 c23e2a8603f5])
* No bug - [PATCH] Bug 1271153 - Adjust automation for the version_display.txt new file on tb - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/06d19bac0eff 06d19bac0eff])
|-
| in production
| 2016-08-01 06:14 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1278260&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1278260}} - Tracking bug for Aug-1-2016 migration work - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/559b89156599 559b89156599])
|-
| in production
| 2016-07-28 04:56 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1289514&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1289514}} - Remove unused import - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/e939e50055c5 e939e50055c5])
* {{bug|1289514}} - Remove bin/hgpoller.py - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/97fe1b0d7fcb 97fe1b0d7fcb])
* {{bug|1289514}} - Consume pushlog version 2 - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/f543c1e5f30a f543c1e5f30a])
* {{bug|1289514}} - Change API and semantics of pushlog parsing - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/528c39fe4a5a 528c39fe4a5a])
* {{bug|1289514}} - Factor common test code into base class - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/93f15cf0f724 93f15cf0f724])
* {{bug|1289514}} - Detect empty repos from empty lastpushid - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/d76e0a6a3900 d76e0a6a3900])
* {{bug|1289514}} - React intelligently to reset pushlogs - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/3041d0f3079f 3041d0f3079f])
* {{bug|1289514}} - Remove empty dataFailed method - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/604839f1803e 604839f1803e])
* {{bug|1289514}} - Poll pushlog from last consumed push ID - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/2b20e69710ab 2b20e69710ab])
* {{bug|1289514}} - Handle empty changesets in push entry - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/86c0a3b7c99c 86c0a3b7c99c])
|-
| in production
| 2016-07-27 09:54 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* No bug - Update release config for Fennec-48.0b11-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/9c1804c74761 9c1804c74761])
* No bug - Update release config for Fennec-48.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/49e72329db05 49e72329db05])
|-
| in production
| 2016-07-25 07:30 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1278260,1278261&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1278260}} - "Tracking bug for Aug-1-2016 migration work" - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/21ae1b8c2405 21ae1b8c2405])
* {{bug|1278261}} - drop api-9 mozilla-release release support, remove all api-9 and api-11 references - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/707b1871d7dc 707b1871d7dc])
|-
| in production
| 2016-07-22 09:39 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1233953,1282849&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1233953}} - Intermittent Linux debug command timed out: 7200 seconds elapsed running - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/2f53b99fd268 2f53b99fd268])
* {{bug|1282849}} - disable fennec debug builds and tests in buildbot - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/19f3d53951ae 19f3d53951ae])
* No bug - Update release config for Fennec-48.0b8-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/da62abc85e9d da62abc85e9d])
* No bug - Update release config for Fennec-48.0b9-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/2b19aa948928 2b19aa948928])
* No bug - Update release config for Fennec-48.0b10-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/ad5f8c375bcd ad5f8c375bcd])
|-
| in production
| 2016-07-12 16:12 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1055707,1277435&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* No bug - Update release config for Thunderbird-48.0b1-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/205cec6585a1 205cec6585a1])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1055707}} - set MOZ_NODE_PATH for Mac - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/64450127bc26 64450127bc26])
* {{bug|1277435}} - Remove funfuzz private repo reference ([https://hg.mozilla.org/build/buildbotcustom/rev/2d29e0ad789e 2d29e0ad789e])
|-
| in production
| 2016-07-07 11:34 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1267262,1278611,1285303&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1267262}} - Kill unused release config files - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/fc34658eac8e fc34658eac8e])
* {{bug|1278611}} - move valgrind tc builds to tier1 and stop building buildbot binaries - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/e582005ba6bd e582005ba6bd])
* {{bug|1285303}} - Reset projects/date configs - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/5db89ab28e80 5db89ab28e80])
* No bug - Update release config for Fennec-48.0b6-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/42a5ebf91840 42a5ebf91840])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1278611}} - move valgrind tc builds to tier1 and stop building buildbot binaries - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/ee757cc22d46 ee757cc22d46])
|-
| in production
| 2016-06-27 18:57 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1281869&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1281869}} - Do pgo builds on-checkin for larch - r=nthomas ([https://hg.mozilla.org/build/buildbot-configs/rev/4887a9a7894d 4887a9a7894d])
* No bug - Update release config for Fennec-48.0b4-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/8c97d6107105 8c97d6107105])
|-
| in production
| 2016-06-07 13:55 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1262760&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1262760}} - Use the objdir's virtualenv's python executable. -- fixes typo - r=rail (hopefully) post-landing ([https://hg.mozilla.org/build/buildbotcustom/rev/6f6a9824ca4d 6f6a9824ca4d])
|-
| in production
| 2016-06-07 11:56 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1262760&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1262760}} - Use the objdir's virtualenv's python executable ([https://hg.mozilla.org/build/buildbotcustom/rev/370d289d34a0 370d289d34a0])
* {{bug|1262760}} - Use the objdir's virtualenv's python executable ([https://hg.mozilla.org/build/buildbotcustom/rev/646b380589aa 646b380589aa])
|-
| in production
| 2016-06-02 08:45 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1276474,1277200,1277389&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1276474}} - Enable all e10s test suites for Windows 7 on Gecko 48+ - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/6b5e33436b61 6b5e33436b61])
* {{bug|1277200}} - Bump update_verify_chunks - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/63ab033a8991 63ab033a8991])
* {{bug|1277389}} - Schedule PGO builds on Ash at 9am and 9pm if the repo has changed - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/51c2e105e9de 51c2e105e9de])
* No bug - Update release config for Fennec-47.0b10-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/b641f8dd3c7f b641f8dd3c7f])
|-
| in production
| 2016-05-30 06:58 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1276389&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1276389}} - Remove esr38 from the buildbot configs - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/1831040dc4bf 1831040dc4bf])
|-
| in production
| 2016-05-29 14:55 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1253757&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1253757}} - Switch mozilla-esr45 to release build promotion - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/d3363d0677d3 d3363d0677d3])
|-
| in production
| 2016-05-27 12:05 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1271355&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1271355}} - Disable web-platform-e10s tests on IX instances - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/4422dacf9643 4422dacf9643])
|-
| in production
| 2016-05-27 09:48 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1271355&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1271355}} - Disable more suites on IX instances - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/9c52931d383c 9c52931d383c])
|-
| in production
| 2016-05-27 07:55 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1274552&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1274552}} - Disable extra jobs on IX / GFX instances - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/c1e529438581 c1e529438581])
|-
| in production
| 2016-05-27 05:59 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* No bug - Backed out changeset 899b41a5e8e5, for xpcshellcommandline bustage ([https://hg.mozilla.org/build/buildbotcustom/rev/c543cd26f4ad c543cd26f4ad])
|-
| in production
| 2016-05-27 05:51 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1274552,1275573,1275775&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1274552}} - Move devtools, jetpack and marionette jobs to Win7 VMs ([https://hg.mozilla.org/build/buildbot-configs/rev/d15d0f3bf6aa d15d0f3bf6aa])
* {{bug|1275573}} - Adjust buildbot configs to make jamun track esr45 - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/b8bcdf1032fe b8bcdf1032fe])
* {{bug|1275775}} - Remove most spidermonkey builds, add osx builds for greening - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/e760aa2e4016 e760aa2e4016])
* {{bug|1275775}} - followup, syntax error fix - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/8ffe3a22fc9c 8ffe3a22fc9c])
* {{bug|1275775}} - Remove most spidermonkey builds, add osx builds for greening - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/be79d77a7c33 be79d77a7c33])
* No bug - Backed out changeset 8ffe3a22fc9c (bug 1275775) ([https://hg.mozilla.org/build/buildbot-configs/rev/70d9c5da9f95 70d9c5da9f95])
* No bug - Backed out changeset e760aa2e4016 (bug 1275775) ([https://hg.mozilla.org/build/buildbot-configs/rev/06ca6db17cf4 06ca6db17cf4])
* No bug - Update release config for Thunderbird-45.1.1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/7bf4cbff80e8 7bf4cbff80e8])
|-
| in production
| 2016-05-26 01:20 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1262760,1275738&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1275738}} - Stop running e10s tests on OSX 10.6 - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/f5fa52ba7df1 f5fa52ba7df1])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1262760}} - Use the objdir's virtualenv's python executable - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/899b41a5e8e5 899b41a5e8e5])
|-
| in production
| 2016-05-24 10:56 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1267712,1269872&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1267712}} - Enable build automation on the autoland repo - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/2c50bb75bc0a 2c50bb75bc0a])
* {{bug|1269872}} - enable gpu/clipboard on 49+ - r=armenzg ([https://hg.mozilla.org/build/buildbot-configs/rev/7d5af12079a6 7d5af12079a6])
* No bug - Update release config for Fennec-47.0b8-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/e765101f1d77 e765101f1d77])
|-
| in production
| 2016-05-23 09:45 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1269872&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1269872}} - add config lines for gpu/clipboard asan jobs - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/c2db2be3fd78 c2db2be3fd78])
|-
| in production
| 2016-05-20 08:37 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1255533,1269872,1274300&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1255533}} - Disable win8 by default on try - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/5f7d99a176f1 5f7d99a176f1])
* {{bug|1269872}} - Create GPU/clipboard jobs for ASAN - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/1ad733905a5b 1ad733905a5b])
* {{bug|1274300}} - Remove b-2008-ix, talos-linux32 from buildbot-configs and slavealloc - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/1995d0e79cab 1995d0e79cab])
* No bug - Update release config for Thunderbird-45.2b1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/b76200a19cd1 b76200a19cd1])
* No bug - Update release config for Thunderbird-45.2b1-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/c4f4ea55bd30 c4f4ea55bd30])
|-
| in production
| 2016-05-19 10:47 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1269872,1271394&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1269872}} - android jobs should be for 4.3, not 2.3 - r=gbrown ([https://hg.mozilla.org/build/buildbot-configs/rev/8aa5b6e8e7c1 8aa5b6e8e7c1])
* {{bug|1269872}} - remove winxp e10 jobs and win8 debug e10s jobs - r=ryanvm ([https://hg.mozilla.org/build/buildbot-configs/rev/800f339e74f1 800f339e74f1])
* {{bug|1271394}} - Enable all standard tests on projects/larch branch - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/151b97c26b65 151b97c26b65])
* No bug - No bug, fix api-11 references in staging release configs, which break staging build masters ([https://hg.mozilla.org/build/buildbot-configs/rev/f00e825ae12e f00e825ae12e])
|-
| in production
| 2016-05-17 09:46 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1223509,1269872,1273135&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1223509}} - disable w732-spot-101 for issues - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/e2b015fe1d6f e2b015fe1d6f])
* {{bug|1269872}} - remove --disable-e10s and use --e10s - r=armenzg ([https://hg.mozilla.org/build/buildbot-configs/rev/10b8e6c3c984 10b8e6c3c984])
* {{bug|1273135}} - Migrate more tests to AWS instances - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/c6b4172b5b96 c6b4172b5b96])
* No bug - Update release config for Fennec-47.0b6-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/a9a4d80011e1 a9a4d80011e1])
|-
| in production
| 2016-05-16 11:22 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1272723&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1272723}} - Remove b2g - r=nthomas ([https://hg.mozilla.org/build/buildbot-configs/rev/f373116efc7d f373116efc7d])
* {{bug|1272723}} - Remove b2g - r=nthomas ([https://hg.mozilla.org/build/buildbot-configs/rev/f373116efc7d f373116efc7d])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1272723}} - Remove b2g - r=nthomas ([https://hg.mozilla.org/build/buildbotcustom/rev/14de2833fc34 14de2833fc34])
|-
| in production
| 2016-05-16 08:15 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1194533,1236954,1269872,1270962,1271355,1272005&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1194533}} - Enable all e10s tests for OSX 10.10 on Gecko 48+ - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/343ac36a1864 343ac36a1864])
* {{bug|1269872}} - Remove mochitest-chrome from Android gpu jobs - r=jmaher ([https://hg.mozilla.org/build/buildbot-configs/rev/d99a2e77873e d99a2e77873e])
* {{bug|1270962}} - create a new clipboard job, or fix clipboard to work on windows 7 vm - r=armenzg ([https://hg.mozilla.org/build/buildbot-configs/rev/ffb96494328b ffb96494328b])
* {{bug|1271355}} - Enable win7_vm by default on try - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/af3b8066baf8 af3b8066baf8])
* {{bug|1272005}} - Disable cedar branch - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/cfbd735fa485 cfbd735fa485])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1236954}} - Add standalone partner repack factory - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/546df4a98723 546df4a98723])
|-
| in production
| 2016-05-12 08:43 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1252003,1271355,1271454&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1271355}} - Migrate some win7 test suites to AWS - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/f3e3592dffd1 f3e3592dffd1])
* {{bug|1271454}} - Create entries for 600 t-w732-spot machines in AWS - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/87d2d8f7fcd9 87d2d8f7fcd9])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1252003}} - Add publish to balrog builder - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/efdb41734f8b efdb41734f8b])
|-
| in production
| 2016-05-12 02:03 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1253341,1262760,1268542,1271550&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1253341}} - Run duplicate Talos jobs in AWS for Linux (reverted) - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/2302dc47cb21 2302dc47cb21])
* {{bug|1268542}} - Disable Linux64 Debug builds and tests in buildbot - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/cfe4c07f8161 cfe4c07f8161])
* {{bug|1271550}} - Add new 10.10.5 machines to releng configs ([https://hg.mozilla.org/build/buildbot-configs/rev/750ab5df5a06 750ab5df5a06])
* No bug - Update release config for Thunderbird-45.1.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/48e0db6d4d22 48e0db6d4d22])
* No bug - Update release config for Thunderbird-45.1.0-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/c199b13b84a3 c199b13b84a3])
* No bug - Backed out changeset c199b13b84a3 to let TB 45.1.0 build3 to restart ([https://hg.mozilla.org/build/buildbot-configs/rev/a55a7e24bba1 a55a7e24bba1])
* No bug - Update release config for Thunderbird-45.1.0-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/3a7c1d3dbace 3a7c1d3dbace])
* No bug - Update release config for Fennec-47.0b4-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/f7b8a43eed4f f7b8a43eed4f])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1262760}} - Use the objdir's virtualenv's python executable - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/a15cf826c0cb a15cf826c0cb])
|-
| in production
| 2016-05-06 09:58 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1262760,1268542,1269543,1270881&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1268542}} - Disable Linux64 Debug builds and tests in buildbot - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/eefa2e45cd9e eefa2e45cd9e])
* {{bug|1269543}} - Stop running tests on OS X 10.6 on Firefox 49+ - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/f88f7f57f714 f88f7f57f714])
* {{bug|1270881}} - Make the mochitest-gl timeout 5400s for both variants - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/29f4c35c876b 29f4c35c876b])
* No bug - Update release config for Thunderbird-45.1.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/ffcc165163a8 ffcc165163a8])
* No bug - backout bug 1268542 - Disable Linux64 Debug builds and tests in buildbot ([https://hg.mozilla.org/build/buildbot-configs/rev/9f885780ab86 9f885780ab86])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1262760}} - Use the virtualenv's python with printconfigsetting.py - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/a6b085c766b5 a6b085c766b5])
|-
| in production
| 2016-05-02 13:55 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1214487&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1214487}} - Add mapping in common for uptake monitoring - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/7d70b0149b14 7d70b0149b14])
|-
| in production
| 2016-05-02 13:00 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1214487,1268895&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1214487}} - Add uptake monitoring builder config - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/07af46009466 07af46009466])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1214487}} - Add uptake monitoring builder - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/6a21eadc4deb 6a21eadc4deb])
* {{bug|1268895}} - Remove unused l10n dep logic - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/a0def3e44f74 a0def3e44f74])
|-
| in production
| 2016-04-27 10:34 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=787208&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|787208}} - Fix cypress repo_path by using common defaults - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/6403d9027a45 6403d9027a45])
* {{bug|787208}} - fix busted patch again - r=bustage ([https://hg.mozilla.org/build/buildbot-configs/rev/af5f7251deeb af5f7251deeb])
* No bug - Update release config for Fennec-47.0b1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/b47a0268849d b47a0268849d])
|-
| in production
| 2016-04-19 06:35 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1264407,1265425&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1264407}} - Switch Thunderbird ESR45 testing to r7 minis - r=aselagea ([https://hg.mozilla.org/build/buildbot-configs/rev/7fe1919120db 7fe1919120db])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1265425}} - Bump partner repacks script_maxtime - r=coop ([https://hg.mozilla.org/build/buildbotcustom/rev/700093ba063c 700093ba063c])
* {{bug|1265425}} - Use slave platform instead of target platform - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/901cbfd439c0 901cbfd439c0])
|-
| in production
| 2016-04-19 05:32 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1260415&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1260415}} - Explicitly set basedir for all platforms - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/5557a28843c5 5557a28843c5])
|-
| in production
| 2016-04-13 06:07 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1212993,1253341,1260415,1262533,1263279,1263385,1263402,1263918&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1253341}} - Run duplicate Talos jobs in AWS for Linux (large instances). r?rail ([https://hg.mozilla.org/build/buildbot-configs/rev/609fef319188 609fef319188])
* {{bug|1263279}} - add balrog rules for TB 45.0ESR builds in release configs - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/4a4c57503630 4a4c57503630])
* {{bug|1263385}} - Increase the number of wpt[-e10s] opt chunks to 5 and debug to 10 - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/ee9a45746a10 ee9a45746a10])
* {{bug|1263402}} - Remove symbol server config for TB builds - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/312fc115e5ab 312fc115e5ab])
* {{bug|1263918}} - Enable release-signing on jamun to run full tests - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/f35a9ae54e4d f35a9ae54e4d])
* No bug - Update release config for Fennec-46.0b10-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/f22ead008630 f22ead008630])
* No bug - Update release config for Firefox-45.0.2esr-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/f44ca27a5aa3 f44ca27a5aa3])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1212993}} - remove get basedir step - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/493648951b51 493648951b51])
* {{bug|1260415}} - Schedule "partner repacks" builder - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/02f5c29c5b11 02f5c29c5b11])
* {{bug|1262533}} - bbot-custom - add .idea dir to list of hgignore files for buildbot and tools repos, r?rail ([https://hg.mozilla.org/build/buildbotcustom/rev/d810ea7559de d810ea7559de])
|-
| in production
| 2016-04-04 13:40 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1253341,1259524,1260415,1260879,1261045&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1253341}} - Disable duplicate talos jobs in AWS - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/c06fa02837e9 c06fa02837e9])
* {{bug|1259524}} - release promotion - add staging mozilla-release configs to buildbot - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/fe28dc4e3993 fe28dc4e3993])
* {{bug|1260415}} - buildboti-config: add "partner repacks" builder - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/3e0f1271d6b1 3e0f1271d6b1])
* {{bug|1260879}} - removes bouncer_branch and add postrelease_bouncer_aliases items to jamun, r?kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/ce584e727954 ce584e727954])
* {{bug|1261045}} - remove symobl-server configuration - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/fa3ae48d4723 fa3ae48d4723])
* No bug - Update release config for Fennec-46.0b7-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/3fb326e95b27 3fb326e95b27])
* No bug - Update release config for Thunderbird-38.7.2-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/78ef71c34627 78ef71c34627])
* No bug - Update release config for Thunderbird-45.0b4-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/653739ef781d 653739ef781d])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1260415}} - buildbotcustom: add "partner repacks" builder - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/430fd7c357f7 430fd7c357f7])
* {{bug|1260879}} - bbotcustom removes bouncer_branch from bouncer submitter factory, r?kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/4461354684f3 4461354684f3])
|-
| in production
| 2016-03-29 13:32 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1256063&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1256063}} - Add update bouncer aliases builder - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/3d7b8b58d31e 3d7b8b58d31e])
* No bug - Update release config for Thunderbird-45.0b3-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/f1695648974e f1695648974e])
* No bug - Update release config for Fennec-46.0b6-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/07fe251fbd29 07fe251fbd29])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1256063}} - Add update bouncer aliases builder - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/fbdc1775117a fbdc1775117a])
|-
| in production
| 2016-03-21 17:41 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1249528&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1249528}} - releaserunner should handle RC channels. Fix product - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/40258e329c78 40258e329c78])
|-
| in production
| 2016-03-21 14:15 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1249528&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1249528}} - releaserunner should handle RC channels - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/42e571aa53d8 42e571aa53d8])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1249528}} - Unify update verify builders - r=Callek ([https://hg.mozilla.org/build/buildbotcustom/rev/139b15174c5d 139b15174c5d])
|-
| in production
| 2016-03-18 12:37 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1253341&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1253341}} - Allow running talos on AWS machines - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/ddb55938d842 ddb55938d842])
|-
| in production
| 2016-03-16 10:35 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1252292,1253341,1257230&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1252292}} - Don't run Ru-e10s on debug builds - r=me ([https://hg.mozilla.org/build/buildbot-configs/rev/04c8fe62d3a4 04c8fe62d3a4])
* {{bug|1253341}} - Enable talos on linux64 AWS machines on try - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/2d6158d681e0 2d6158d681e0])
* {{bug|1257230}} - Incorrect branding for 45.0.1esr build - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/b5163b4a7814 b5163b4a7814])
* No bug - Update release config for Fennec-46.0b2-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/216ac6505867 216ac6505867])
* No bug - Update release config for Firefox-38.7.1esr-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/2b0b8272dda2 2b0b8272dda2])
* No bug - backouts release change for 38.7.1esr - rev: 2b0b8272dda2 ([https://hg.mozilla.org/build/buildbot-configs/rev/ea85259cf436 ea85259cf436])
* No bug - merges backout 2b0b8272dda2 -> default ([https://hg.mozilla.org/build/buildbot-configs/rev/7b0276c224b4 7b0276c224b4])
* No bug - Update release config for Firefox-38.7.1esr-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/b735b52c0d5a b735b52c0d5a])
* No bug - Update release config for Firefox-45.0.1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/b3267390eb7d b3267390eb7d])
* No bug - Update release config for Firefox-45.0.1esr-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/1f7801999788 1f7801999788])
* No bug - Update release config for Fennec-45.0.1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/e759c66d311c e759c66d311c])
* No bug - Backed out 2d6158d681e0 for suspicion of causing intermittent checkconfig ([https://hg.mozilla.org/build/buildbot-configs/rev/7c463958098d 7c463958098d])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1253341}} - Allow running talos on AWS machines - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/42184dbdf8f3 42184dbdf8f3])
* No bug - Backed out changeset 42184dbdf8f3 for suspicion of causing intermittent checkconfig failureis ([https://hg.mozilla.org/build/buildbotcustom/rev/046c46eb099d 046c46eb099d])
|-
| in production
| 2016-03-15 07:43 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1249528,1252292,1255812,1256184&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1249528}} - releaserunner should handle RC channels - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/5235b4ceb15e 5235b4ceb15e])
* {{bug|1249528}} - Bustage fix - r=me ([https://hg.mozilla.org/build/buildbot-configs/rev/806fd2074e58 806fd2074e58])
* {{bug|1249528}} - backout - r=me ([https://hg.mozilla.org/build/buildbot-configs/rev/fc1e57afc013 fc1e57afc013])
* {{bug|1252292}} - Set Ash to the same branch priority as mozilla-central - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/50c1fdd4ff3e 50c1fdd4ff3e])
* {{bug|1255812}} - Move 10 XP test machines to the Windows 8 pool - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/36eb260ec81c 36eb260ec81c])
* {{bug|1256184}} - Remove code that unconditionally enables Talos g2 irrespective of the branch's enable_talos setting - r=jmaher ([https://hg.mozilla.org/build/buildbot-configs/rev/ec3937ad8bd7 ec3937ad8bd7])
|-
| in production
| 2016-03-11 09:50 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1186617,1242682,1245585,1250999,1255078&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1242682}} - Separate dom/media into its own subsuite - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/cc9e268a0036 cc9e268a0036])
* {{bug|1245585}} - turn off android talos tests which run on the panda boards - r=aselagea ([https://hg.mozilla.org/build/buildbot-configs/rev/771999c6f4ef 771999c6f4ef])
* {{bug|1250999}} - releng - remove api-9-10 from trunk - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/1bbe091937b4 1bbe091937b4])
* No bug - Update release config for Thunderbird-38.7.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/cf9b859c9d8c cf9b859c9d8c])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1186617}} - releng changes to decommission pandas - r=aselagea ([https://hg.mozilla.org/build/buildbotcustom/rev/695299094ba0 695299094ba0])
* {{bug|1255078}} - Android Beta tests broken: Found installer url ~/bouncer.apk - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/b4c6ed50c14c b4c6ed50c14c])
|-
| in production
| 2016-03-10 06:17 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1245585,1253858,1255088&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1245585}} - turn off android talos tests which run on the panda boards - r=aselagea ([https://hg.mozilla.org/build/buildbot-configs/rev/c58e5674bae2 c58e5674bae2])
* {{bug|1245585}} - turn off android talos tests which run on the panda boards - r=aselagea ([https://hg.mozilla.org/build/buildbot-configs/rev/87168a6da9f1 87168a6da9f1])
* {{bug|1253858}} - Turn off mochitest-push on esr45 - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/95e902ea4772 95e902ea4772])
* {{bug|1255088}} - turn on talos e10s for mozilla-beta, remove some other misconfigurations - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/8a6d6e31623f 8a6d6e31623f])
|-
| in production
| 2016-03-09 06:14 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1166420,1242682,1251351,1251477,1254902&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1166420}} - Add relengapi.tok to TB bb configs - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/d7776ec49bb4 d7776ec49bb4])
* {{bug|1242682}} - Separate dom/media into its own subsuite - r=armenzg ([https://hg.mozilla.org/build/buildbot-configs/rev/dd518a784a46 dd518a784a46])
* {{bug|1251351}} - Add config file options for marionette-e10s on Windows - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/1ce5d30d8e7c 1ce5d30d8e7c])
* {{bug|1251477}} - rename update verify configs - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/aed2ba7655a0 aed2ba7655a0])
* {{bug|1254902}} - Add relengapi.tok to buildbot configs - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/ddf4c824ca46 ddf4c824ca46])
|-
| in production
| 2016-03-07 19:31 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1254337&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1254337}} - fix tuple typo for beta releasetasks items, bbotcfgs, r?rail ([https://hg.mozilla.org/build/buildbot-configs/rev/b9a2659ad8fe b9a2659ad8fe])
|-
| in production
| 2016-03-07 16:12 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1251351,1251936,1254337&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1251351}} - Schedule Mn-e10s tests on win7 - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/8a111c694d59 8a111c694d59])
* {{bug|1251936}} - Part 3: Add buildbot configs for Windows x86 static analysis buillds - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/ae056e1b3ad7 ae056e1b3ad7])
* {{bug|1254337}} - use release-signing on mozilla-beta and disable push2releases automatic, bbotcfgs, r?rail ([https://hg.mozilla.org/build/buildbot-configs/rev/75be1891c7f8 75be1891c7f8])
* No bug - Update release config for Fennec-46.0b1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/236b949b7359 236b949b7359])
|-
| in production
| 2016-03-07 05:36 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1243796&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1243796}} - Tracking bug for Mar-7-2016 migration work - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/f03990b8f382 f03990b8f382])
|-
| in production
| 2016-03-04 13:23 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1253492,1253508,1253710,1253714&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1253492}} - Add missing e10s defines to un-break Mn-e10s and M(gl) jobs - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/7b8c9c7698ed 7b8c9c7698ed])
* {{bug|1253508}} - Let e10s jobs ride the trains - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/1e47404949d2 1e47404949d2])
* {{bug|1253710}} - Enable OSX 10.10 opt e10s tests on Gecko 47 - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/15dc4b9def85 15dc4b9def85])
* {{bug|1253714}} - Enable OSX 10.10 debug e10s browser-chrome and Marionette tests on Gecko 47 - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/e5cc6f88dcf5 e5cc6f88dcf5])
|-
| in production
| 2016-03-04 12:26 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1251477&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1251477}} - prep buildbot configs for release promotion in production on beta - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/57c166d8109b 57c166d8109b])
|-
| in production
| 2016-03-04 09:03 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1242682,1253410,1253437,1253492&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1242682}} - Separate dom/media into its own subsuite [fix e10s config-file] - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/f0acc9241544 f0acc9241544])
* {{bug|1253410}} - Enable ASAN builds/tests on ESR45 - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/e254fe53e290 e254fe53e290])
* {{bug|1253437}} - Split Linux debug/ASAN xpcshell into 4 chunks - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/9a94815dc367 9a94815dc367])
* {{bug|1253492}} - Enable more e10s test suites on Linux - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/8fb9139950ba 8fb9139950ba])
* {{bug|1253492}} - Fix up duplicate builder bustage - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/6535597727b5 6535597727b5])
|-
| in production
| 2016-03-04 07:06 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1237094,1242682&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1237094}} - tracking bug for build and release of Firefox 45.0 ESR - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/f65f8e566a68 f65f8e566a68])
* {{bug|1242682}} - Separate dom/media into its own subsuite - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/ad171d57880d ad171d57880d])
|-
| in production
| 2016-03-03 16:54 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1192994,1252248,1252809,1253364,1253429&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1192994}} - Handle Linux64 data from SETA - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/8382286ea52c 8382286ea52c])
* {{bug|1252248}} - Add more tst-linux64, tst-emulator64 capacity in AWS - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/bc958535ce5a bc958535ce5a])
* {{bug|1252809}} - Only test pretty names on comm train branches, not comm-central - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/18c0c4c99064 18c0c4c99064])
* {{bug|1253364}} - Clean up list of machines that aren't in slavealloc - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/b86333d2e140 b86333d2e140])
* {{bug|1253429}} - production-ify release promotion configs, use existing bucket - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/858343665d94 858343665d94])
* No bug - Update release config for Firefox-38.7.0esr-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/39239fcf0362 39239fcf0362])
* No bug - Update release config for Fennec-45.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/20518d5e75df 20518d5e75df])
* No bug - Update release config for Fennec-45.0-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/92ab47cbc53a 92ab47cbc53a])
* No bug - Update release config for Firefox-45.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/7f197028d1d9 7f197028d1d9])
* No bug - Update release config for Firefox-45.0esr-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/08355f5f053b 08355f5f053b])
* No bug - Backed out 3 changesets on default to restart releases ([https://hg.mozilla.org/build/buildbot-configs/rev/436d072f5952 436d072f5952])
* No bug - Update release config for Fennec-45.0-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/fc5ddedd1c1d fc5ddedd1c1d])
* No bug - Update release config for Firefox-45.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/b51aad4f9e4b b51aad4f9e4b])
* No bug - Update release config for Firefox-45.0esr-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/13e413a55b5b 13e413a55b5b])
|-
| in production
| 2016-03-02 13:51 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1237094,1251381,1252756&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1237094}} - tracking bug for build and release of Firefox 45.0 ESR - r=nthomas ([https://hg.mozilla.org/build/buildbot-configs/rev/91b85a6e4c1e 91b85a6e4c1e])
* {{bug|1251381}} - Switch gtests to riding 46 - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/2a06d32c80b9 2a06d32c80b9])
* {{bug|1252756}} - Shut off periodic PGO on b2g-inbound - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/6c131bed7fae 6c131bed7fae])
* No bug - No bug - shorten travis results to be less spammy - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/fb1af69edbec fb1af69edbec])
* No bug - Update release config for Fennec-45.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/38e4592e3d57 38e4592e3d57])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* No bug - No Bug - shorten travis results notification to be less spammy - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/4b96f5cb0d0f 4b96f5cb0d0f])
|-
| in production
| 2016-03-02 06:25 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1237094,1252258,1252328&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1237094}} - tracking bug for Firefox 45.0 ESR, channel rules - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/4b855f6b6eae 4b855f6b6eae])
* {{bug|1252258}} - Move 8 w10 test machines to the Windows 7 pool - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/25b7652be68b 25b7652be68b])
* {{bug|1252328}} - Only run e10s tests on Gecko 46+ - r= ([https://hg.mozilla.org/build/buildbot-configs/rev/e9147a125ebc e9147a125ebc])
|-
| in production
| 2016-03-01 07:30 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1242682,1249376&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1242682}} - Separate dom/media into its own subsuite - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/bf7f47ae7e0f bf7f47ae7e0f])
* {{bug|1249376}} - Move 20 XP test machines to the Windows 7 pool - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/dee25cbffdcd dee25cbffdcd])
* No bug - Update release config for Fennec-45.0b11-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/b42fec100a37 b42fec100a37])
* No bug - Update release config for Firefox-45.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/e172a9923d43 e172a9923d43])
|-
| in production
| 2016-02-29 08:10 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1249682&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1249682}} - Remove addon-sdk buildbot configs - r=Callek ([https://hg.mozilla.org/build/buildbotcustom/rev/01d5bda06fc9 01d5bda06fc9])
|-
| in production
| 2016-02-25 09:50 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1237094,1251144,1251285&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1237094}} - tracking bug for build and release of Firefox 45.0 ESR - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/8712d19cff79 8712d19cff79])
* {{bug|1251144}} - Don't merge release promotion builders - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/1668779e0c88 1668779e0c88])
* {{bug|1251285}} - Disable alder project branch ([https://hg.mozilla.org/build/buildbot-configs/rev/81db79611afa 81db79611afa])
* {{bug|1251285}} - Disable alder project branch - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/0441563764d3 0441563764d3])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1251144}} - Don't merge release promotion builders - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/0cbca385a7e2 0cbca385a7e2])
|-
| in production
| 2016-02-24 13:45 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1250585&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1250585}} - adds push to mirrors automatic date, buildbot-configs - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/8ff751a38618 8ff751a38618])
* No bug - Update release config for Fennec-45.0b9-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/fc63a5db2c0c fc63a5db2c0c])
* No bug - Update release config for Firefox-45.0b9-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/884497cab040 884497cab040])
|-
| in production
| 2016-02-23 13:53 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1245559,1250306&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1245559}} - Enable wpt e10s on Windows 7 - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/17cc02d08629 17cc02d08629])
* {{bug|1245559}} - Enable wpt e10s on Windows 7 - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/646145351982 646145351982])
* {{bug|1250306}} - Point date tools repo at rail's fork - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/9bae3c5f95f2 9bae3c5f95f2])
* No bug - Backed out changeset 17cc02d08629 ([https://hg.mozilla.org/build/buildbot-configs/rev/a06e78e32bcb a06e78e32bcb])
* No bug - [mq]: tb.cypress.tests ([https://hg.mozilla.org/build/buildbot-configs/rev/4218b4cabec6 4218b4cabec6])
|-
| in production
| 2016-02-22 13:17 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1248474&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* No bug - Update release config for Firefox-45.0b8-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/5f9a37958eaf 5f9a37958eaf])
* No bug - Update release config for Firefox-45.0b8-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/5f9a37958eaf 5f9a37958eaf])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1248474}} - Don't use a slash in buildername - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/c87b90ed718d c87b90ed718d])
* {{bug|1248474}} - Don't use a slash in buildername - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/c87b90ed718d c87b90ed718d])
|-
| in production
| 2016-02-19 13:00 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1248474&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1248474}} - [RelPromo] Run Mac/Windows update verification via BBB - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/6035a0b57bfb 6035a0b57bfb])
|-
| in production
| 2016-02-19 07:03 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1241955,1246538,1248474,1248729,1249383&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1241955}} - Enable tests on date - r=callek ([https://hg.mozilla.org/build/buildbot-configs/rev/cc561d788755 cc561d788755])
* {{bug|1248474}} - [RelPromo] Run Mac/Windows update verification via BBB - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/437d703fad2b 437d703fad2b])
* {{bug|1248729}} - Schedule e10s mochitest-push on Windows 7 patch by aselagea - r=jgriffin ([https://hg.mozilla.org/build/buildbot-configs/rev/2999a899e438 2999a899e438])
* {{bug|1249383}} - disable talos on b2g-inbound - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/279753fc211c 279753fc211c])
* No bug - Update release config for Firefox-45.0b7-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/87671b77f024 87671b77f024])
* No bug - Update release config for Thunderbird-45.0b2-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/2706c3b66654 2706c3b66654])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1246538}} - mochitest-o should trigger mochitest-a11y and mochitest-chrome-* patch by aselagea - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/926a0351a0d7 926a0351a0d7])
* {{bug|1248474}} - [RelPromo] Run Mac/Windows update verification via BBB - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/eb8290943fcf eb8290943fcf])
|-
| in production
| 2016-02-17 08:46 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1236954,1246627,1247078&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1246627}} - Schedule e10s jsreftests on win7 patch by aselagea - r=jgriffin ([https://hg.mozilla.org/build/buildbot-configs/rev/cc37bdb15492 cc37bdb15492])
* No bug - revert Bug 1246627 - Schedule e10s jsreftests on win7 ([https://hg.mozilla.org/build/buildbot-configs/rev/71b9ccb8abf1 71b9ccb8abf1])
* No bug - Update release config for Fennec-45.0b6-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/69f84fb1f35b 69f84fb1f35b])
* No bug - Update release config for Firefox-45.0b6-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/df4ed20df968 df4ed20df968])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1236954}} - add standalone partner repack factory - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/a7bd1793a7ab a7bd1793a7ab])
* {{bug|1247078}} - [tryparser] Don't schedule any jobs if no try syntax is detected - r=catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/46fb74d54d79 46fb74d54d79])
|-
| in production
| 2016-02-14 08:43 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1244510,1245605,787208&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1245605}} - Use 'dumb' source mozconfig for TB beta builds following bug 123448 - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/22f1e00d700c 22f1e00d700c])
* {{bug|787208}} - Followup, don't set mozilla_dir - r=no-reviewer ([https://hg.mozilla.org/build/buildbot-configs/rev/8c7677a7cdb6 8c7677a7cdb6])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1244510}} - Add a try_parser alias for mochitest-e10s-(bc|browser-chrome) patch by aselagea - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/c994ec64ba03 c994ec64ba03])
|-
| in production
| 2016-02-12 07:43 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1218589,1229532,1246627,787208&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1218589}} - Nightly mozilla-central builds of graphene patch by aselagea - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/f88d45c3fbe6 f88d45c3fbe6])
* {{bug|1229532}} - remove hg bundle builder config - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/f0c9c5b5a7b8 f0c9c5b5a7b8])
* {{bug|1246627}} - Schedule e10s jsreftests on win7 patch by aselagea - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/66bc120b896c 66bc120b896c])
* {{bug|787208}} - In support of merging comm-central and mozilla-central together, configure cypress as a Thunderbird branch. r=rail - r=Fallen f=jcranmer ([https://hg.mozilla.org/build/buildbot-configs/rev/b92aaf935173 b92aaf935173])
* No bug - Update release config for Firefox-38.6.1esr-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/33fe33d65478 33fe33d65478])
* No bug - Update release config for Firefox-44.0.2-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/09cd8415adb9 09cd8415adb9])
* No bug - Update release config for Thunderbird-38.6.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/91cf6aa50268 91cf6aa50268])
* No bug - Update release config for Firefox-45.0b5-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/6376fd9567c4 6376fd9567c4])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1229532}} - remove hg bundle builders - r=bhearsum ([https://hg.mozilla.org/build/buildbotcustom/rev/1b880a68c991 1b880a68c991])
|-
| in production
| 2016-02-02 14:53 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1242103&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1242103}} - Add browser-chrome-screenshots to m-c and other try - r=armenzg ([https://hg.mozilla.org/build/buildbot-configs/rev/63614b6564ee 63614b6564ee])
* No bug - Update release config for Fennec-45.0b2-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/7a121e3abe60 7a121e3abe60])
* No bug - Update release config for Firefox-45.0b2-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/a70a9918a489 a70a9918a489])
* No bug - Update release config for Thunderbird-45.0b1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/39642e4c3ba4 39642e4c3ba4])
|-
| in production
| 2016-01-26 20:05 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1190860&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1190860}} - fix valgrind by setting up mock before running tooltool, r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/28864afc6409 28864afc6409])
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1190860}} - Copy in tooltool.py, and run it from in-mock location (with TB too), r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/10af9c3220aa 10af9c3220aa])
|-
| in production
| 2016-01-26 18:05 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1190860&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1190860}} - followup fix r=Callek over irc ([https://hg.mozilla.org/build/buildbotcustom/rev/a4bc25f76f2d a4bc25f76f2d])
* {{bug|1166464}} - followup to fix reconfig, r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/de981e55b123 de981e55b123])
|-
| in production
| 2016-01-26 16:00 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1242374,1166464,1242641,1190860,1211889&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1242374}} - set an sdkUrl to avoid interfering with packageUrl; r=bhearsum ([https://hg.mozilla.org/build/buildbotcustom/rev/34f988f27f10 34f988f27f10])
* {{bug|1166464}} - Bump version after shipping release builds r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/97febf14fe8d 97febf14fe8d])
* {{bug|1242641}} - backout 3b001e0b8438 ([https://hg.mozilla.org/build/buildbotcustom/rev/360f6f0cf623 360f6f0cf623])
* {{bug|1242641}} - backout 31ff0be74400 ([https://hg.mozilla.org/build/buildbotcustom/rev/9b9b645d00a1 9b9b645d00a1])
* {{bug|1190860}} - Run tooltool inside mock, r=Callek ([https://hg.mozilla.org/build/buildbotcustom/rev/889eb503d966 889eb503d966])
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1211889}} - split mochitest-oth into- mochitest-a11y (a), mochitest-chrome (c1, c2, c3) for linux32 debug. r=armenzg ([https://hg.mozilla.org/build/buildbot-configs/rev/d9e027f12297 d9e027f12297])
* {{bug|1166464}} - Bump version after shipping release builds r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/bb2664d7bf15 bb2664d7bf15])

|-
| in production
| 2016-01-25 20:47 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1242641&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1242641}} - followup to fix cwd for mock call, and use python from the virtualenv ([https://hg.mozilla.org/build/buildbotcustom/rev/3b001e0b8438 3b001e0b8438])
|-
| in production
| 2016-01-25 14:54 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1242641&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1242641}} - fix printconfigsetting.py calls for Firefox on m-b and higher, r=Callek ([https://hg.mozilla.org/build/buildbotcustom/rev/31ff0be74400 31ff0be74400])
|-
| in production
| 2016-01-25 05:50 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1232329&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1232329}} - Tracking bug for Jan-25-2016 migration work - r=trivial ([https://hg.mozilla.org/build/buildbot-configs/rev/c38c7f7b76e9 c38c7f7b76e9])
* No bug - Update release config for Firefox-44.0-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/3941c0a6daf6 3941c0a6daf6])
|-
| in production
| 2016-01-21 12:12 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1235961&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1235961}} - set tools_repo_cache for mac platforms - r=bustage ([https://hg.mozilla.org/build/buildbot-configs/rev/fa16bb751b50 fa16bb751b50])
|-
| in production
| 2016-01-21 10:56 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1231408,1233005,1236951,1241159,1241481&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1231408}} - Add browser-chrome-screenshots to yosemite_r7 on Try - r=armenzg ([https://hg.mozilla.org/build/buildbot-configs/rev/b74388666f8e b74388666f8e])
* {{bug|1233005}} - Enable packageSDK in release configs - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/9f0795fa7184 9f0795fa7184])
* No bug - Update release config for Fennec-44.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/2b71401d73e8 2b71401d73e8])
* No bug - Update release config for Firefox-44.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/0154167166dd 0154167166dd])
* No bug - Update release config for Firefox-38.6.0esr-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/fa10d55edf12 fa10d55edf12])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1233005}} - Set packageSDK if enabled in the config - r=bhearsum ([https://hg.mozilla.org/build/buildbotcustom/rev/5d15ab1c5ddd 5d15ab1c5ddd])
* {{bug|1236951}} - set tools_repo_cache for partner repacks - r=bustage ([https://hg.mozilla.org/build/buildbotcustom/rev/c06e740a09e4 c06e740a09e4])
* {{bug|1241159}} - handle mochites-e10s-devtools-chrome as an alias for the e10s devtools chunks - r=Callek ([https://hg.mozilla.org/build/buildbotcustom/rev/25315f9d0e2e 25315f9d0e2e])
* {{bug|1241481}} - Get rid of script_revlink - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/dc9b1cf7b9e1 dc9b1cf7b9e1])
|-
| in production
| 2016-01-20 06:16 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1236951&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* No bug - Update release config for Fennec-44.0b11-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/370703910927 370703910927])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1236951}} - use tools_repo_cache for toolsdir when set - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/6368247d8c7a 6368247d8c7a])
* {{bug|1236951}} - use WithProperties in case we actually have a revision - r=bustage ([https://hg.mozilla.org/build/buildbotcustom/rev/b049ab7ef9a7 b049ab7ef9a7])
|-
| in production
| 2016-01-19 07:36 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1225899,1236835,1236951,1239785&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1225899}} - add beetmover candidates flag to date config - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/fca43032761e fca43032761e])
* {{bug|1236951}} - Update partner repack stanza in release templates - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/2f6ad8dd5b7e 2f6ad8dd5b7e])
* {{bug|1239785}} - Move some XP test machines to the Windows 7 pool - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/31b0f9f1195c 31b0f9f1195c])
* No bug - Update release config for Fennec-44.0b10-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/cfff00d1a9f5 cfff00d1a9f5])
* No bug - Update release config for Firefox-44.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/8a87f5fe2b16 8a87f5fe2b16])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1236835}} - Disable all B2G Desktop Builds & Tests on All trees. Cleanup - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/e0ab5efe7082 e0ab5efe7082])
|-
| in production
| 2016-01-18 06:19 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1151954,1232329,1239025&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1232329}} - Tracking bug for Jan-25-2016 migration work - r=mtabara ([https://hg.mozilla.org/build/buildbot-configs/rev/a0007d5cd72b a0007d5cd72b])
* {{bug|1239025}} - Turn on reftest for e10s on win7 - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/c257ac5cf785 c257ac5cf785])
[https://hg.mozilla.org/build/mozharness mozharness]
* {{bug|1151954}} - move get_apk.sh to a mozharness file - r=sylvestre ([https://hg.mozilla.org/build/mozharness/rev/15845edae9a1 15845edae9a1])
|-
| in production
| 2016-01-15 08:55 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1236951&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* No bug - Update release config for Fennec-44.0b9-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/15dc357fc5bd 15dc357fc5bd])
* No bug - Update release config for Firefox-44.0b9-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/df3ee52930d6 df3ee52930d6])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1236951}} - Set script_repo_revision for ScriptFactory when archiver is used - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/aa885f5cdbc3 aa885f5cdbc3])
|-
| in production
| 2016-01-13 15:03 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1226180,1236835,1236951,1239074&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1226180}} - decommission t-yosemite-r5 machines - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/b05f8de5ec9d b05f8de5ec9d])
* {{bug|1236835}} - Disable all B2G Desktop Builds & Tests on All trees - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/66b14e07db81 66b14e07db81])
* {{bug|1236951}} - Re-enable partner repacks on beta and release - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/5bdbf1ac3f5f 5bdbf1ac3f5f])
* {{bug|1239074}} - Add trailing slash to ${MOZILLABUILD} so that PATH gets properly created - r=catlee ([https://hg.mozilla.org/build/buildbot-configs/rev/388b403428e2 388b403428e2])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1236951}} - Replace PartnerRepackFactory with a SigningScriptFactory instance - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/089b94bab05c 089b94bab05c])
|-
| in production
| 2016-01-12 09:04 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1238947&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1238947}} - Disable b2g37_v2.2 automation - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/4989f1200474 4989f1200474])
* No bug - Update release config for Fennec-44.0b8-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/e8d645fd93cd e8d645fd93cd])
* No bug - Update release config for Firefox-44.0b8-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/6196d2a86352 6196d2a86352])
* No bug - Update release config for Thunderbird-44.0b1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/6580fa6c368b 6580fa6c368b])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1238947}} - Disable b2g37_v2.2 automation - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/2135d176b8e8 2135d176b8e8])
[https://hg.mozilla.org/build/mozharness mozharness]
* {{bug|1238947}} - Disable b2g37_v2.2 automation - r=rail ([https://hg.mozilla.org/build/mozharness/rev/bcdac5ca4d07 bcdac5ca4d07])
|-
| in production
| 2016-01-11 08:10 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1238083&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1238083}} - Clean up stage tasks - r=bhearsum ([https://hg.mozilla.org/build/buildbot-configs/rev/0d695ad18d3f 0d695ad18d3f])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1238083}} - Clean up stage tasks - r=bhearsum ([https://hg.mozilla.org/build/buildbotcustom/rev/65c073287d56 65c073287d56])
|-
| in production
| 2016-01-11 07:41 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1073442,1237985&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1073442}} - adjust chunks from 5 to 8 for debug mochitests - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/5ae05a86c3ab 5ae05a86c3ab])
* {{bug|1237985}} - Disable b2g37_v2.2R automation - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/00c4b51cb1fb 00c4b51cb1fb])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1237985}} - Disable b2g37_v2.2R automation - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/7eb568e273d8 7eb568e273d8])
[https://hg.mozilla.org/build/mozharness mozharness]
* {{bug|1237985}} - Disable b2g37_v2.2R automation - r=rail ([https://hg.mozilla.org/build/mozharness/rev/9effeabb1c18 9effeabb1c18])
* {{bug|1237985}} - Disable b2g37_v2.2R automation - r=rail ([https://hg.mozilla.org/build/mozharness/rev/8da7cb911d70 8da7cb911d70])
* No bug - Revert Bug 1237985 - Disable b2g37_v2.2R automation - r=rail ([https://hg.mozilla.org/build/mozharness/rev/cdc0ebfcef83 cdc0ebfcef83])
|-
| in production
| 2016-01-07 06:39 PT
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1211770,1215204,1221963,1229790,1232780&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1211770}} - Disable HSTS/HPKP updates on b2g 2.2 branches - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/65d3648d9b27 65d3648d9b27])
* {{bug|1221963}} - Schedule media-youtube-tests on trunk branches - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/f7d9ee4eab35 f7d9ee4eab35])
* {{bug|1229790}} - add ec2 windows slaves to configs, restricted to try - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/4a61c10820eb 4a61c10820eb])
* {{bug|1232780}} - Schedule e10s mochitest-plain on win7-opt - r=jgriffin ([https://hg.mozilla.org/build/buildbot-configs/rev/ea429699060a ea429699060a])
* No bug - Update release config for Firefox-43.0.4-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/1bc16691f520 1bc16691f520])
* No bug - Update release config for Firefox-43.0.4-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/4b67594abc24 4b67594abc24])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1215204}} - Add bouncer submitter to release promotion graph - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/93196910e3df 93196910e3df])
* {{bug|1215204}} - Add bouncer submitter to release promotion graph - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/4cda855c1404 4cda855c1404])
* No bug - revert Bug 1215204 - Add bouncer submitter to release promotion graph - r=rail ([https://hg.mozilla.org/build/buildbotcustom/rev/8dda9c1cce64 8dda9c1cce64])
|}

ReleaseEngineering/Day 1 Checklist

2016-10-16T22:11:37Z

NThomas: /* Wiki */ Add yourself to list of team members

__TOC__

Welcome to Release Engineering!

This page is meant to get new hires, interns, or interested community members up to speed with the right software, configurations, and communication channels to contribute effectively to the release engineering pipeline.

= Video Introductions =

* [http://youtu.be/7j0NDGJVROI Release Engineering as a Force Multiplier] by John O'Duinn (former director) about how Mozilla Release Engineering handles our business and how we work to improve each and every day.
* This [https://etherpad.mozilla.org/vikibank crowd-sourced video list] contains other videos about various aspects of working at Mozilla, including important information such as [http://videos.mozilla.org/serv/air_mozilla/making_espresso_at_mozilla_toronto.webm how to operate the espresso machine].

= Development Best Practices =

* Read and keep up to date with: [[ReleaseEngineering/Development_Best_Practices|Development Best Practices]]
* Read and keep up to date with: http://moz-releng-docs.readthedocs.org/en/latest/
** Please fix any issues you find: https://github.com/mozilla/build-relengdocs As someone new to the project, your fresh perspective helps shine a light on missing information or assumptions we take for granted.

= Single Sign-On (SSO) =
Many Mozilla services use SSO now, including Gmail, Drive, and ServiceNow. The [https://mozilla.okta.com/app/UserHome?fromLogin=true Single Sign-On homepage] has a list of the available services.

'''NOTE''': SSO requires that [[#LDAP.2C_SSH.2C_VPN|LDAP be setup first]].

= Mail =
Mozilla mail is handled by [https://mail.google.com/ Gmail] now.

You should be added to the release@mozilla.com google group as a new hire/intern. This mailing list is managed by Google groups. Owners of this group will be able to add you. Send a test message to release@m.c to verify that your address has been added/subscribed. Talk to your manager if it is not working.

'''WARNING''': release@m.c can contain security-sensitive information. Do not automatically forward your email to a system that is not under Mozilla's control.

== Mailing lists ==
You'll need to manually subscribe to:
* [https://lists.mozilla.org/listinfo/release-engineering release-engineering] public mailing list
* [https://lists.mozilla.org/listinfo/dev-planning mozilla.dev.planning]
* [https://lists.mozilla.org/listinfo/dev-tree-management mozilla.dev.tree-management]

These are available as [news://news.mozilla.org newsgroups], google groups, and [https://lists.mozilla.org/listinfo Mailman lists]

== Mail Filtering ==

With all that new email, you will want to set up some filters in Gmail (https://mail.google.com/mail/u/0/#settings/filters) to filter some of the higher-volume automated mail into a folder. You may eventually want to handle this information, but on day one hundreds of nagios notifications are not going to be educational.

Here is [http://people.mozilla.org/~coop/mozillaMailFilters.xml an imperfect set of Gmail filters] that you can import to get you started.

A list of new (and some older) automated emails are indexed by subject, along with relevant actions, [https://wiki.mozilla.org/ReleaseEngineering/How_To/Process_release_email here].

If you are going to working on puppet, you should also look at this page on [https://intranet.mozilla.org/RelEngWiki/index.php/How_To/Read_Releng-Shared_Emails how to read releng shared emails].

== Calendar ==
Like mail, we now use [https://www.google.com/calendar/ Google calendar].

You'll want to subscribe to the following public calendars:
* [https://www.google.com/calendar/feeds/mozilla.com_2d32343333353036312d393737%40resource.calendar.google.com/public/basic Release Engineering - Public]
* [https://www.google.com/calendar/feeds/mozilla.com_toi1svbfjd878aslutkgj32dco%40group.calendar.google.com/public/basic Releng PTO]

Talk to your manager/mentor to get added to the various other private calendars as appropriate.

= Access =

== Bugzilla ==
Almost everything at Mozilla goes through Bugzilla. [https://bugzilla.mozilla.org/createaccount.cgi Create a Bugzilla account] if you have not already.

You'll need a few tweaks to your account to get access to everything releng-related:
* Add privileges for bugzilla group "build" (Mozilla Build Team) (Can be done by coop, catlee, hwine, or bugzilla admin.)
* Add your irc nickname & ldap username as "aliases" for your account
** log into bugzilla & follow links "Preferences" -> "Account Information"
** append the aliases, with a leading ':' and enclosed in brackets ('[]') to the "Real Name" field
** e.g.: "<tt>Hal Wine [:hwine]</tt>"
* Excellent video showing how to do power searches: [http://people.mozilla.org/~johnath/bugzilla/BeltznerDoesQuicksearch.ogv Bugzilla Quick Search] (5 minutes you NEED to give yourself)
** [https://bugzilla.mozilla.org/page.cgi?id=quicksearch.html QuickSearch help]

== Filing bugs against Release Engineering ==
The product to use is, unsurprisingly, "Release Engineering." There are multiple possible components under that product, so take your best guess or ask for guidance in IRC.

== LDAP, SSH, VPN ==

'''''Warning for people who already have an LDAP account:''''' '''Change your password.''' Otherwise, adding you to the releng group may lock your account without further notice. There is a stronger password policy in releng: users must change their password every 3 months. If you don't change your password, the only symptom will be that one day or another (already observed after 8 days), your regular password won't work anymore. If this happens to you, contact people in #servicedesk, they will be able to reset your password.

This is mostly applicable only to employees and interns, although it *is* possible for other contributors to acquire some limited LDAP access. Speak to someone you work with on the releng team if you would like to investigate this.

SSH keys are used for access to most machines, and some of them are stored in your LDAP record:
* Add your SSH public key to your LDAP account, at https://login.mozilla.com/

You'll need a number of other bits set in your LDAP object to access releng networks and systems via the VPN.
* Full instructions for [https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=30769829#MozillaVPN%28DatacenterVPN%29 accessing the VPN] are on [https://mana.mozilla.org mana].
** this usually starts by generating a certificate at https://login.mozilla.com/
* file a bug against Product/Component: Infrastructure & Operations/MOC: Service Requests asking to be added to the '''releng''' and '''vpn_releng''' groups. Something similar to {{bug|1223449}}
* If you're on MacOSX or Windows, you might want to request a license for Viscosity (A VPN client). To do so from [https://mozilla.service-now.com/ service-now].
** 'Order Stuff' > 'Software Applications' > 'Viscosity VPN' > 'Buisness Support && follow onscreen instructions
* Your manager is responsible for making sure you belong to the following LDAP groups, ideally before your first day:
cn=balrog,ou=groups,dc=mozilla
cn=corp-vpn,ou=groups,dc=mozilla
cn=CrashReportsAdmin,ou=groups,dc=mozilla
cn=GraphsAdmin,ou=groups,dc=mozilla
cn=IntranetWiki,ou=groups,dc=mozilla
cn=inventory,ou=groups,dc=mozilla
cn=irccloud,ou=groups,dc=mozilla
cn=mfa,ou=groups,dc=mozilla
cn=nagiosadmin,ou=groups,dc=mozilla
cn=phonebook_access,ou=groups,dc=mozilla
cn=releng,ou=groups,dc=mozilla
cn=StatsDashboard,ou=groups,dc=mozilla
cn=tableau_users,ou=groups,dc=mozilla
cn=team_moco,ou=groups,dc=mozilla
cn=vpn_corp,ou=groups,dc=mozilla
cn=vpn_default,ou=groups,dc=mozilla
cn=vpn_releng,ou=groups,dc=mozilla
cn=vpn_releng_loan,ou=groups,dc=mozilla
cn=vpn_releng_self_serve,ou=groups,dc=mozilla
cn=vpn_releng_slaveapi_dev,ou=groups,dc=mozilla
cn=vpn_relengwiki,ou=groups,dc=mozilla
cn=vpn_tooltooleditor,ou=groups,dc=mozilla

=== Common Machines ===
Once your SSH public key is associated with your LDAP record, you should have access a number of shared Mozilla hosts:
* people.mozilla.org (account is not automatic, use [https://mozilla.service-now.com/ service-now] to request).
* hg.mozilla.org (use full email address as username) (commit access is not automatic, see below for access)

=== Build Machines ===
Releng runs many, many machines, including:
* staging and production buildbot masters
* staging and production slaves
* servers hosting a variety of services
* a handful of shared releng machines for development

We access these machines via SSH (all), VNC (all), or remote desktop (Windows), depending on the context of the activity we're performing.

To add your SSH key to the buildslaves, you'll need to do something like {{bug|899315}}.
# File a bug in "Release Engineering: Platform Support"
# Checkout the puppet repo and create a patch
# Ask for a reviewer. Your mentor/manager are good options here.

If you need an account on cruncher (shared releng machine), ask catlee, and send him your SSH public key. (If you aren't sure you need the account, you probably do not for now.)

=== SSH Config ===
You'll want to update your ~/.ssh/config accordingly:

<pre class="_fck_mw_lspace">Host hg.mozilla.org git.mozilla.org
User <short-ldap-name-here>@mozilla.com
Compression yes
ServerAliveInterval 300

Host *.mozilla.com
User <short-ldap-name-here>
Compression yes
ServerAliveInterval 300

Host *.build.mozilla.org
Compression yes
User cltbld
ServerAliveInterval 300</pre>

== Mercurial (hg) ==
Most development in releng (and at Mozilla writ-large) is stored in version control using [http://mercurial.selenic.com/ hg].

There is an excellent step-by-step guide for setting up and using hg: [https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html Mercurial for Mozillians]

The root webview of the Mozilla hg repositories is here: https://hg.mozilla.org/

Most releng code lives in repos under https://hg.mozilla.org/build

There are 3 levels of commit access:
* Level 1 access allows you to use the [[ReleaseEngineering/TryServer|Try Server]] and setup user repos. As a new contributor, you should request this on day one.
* Level 2 access is required to land code in the build and project repos. Once you have a proven track record of successful patches, you can ask your manager/mentor to vouch for your Level 2 access. Your manager/mentor can also land patches for you until you receive Level 2 access.
* Level 3 access is required to land code directly on [https://hg.mozilla.org/mozilla-central mozilla-central] and its derived release branches. At some point in your Mozilla contribution story, you may need Level 3 access but many contributors never do. Talk to your manager/mentor if you think you need this access. You should already have Level 2 access when you request Level 3.

You need to file an IT bug to get hg commit access. Follow the instructions for [https://www.mozilla.org/en-US/about/governance/policies/commit/ Becoming a Mozilla Committer], and for Level 2, specify you need access to (at least) hg.mozilla.org/build/* (Product/Component: mozilla.org/Repository Account Requests).
* example request: {{bug|703351}}

== Git & Github ==
There are git mirrors of many popular Mozilla repositories. One of the Mozilla github admins (catlee, coop) can add you to the following GitHub groups:
* [https://github.com/mozilla-releng Mozilla-Releng organization]
* [https://github.com/orgs/mozilla/teams/releng Releng team within the Mozilla organization]

There are also a handful of git repos hosted directly by Mozilla. Your manager/mentor will let you know if you need access to one of these. ([[Github|See also]])

== PGP ==
PGP keys are used for encrypting sensitive data - you are responsible for generating and managing you own keys. A [https://intranet.mozilla.org/AndroidMarket#How_To:_GPG_Encryption.2FDecryption quick, Mac-focused primer is available on the intranet] or you can use the [https://www.gnupg.org/gph/en/manual.html The GNU Privacy Handbook] for reference. After you have created your keys, do try to get your key signed by other team members, and uploaded to [https://gpg.mozilla.org gpg.mozilla.org].
* It is okay to reuse an existing gpg key, provided you add an additional ID for your canonical LDAP name (e.g. 'hwine [at] mozilla.com', not an alias like 'Hal_9001 [at] mozilla.com').

== Other Services ==
For access to other services, you'll need file a couple of bugs:
* Access to [http://nagios.mozilla.org/nagios/ Nagios]
** File a bug in bugzilla under 'MOC: Service Requests'
* Access to [https://inventory.mozilla.org/en-US/#inventory inventory]
** File a bug under 'Infrastructure & Operations::WebOps: Inventory'

= Communication =

== IRC ==
The majority of day-to-day communication in releng happens on IRC:
* https://wiki.mozilla.org/IRC irc://irc.mozilla.org
* irc.mozilla.org:6697 #mozbuild (use SSL, ask for the channel key; your nick needs to be registered)
** private team channel
<pre class="_fck_mw_lspace"> * /attach ircs://irc.mozilla.org:6697/?pass=[your_irc_pass]
* /join #mozbuild [access_key]</pre>

* irc.mozilla.org #releng
** public channel for discussion with sherrifs, and continuous integration bot
* keywords to hilite on:
** "!squirrel" is "I need eyes on this now" keyword in #mozbuild & #releng- please set your client to alert on it.
** "r?" - team member looking for a review, perhaps not in but (irc/pastebin)

* also useful to join
** #developers, #airmozilla, #sf, #mobile, #planning, #release-drivers, #ateam,
** #infra #sysadmins access_key is here https://mana.mozilla.org/wiki/display/SYSADMIN/IRC+use+within+IT
** #moco [access_key is here [https://intranet.mozilla.org/StaffMeeting [n]]]
** #firebot for hiliting when you're mentioned in a bug, review request, etc.

* http://chatzilla.hacksrus.com/faq/#connect

Many folks run a BNC (an IRC bouncer) (e.g. znc) and/or <tt>irssi</tt> under screen to get continuous view of traffic. Ask around. (some [https://mana.mozilla.org/wiki/display/SYSADMIN/IRC+use+within+IT#IRCusewithinIT-irssi good irssi notes])

See [[#pastebin|below]] for a local pastebin install, so you don't paste huge amounts in channel.

IT now also provides a hosted [https://mana.mozilla.org/wiki/display/SD/IRCCloud+Account+Setup IRCCloud cloud] instance you can partake of.
* NOTE: if you're using IRCCloud, and are invited to a password protected channel, IRCCloud will not have the password available to re-connect you. Do the following as soon as you enter the channel:
*# Click the "Options" button
*# Copy the password
*# Click the "Leave" button
*# Click the "Rejoin" button
*# Paste the password into the text box.

== Wiki ==
If you're reading this now, you found the wiki! ;)

Please add yourself and details to the [[ReleaseEngineering#Team | list of team members]].

Don't be shy about making improvements to releng pages based on your experiences. Getting someone in releng to review your changes first is good practice. Just ask in #releng.
* Useful [https://wiki.mozilla.org/Special:Templates templates] (aka "macros")
** for releng, use <tt>{{bug|999}}</tt> for bugzilla references; <tt>{{rev|releases/mozilla-release|6e453b4f7056}}</tt> for hg revisions is also quite nice.

== Pastebin ==
There's a local pastebin instance at http://pastebin.mozilla.org/, so you don't need to paste multi-lines into a channel. You'll find a helper script in the [http://hg.mozilla.org/build/braindump/ braindump] repo at <tt>utils/pastebin</tt>.

* it creates a new page from stdin, and outputs the url to stdout
* an argument will be interpretted as the syntax highlighting format to use
* sample usage: <tt>hg diff | pastebin python | pbcopy</tt> (last part mac specific)
* set & export <tt>PASTEBIN_NAME</tt> if you don't want to post as anonymous.

(Many folks now use the pastebin integrated into IRCCloud.)

== Google Drive ==
Google Drive (formerly Google docs) is a preferred way to share things these days. This includes spreadsheets and documents that will change a great deal over time.

Google Drive access should be enabled with your email account when you start. If you need access to a particular document, talk to the document owner or your manager/mentor.

== Mana ==
Some internal Mozilla systems (IT, HR) are documented on [https://mana.mozilla.org mana]. File a [https://mozilla.service-now.com/ ServiceNow] ticket if you don't have access when you start.

== Vidyo Services ==
Our primary two way video meeting platform is Vidyo. Basic usage instructions are [https://mana.mozilla.org/wiki/display/SD/Vidyo+Desktop+3 here]. Especially if you are running linux, it is highly recommended that you install the client and make test calls prior to any meeting. Many of our team meetings are held in the '''ReleaseEngineering''' room.
* ''Pro tip: many folks have found the mobile client useful to have preinstalled as a backup device.''
* If you're going to record a meeting, practice first. (Instructions are linked from [https://mana.mozilla.org/wiki/display/SD/Vidyo+Desktop+3 mana page].)
* Ask team members for details on recording in the '''ReleaseEngineering''' room.

== Other Resources ==
=== Short-term ===
* [https://docs.google.com/document/d/1VcEjW82jBxr77aYi3TVaha9S4uJwAkwEapXnhhMwcgg RelEng crowd-sourced Glossary of Terms]
* Join https://mozillians.org/ (public, and yet-another-set-of-credentials)
** join the "release engineering" group
* [https://intranet.mozilla.org/ Mozilla intranet]
** https://intranet.mozilla.org/StaffMeeting
* https://mana.mozilla.org/wiki/display/RelEng/Release+Engineering+Home (for sensitive information only) (LDAP Note: Req attribute RelEng)
* https://trello.com/b/KwwYSXE1/release-engineering-status-board (Projects and deliverables status)
* Sign up for [https://wiki.mozilla.org/Safari_Books O'Reilly's online library] of their boooks - a great resource
* Overview of dev cycle: http://k0s.org/mozilla/workflow.svg (slightly dated). And browse http://k0s.org/mozilla

=== Longer-term ===
Talk to your mentor/manager to see which of these make sense.
* https://build.inventory.mozilla.org/build
* https://nagios.mozilla.org/nagios/
* https://metrics.mozilla.com/
* background for starting to understand buildbot releases (stale, but s/b roughly accurate): [http://people.mozilla.com/~bhearsum/mozilla-beta-schedulers.png Beta schedulers]
* How teams coordinate during a release: [http://moz-releng-docs.readthedocs.org/en/latest/release_workflows release work flows]

Firefox Desktop + Firefox Mobile release process docs:
* [[Release_Management/Release_Process]]
* [[Release_Management/Release_Process/FAQ]]
* [https://intranet.mozilla.org/TravelPolicies#Corporate_Travel_Accounts Egencia account]: this should be accessible via [https://mana.mozilla.org/wiki/display/SD/SSO+Quick+Links SSO]

ReleaseEngineering

2016-10-16T22:05:18Z

NThomas: /* Team */ get better @ typos dude

<div style="float: left;">__TOC__</div>
<div style="float: right; width: 60%;">
<div class="firstHeading toc" style="border: thin black solid; text-size: 80%; width: 50%;">
<h2>Quick Links For You</h2>
* [https://treeherder.mozilla.org/ Treeherder]
* [https://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering&component=Other File a new RelEng bug]
* [[ReleaseEngineering/Maintenance | RelEng Changes]] - what did RelEng break?
* [https://secure.pub.build.mozilla.org Mozilla Release Engineering Resources]
</div>
<br clear="all" />
<div class="firstHeading toc" style="border: thin black solid; text-size: 80%; width: 50%;">
<h2>Quick Links For Us</h2>
* [[ReleaseEngineering/How To|How Tos]]
* [[ReleaseEngineering/Buildduty|Buildduty]]
</div>
</div>

<br clear="all"/>

= About Us =
We ship Firefox to the world!

Release Engineering does this by:

* Developing world-class, scalable, open, secure, and maintainable infrastructure for application builds and tests, on a diverse set of desktop and mobile platforms

* Building fast and reliable pipelines to transform source code into shipped products

= Contacting Release Engineering =
Our team is located on 3 continents and in 4 time zones. This allows us to provide better service, especially if you contact us using one of the following methods:
* To request a new service, please [https://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering file a bug]
* To ask a general question, ping in #releng on IRC, or email release@m.c
* To request status on an existing request, set the "needinfo" flag in the bug.
Our team works normal business hours for our respective time zones. If you need to escalate an issue that is both '''urgent''' ''and'' '''important''', use the following methods:
# Use #releng in IRC or email release@m.c, starting your message with 'URGENT:'. If no response in 15 minutes, then
# Be sure the bug severity is set to "<tt>blocker</tt>"
# Call one or more of the following people (nagios page preferred if you have access):
#* [https://phonebook.mozilla.org/#search/ccooper Chris Cooper] Eastern Time
#* [https://phonebook.mozilla.org/#search/catlee Chris AtLee] Eastern Time
#* [https://phonebook.mozilla.org/#search/hwine Hal Wine] Pacific Time
If you get voice mail, please leave your callback number.

= Contributing =
If you are interested in getting involved with Release Engineering, please see our [[ReleaseEngineering/Contribute|contributing page]]! We'd love to hear from you.

= Team =

Mozilla's Release Engineering:
* twitter: [https://twitter.com/MozRelEng @MozRelEng]
* blogs: [http://planet.mozilla.org/releng Planet Release Engineering]
* calendar: [https://www.google.com/calendar/embed?src=mozilla.com_2d32343333353036312d393737%40resource.calendar.google.com&ctz=America/Los_Angeles Google Calendar]
* mailing list: [https://groups.google.com/forum/#!forum/mozilla.release.engineering release-engineering@lists.mozilla.org]

{| border=1
| '''Name'''
| '''Profile'''
| '''Twitter'''
| '''Blog'''
|-
| Ben Hearsum
| [https://mozillians.org/u/bhearsum bhearsum]
| [https://twitter.com/bhearsum @bhearsum]
| [http://hearsum.ca/blog Ben's blog]
|-
| Justin Wood
| [https://mozillians.org/en-US/u/Callek/ Callek]
|
| [http://blog.drapostles.org/ "Callek's Blog"]
|-
| Chris AtLee
| [https://mozillians.org/u/catlee catlee]
| [https://twitter.com/chrisatlee @chrisatlee]
| [http://atlee.ca/blog/ chris' random ramblings: programming, photography, media, and anything else that strikes my fancy]
|-
| Chris Cooper
| [https://mozillians.org/u/coop coop]
| [https://twitter.com/ccooper @ccooper]
| [http://coop.deadsquid.com/ Five Different Types of Fried Cheese]
|-
| Hal Wine
| [https://mozillians.org/u/hwine hwine]
| [https://twitter.com/hwine @hwine]
| [http://dtor.com/halfire/ I am the blog of Hal Fire, and I bring you……interesting tidbits of release engineering]
|-
| Jordan Lund
| [https://mozillians.org/u/jlund jlund]
|
|
|-
| Kim Moir
| [https://mozillians.org/u/kmoir kmoir]
| [http://twitter.com/kmoir @kmoir]
| [http://relengofthenerds.blogspot.com Releng of the Nerds]
|-
| Michael Shal
| [https://mozillians.org/u/mshal/ mshal]
|
| [http://gittup.org/blog/ Mike Shal's blog]
|-
| Nick Thomas
| [https://mozillians.org/u/nthomas nthomas]
| [https://twitter.com/nthomasftang @nthomasftang]
| [http://ftangftang.wordpress.com/ F'tang F'tang Release wrangling at the bottom of the world]
|-
| Rail Aliiev
| [https://mozillians.org/u/rail rail]
| [https://twitter.com/railaliiev @railaliiev]
| [http://rail.merail.ca/ Rail's blog]
|-
| Aki Sasaki
| [https://mozillians.org/en-US/u/aki/ aki]
| [https://twitter.com/escapewindow @escapewindow]
| [http://escapewindow.dreamwidth.org/ escapewindow]
|-
| Mihai Tabara
| [https://mozillians.org/en-US/u/mtabara/ mtabara]
| [https://twitter.com/mihaitabara @mihaitabara]
|
|-
| Rok Garbas
|
| [https://twitter.com/garbas @garbas]
| [https://garbas.si/ Rok's blog]
|-
| Johan Lorenzo
|
|
|
|-
| Simon Fraser
| [https://mozillians.org/en-US/u/sfraser/ sfraser]
|
|
|}

== Interns ==
* [[ReleaseEngineering/Interns|Interns]]

= Documentation =
* [[ReleaseEngineering/How Tos|How Tos]] - a collection of instructions for specific tasks
* [http://moz-releng-docs.readthedocs.org/en/latest/ Releng Docs] for overviews, details, and background.
* [[ReleaseEngineering/Development Best Practices|Development Best Practices]] - read this when you're starting something new
* [[ReleaseEngineering/Wiki Guidelines|Wiki Guidelines]] - how to wiki garden for fun and profit
* [[ReleaseEngineering/Applications|Applications]] - Various applications and services that RelEng provides
** [[ReleaseEngineering/Deployment|Deployment]] - How we deploy our tools, and how do we update them.
* [[ReleaseEngineering/OverviewArchitectureDiagram|Overview Architecture Diagram]] - an architecture diagram that gives a high level overview of systems involved in producing releases of Mozilla's software

* [[ReleaseEngineering/Buildbot Migration|Buildbot Migration]] - What builds are produced where? Buildbot, TaskCluster, or other?

== Buildbot ==
* Master
** [[ReleaseEngineering/Master Naming|Master Naming]]
** [[ReleaseEngineering/Master Setup|Master Setup]]
** [[ReleaseEngineering/Upgrading Buildbot|Upgrading Buildbot]]
** [[ReleaseEngineering/Preproduction|Preproduction]]
** [https://intranet.mozilla.org/RelEngWiki/index.php/Masters Masters] (authentication required)
** [[ReleaseEngineering/Queue directories|Queue directories]]
* Slave
** [[ReleaseEngineering/Buildslave Versions|Buildslave Versions]]
** [[ReleaseEngineering/Buildslave Startup Process|Buildslave Startup Process]]
** [[ReleaseEngineering/Slave_Naming|Slave naming]]
* Development
** [[ReleaseEngineering/Virtualenv|Virtualenv]] - How to set up and use python virtual environments
** [[ReleaseEngineering:TestingTechniques|Methods for testing your changes]]
** [[ReleaseEngineering/Buildbot Best Practices|Buildbot Best Practices]]
** [[ReleaseEngineering/Landing Buildbot Master Changes|Landing Buildbot Master Changes]]
* References
** [[ReleaseEngineering/Buildbot_Database_Schemas|Buildbot Database Schemas]]

== Releases ==
* [[Releases/RelEngChecklist | Release Checklist]] and [[Releases/BuildNotesIndex | Build Notes]]
* [[:Category:Releases:Post_Mortem_Meetings | Releng Post Mortem Meetings]]
* [[ReleaseEngineering/Releaseduty | What is releaseduty?]]
* [[Release:Release_Automation_on_Mercurial:Documentation|Release Automation]]
* [[UpdateGeneration|Update Generation]]
** [[ReleaseEngineering/PatcherTags|Patcher tags for release updates]]

== Platform Support ==
* [[ReferencePlatforms|Reference Platforms]]
* [[ReleaseEngineering/Applications#slavealloc|Slave Allocator]]
* [[ReleaseEngineering/Official Platform Support Checklist|Official Platform Support Checklist]]

== Configuration Management ==
* [[ReleaseEngineering/PuppetAgain | PuppetAgain]]

== Buildduty ==
* [[ReleaseEngineering/Buildduty| How To: So you're on buildduty...]]
* [[ReleaseEngineering/Managing Buildbot with Fabric|Managing Buildbot with Fabric]]
* [[Auto-tools/Projects/Treeherder|Treeherder wiki page]]
* [[Clobbering_the_Tree | Clobbering the Tree]]
* [[ReleaseEngineering/Bugzilla/Triage | Bug Triage]]
* [[ReleaseEngineering:RelEngITSharedDowntime | RelEng+IT shared downtime]]
* [[ReleaseEngineering:ProjectBranchPlanning|Project Branch Planning: how to request a new project branch]]
** [[ReleaseEngineering/DisposableProjectBranches | Disposable Project Branch Bookings]]
* [[Build:TinderboxErrors| Build and Test Log Error/Warning Reference: Debugging red and orange in results]]

== Tryserver ==
* [[ReleaseEngineering/TryserverAsBranch|Tryserver As Branch]]
* [[ReleaseEngineering/TryServer | TryServer]]
* [[ReleaseEngineering/TryChooser | TryChooser]]

== Testing ==
* [[ReleaseEngineering:GraphServer | Graph Server Notes]]
* [[ReleaseEngineering:IntermittentErrors | Intermittent Errors]]
* [[ReleaseEngineering/RequestingASlave | Requesting A Slave]]
* [[ReleaseEngineering/Testing a new Signing Server | Testing a new Signing Server]]

== Bugzilla ==
* [[ReleaseEngineering/Bugzilla/Flags|Flags]]
* [[ReleaseEngineering/Bugzilla/Whiteboard|Whiteboard]]
* [[ReleaseEngineering/Bugzilla/Triage|Triage]]

== VCS Sync ==
* [[ReleaseEngineering/VCSSync/HowTo|HowTo]]
* [[ReleaseEngineering/VCSSync/History|History]]

= Subpages =
* [[ReleaseEngineering/Breakout Sessions]]
* [[ReleaseEngineering/Blackbox Sessions]]
* [[ReleaseEngineering/Presentations]]
* [[ReleaseEngineering/Obsolete Pages]]
* [[Special:PrefixIndex/{{FULLPAGENAME}}/]] - all subpages of [[{{FULLPAGENAME}}]] in the wiki
* [[ReleaseEngineering/NoReboots]]

ReleaseEngineering

2016-10-16T22:04:21Z

NThomas: /* Team */ Twitter is where it's @

<div style="float: left;">__TOC__</div>
<div style="float: right; width: 60%;">
<div class="firstHeading toc" style="border: thin black solid; text-size: 80%; width: 50%;">
<h2>Quick Links For You</h2>
* [https://treeherder.mozilla.org/ Treeherder]
* [https://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering&component=Other File a new RelEng bug]
* [[ReleaseEngineering/Maintenance | RelEng Changes]] - what did RelEng break?
* [https://secure.pub.build.mozilla.org Mozilla Release Engineering Resources]
</div>
<br clear="all" />
<div class="firstHeading toc" style="border: thin black solid; text-size: 80%; width: 50%;">
<h2>Quick Links For Us</h2>
* [[ReleaseEngineering/How To|How Tos]]
* [[ReleaseEngineering/Buildduty|Buildduty]]
</div>
</div>

<br clear="all"/>

= About Us =
We ship Firefox to the world!

Release Engineering does this by:

* Developing world-class, scalable, open, secure, and maintainable infrastructure for application builds and tests, on a diverse set of desktop and mobile platforms

* Building fast and reliable pipelines to transform source code into shipped products

= Contacting Release Engineering =
Our team is located on 3 continents and in 4 time zones. This allows us to provide better service, especially if you contact us using one of the following methods:
* To request a new service, please [https://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering file a bug]
* To ask a general question, ping in #releng on IRC, or email release@m.c
* To request status on an existing request, set the "needinfo" flag in the bug.
Our team works normal business hours for our respective time zones. If you need to escalate an issue that is both '''urgent''' ''and'' '''important''', use the following methods:
# Use #releng in IRC or email release@m.c, starting your message with 'URGENT:'. If no response in 15 minutes, then
# Be sure the bug severity is set to "<tt>blocker</tt>"
# Call one or more of the following people (nagios page preferred if you have access):
#* [https://phonebook.mozilla.org/#search/ccooper Chris Cooper] Eastern Time
#* [https://phonebook.mozilla.org/#search/catlee Chris AtLee] Eastern Time
#* [https://phonebook.mozilla.org/#search/hwine Hal Wine] Pacific Time
If you get voice mail, please leave your callback number.

= Contributing =
If you are interested in getting involved with Release Engineering, please see our [[ReleaseEngineering/Contribute|contributing page]]! We'd love to hear from you.

= Team =

Mozilla's Release Engineering:
* twitter: [https://twitter.com/MozRelEng @MozRelEng]
* blogs: [http://planet.mozilla.org/releng Planet Release Engineering]
* calendar: [https://www.google.com/calendar/embed?src=mozilla.com_2d32343333353036312d393737%40resource.calendar.google.com&ctz=America/Los_Angeles Google Calendar]
* mailing list: [https://groups.google.com/forum/#!forum/mozilla.release.engineering release-engineering@lists.mozilla.org]

{| border=1
| '''Name'''
| '''Profile'''
| '''Twitter'''
| '''Blog'''
|-
| Ben Hearsum
| [https://mozillians.org/u/bhearsum bhearsum]
| [https://twitter.com/bhearsum @bhearsum]
| [http://hearsum.ca/blog Ben's blog]
|-
| Justin Wood
| [https://mozillians.org/en-US/u/Callek/ Callek]
|
| [http://blog.drapostles.org/ "Callek's Blog"]
|-
| Chris AtLee
| [https://mozillians.org/u/catlee catlee]
| [https://twitter.com/chrisatlee @chrisatlee]
| [http://atlee.ca/blog/ chris' random ramblings: programming, photography, media, and anything else that strikes my fancy]
|-
| Chris Cooper
| [https://mozillians.org/u/coop coop]
| [https://twitter.com/ccooper @ccooper]
| [http://coop.deadsquid.com/ Five Different Types of Fried Cheese]
|-
| Hal Wine
| [https://mozillians.org/u/hwine hwine]
| [https://twitter.com/hwine @hwine]
| [http://dtor.com/halfire/ I am the blog of Hal Fire, and I bring you……interesting tidbits of release engineering]
|-
| Jordan Lund
| [https://mozillians.org/u/jlund jlund]
|
|
|-
| Kim Moir
| [https://mozillians.org/u/kmoir kmoir]
| [http://twitter.com/kmoir @kmoir]
| [http://relengofthenerds.blogspot.com Releng of the Nerds]
|-
| Michael Shal
| [https://mozillians.org/u/mshal/ mshal]
|
| [http://gittup.org/blog/ Mike Shal's blog]
|-
| Nick Thomas
| [https://mozillians.org/u/nthomas nthomas]
| [https://twitter.com/nthomasftang @nthomasftang]
| [http://ftangftang.wordpress.com/ F'tang F'tang Release wrangling at the bottom of the world]
|-
| Rail Aliiev
| [https://mozillians.org/u/rail rail]
| [https://twitter.com/railaliiev @railaliiev]
| [http://rail.merail.ca/ Rail's blog]
|-
| Aki Sasaki
| [https://mozillians.org/en-US/u/aki/ aki]
| [https://twitter.com/escapewindow escapewindow]
| [http://escapewindow.dreamwidth.org/ escapewindow]
|-
| Mihai Tabara
| [https://mozillians.org/en-US/u/mtabara/ mtabara]
| [https://twitter.com/mihaitabara @mihaitabara]
|
|-
| Rok Garbas
|
| [https://twitter.com/garbas @garbas]
| [https://garbas.si/ Rok's blog]
|-
| Johan Lorenzo
|
|
|
|-
| Simon Fraser
| [https://mozillians.org/en-US/u/sfraser/ @sfraser]
|
|
|}

== Interns ==
* [[ReleaseEngineering/Interns|Interns]]

= Documentation =
* [[ReleaseEngineering/How Tos|How Tos]] - a collection of instructions for specific tasks
* [http://moz-releng-docs.readthedocs.org/en/latest/ Releng Docs] for overviews, details, and background.
* [[ReleaseEngineering/Development Best Practices|Development Best Practices]] - read this when you're starting something new
* [[ReleaseEngineering/Wiki Guidelines|Wiki Guidelines]] - how to wiki garden for fun and profit
* [[ReleaseEngineering/Applications|Applications]] - Various applications and services that RelEng provides
** [[ReleaseEngineering/Deployment|Deployment]] - How we deploy our tools, and how do we update them.
* [[ReleaseEngineering/OverviewArchitectureDiagram|Overview Architecture Diagram]] - an architecture diagram that gives a high level overview of systems involved in producing releases of Mozilla's software

* [[ReleaseEngineering/Buildbot Migration|Buildbot Migration]] - What builds are produced where? Buildbot, TaskCluster, or other?

== Buildbot ==
* Master
** [[ReleaseEngineering/Master Naming|Master Naming]]
** [[ReleaseEngineering/Master Setup|Master Setup]]
** [[ReleaseEngineering/Upgrading Buildbot|Upgrading Buildbot]]
** [[ReleaseEngineering/Preproduction|Preproduction]]
** [https://intranet.mozilla.org/RelEngWiki/index.php/Masters Masters] (authentication required)
** [[ReleaseEngineering/Queue directories|Queue directories]]
* Slave
** [[ReleaseEngineering/Buildslave Versions|Buildslave Versions]]
** [[ReleaseEngineering/Buildslave Startup Process|Buildslave Startup Process]]
** [[ReleaseEngineering/Slave_Naming|Slave naming]]
* Development
** [[ReleaseEngineering/Virtualenv|Virtualenv]] - How to set up and use python virtual environments
** [[ReleaseEngineering:TestingTechniques|Methods for testing your changes]]
** [[ReleaseEngineering/Buildbot Best Practices|Buildbot Best Practices]]
** [[ReleaseEngineering/Landing Buildbot Master Changes|Landing Buildbot Master Changes]]
* References
** [[ReleaseEngineering/Buildbot_Database_Schemas|Buildbot Database Schemas]]

== Releases ==
* [[Releases/RelEngChecklist | Release Checklist]] and [[Releases/BuildNotesIndex | Build Notes]]
* [[:Category:Releases:Post_Mortem_Meetings | Releng Post Mortem Meetings]]
* [[ReleaseEngineering/Releaseduty | What is releaseduty?]]
* [[Release:Release_Automation_on_Mercurial:Documentation|Release Automation]]
* [[UpdateGeneration|Update Generation]]
** [[ReleaseEngineering/PatcherTags|Patcher tags for release updates]]

== Platform Support ==
* [[ReferencePlatforms|Reference Platforms]]
* [[ReleaseEngineering/Applications#slavealloc|Slave Allocator]]
* [[ReleaseEngineering/Official Platform Support Checklist|Official Platform Support Checklist]]

== Configuration Management ==
* [[ReleaseEngineering/PuppetAgain | PuppetAgain]]

== Buildduty ==
* [[ReleaseEngineering/Buildduty| How To: So you're on buildduty...]]
* [[ReleaseEngineering/Managing Buildbot with Fabric|Managing Buildbot with Fabric]]
* [[Auto-tools/Projects/Treeherder|Treeherder wiki page]]
* [[Clobbering_the_Tree | Clobbering the Tree]]
* [[ReleaseEngineering/Bugzilla/Triage | Bug Triage]]
* [[ReleaseEngineering:RelEngITSharedDowntime | RelEng+IT shared downtime]]
* [[ReleaseEngineering:ProjectBranchPlanning|Project Branch Planning: how to request a new project branch]]
** [[ReleaseEngineering/DisposableProjectBranches | Disposable Project Branch Bookings]]
* [[Build:TinderboxErrors| Build and Test Log Error/Warning Reference: Debugging red and orange in results]]

== Tryserver ==
* [[ReleaseEngineering/TryserverAsBranch|Tryserver As Branch]]
* [[ReleaseEngineering/TryServer | TryServer]]
* [[ReleaseEngineering/TryChooser | TryChooser]]

== Testing ==
* [[ReleaseEngineering:GraphServer | Graph Server Notes]]
* [[ReleaseEngineering:IntermittentErrors | Intermittent Errors]]
* [[ReleaseEngineering/RequestingASlave | Requesting A Slave]]
* [[ReleaseEngineering/Testing a new Signing Server | Testing a new Signing Server]]

== Bugzilla ==
* [[ReleaseEngineering/Bugzilla/Flags|Flags]]
* [[ReleaseEngineering/Bugzilla/Whiteboard|Whiteboard]]
* [[ReleaseEngineering/Bugzilla/Triage|Triage]]

== VCS Sync ==
* [[ReleaseEngineering/VCSSync/HowTo|HowTo]]
* [[ReleaseEngineering/VCSSync/History|History]]

= Subpages =
* [[ReleaseEngineering/Breakout Sessions]]
* [[ReleaseEngineering/Blackbox Sessions]]
* [[ReleaseEngineering/Presentations]]
* [[ReleaseEngineering/Obsolete Pages]]
* [[Special:PrefixIndex/{{FULLPAGENAME}}/]] - all subpages of [[{{FULLPAGENAME}}]] in the wiki
* [[ReleaseEngineering/NoReboots]]

ReleaseEngineering

2016-10-16T21:59:40Z

NThomas: /* Team */ Add "new" team members

<div style="float: left;">__TOC__</div>
<div style="float: right; width: 60%;">
<div class="firstHeading toc" style="border: thin black solid; text-size: 80%; width: 50%;">
<h2>Quick Links For You</h2>
* [https://treeherder.mozilla.org/ Treeherder]
* [https://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering&component=Other File a new RelEng bug]
* [[ReleaseEngineering/Maintenance | RelEng Changes]] - what did RelEng break?
* [https://secure.pub.build.mozilla.org Mozilla Release Engineering Resources]
</div>
<br clear="all" />
<div class="firstHeading toc" style="border: thin black solid; text-size: 80%; width: 50%;">
<h2>Quick Links For Us</h2>
* [[ReleaseEngineering/How To|How Tos]]
* [[ReleaseEngineering/Buildduty|Buildduty]]
</div>
</div>

<br clear="all"/>

= About Us =
We ship Firefox to the world!

Release Engineering does this by:

* Developing world-class, scalable, open, secure, and maintainable infrastructure for application builds and tests, on a diverse set of desktop and mobile platforms

* Building fast and reliable pipelines to transform source code into shipped products

= Contacting Release Engineering =
Our team is located on 3 continents and in 4 time zones. This allows us to provide better service, especially if you contact us using one of the following methods:
* To request a new service, please [https://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering file a bug]
* To ask a general question, ping in #releng on IRC, or email release@m.c
* To request status on an existing request, set the "needinfo" flag in the bug.
Our team works normal business hours for our respective time zones. If you need to escalate an issue that is both '''urgent''' ''and'' '''important''', use the following methods:
# Use #releng in IRC or email release@m.c, starting your message with 'URGENT:'. If no response in 15 minutes, then
# Be sure the bug severity is set to "<tt>blocker</tt>"
# Call one or more of the following people (nagios page preferred if you have access):
#* [https://phonebook.mozilla.org/#search/ccooper Chris Cooper] Eastern Time
#* [https://phonebook.mozilla.org/#search/catlee Chris AtLee] Eastern Time
#* [https://phonebook.mozilla.org/#search/hwine Hal Wine] Pacific Time
If you get voice mail, please leave your callback number.

= Contributing =
If you are interested in getting involved with Release Engineering, please see our [[ReleaseEngineering/Contribute|contributing page]]! We'd love to hear from you.

= Team =

Mozilla's Release Engineering:
* twitter: [https://twitter.com/MozRelEng @MozRelEng]
* blogs: [http://planet.mozilla.org/releng Planet Release Engineering]
* calendar: [https://www.google.com/calendar/embed?src=mozilla.com_2d32343333353036312d393737%40resource.calendar.google.com&ctz=America/Los_Angeles Google Calendar]
* mailing list: [https://groups.google.com/forum/#!forum/mozilla.release.engineering release-engineering@lists.mozilla.org]

{| border=1
| '''Name'''
| '''Profile'''
| '''Twitter'''
| '''Blog'''
|-
| Ben Hearsum
| [https://mozillians.org/u/bhearsum bhearsum]
| [https://twitter.com/bhearsum @bhearsum]
| [http://hearsum.ca/blog Ben's blog]
|-
| Justin Wood
| [https://mozillians.org/en-US/u/Callek/ Callek]
|
| [http://blog.drapostles.org/ "Callek's Blog"]
|-
| Chris AtLee
| [https://mozillians.org/u/catlee catlee]
| [https://twitter.com/chrisatlee @chrisatlee]
| [http://atlee.ca/blog/ chris' random ramblings: programming, photography, media, and anything else that strikes my fancy]
|-
| Chris Cooper
| [https://mozillians.org/u/coop coop]
| [https://twitter.com/ccooper @ccooper]
| [http://coop.deadsquid.com/ Five Different Types of Fried Cheese]
|-
| Hal Wine
| [https://mozillians.org/u/hwine hwine]
| [https://twitter.com/hwine @hwine]
| [http://dtor.com/halfire/ I am the blog of Hal Fire, and I bring you……interesting tidbits of release engineering]
|-
| Jordan Lund
| [https://mozillians.org/u/jlund jlund]
|
|
|-
| Kim Moir
| [https://mozillians.org/u/kmoir kmoir]
| [http://twitter.com/kmoir @kmoir]
| [http://relengofthenerds.blogspot.com Releng of the Nerds]
|-
| Michael Shal
| [https://mozillians.org/u/mshal/ mshal]
|
| [http://gittup.org/blog/ Mike Shal's blog]
|-
| Nick Thomas
| [https://mozillians.org/u/nthomas nthomas]
| [https://twitter.com/nthomasftang @nthomasftang]
| [http://ftangftang.wordpress.com/ F'tang F'tang Release wrangling at the bottom of the world]
|-
| Rail Aliiev
| [https://mozillians.org/u/rail rail]
| [https://twitter.com/railaliiev @railaliiev]
| [http://rail.merail.ca/ Rail's blog]
|-
| Aki Sasaki
| [https://mozillians.org/en-US/u/aki/ aki]
| [https://twitter.com/escapewindow escapewindow]
| [http://escapewindow.dreamwidth.org/ escapewindow]
|-
| Mihai Tabara
| [https://mozillians.org/en-US/u/mtabara/ mtabara]
| [https://twitter.com/mihaitabara mihaitabara]
|
|-
| Rok Garbas
|
| [https://twitter.com/garbas garbas]
| [https://garbas.si/ Rok's blog]
|-
| Johan Lorenzo
|
|
|
|-
| Simon Fraser
| [https://mozillians.org/en-US/u/sfraser/ sfraser]
|
|
|}

== Interns ==
* [[ReleaseEngineering/Interns|Interns]]

= Documentation =
* [[ReleaseEngineering/How Tos|How Tos]] - a collection of instructions for specific tasks
* [http://moz-releng-docs.readthedocs.org/en/latest/ Releng Docs] for overviews, details, and background.
* [[ReleaseEngineering/Development Best Practices|Development Best Practices]] - read this when you're starting something new
* [[ReleaseEngineering/Wiki Guidelines|Wiki Guidelines]] - how to wiki garden for fun and profit
* [[ReleaseEngineering/Applications|Applications]] - Various applications and services that RelEng provides
** [[ReleaseEngineering/Deployment|Deployment]] - How we deploy our tools, and how do we update them.
* [[ReleaseEngineering/OverviewArchitectureDiagram|Overview Architecture Diagram]] - an architecture diagram that gives a high level overview of systems involved in producing releases of Mozilla's software

* [[ReleaseEngineering/Buildbot Migration|Buildbot Migration]] - What builds are produced where? Buildbot, TaskCluster, or other?

== Buildbot ==
* Master
** [[ReleaseEngineering/Master Naming|Master Naming]]
** [[ReleaseEngineering/Master Setup|Master Setup]]
** [[ReleaseEngineering/Upgrading Buildbot|Upgrading Buildbot]]
** [[ReleaseEngineering/Preproduction|Preproduction]]
** [https://intranet.mozilla.org/RelEngWiki/index.php/Masters Masters] (authentication required)
** [[ReleaseEngineering/Queue directories|Queue directories]]
* Slave
** [[ReleaseEngineering/Buildslave Versions|Buildslave Versions]]
** [[ReleaseEngineering/Buildslave Startup Process|Buildslave Startup Process]]
** [[ReleaseEngineering/Slave_Naming|Slave naming]]
* Development
** [[ReleaseEngineering/Virtualenv|Virtualenv]] - How to set up and use python virtual environments
** [[ReleaseEngineering:TestingTechniques|Methods for testing your changes]]
** [[ReleaseEngineering/Buildbot Best Practices|Buildbot Best Practices]]
** [[ReleaseEngineering/Landing Buildbot Master Changes|Landing Buildbot Master Changes]]
* References
** [[ReleaseEngineering/Buildbot_Database_Schemas|Buildbot Database Schemas]]

== Releases ==
* [[Releases/RelEngChecklist | Release Checklist]] and [[Releases/BuildNotesIndex | Build Notes]]
* [[:Category:Releases:Post_Mortem_Meetings | Releng Post Mortem Meetings]]
* [[ReleaseEngineering/Releaseduty | What is releaseduty?]]
* [[Release:Release_Automation_on_Mercurial:Documentation|Release Automation]]
* [[UpdateGeneration|Update Generation]]
** [[ReleaseEngineering/PatcherTags|Patcher tags for release updates]]

== Platform Support ==
* [[ReferencePlatforms|Reference Platforms]]
* [[ReleaseEngineering/Applications#slavealloc|Slave Allocator]]
* [[ReleaseEngineering/Official Platform Support Checklist|Official Platform Support Checklist]]

== Configuration Management ==
* [[ReleaseEngineering/PuppetAgain | PuppetAgain]]

== Buildduty ==
* [[ReleaseEngineering/Buildduty| How To: So you're on buildduty...]]
* [[ReleaseEngineering/Managing Buildbot with Fabric|Managing Buildbot with Fabric]]
* [[Auto-tools/Projects/Treeherder|Treeherder wiki page]]
* [[Clobbering_the_Tree | Clobbering the Tree]]
* [[ReleaseEngineering/Bugzilla/Triage | Bug Triage]]
* [[ReleaseEngineering:RelEngITSharedDowntime | RelEng+IT shared downtime]]
* [[ReleaseEngineering:ProjectBranchPlanning|Project Branch Planning: how to request a new project branch]]
** [[ReleaseEngineering/DisposableProjectBranches | Disposable Project Branch Bookings]]
* [[Build:TinderboxErrors| Build and Test Log Error/Warning Reference: Debugging red and orange in results]]

== Tryserver ==
* [[ReleaseEngineering/TryserverAsBranch|Tryserver As Branch]]
* [[ReleaseEngineering/TryServer | TryServer]]
* [[ReleaseEngineering/TryChooser | TryChooser]]

== Testing ==
* [[ReleaseEngineering:GraphServer | Graph Server Notes]]
* [[ReleaseEngineering:IntermittentErrors | Intermittent Errors]]
* [[ReleaseEngineering/RequestingASlave | Requesting A Slave]]
* [[ReleaseEngineering/Testing a new Signing Server | Testing a new Signing Server]]

== Bugzilla ==
* [[ReleaseEngineering/Bugzilla/Flags|Flags]]
* [[ReleaseEngineering/Bugzilla/Whiteboard|Whiteboard]]
* [[ReleaseEngineering/Bugzilla/Triage|Triage]]

== VCS Sync ==
* [[ReleaseEngineering/VCSSync/HowTo|HowTo]]
* [[ReleaseEngineering/VCSSync/History|History]]

= Subpages =
* [[ReleaseEngineering/Breakout Sessions]]
* [[ReleaseEngineering/Blackbox Sessions]]
* [[ReleaseEngineering/Presentations]]
* [[ReleaseEngineering/Obsolete Pages]]
* [[Special:PrefixIndex/{{FULLPAGENAME}}/]] - all subpages of [[{{FULLPAGENAME}}]] in the wiki
* [[ReleaseEngineering/NoReboots]]

ReleaseEngineering/TryServer

2016-10-04T03:00:31Z

NThomas: /* Configuration */ selective pulls are ok

= Try Server =
The [https://treeherder.mozilla.org/#/jobs?repo=try try server] is an easy way to test a patch without actually checking the patch into the core repository. Your code will go through the same tests as a mozilla-central push, and you'll be able to download builds if you wish.

== Getting access to the Try Server ==
To use the try server, you'll need [http://www.mozilla.org/hacking/commit-access-policy/ Level 1 Commit Access]. You can learn more about Mozilla's commit access policies and start the process of signing up for an account here: [http://www.mozilla.org/hacking/committer/ Becoming a Mozilla Contributor]

After you have level 1 commit access, you'll need to do a bit of setup before you'll be able to push. See [https://developer.mozilla.org/en-US/docs/Mercurial/Using_Mercurial#How_do_I_check_stuff_in.3F this guide] for instructions. Assuming you only have level 1 access, you won't be able to push to mozilla-central, but you can replace 'hg.mozilla.org/mozilla-central' with 'hg.mozilla.org/try'.

== Configuration ==
Before using try, there is some recommended configuration you should set up. This can be accomplished by running:

$ ./mach mercurial-setup

Be sure to at least enable the [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/firefoxtree.html firefoxtree extension] and the push-to-try extension. The push-to-try extension is required if you wish to use the |mach try| command (see below). Firefoxtree will give you a handy 'try' alias you can use for pushing, and prevent you from accidentally pushing multiple heads. If for some reason you prefer not to use firefoxtree, you can set the same alias up manually in your hgrc:

[paths]
try = ssh://hg.mozilla.org/try

Note: Never pull the whole try repo. You'll end up with hundreds of heads containing everyone's half baked and broken pushes.

For more information, see [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html mercurial for mozillians].

== How to push to try ==
There are two ways to schedule jobs on try. You can either push like normal, and select which jobs you want using [https://wiki.mozilla.org/EngineeringProductivity/Projects/Treeherder treeherder], or you can specify a try syntax in your commit message to schedule jobs automatically.

=== Scheduling jobs with Treeherder ===

Note: The "Add New Jobs" feature doesn't show TaskCluster jobs for non-try pushes. See {{bug|1286813}}

With this method, just push to try like you would to inbound (don't create any try syntax):

$ hg push -r . try

You should see a warning about no jobs being scheduled automatically, followed by a link to your push in treeherder. Open the link, select the drop-down arrow at the top right, and choose "Add new Jobs". It might take upto 6 seconds for the jobs to load ({{bug|1288028}}) :

https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042414/TH_dropdown.png

Click the job symbols you wish to schedule. If you select a test job, the required build will automatically be scheduled:

https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042426/TH_with_jobs.png

Finally, click "Trigger New Jobs" near the top right of your push.

NOTE: The outcome of your request will show up as a 'Sch' job ("Scheduling" request) or as an 'A' ("Action Task") job depending if you requested Buildbot (old CI) or TaskCluster (new CI) jobs. [https://armenzg.blogspot.ca/2016/06/adding-new-jobs-to-treeherder-is-now.html Blog post]. In the logs you can see the outcome of your request. If there are any issues you will be able to file it with that information. A failed 'Sch' or 'A' job would mean "Add New Jobs" has failed.

http://people.mozilla.org/~armenzg/sattap/034f72da.png

=== Scheduling jobs with Try Syntax ===
If you know exactly what you want to run, you can use [[Build:TryChooser]] to select which jobs run directly in your commit message. Make sure the commit message containing try syntax is placed in your ''topmost commit''. The [http://trychooser.pub.build.mozilla.org/ TryChooser] web page can help you build a commit message for custom requests. It will generate both a syntax string as well as a mach command you can simply paste into your terminal.

==== Using mach ====
The recommended way to push with a try syntax, is to use |mach try|. It works with both mercurial and git, and you can use the [http://trychooser.pub.build.mozilla.org/ TryChooser] web page to generate the command. For example:

$ mach try -b o -p linux -u mochitest-1 -t none

The |mach try| command also has some advanced features. It can actually change how the job gets run within automation. For example, to run only mochitests under 'dom/indexedDB':

$ mach try -b o -p linux -u mochitests --and dom/indexedDB

For more information see:

$ mach try --help

Note: This method doesn't work well with mq.

==== Using the Trychooser Extension ====
You can also use the [https://bitbucket.org/sfink/trychooser mercurial trychooser extension] from sfink. It has some neat features, like a curses ui that lets you graphically choose a try syntax. Though {{bug|1197868}} tracks implementing this feature in |mach try|. Eventually trychooser will be deprecated in favor of |mach try|.

==== Using mq ====
The aforementioned tools may not play all that nicely with [https://www.mercurial-scm.org/wiki/MqExtension mercurial queues]. If you use mq, you can still push to try manually. First build your try syntax with the [http://trychooser.pub.build.mozilla.org/ TryChooser] web page. Then run:

$ hg qref --message "try: <your-computed-syntax-here>"
$ hg push -f try

== Viewing the results ==
You can see the results of your tryserver build in a number of ways:

* You'll get an email on a successful push with a link to treeherder for your revision as well as optional emails on any non-successful build/test/talos results (this setting can be adjusted using [[Build:TryChooser]] args for email notification)
* You can have the results of your try run posted to bug(s) automatically at the completion of the run using the --post-to-bugzilla flag in your try syntax (see: [[Build:TryChooser]] for examples)
* The link to treeherder will be printed on the command line.
* Look for your changeset on [https://treeherder.mozilla.org/#/jobs?repo=try Treeherder]. You can add '''&author=YOUR.EMAIL''' to only see your pushes.
* Download your completed builds from [http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/?C=M;O=D firefox/tryserver-builds on ftp.m.o].

== Using a custom mozconfig ==
The mozconfigs for recent mozilla-central clones are located in the browser/config/mozconfigs directory. Edit those as you please.

If you want to apply the same mozconfig changes to multiple platforms, you can edit <tt>build/mozconfig.common.override</tt> instead. This file is included at the end of each of the in-tree mozconfig files.

Android mozconfigs are in mobile/android/config/mozconfigs.

Note:
* TryServer purpose is to tell what will happen on Tinderbox, not to check every possible build option/configuration.
** Any non-standard feature is implicitly unsupported. You may try them, but don't complain if they break.

== Getting debug symbols ==
By default native debug symbols are not uploaded for Try server builds because of their size. If you want to debug your builds locally you must add <tt>MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS=1</tt> to the in-tree mozconfigs. You can do this for all platforms by importing [http://hg.mozilla.org/users/tmielczarek_mozilla.com/mq/raw-file/b44faf6cd177/enable-full-symbols this patch] into your mq and pushing it along with your changes to Try. This will cause a ...crashreporter-symbols-full.zip package to be uploaded to the builds directory for each platform built.

== Adding new jobs ==
It's possible to create new jobs (or modify existing ones) directly *in* your try push, provided you use taskcluster. Just edit the relevant configuration in testing/taskcluster. For more information on creating jobs, see the [http://docs.taskcluster.net/ taskcluster docs].

== Desktop l10n jobs ==
You can use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.

The jobs can be customized by modifying files prior to pushing:
* reducing the number of locales by limiting <tt>browser/locales/all-locales</tt> (eg top-locales like de fr ja ja-JP-mac ru zh-TW). Leaving a full list of locales is likely to hit a timeout on Mac and Windows
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#4 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario
* use the Aurora l10n repos by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#10 hg_l10n_base]

The resulting builds are uploaded to the same sub-directory as en-US builds, eg try-linux/ for 32bit linux.

== Desktop l10n jobs (on Taskcluster) ==

To get Desktop l10n jobs on taskcluster on try, simply pass `-p linux64-l10n,linux-l10n` with your try push. You'll also get these jobs automatically with `-p linux64,linux` if you are touching one of the files known to be heavily involved in l10n jobs. (Taskcluster support for other platforms is not yet available)

You can also use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.

The jobs can be customized by modifying files prior to pushing:
* reducing the number of locales by limiting <tt>browser/locales/all-locales</tt> (eg top-locales like de fr ja ja-JP-mac ru zh-TW).
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#4 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario
* use the Aurora l10n repos by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#10 hg_l10n_base]

The resulting builds are uploaded as a task artifact, and are not yet signed.

== Android Single-Locale l10n jobs (on Taskcluster) ==

(NOTE: Only supported on Gecko 51a1 and above)

To get Android l10n jobs on taskcluster on try, simply pass `-p android-api-15-l10n` with your try push. You'll also get these jobs automatically with `-p android-api-15` if you are touching one of the files known to be heavily involved in l10n jobs.

You can also use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.

The jobs can be customized by modifying files prior to pushing:
* reducing the number of locales by limiting <tt>mobile/android/locales/all-locales</tt> (eg top-locales like de fr ru zh-TW).
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try_android-api-15.py#3 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario
* use the Aurora l10n repos by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try_android-api-15.py#42 hg_l10n_base]

The resulting builds are uploaded as a task artifact, and are not yet signed.

== Server Status ==
* Pending builds by revision are at https://secure.pub.build.mozilla.org/buildapi/pending
* In-progress builds by revision are are https://secure.pub.build.mozilla.org/buildapi/running

== Other Notes ==
* Finished builds will live in http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/<your_ldap_email>-<revision> for 14 days before deletion
* Treeherder data is purged after 4 months.
* If you have any problems please [https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org&component=Release%20Engineering&status_whiteboard=tryserver file a bug]

* Suggestions for the future can be made [[Build:TryServer:Suggestions|here]] or file a blocking bug against {{bug|try_enhancements}}

== Other Mozilla Try Servers ==
* [[ReleaseEngineering/ThunderbirdTryServer|Thunderbird Try Server]] for the comm-central repository

== Problem Diagnosis ==
=== Can not access try server ===
Test your account & configuration
* <code>ssh hg.mozilla.org</code>, response: "No Interactive shells allowed here!"
* <code>ssh hg.mozilla.org clone invalid_sandbox</code>, response: menu display and interactive prompting.

<div id="long_try_push"></div>
=== Pushes to try take a very long time ===
Note: if a fellow developer cancelled their try push, they have saddled you with the cost of rebuilding the cache. (See [http://dtor.com/halfire/2014/07/02/2014_06_try_server_update.html#caching caching] details.)

If you're experiencing excessive wait times (> 45min) pushing to try, please file a bug asking IT to reset the try repository using [https://bugzilla.mozilla.org/enter_bug.cgi?comment=please%20schedule%20a%20reset%20of%20the%20try%20repository%20ASAP%20with%20sheriffs%20%26%20releng.&component=WebOps%3A%20Source%20Control&op_sys=All&product=Infrastructure%20%26%20Operations&rep_platform=All&short_desc=Push%20to%20try%20taking%20XXX%20minutes this template] (please include specifics of your experience). They will coordinate with sheriffs and release engineering as needed.

=== Waiting for Lock ===
If you get a message similar to:
remote: waiting for lock on repository /repo/hg/mozilla/try/ held by 'hgssh1.dmz.scl3.mozilla.com:23974'
remote: abort: repository /repo/hg/mozilla/try/: timed out waiting for lock held by hgssh1.dmz.scl3.mozilla.com:30549
It means several developers are trying to push to try at the same time. In the case above, nothing appears to be wrong, as the PID changes between the messages.

=== Waiting for Lock multiple times with the same pid ===
Similar to the above case, but with the same pid when you retry over and over again.

Please retry your push. If you see messages indicating the same process has been pushing for more than 15 minutes, treat as [[#long_try_push|above]].

== Buildduty issues ==
=== How do I trigger additional talos/test runs for a given try build? ===
If your trychooser syntax included the tests you'd like more of, then select the job you want on Treeherder and use the + button. For test suites you didn't request originally you can use the "Add New Jobs" interface in Treeherder.

=== How do I cancel existing jobs? ===
For individual jobs, select the relevant one on Treeherder and use the cancel button. To cancel all jobs, use the menu arrow shown on the header row for each push, and then the "Cancel all" option.

=== TryChooser ===
See the [[ReleaseEngineering/TryChooser#Buildduty_Issues|TryChooser]] wiki page.

== See Also ==
* https://developer.mozilla.org/en/Creating_Mercurial_User_Repositories
* http://trychooser.pub.build.mozilla.org/
* https://wiki.mozilla.org/Sheriffing/How:To:Recommended_Try_Practices
* https://wiki.mozilla.org/Build:TryChooser
* https://wiki.mozilla.org/ReleaseEngineering:Autoland
* [http://build.mozilla.org/buildapi/self-serve Manage Submissions] [[https://build.mozilla.org/buildapi/self-serve/try try]]
* [https://secure.pub.build.mozilla.org/builddata/reports/reportor/daily/highscores/highscores.html Scoreboard]

Balrog

2016-09-08T22:13:31Z

NThomas: Fix repo

= Overview =
Balrog is the software that runs the server side component of the update system used by Firefox and other Mozilla products. It is the successor to [https://wiki.mozilla.org/AUS AUS (Application Update Service)], which did not scale to our current needs nor allow us to adapt to more recent business requirements. Balrog helps us ship updates faster and with much more flexability than we've had in the past.

Bugs and feature requests should be filed in the [https://bugzilla.mozilla.org/enter_bug.cgi?component=Balrog%3A%20Backend&product=Release%20Engineering Backend] or [https://bugzilla.mozilla.org/enter_bug.cgi?component=Balrog%3A%20Frontend&product=Release%20Engineering Frontend] components of Bugzilla. You can find the Balrog developers in [irc://irc.mozilla.org/#balrog #balrog].

= Domains =
Balrog admin interface is accessible at [https://aus4-admin.mozilla.org/ aus4-admin.mozilla.org] (vpn required).

The public interface that Firefox and other applications talk to is at aus5.mozilla.org. Many older update domains are also served by Balrog, including aus4.mozilla.org, aus3.mozilla.org, and aus2.mozilla.org. More details on these can be found on the [[Balrog/Client_Domains | Client Domains page]].

= Database Model =
Balrog's model centres around two concepts: Rules and Releases. When a request for update from an application is received it is matched up against the rules. Once the correct rule has been found, it contains a pointer to a Release, which contains all of the metadata needed to construct a proper update response. Rules and Releases are described in greater detail below:

== Rules ==
The most important part of Balrog to understand is its rules. When a request comes in it is matched against each of Balrog's rule to find the one that best suits it (more in this below). Once found, Balrog looks at that rule's "mapping", which points at a release that has the required information to serve an update back to the client. Without any rules, Balrog will never serve an update. With badly configured rules Balrog could do bad things like serve Firefox updates to B2G devices.

In addition to the information found in the request, each rule also has a priority, which allows us to override updates for specific things while letting the rest "fall down" to a more general rule. For example, we block unsupported versions of OS' with a rule of the highest priority but continue to serve updates for users of other OS' with a rule of slightly lower priority. The ability to override and fallback by setting the priority is one of the reasons Balrog is so flexible, and a key way that it distinguishes itself from its predecessor.

=== What's in a rule? ===
Each rule has quite a lot of columns, but they all fall into one of the buckets below:
* '''Matchable''' - these correspond to information provided in the update request, and are used to filter out rules that don't apply to the request.
* '''Decision''' - these are also used to filter rules, but do not correspond to information in the request.
* '''Response''' - these contain information that ends up in the response
* '''Info''' - informational columns, not used as part of serving updates

Individual columns are detailed in the table below:
{| class="wikitable sortable"
|-
! Attribute
! Category
! Description
! Matching logic
! Examples
|- style="background-color: palegreen"
| '''Product'''
| Matchable
| The name of the application requesting an update.
| Exact string match only
| "Firefox" or "B2G"
|- style="background-color: palegreen"
| '''Version'''
| Matchable
| The version of the application requesting an update.
| Exact string match or operator plus version to compare the incoming one against
| "36.0" or ">=38.0a1"
|- style="background-color: palegreen"
| '''Channel'''
| Matchable
| The update channel of the application request an update.
| Exact string match or a string with "*" character to glob
| "nightly" or "beta*"
|- style="background-color: palegreen"
| '''buildTarget'''
| Matchable
| The "build target" of the application requesting an update. This is usually related to the target platform the app was built for.
| Exact string match only
| "Darwin_x86_64-gcc3-u-i386-x86_64" or "flame-kk-userdebug"
|- style="background-color: palegreen"
| '''buildID'''
| Matchable
| The build ID of the application requesting an update.
| Exact string match or operator plus buildid to compare the incoming one against
| "201410010830" or "<201512010830"
|- style="background-color: palegreen"
| '''Locale'''
| Matchable
| The locale of the application requesting an update.
| Exact string match or comma separated list of locales to do an exact match on
| "de" or "en-US,en-GB,id"
|- style="background-color: palegreen"
| '''osVersion'''
| Matchable
| The OS Version of the application requesting an update. This field is primarily used to point desupported operating systems to their last supported build.
| Partial string match or comma separated list of partial strings to match on
| "Windows_NT 5.0" or "Darwin 6,Darwin 7,Darwin 8"
|- style="background-color: palegreen"
| '''systemRequirements'''
| Matchable
| The supported hardware features of the application requesting an update. This field is primarily used to point desupported users based on their hardware. Eg: users who do not support SSE2
| Full string match or comma separated list of full strings to match on
| "SSE" or "MMX,SSE"
|- style="background-color: palegreen"
| '''distribution'''
| Matchable
| The partner distribution name of the application requesting an update or "default" if the application is not a partner build.
| Exact string match only
| "default" or "yahoo"
|- style="background-color: palegreen"
| '''distVersion'''
| Matchable
| The version of the partner distribution of the application requesting an update or "default" if the application is not a partner build.
| Exact string match only
| "default" or "1.19"
|- style="background-color: palegreen"
| '''headerArchitecture'''
| Matchable
| The architecture of the OS of the client as guessed based on build target. This field is mostly deprecated now that this information is included in the build target.
| Exact string match only
| "PPC" and "Intel" are the only possible values
|- style="background-color: palegreen"
| '''Whitelist'''
| Matchable
| A pointer to a Whitelist blob (stored in the Releases table) that can determine whether an update request is authorized to have the Release this rule is mapped to. Commonly used in whitelisting IMEIs for FirefoxOS updates.
| If a whitelist is present, its shouldServeUpdate is called. If it returns True, this rule is considered to be matching. If it returns False, this rule is thrown out.
| Any valid release name, or NULL
|- style="background-color: lightskyblue"
| '''Priority'''
| Decision
| The priority of the rule, relative to other rules. If multiple rules match an incoming request based on the Matchable columns, the rule with the highest priority is chosen.
| N/A
| Any number, by convention positive integers.
|- style="background-color: lightskyblue"
| '''backgroundRate'''
| Decision
| The percentage of background update requests that should receive an update if they match this rule. Generally this is used as a throttle to increase or decrease the rate at which the majority of users receive an update.
| N/A
| Any number 0 to 10
|- style="background-color: crimson"
|- style="background-color: orange"
| '''Mapping'''
| Response
| The Release to construct an update out of. This is a foreign key to the "name" column of the Releases table.
| N/A
| Any valid release name, or NULL.
|- style="background-color: orange"
| '''update_type'''
| Response
| The update_type to use in the XML response. It's very rare for a rule to use anything other than "minor" these days.
| N/A
| "minor" or "major"
|- style="background-color: crimson"
| '''id'''
| Info
| The id of the rule. This id is necessary to make changes to the rule through the REST API.
| N/A
| Autoincrementing integer
|- style="background-color: crimson"
| '''Alias'''
| Info
| A unique alias for the rule. Can be used in place of id in any REST operation that doesn't involve rule history.
| N/A
| "firefox-release-betatest", "firefox-nightly"
|- style="background-color: crimson"
| '''Comment'''
| Info
| A string describing the purpose of the rule. Not always necessary for obvious rules.
| N/A
| Any string
|}

=== How are requests matched up to rules? ===
The incoming request parts match up directly to incoming URL parts. For example, most update requests will send an URL in the following format:
/update/3/<product>/<version>/<buildID>/<buildTarget>/<locale>/<channel>/<osVersion>/<distribution>/<distVersion>/update.xml?force=1 # force can also be left off

The following logic is used to figure out which rule an update matches and what to respond with:
# If a rule specifies one of these fields and a request's field doesn't match it, the rule is considered not to be a match and the rule is ignored for that request. See above for details on how specific columns perform matching.
# If "force" wasn't specified, the backgroundRate of the selected rule is looked at.
# If we still choose serve an update after accounting for backgroundRate we look at the rule's mapping. This is a foreign key that points at an entry in the releases table. That row has most of the information we need to construct the update.
# Using the update_type and release that the mapping points to, construct and return an XML response with the details of the update for the client.

== Releases ==
'''needs fleshing out'''<br/>
To Balrog, a "release" is data about a related set of builds. This does _not_ match up with the concept of a "release" being on the "beta", "release" or "esr" channel elsewhere. In Balrog, each set of nightlies on any branch is considered a release.

While there's no enforced format on release names, there are a few conventions that we use:
* Nightly-style builds submit to releases named by product and branch. Each nightly generally submits to two different releases, one "dated" (eg: Firefox-mozilla-central-nightly-20150513010203) and one "latest" (eg: Firefox-mozilla-central-nightly-latest).
* Release-style builds submit to releases named by product, version number, and build number, eg: Firefox-38.0-build1
* GMP blobs are created by hand and generally named with the version of each plugin they contain in the name, eg: GMP-20150423-CDM-v4-OpenH264-v1.4

More details on blobs can be found on [[Balrog/Blobs | the Blobs page]].

== Permissions ==
The permissions table is a simple list of usernames and the ACLs that they have. A user could be an "admin", giving them write access to everything, or could have one or more specific permissions. For example, our "ffxbld" system account has access to make create or modify "Firefox" or "Fennec" releases. These specific ACLs let us do things such as give Go Faster folks access to Balrog without the risk of them or their tools accidentally messing up Firefox updates (or visa versa).

The table below describe all possible permissions:
{| class="wikitable sortable"
|-
! Object
! Action
! Options
! Comments
|-
| admin
| No supported actions
| No supported options
| An admin user has completely unrestricted access to Balrog.
|-
| rowspan="3" | rule
| create
| rowspan="9" | products - If specified, the user only has permission for the object and action if the changes they are making only affect the product specified.
| rowspan="12" |
|-
| modify
|-
| delete
|-
| rowspan="3" | release
| create
|-
| modify
|-
| delete
|-
| rowspan="2" | release_read_only
| set
|-
| unset
|-
| release_locale
| modify
|-
| rowspan="3" | permission
| create
| rowspan="3" | No supported options.
|-
| modify
|-
| delete
|}

== History Tables ==
Change attribution and recording is embedded deeply into Balrog. The rules, releases, and permissions tables all have a corresponding history table that records the time a change was made and who made it. This allows us to look back in time when debugging issues, attribute changes to people (aka blame), and quickly roll back bad changes.

= Admin App =
== API ==
See the [[Balrog/Admin API | Admin API page]].
== UI Use Cases ==
=== Locking/Unlocking Nightlies ===
One of the most common uses for the Balrog UI is to lock a nightly update channel to a specific release for a period of time, and then unlock it later (so that users on that channel start receiving the latest available build again). This is often done if a serious bug is introduced to minimize the number of users affected by it.

Taking the B2G nightly channel as an example, let's see how we would lock it to the nightlies from 20150505160203:
# Log in to https://aus4-admin.mozilla.org
# Click on the "Rules" link at the top of the page
# Use the filter in the top right to narrow down the rules to "product:B2G channel:nightly"
# Locate the rule (or rules) on the "nightly" channel
#* Changing the sort to "Product, Channel" will group things together better.
# For each rule on the channel:
#* Click the "Update" button to enter edit mode
#* Find the mapping field and replace the "-latest" part with "-20150505160203" (the UI will autocomplete this for you if you start typing).
#* Scroll down and click "Save Changes"

When you're ready to unlock the updates, follow the same steps as above but replace the "-20150505160203" part of the mapping with "-latest" again.

=== Adding a rule for a new update channel ===
When nightly builds are set up on a new branch, rules need to be added to Balrog for updates to be served. Note that the nightly build automation is responsible for providing metadata about each new set of builds to Balrog.

As an example, here is how B2G updates could be set up on a hypothetical "mozilla-b2g40" branch:
# Log in to https://aus4-admin.mozilla.org
# Click on the "Rules" link at the top of the page
# Click on "Add a new Rule" near the top left of the page
# Fill out the form as follows:
#* Product: B2G
#* Channel: nightly-b2g40
#* Mapping: B2G-mozilla-b2g40-nightly-latest
#* Rate: 100
#* Priority: 90
# Click "Save Changes"

=== Modify an existing release ===
Most modifications to releases are done by automation, but sometimes we need to tweak them by hand. For example, when adjusting What's New page configuration.

For example, if you wanted to modify the "Firefox-38.0-build3" release, follow these steps:
# Log in to https://aus4-admin.mozilla.org/
# Click the "Releases" link at the top of the page
# Find the "Firefox-38.0-build3" release and click the "Download" link
# Save the file locally and modify it to your liking
# Click the "Update" link for "Firefox-38.0-build3"
# Click "Browse" and choose your new local version
# Click "Save Changes"

= Code Overview =
Balrog's code is organized into roughly the following parts:
* [https://github.com/mozilla/balrog/tree/master/auslib/blobs The blobs] - These contain most of the brains (business logic) behind Balrog. They know how to validate new data coming into the system and translate existing data into useful responses to update requests.
* [https://github.com/mozilla/balrog/blob/master/auslib/db.py The database abstraction layer] - This layer sits between the actual database and the applications. It defines the database schema, performs permissions checking, and ensures all changes are written to history tables. Application should never touch the database directly - they should always go through this layer.
* [https://github.com/mozilla/balrog/tree/master/auslib/web The user-facing application] - The entry point to requests from applications looking for updates.
* [https://github.com/mozilla/balrog/tree/master/auslib/admin The admin API] - A simple RESTful API that allows the Admin UI and automation to make changes to Balrog's database.
* [https://github.com/mozilla/balrog-ui The admin UI] A human friendly interface to manage updates.

= Hacking =
Balrog's code is split between [https://github.com/mozilla/balrog the backend server] and the [https://github.com/mozilla/balrog-ui admin ui]. Both Github repositories use Travis for continuous integration and accept pull requests. To get both repositories do the following:
git clone https://github.com/mozilla/balrog
cd balrog
git submodule init
git submodule update

== Docker ==
You can run a complete Balrog environment locally with Docker, configured very similarly to production. You'll need Docker and Docker Compose installed; on Mac or Windows we require Docker for Mac/Docker for Windows v1.12 or higher. Then run the following in the root of the Balrog repo:
docker-compose up

Once it completes, you should be able to access
* the admin interface at [http://127.0.0.1:8080 http://127.0.0.1:8080]
* the public interface on port 9090, for example [http://127.0.0.1:9090/update/3/Firefox/33.0/20141202185629/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/default/default/default/update.xml?force=1 http://127.0.0.1:9090/update/3/Firefox/33.0/20141202185629/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/default/default/default/update.xml?force=1]

== UI hacking ==

If you want to hack on the UI, you'll also need to install some npm modules:
cd ui
npm install
npm install -g lineman
cd ..

To make code changes visible to docker you need to run:
cd ui
lineman build

lineman build minifies the js & css, which can make debugging more difficult, but you can use the ''run'' mode of lineman instead. First make sure docker-compose is up, then
cd ui
lineman run
The UI will be available at [http://127.0.0.1:8000 http://127.0.0.1:8000]. lineman will watch for changes to the source so you need only refresh the browser after saving your work.

== Unit Tests ==
=== Backend ===
The recommended way to run backend tests is through Docker. Doing so will ensure your tests will run the same way they do in CI. To run them, use this command in the root of your Balrog repository:
./run-tests.sh

If you can't or don't want to run tests within Docker for some reason, you can run them with directly with:
tox

Tests run fine on any posix-like environment, but are only run regularly within the Docker image, so it's possible to have failures that aren't related to Balrog code or your changes when running directly with tox.

=== Frontend ===
To run the unit tests use:
cd ui
lineman spec-ci

More options can be found in [https://github.com/mozilla/balrog-ui/blob/master/README.md ui/README.md]

= Getting Involved =
If you like to get involved in the development of Balrog there's lots of areas where we could use some help. Below is a list of specific tasks that would be good first choices. Come talk to us in [irc://irc.mozilla.org/#balrog #balrog] if you're interested!

<bugzilla>
{
"quicksearch": "status:new,assigned,reopened,unconfirmed comp:balrog mentor:'@' assignee:nobody@mozilla.org whiteboard:[good first bug]",
"product": "Release Engineering",
"include_fields": "id,component,summary,status"
}
</bugzilla>

= Meeting Notes =
See the [[Balrog/Meetings | Meetings page]].

= Running without Docker Compose =
== Creating a database ==
Balrog's database is controlled through sqlalchemy-migrate. To initialize a new Balrog database, run the following:
docker run --entrypoint python mozilla/balrog /app/scripts/manage-db.py -d DBURI create

Similarly, to upgrade the schema of an existing Balrog database, run the following:
docker run --entrypoint python mozilla/balrog /app/scripts/manage-db.py -d DBURI upgrade

See the [[#Environment Variables | "Environment Variables" section below]] for DBURI format. If your testing out local changes that affect database creation or upgrades, you should replace "mozilla/balrog" with your local image.

== Environment Variables ==
The following environment variables are required by the Balrog WSGI apps:
* DBURI - The database to use, in the format: driver://user:password@host/database.
* SECRET_KEY - A pseudorandom string to use when generating CSRF tokens. Only used for the admin app.

These are optional:
* LOG_LEVEL - Controls the python level the app logs at. Set to INFO by default.
* LOG_FORMAT - Controls the log format. If unset, mozlog format (json) will be used. Can be overridden with "plain" to log simple plain-text messages. The former is recommended for production, the latter for local development.
* NOTIFY_TO_ADDR - An address to send e-mail to if a Rule or Permission changes. Unset by default, and only used for the admin app. If set, the following additional variables are required:
** SMTP_HOST, SMTP_PORT, SMTP_USERNAME, SMTP_PASSWORD - Information about the SMTP relay to send mail through.
** NOTIFY_FROM_ADDR - The "from" address to use when sending mail.

= Infrastructure =
== Support & Escalation ==
''If the issue may be visible to users, please make sure [irc://irc.mozilla.org/#moc #moc] is also notified. They can also assist with the notifications below.''

RelEng is the first point of contact for issues. To contact them, follow [[ReleaseEngineering#Contacting_Release_Engineering|the standard RelEng escalation path]].

If RelEng is unable to correct the issue, they may [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations escalate to CloudOps].

== Monitoring & Metrics ==
TODO: add more, flesh out.

* [https://app.datadoghq.com/dash/156924/balrog-web-aus5mozillaorg?live=true&page=0&is_auto=false&tile_size=m&fullscreen=false Datadog Dashboard]

== Backups ==
Balrog uses the built-in RDS backups. The database in snapshotted nightly, and incremental backups are done throughout the day. If necessary, we have the ability to recover to within a 5 minute window. Database restoration is done by CloudOps, and they should be contacted immediately if needed.

== Deploying Changes ==
Balrog's [https://mana.mozilla.org/wiki/display/SVCOPS/Firefox+Update+Service stage and production infrastructure] is managed by [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations the Cloud Operations team]. This section describes how to go from a reviewed patch to deploying it in production.

=== Is now a good time? ===
Before you deploy, consider whether or not it's an appropriate time to. Some factors to consider:
* Are we in the middle of an important release such as a chemspill? If so, it's probably not a good time to deploy.
* How risky are your changes? If they're high risk, deploying on a Friday is probably a bad idea.
* Do you need to migrate any data? If you do, make sure you have time to do so right after deploying.
* Do you have enough time to safely do a push? Most pushes take at most 60 minutes to complete after the stage push has been done. This time is mostly affected by how long it takes you to verify your changes in stage and production.

=== Landing ===
==== UI Changes ====
If you've made a change to the [https://github.com/mozilla/balrog-ui ui repository], make sure you run "lineman build" and commit the result before pushing that change. This will rebuild the UI application and put the result into "dist", which is where the deployed instances run out of. You also need to commit the subrepository change to the [https://github.com/mozilla/balrog/ Balrog repository] and push that back. Generally, the workflow for this is something like:
<pre>
cd /your/balrog/repo
cd ui
lineman build
git commit -a -m "Rebuild UI."
git push origin
cd ..
git commit -a -m "Update to latest UI."
git push origin
</pre>

==== Backend Changes ====
Just push your change to the master branch of the [https://github.com/mozilla/balrog/ Balrog repository].

==== Schema Upgrades ====
If you need to do a schema change you '''must''' ensure that either the current production code can run with your schema change applied, or that your new code can run with the old schema. Code and schema changes '''cannot''' be done at the same instant, so you must be able to support one of these scenarios. Generally, additive changes (column or table additions) should do the schema change first, while destructive changes (column or table deletions) should do the schema change second. You can simulate the upgrade with your local Docker containers to verify which is right for you.

When you file the deployment bug (see below), include a note about the schema change in it. Something like:
This push requires a schema change that needs to be done _prior_ to the new code going out. Thas can be performed by running the Docker image with the "upgrade-db" command, with DBURI set.

{{bug|1295678}} is an example of a push with a schema change.

=== Testing ===
Before asking for a push, you should do some functional testing on your local machine with the [[#Docker | Docker]] images. You should do this against [https://github.com/mozilla/balrog the master branch of the upstream repository] to ensure you're testing the exact code that is to be deployed. At the very least, you should do explicit testing of all the new code that would be included in the push. Eg: if you're changing the format of a blob, make sure that you can add a new blob of that type, and that the XML response looks correct.

=== Pushing to stage and production ===
Pushing live is a two step process. First, you must push to the stage environment and ensure things are working there. Then, you can push live.
# Bump the [https://github.com/mozilla/balrog/blob/master/version.txt in-repo version].
# [https://github.com/mozilla/balrog/releases Tag the repository] with a "vX.Y" tag. Eg: "git tag -s vX.Y"
# Wait for CI jobs to complete. Unit tests must pass and a [https://hub.docker.com/r/mozilla/balrog/tags/ new Docker Image must be pushed to Dockerhub] before you proceed.
# [https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=nobody%40mozilla.org&bug_file_loc=http%3A%2F%2F&bug_ignored=0&bug_severity=normal&bug_status=NEW&cc=bwong%40mozilla.com&cc=ckolos%40mozilla.com&cc=bhearsum%40mozilla.com&cf_blocking_b2g=---&cf_blocking_fennec=---&cf_fx_iteration=---&cf_fx_points=---&cf_status_b2g_2_6=---&cf_status_firefox48=---&cf_status_firefox49=---&cf_status_firefox50=---&cf_status_firefox51=---&cf_status_firefox_esr45=---&cf_tracking_b2g=---&cf_tracking_e10s=---&cf_tracking_firefox48=---&cf_tracking_firefox49=---&cf_tracking_firefox50=---&cf_tracking_firefox51=---&cf_tracking_firefox_esr45=---&cf_tracking_firefox_relnote=---&comment=Balrog%20version%20X.Y%20is%20ready%20to%20be%20pushed%20to%20stage.%20Please%20deploy%20the%20new%20Docker%20image%20%28TAG_NAME%29.%0D%0A%0D%0AWe%27d%20like%20the%20production%20push%20for%20this%20to%20happen%20sometime%20between%2011am%20and%201pm%20pacific%20on%20%28DATE%20GOES%20HERE%29&component=Operations%3A%20Deployment%20Requests&contenttypemethod=autodetect&contenttypeselection=text%2Fplain&defined_groups=1&flag_type-37=X&flag_type-4=X&flag_type-5=X&flag_type-607=X&flag_type-708=X&flag_type-720=X&flag_type-721=X&flag_type-737=X&flag_type-781=X&flag_type-787=X&flag_type-800=X&flag_type-803=X&flag_type-846=X&flag_type-864=X&flag_type-875=X&flag_type-892=X&flag_type-901=X&flag_type-905=X&flag_type-908=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=--&product=Cloud%20Services&rep_platform=Unspecified&short_desc=please%20deploy%20balrog%20X.Y%20to%20stage%2Fprod&target_milestone=---&version=unspecified File a bug] to have the new version pushed to stage. Be sure to include the new version number, and Docker image tag you want deployed.
# Once stage has been updated, verify your changes again. Even though you've tested locally, it's important to retest in stage to make sure there's no deployment-specific issues.
# When stage looks good, you're ready to comment in the bug to ask for production to be updated.
# Reverify in production. When production has been updated, verify your changes again there. If you need to tweak rules or releases to do so, be careful not to touch any live channels (create new rules or releases if necessary). This final verification is as more about making sure the right thing got deployed than whether or not your code is correct.

= Scripts =
These tools all live in [https://hg.mozilla.org/build/tools/file/default/scripts/updates tools/scripts/updates]. We use them to programmatically adjust the update server.

== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-submitter.py balrog-submitter.py] ==

Used to submit nightly and release style builds into release blobs. Called on build slaves, once per each combination of platform-locale.

== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-release-pusher.py balrog-release-pusher.py] ==
Used by the release automation to add the metadata to a release blob, and to push the new release onto the test channels. Example builder name - release-mozilla-beta-firefox_updates.

== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-release-shipper.py balrog-release-shipper.py] ==
Used by the release automation to push a release to the production chanenl. Example builder name - release-mozilla-beta-update_shipping.

== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-tweaker.py balrog-tweaker.py] ==
We don't have UI support for modifying blobs, so this is a helper to submit a blob fragment to replace content on the server. Doesn't handle removing keys.

Example:
<pre>python scripts/updates/balrog-tweaker.py --json json -b 'Firefox-33.0-build1' --api-root 'https://aus4-admin.mozilla.org' --credentials-file cred -u 'nthomas@mozilla.com' -v</pre>
where json is a file containing the fragment of json to submit, cred is a file containing the password for the -u argument, with format
<pre>balrog_credentials = {
'username': 'password'
}</pre>

== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-nightly-locker.py balrog-nightly-locker.py] ==
Used to 'freeze' nightly updates by pointing to a dated release blob instead of the latest, eg for big code landings, merges. Also to unfreeze afterwards. More details on usage at [[ReleaseEngineering/How_To/Enable_or_Disable_Updates_on_Aurora | Enable/disable updates on Aurora]]

== Common code ==
[https://hg.mozilla.org/build/tools/file/default/lib/python/balrog/submitter/cli.py cli.py] and [https://hg.mozilla.org/build/tools/file/default/lib/python/balrog/submitter/api.py api.py] in tools/lib/python/balrog/submitter/ provide the shared code.

Balrog

2016-09-08T22:12:54Z

NThomas: Rework hacking section

= Overview =
Balrog is the software that runs the server side component of the update system used by Firefox and other Mozilla products. It is the successor to [https://wiki.mozilla.org/AUS AUS (Application Update Service)], which did not scale to our current needs nor allow us to adapt to more recent business requirements. Balrog helps us ship updates faster and with much more flexability than we've had in the past.

Bugs and feature requests should be filed in the [https://bugzilla.mozilla.org/enter_bug.cgi?component=Balrog%3A%20Backend&product=Release%20Engineering Backend] or [https://bugzilla.mozilla.org/enter_bug.cgi?component=Balrog%3A%20Frontend&product=Release%20Engineering Frontend] components of Bugzilla. You can find the Balrog developers in [irc://irc.mozilla.org/#balrog #balrog].

= Domains =
Balrog admin interface is accessible at [https://aus4-admin.mozilla.org/ aus4-admin.mozilla.org] (vpn required).

The public interface that Firefox and other applications talk to is at aus5.mozilla.org. Many older update domains are also served by Balrog, including aus4.mozilla.org, aus3.mozilla.org, and aus2.mozilla.org. More details on these can be found on the [[Balrog/Client_Domains | Client Domains page]].

= Database Model =
Balrog's model centres around two concepts: Rules and Releases. When a request for update from an application is received it is matched up against the rules. Once the correct rule has been found, it contains a pointer to a Release, which contains all of the metadata needed to construct a proper update response. Rules and Releases are described in greater detail below:

== Rules ==
The most important part of Balrog to understand is its rules. When a request comes in it is matched against each of Balrog's rule to find the one that best suits it (more in this below). Once found, Balrog looks at that rule's "mapping", which points at a release that has the required information to serve an update back to the client. Without any rules, Balrog will never serve an update. With badly configured rules Balrog could do bad things like serve Firefox updates to B2G devices.

In addition to the information found in the request, each rule also has a priority, which allows us to override updates for specific things while letting the rest "fall down" to a more general rule. For example, we block unsupported versions of OS' with a rule of the highest priority but continue to serve updates for users of other OS' with a rule of slightly lower priority. The ability to override and fallback by setting the priority is one of the reasons Balrog is so flexible, and a key way that it distinguishes itself from its predecessor.

=== What's in a rule? ===
Each rule has quite a lot of columns, but they all fall into one of the buckets below:
* '''Matchable''' - these correspond to information provided in the update request, and are used to filter out rules that don't apply to the request.
* '''Decision''' - these are also used to filter rules, but do not correspond to information in the request.
* '''Response''' - these contain information that ends up in the response
* '''Info''' - informational columns, not used as part of serving updates

Individual columns are detailed in the table below:
{| class="wikitable sortable"
|-
! Attribute
! Category
! Description
! Matching logic
! Examples
|- style="background-color: palegreen"
| '''Product'''
| Matchable
| The name of the application requesting an update.
| Exact string match only
| "Firefox" or "B2G"
|- style="background-color: palegreen"
| '''Version'''
| Matchable
| The version of the application requesting an update.
| Exact string match or operator plus version to compare the incoming one against
| "36.0" or ">=38.0a1"
|- style="background-color: palegreen"
| '''Channel'''
| Matchable
| The update channel of the application request an update.
| Exact string match or a string with "*" character to glob
| "nightly" or "beta*"
|- style="background-color: palegreen"
| '''buildTarget'''
| Matchable
| The "build target" of the application requesting an update. This is usually related to the target platform the app was built for.
| Exact string match only
| "Darwin_x86_64-gcc3-u-i386-x86_64" or "flame-kk-userdebug"
|- style="background-color: palegreen"
| '''buildID'''
| Matchable
| The build ID of the application requesting an update.
| Exact string match or operator plus buildid to compare the incoming one against
| "201410010830" or "<201512010830"
|- style="background-color: palegreen"
| '''Locale'''
| Matchable
| The locale of the application requesting an update.
| Exact string match or comma separated list of locales to do an exact match on
| "de" or "en-US,en-GB,id"
|- style="background-color: palegreen"
| '''osVersion'''
| Matchable
| The OS Version of the application requesting an update. This field is primarily used to point desupported operating systems to their last supported build.
| Partial string match or comma separated list of partial strings to match on
| "Windows_NT 5.0" or "Darwin 6,Darwin 7,Darwin 8"
|- style="background-color: palegreen"
| '''systemRequirements'''
| Matchable
| The supported hardware features of the application requesting an update. This field is primarily used to point desupported users based on their hardware. Eg: users who do not support SSE2
| Full string match or comma separated list of full strings to match on
| "SSE" or "MMX,SSE"
|- style="background-color: palegreen"
| '''distribution'''
| Matchable
| The partner distribution name of the application requesting an update or "default" if the application is not a partner build.
| Exact string match only
| "default" or "yahoo"
|- style="background-color: palegreen"
| '''distVersion'''
| Matchable
| The version of the partner distribution of the application requesting an update or "default" if the application is not a partner build.
| Exact string match only
| "default" or "1.19"
|- style="background-color: palegreen"
| '''headerArchitecture'''
| Matchable
| The architecture of the OS of the client as guessed based on build target. This field is mostly deprecated now that this information is included in the build target.
| Exact string match only
| "PPC" and "Intel" are the only possible values
|- style="background-color: palegreen"
| '''Whitelist'''
| Matchable
| A pointer to a Whitelist blob (stored in the Releases table) that can determine whether an update request is authorized to have the Release this rule is mapped to. Commonly used in whitelisting IMEIs for FirefoxOS updates.
| If a whitelist is present, its shouldServeUpdate is called. If it returns True, this rule is considered to be matching. If it returns False, this rule is thrown out.
| Any valid release name, or NULL
|- style="background-color: lightskyblue"
| '''Priority'''
| Decision
| The priority of the rule, relative to other rules. If multiple rules match an incoming request based on the Matchable columns, the rule with the highest priority is chosen.
| N/A
| Any number, by convention positive integers.
|- style="background-color: lightskyblue"
| '''backgroundRate'''
| Decision
| The percentage of background update requests that should receive an update if they match this rule. Generally this is used as a throttle to increase or decrease the rate at which the majority of users receive an update.
| N/A
| Any number 0 to 10
|- style="background-color: crimson"
|- style="background-color: orange"
| '''Mapping'''
| Response
| The Release to construct an update out of. This is a foreign key to the "name" column of the Releases table.
| N/A
| Any valid release name, or NULL.
|- style="background-color: orange"
| '''update_type'''
| Response
| The update_type to use in the XML response. It's very rare for a rule to use anything other than "minor" these days.
| N/A
| "minor" or "major"
|- style="background-color: crimson"
| '''id'''
| Info
| The id of the rule. This id is necessary to make changes to the rule through the REST API.
| N/A
| Autoincrementing integer
|- style="background-color: crimson"
| '''Alias'''
| Info
| A unique alias for the rule. Can be used in place of id in any REST operation that doesn't involve rule history.
| N/A
| "firefox-release-betatest", "firefox-nightly"
|- style="background-color: crimson"
| '''Comment'''
| Info
| A string describing the purpose of the rule. Not always necessary for obvious rules.
| N/A
| Any string
|}

=== How are requests matched up to rules? ===
The incoming request parts match up directly to incoming URL parts. For example, most update requests will send an URL in the following format:
/update/3/<product>/<version>/<buildID>/<buildTarget>/<locale>/<channel>/<osVersion>/<distribution>/<distVersion>/update.xml?force=1 # force can also be left off

The following logic is used to figure out which rule an update matches and what to respond with:
# If a rule specifies one of these fields and a request's field doesn't match it, the rule is considered not to be a match and the rule is ignored for that request. See above for details on how specific columns perform matching.
# If "force" wasn't specified, the backgroundRate of the selected rule is looked at.
# If we still choose serve an update after accounting for backgroundRate we look at the rule's mapping. This is a foreign key that points at an entry in the releases table. That row has most of the information we need to construct the update.
# Using the update_type and release that the mapping points to, construct and return an XML response with the details of the update for the client.

== Releases ==
'''needs fleshing out'''<br/>
To Balrog, a "release" is data about a related set of builds. This does _not_ match up with the concept of a "release" being on the "beta", "release" or "esr" channel elsewhere. In Balrog, each set of nightlies on any branch is considered a release.

While there's no enforced format on release names, there are a few conventions that we use:
* Nightly-style builds submit to releases named by product and branch. Each nightly generally submits to two different releases, one "dated" (eg: Firefox-mozilla-central-nightly-20150513010203) and one "latest" (eg: Firefox-mozilla-central-nightly-latest).
* Release-style builds submit to releases named by product, version number, and build number, eg: Firefox-38.0-build1
* GMP blobs are created by hand and generally named with the version of each plugin they contain in the name, eg: GMP-20150423-CDM-v4-OpenH264-v1.4

More details on blobs can be found on [[Balrog/Blobs | the Blobs page]].

== Permissions ==
The permissions table is a simple list of usernames and the ACLs that they have. A user could be an "admin", giving them write access to everything, or could have one or more specific permissions. For example, our "ffxbld" system account has access to make create or modify "Firefox" or "Fennec" releases. These specific ACLs let us do things such as give Go Faster folks access to Balrog without the risk of them or their tools accidentally messing up Firefox updates (or visa versa).

The table below describe all possible permissions:
{| class="wikitable sortable"
|-
! Object
! Action
! Options
! Comments
|-
| admin
| No supported actions
| No supported options
| An admin user has completely unrestricted access to Balrog.
|-
| rowspan="3" | rule
| create
| rowspan="9" | products - If specified, the user only has permission for the object and action if the changes they are making only affect the product specified.
| rowspan="12" |
|-
| modify
|-
| delete
|-
| rowspan="3" | release
| create
|-
| modify
|-
| delete
|-
| rowspan="2" | release_read_only
| set
|-
| unset
|-
| release_locale
| modify
|-
| rowspan="3" | permission
| create
| rowspan="3" | No supported options.
|-
| modify
|-
| delete
|}

== History Tables ==
Change attribution and recording is embedded deeply into Balrog. The rules, releases, and permissions tables all have a corresponding history table that records the time a change was made and who made it. This allows us to look back in time when debugging issues, attribute changes to people (aka blame), and quickly roll back bad changes.

= Admin App =
== API ==
See the [[Balrog/Admin API | Admin API page]].
== UI Use Cases ==
=== Locking/Unlocking Nightlies ===
One of the most common uses for the Balrog UI is to lock a nightly update channel to a specific release for a period of time, and then unlock it later (so that users on that channel start receiving the latest available build again). This is often done if a serious bug is introduced to minimize the number of users affected by it.

Taking the B2G nightly channel as an example, let's see how we would lock it to the nightlies from 20150505160203:
# Log in to https://aus4-admin.mozilla.org
# Click on the "Rules" link at the top of the page
# Use the filter in the top right to narrow down the rules to "product:B2G channel:nightly"
# Locate the rule (or rules) on the "nightly" channel
#* Changing the sort to "Product, Channel" will group things together better.
# For each rule on the channel:
#* Click the "Update" button to enter edit mode
#* Find the mapping field and replace the "-latest" part with "-20150505160203" (the UI will autocomplete this for you if you start typing).
#* Scroll down and click "Save Changes"

When you're ready to unlock the updates, follow the same steps as above but replace the "-20150505160203" part of the mapping with "-latest" again.

=== Adding a rule for a new update channel ===
When nightly builds are set up on a new branch, rules need to be added to Balrog for updates to be served. Note that the nightly build automation is responsible for providing metadata about each new set of builds to Balrog.

As an example, here is how B2G updates could be set up on a hypothetical "mozilla-b2g40" branch:
# Log in to https://aus4-admin.mozilla.org
# Click on the "Rules" link at the top of the page
# Click on "Add a new Rule" near the top left of the page
# Fill out the form as follows:
#* Product: B2G
#* Channel: nightly-b2g40
#* Mapping: B2G-mozilla-b2g40-nightly-latest
#* Rate: 100
#* Priority: 90
# Click "Save Changes"

=== Modify an existing release ===
Most modifications to releases are done by automation, but sometimes we need to tweak them by hand. For example, when adjusting What's New page configuration.

For example, if you wanted to modify the "Firefox-38.0-build3" release, follow these steps:
# Log in to https://aus4-admin.mozilla.org/
# Click the "Releases" link at the top of the page
# Find the "Firefox-38.0-build3" release and click the "Download" link
# Save the file locally and modify it to your liking
# Click the "Update" link for "Firefox-38.0-build3"
# Click "Browse" and choose your new local version
# Click "Save Changes"

= Code Overview =
Balrog's code is organized into roughly the following parts:
* [https://github.com/mozilla/balrog/tree/master/auslib/blobs The blobs] - These contain most of the brains (business logic) behind Balrog. They know how to validate new data coming into the system and translate existing data into useful responses to update requests.
* [https://github.com/mozilla/balrog/blob/master/auslib/db.py The database abstraction layer] - This layer sits between the actual database and the applications. It defines the database schema, performs permissions checking, and ensures all changes are written to history tables. Application should never touch the database directly - they should always go through this layer.
* [https://github.com/mozilla/balrog/tree/master/auslib/web The user-facing application] - The entry point to requests from applications looking for updates.
* [https://github.com/mozilla/balrog/tree/master/auslib/admin The admin API] - A simple RESTful API that allows the Admin UI and automation to make changes to Balrog's database.
* [https://github.com/mozilla/balrog-ui The admin UI] A human friendly interface to manage updates.

= Hacking =
Balrog's code is split between [https://github.com/mozilla/balrog the backend server] and the [https://github.com/mozilla/balrog-ui admin ui]. Both Github repositories use Travis for continuous integration and accept pull requests. To get both repositories do the following:
git clone https://github.com/mozilla/balrog
cd balrog
git submodule init
git submodule update

== Docker ==
You can run a complete Balrog environment locally with Docker, configured very similarly to production. You'll need Docker and Docker Compose installed; on Mac or Windows we require Docker for Mac/Docker for Windows v1.12 or higher. Then run the following in the root of the Balrog repo:
docker-compose up

Once it completes, you should be able to access
* the admin interface at [http://127.0.0.1:8080 http://127.0.0.1:8080]
* the public interface on port 9090, for example [http://127.0.0.1:9090/update/3/Firefox/33.0/20141202185629/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/default/default/default/update.xml?force=1 http://127.0.0.1:9090/update/3/Firefox/33.0/20141202185629/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/default/default/default/update.xml?force=1]

== UI hacking ==

If you want to hack on the UI, you'll also need to install some npm modules:
cd ui
npm install
npm install -g lineman
cd ..

To make code changes visible to docker you need to run:
cd ui
lineman build

lineman build minifies the js & css, which can make debugging more difficult, but you can use the ''run'' mode of lineman instead. First make sure docker-compose is up, then
cd ui
lineman run
The UI will be available at [http://127.0.0.1:8000 http://127.0.0.1:8000]. lineman will watch for changes to the source so you need only refresh the browser after saving your work.

== Unit Tests ==
=== Backend ===
The recommended way to run backend tests is through Docker. Doing so will ensure your tests will run the same way they do in CI. To run them, use this command in the root of your Balrog repository:
./run-tests.sh

If you can't or don't want to run tests within Docker for some reason, you can run them with directly with:
tox

Tests run fine on any posix-like environment, but are only run regularly within the Docker image, so it's possible to have failures that aren't related to Balrog code or your changes when running directly with tox.

=== Frontend ===
To run the unit tests use:
cd ui
lineman spec-ci

More options can be found in [https://github.com/mozbhearsum/balrog-ui/blob/master/README.md ui/README.md]

= Getting Involved =
If you like to get involved in the development of Balrog there's lots of areas where we could use some help. Below is a list of specific tasks that would be good first choices. Come talk to us in [irc://irc.mozilla.org/#balrog #balrog] if you're interested!

<bugzilla>
{
"quicksearch": "status:new,assigned,reopened,unconfirmed comp:balrog mentor:'@' assignee:nobody@mozilla.org whiteboard:[good first bug]",
"product": "Release Engineering",
"include_fields": "id,component,summary,status"
}
</bugzilla>

= Meeting Notes =
See the [[Balrog/Meetings | Meetings page]].

= Running without Docker Compose =
== Creating a database ==
Balrog's database is controlled through sqlalchemy-migrate. To initialize a new Balrog database, run the following:
docker run --entrypoint python mozilla/balrog /app/scripts/manage-db.py -d DBURI create

Similarly, to upgrade the schema of an existing Balrog database, run the following:
docker run --entrypoint python mozilla/balrog /app/scripts/manage-db.py -d DBURI upgrade

See the [[#Environment Variables | "Environment Variables" section below]] for DBURI format. If your testing out local changes that affect database creation or upgrades, you should replace "mozilla/balrog" with your local image.

== Environment Variables ==
The following environment variables are required by the Balrog WSGI apps:
* DBURI - The database to use, in the format: driver://user:password@host/database.
* SECRET_KEY - A pseudorandom string to use when generating CSRF tokens. Only used for the admin app.

These are optional:
* LOG_LEVEL - Controls the python level the app logs at. Set to INFO by default.
* LOG_FORMAT - Controls the log format. If unset, mozlog format (json) will be used. Can be overridden with "plain" to log simple plain-text messages. The former is recommended for production, the latter for local development.
* NOTIFY_TO_ADDR - An address to send e-mail to if a Rule or Permission changes. Unset by default, and only used for the admin app. If set, the following additional variables are required:
** SMTP_HOST, SMTP_PORT, SMTP_USERNAME, SMTP_PASSWORD - Information about the SMTP relay to send mail through.
** NOTIFY_FROM_ADDR - The "from" address to use when sending mail.

= Infrastructure =
== Support & Escalation ==
''If the issue may be visible to users, please make sure [irc://irc.mozilla.org/#moc #moc] is also notified. They can also assist with the notifications below.''

RelEng is the first point of contact for issues. To contact them, follow [[ReleaseEngineering#Contacting_Release_Engineering|the standard RelEng escalation path]].

If RelEng is unable to correct the issue, they may [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations escalate to CloudOps].

== Monitoring & Metrics ==
TODO: add more, flesh out.

* [https://app.datadoghq.com/dash/156924/balrog-web-aus5mozillaorg?live=true&page=0&is_auto=false&tile_size=m&fullscreen=false Datadog Dashboard]

== Backups ==
Balrog uses the built-in RDS backups. The database in snapshotted nightly, and incremental backups are done throughout the day. If necessary, we have the ability to recover to within a 5 minute window. Database restoration is done by CloudOps, and they should be contacted immediately if needed.

== Deploying Changes ==
Balrog's [https://mana.mozilla.org/wiki/display/SVCOPS/Firefox+Update+Service stage and production infrastructure] is managed by [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations the Cloud Operations team]. This section describes how to go from a reviewed patch to deploying it in production.

=== Is now a good time? ===
Before you deploy, consider whether or not it's an appropriate time to. Some factors to consider:
* Are we in the middle of an important release such as a chemspill? If so, it's probably not a good time to deploy.
* How risky are your changes? If they're high risk, deploying on a Friday is probably a bad idea.
* Do you need to migrate any data? If you do, make sure you have time to do so right after deploying.
* Do you have enough time to safely do a push? Most pushes take at most 60 minutes to complete after the stage push has been done. This time is mostly affected by how long it takes you to verify your changes in stage and production.

=== Landing ===
==== UI Changes ====
If you've made a change to the [https://github.com/mozilla/balrog-ui ui repository], make sure you run "lineman build" and commit the result before pushing that change. This will rebuild the UI application and put the result into "dist", which is where the deployed instances run out of. You also need to commit the subrepository change to the [https://github.com/mozilla/balrog/ Balrog repository] and push that back. Generally, the workflow for this is something like:
<pre>
cd /your/balrog/repo
cd ui
lineman build
git commit -a -m "Rebuild UI."
git push origin
cd ..
git commit -a -m "Update to latest UI."
git push origin
</pre>

==== Backend Changes ====
Just push your change to the master branch of the [https://github.com/mozilla/balrog/ Balrog repository].

==== Schema Upgrades ====
If you need to do a schema change you '''must''' ensure that either the current production code can run with your schema change applied, or that your new code can run with the old schema. Code and schema changes '''cannot''' be done at the same instant, so you must be able to support one of these scenarios. Generally, additive changes (column or table additions) should do the schema change first, while destructive changes (column or table deletions) should do the schema change second. You can simulate the upgrade with your local Docker containers to verify which is right for you.

When you file the deployment bug (see below), include a note about the schema change in it. Something like:
This push requires a schema change that needs to be done _prior_ to the new code going out. Thas can be performed by running the Docker image with the "upgrade-db" command, with DBURI set.

{{bug|1295678}} is an example of a push with a schema change.

=== Testing ===
Before asking for a push, you should do some functional testing on your local machine with the [[#Docker | Docker]] images. You should do this against [https://github.com/mozilla/balrog the master branch of the upstream repository] to ensure you're testing the exact code that is to be deployed. At the very least, you should do explicit testing of all the new code that would be included in the push. Eg: if you're changing the format of a blob, make sure that you can add a new blob of that type, and that the XML response looks correct.

=== Pushing to stage and production ===
Pushing live is a two step process. First, you must push to the stage environment and ensure things are working there. Then, you can push live.
# Bump the [https://github.com/mozilla/balrog/blob/master/version.txt in-repo version].
# [https://github.com/mozilla/balrog/releases Tag the repository] with a "vX.Y" tag. Eg: "git tag -s vX.Y"
# Wait for CI jobs to complete. Unit tests must pass and a [https://hub.docker.com/r/mozilla/balrog/tags/ new Docker Image must be pushed to Dockerhub] before you proceed.
# [https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=nobody%40mozilla.org&bug_file_loc=http%3A%2F%2F&bug_ignored=0&bug_severity=normal&bug_status=NEW&cc=bwong%40mozilla.com&cc=ckolos%40mozilla.com&cc=bhearsum%40mozilla.com&cf_blocking_b2g=---&cf_blocking_fennec=---&cf_fx_iteration=---&cf_fx_points=---&cf_status_b2g_2_6=---&cf_status_firefox48=---&cf_status_firefox49=---&cf_status_firefox50=---&cf_status_firefox51=---&cf_status_firefox_esr45=---&cf_tracking_b2g=---&cf_tracking_e10s=---&cf_tracking_firefox48=---&cf_tracking_firefox49=---&cf_tracking_firefox50=---&cf_tracking_firefox51=---&cf_tracking_firefox_esr45=---&cf_tracking_firefox_relnote=---&comment=Balrog%20version%20X.Y%20is%20ready%20to%20be%20pushed%20to%20stage.%20Please%20deploy%20the%20new%20Docker%20image%20%28TAG_NAME%29.%0D%0A%0D%0AWe%27d%20like%20the%20production%20push%20for%20this%20to%20happen%20sometime%20between%2011am%20and%201pm%20pacific%20on%20%28DATE%20GOES%20HERE%29&component=Operations%3A%20Deployment%20Requests&contenttypemethod=autodetect&contenttypeselection=text%2Fplain&defined_groups=1&flag_type-37=X&flag_type-4=X&flag_type-5=X&flag_type-607=X&flag_type-708=X&flag_type-720=X&flag_type-721=X&flag_type-737=X&flag_type-781=X&flag_type-787=X&flag_type-800=X&flag_type-803=X&flag_type-846=X&flag_type-864=X&flag_type-875=X&flag_type-892=X&flag_type-901=X&flag_type-905=X&flag_type-908=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=--&product=Cloud%20Services&rep_platform=Unspecified&short_desc=please%20deploy%20balrog%20X.Y%20to%20stage%2Fprod&target_milestone=---&version=unspecified File a bug] to have the new version pushed to stage. Be sure to include the new version number, and Docker image tag you want deployed.
# Once stage has been updated, verify your changes again. Even though you've tested locally, it's important to retest in stage to make sure there's no deployment-specific issues.
# When stage looks good, you're ready to comment in the bug to ask for production to be updated.
# Reverify in production. When production has been updated, verify your changes again there. If you need to tweak rules or releases to do so, be careful not to touch any live channels (create new rules or releases if necessary). This final verification is as more about making sure the right thing got deployed than whether or not your code is correct.

= Scripts =
These tools all live in [https://hg.mozilla.org/build/tools/file/default/scripts/updates tools/scripts/updates]. We use them to programmatically adjust the update server.

== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-submitter.py balrog-submitter.py] ==

Used to submit nightly and release style builds into release blobs. Called on build slaves, once per each combination of platform-locale.

== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-release-pusher.py balrog-release-pusher.py] ==
Used by the release automation to add the metadata to a release blob, and to push the new release onto the test channels. Example builder name - release-mozilla-beta-firefox_updates.

== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-release-shipper.py balrog-release-shipper.py] ==
Used by the release automation to push a release to the production chanenl. Example builder name - release-mozilla-beta-update_shipping.

== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-tweaker.py balrog-tweaker.py] ==
We don't have UI support for modifying blobs, so this is a helper to submit a blob fragment to replace content on the server. Doesn't handle removing keys.

Example:
<pre>python scripts/updates/balrog-tweaker.py --json json -b 'Firefox-33.0-build1' --api-root 'https://aus4-admin.mozilla.org' --credentials-file cred -u 'nthomas@mozilla.com' -v</pre>
where json is a file containing the fragment of json to submit, cred is a file containing the password for the -u argument, with format
<pre>balrog_credentials = {
'username': 'password'
}</pre>

== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-nightly-locker.py balrog-nightly-locker.py] ==
Used to 'freeze' nightly updates by pointing to a dated release blob instead of the latest, eg for big code landings, merges. Also to unfreeze afterwards. More details on usage at [[ReleaseEngineering/How_To/Enable_or_Disable_Updates_on_Aurora | Enable/disable updates on Aurora]]

== Common code ==
[https://hg.mozilla.org/build/tools/file/default/lib/python/balrog/submitter/cli.py cli.py] and [https://hg.mozilla.org/build/tools/file/default/lib/python/balrog/submitter/api.py api.py] in tools/lib/python/balrog/submitter/ provide the shared code.

ReleaseEngineering/Applications/SlaveAPI

2016-09-04T22:53:39Z

NThomas: Update for repo move in bug 1277443

= What is it? =
SlaveAPI is a service that is intended to be a single point of entry for all actions related to slaves. Clients interact with it through a simple REST API and it communicates with many other systems on the backend to fulfill the requests (eg, bugzilla, slavealloc, inventory) made.

Its source code lives on [https://github.com/mozilla/build-slaveapi github].

= Where is it? =
SlaveAPI has both a [http://slaveapi-dev1.srv.releng.scl3.mozilla.com:8080 dev] and [https://secure.pub.build.mozilla.org/slaveapi production] instance. Each lives on its own machine within the RelEng network. Unless you're in the process of landing a patch to SlaveAPI, you should be using the production instance.

These machines are managed by PuppetAgain.

= Requirements/dependencies =
SlaveAPI depends on many Python packages to function correctly. Installing it through its setup.py will take care of pulling in everything you need.

= Deployment =
{{Release Engineering Deployment|slaveapi}}
Changes to SlaveAPI's config or credentials file should be done through Puppet. Both instances will pick up the changes and automatically reload the server.

Changes to SlaveAPI's code are a little trickier. SlaveAPI itself is deployed through pip by Puppet, so the first step is to [https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Data#.._add_a_data_file upload a new version of SlaveAPI to PuppetAgain] and update the [https://github.com/mozilla/build-puppet/blob/master/modules/slaveapi/manifests/instance.pp#L5 version in the Puppet manifests]. Once those are done the machine will automatically install the new version. However, the SlaveAPI service must be restarted to pick up the changes. You can do this by logging onto the machine and running the following:
sudo su - cltbld
cd /builds/slaveapi/dev (or prod)
source bin/activate
slaveapi-server.py stop slaveapi.ini && slaveapi-server.py start slaveapi.ini

The manual restart will be done automatically once {{bug|924616}} is fixed.

The full process looks like this:

# bump the package version like in https://github.com/mozilla/build-slaveapi/commit/60d25601e0783bbfb9d5959c4092693fdb026321
# note: package is versioned according to http://semver.org/
$ vim slaveapi/__init__.py
# get review, commit, push, then
$ python setup.py sdist
$ ls dist
slaveapi-1.0.19.tar.gz
$ scp slaveapi-1.0.19.tar.gz root@releng-puppet2.srv.releng.scl3.mozilla.com:
$ ssh root@releng-puppet2.srv.releng.scl3.mozilla.com
$ mv -i slaveapi-1.0.19.tar.gz /data/python/packages/
$ puppetmaster-fixperms
# you should see the new package at http://puppetagain.pub.build.mozilla.org/data/python/packages/
# update slaveapi version in [puppet]/modules/slaveapi/manifests/instance.pp
$ vim modules/slaveapi/manifests/instance.pp
# get review, commit, push
# merge to puppet 'production' branch and push
# need to restart the production daemon
$ ssh <username>@slaveapi1.build.mozilla.org or <username>slaveapi-dev1.srv.releng.scl3.mozilla.com
$ sudo su - cltbld
$ cd /builds/slaveapi/prod or dev
# make sure the local package version has been upgraded by puppet before restarting the daemon
$ source bin/activate
$ pip freeze | grep slaveapi
slaveapi==1.0.19
$ slaveapi-server.py stop slaveapi.ini && slaveapi-server.py start slaveapi.ini

= API Documentation =
The details of the API are documented alongside the code, and an HTML version [http://mozilla-slaveapi.readthedocs.org/en/latest/ is available at mozilla-slaveapi.readthedocs.org].

ReleaseEngineering/Applications/Ship It

2016-09-04T22:51:11Z

NThomas: Update for repo move in bug 1283640

= What is it? =
Ship it is a system that accepts requests for releases from humans and translates them into information and requests that our [https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Documentation Buildbot-based release automation] can process. It consists of a web application/api and a backend script ("release runner") to do much of the actual work.

Source code for the web application lives on [https://github.com/mozilla-releng/ship-it github]. Release runner is part of [https://hg.mozilla.org/build/tools/file/default/buildfarm/release/release-runner.py the build/tools repository]. Release promotion uses the default branch, while legacy automation (eg Thunderbird) uses the <tt>old-release-runner</tt> branch.

= Where is it? =
The web application is deployed on IT's generic web cluster. There are [https://ship-it-dev.allizom.org/ dev], [https://ship-it.allizom.org/ stage], and [https://ship-it.mozilla.org/ production] instances of it. More in-depth, IT-oriented, information is available on [https://mana.mozilla.org/wiki/display/websites/ship-it.mozilla.org mana].

Release runner is deployed through Puppet on buildbot-master85 for release promotion, and on buildbot-master81 for legacy automation. Supervisord manages the running process.

= Requirements/dependencies =
The web application depends on many Python packages. Most are included in a vendor library in the repository. Required binary packages are noted in [https://github.com/mozilla-releng/ship-it/blob/master/requirements/compiled.txt its requirements list].

Release runner's dependencies are described [https://hg.mozilla.org/build/puppet/file/default/modules/releaserunner/manifests/init.pp in its puppet manifests].

= Deployment =
== Web app ==
The dev instance of Ship It automatically pulls the tip of the master branch of the git repository - no human intervention is needed to deploy to it. Stage and production deployment are handled through Chief, which let us push explicit revisions at will:
* [http://genericadm.private.phx1.mozilla.com/chief/shipit.stage Stage's Chief]
* [http://genericadm.private.phx1.mozilla.com/chief/shipit.prod Production's Chief]

When deploying, you should enter your own name in the first field, the deployment password in the second, and the revision you want to push in the third. If you don't have the deployment passwords, ask someone for them.

== Release runner ==
Release runner's repository needs to be updated by hand to deploy a newly checked in version. Run the following on buildbot-master85 (release promotion) or buildbot-master81 (legacy) to do so:
cd /builds/releaserunner/tools
hg pull && hg up
Switch to root and run
supervisorctl restart releaserunner

= Using it =
Standard usage of Ship It is documented as part of the full [https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Starting_a_Release#Submit_to_Ship_It release automation documentation]. [https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Troubleshooting Troubleshooting information] is also available there.

ReleaseEngineering/Repositories/tooltool

2016-09-04T22:46:15Z

NThomas: git.m.o is gone, bug 1277443. So is the hg mirror

{{Repository
|Owner=Releng
|RoR=https://github.com/mozilla/build-tooltool
}}

ReleaseEngineering/Repositories/tools

2016-09-04T22:45:22Z

NThomas: git.m.o is gone, bug 1277443

{{Repository
|Owner=Releng
|RoR=https://hg.mozilla.org/build/tools
|Mirrors=http://github.com/mozilla/build-tools
}}

ReleaseEngineering/Repositories/slaveapi

2016-09-04T22:44:31Z

NThomas: git.m.o is gone, bug 1277443

{{Repository
|Owner=Releng
|RoR=https://github.com/mozilla/build-slaveapi
}}

ReleaseEngineering/Repositories/ship-it

2016-09-04T22:43:56Z

NThomas: Previous edit should have referenced bug 1283640

{{Repository
|Owner=Releng
|RoR=https://github.com/mozilla-releng/ship-it.git
}}

ReleaseEngineering/Repositories/ship-it

2016-09-04T22:42:34Z

NThomas: Repo moved, bug 1277443

{{Repository
|Owner=Releng
|RoR=https://github.com/mozilla-releng/ship-it.git
}}

ReleaseEngineering/Repositories/puppet

2016-09-04T22:40:35Z

NThomas: git.m.o is gone, bug 1277443

{{Repository
|RoR=https://hg.mozilla.org/build/puppet
|Owner=Releng
|Mirrors=https://github.com/mozilla/build-puppet
}}

ReleaseEngineering/Repositories/mapper

2016-09-04T22:39:50Z

NThomas: Repo moved, bug 1277443

{{Repository
|Owner=Releng
|RoR=https://github.com/mozilla/build-mapper
}}

ReleaseEngineering/Repositories/fwunit

2016-09-04T22:37:24Z

NThomas: Repo moved, bug 1277443

{{Repository
|Owner=Releng,Relops
|RoR=https://github.com/mozilla/build-fwunit
}}