CIDuty/Reconfigs

A reconfig (short for "reconfigure the buildbot masters") is how changes to buildbot configurations make it into production. Buildduty is responsible for running reconfigs.

In a nutshell, a reconfig consists of:

• moving the production tag for the buildbot-configs repository and the production-0.8 tag for the buildbotcustom to the current tip (or chosen revision)
• updating the source checkout of both repositories on all the buildbot masters
• updating the tools checkout on each master to the current tip
• executing a buildbot reconfig command on each buildbot master

We also still move the production tag for the mozharness repository until mozharness is moved fully in-tree.

Please see the instructions for how to land buildbot changes for more information.

It is polite to ask in #releng if anyone has further changes to land before starting the reconfig process.

How to reconfig

The current state of the art is to use the end_to_end_reconfig.sh script.

The end_to_end_reconfig.sh script uses the original fabric scripts as it's core, but also takes care of updating the wiki, updating bugs affected by the merge, and updating the tools checkouts on foopies as well. As such, the script has a few, indirect python module dependencies:

• fabric
• requests

However, the script will try to install the dependencies automatically if they are missing.

In order to update the wiki/bugzilla, the script also requires your credentials for those services in a config file:

# Needed if updating wiki - note the wiki does *not* use your LDAP credentials...
export WIKI_USERNAME=XXX
export WIKI_PASSWORD=XXXXXXXX

# Needed if updating Bugzilla bugs to mark them as in production - *no* Persona integration - must be a  native Bugzilla account.. .
# Details for the 'Release Engineering SlaveAPI Service' <slaveapi@mozilla.releng.tld> Bugzilla user can be found in the RelEng
# private repo, in file passwords/slaveapi-bugzilla.txt.gpg (needs decrypting with an approved gpg key).
export BUGZILLA_USERNAME='XXX@mozilla.com'
export BUGZILLA_PASSWORD='XXXXXXXXX'

# Used for slaveapi actions
export LDAP_USERNAME='XXX@mozilla.com'
export LDAP_PASSWORD='XXXXXXXX'

Again, the script will create a template config file (default is ~/.reconfig/config) if it does not exist, but you'll need to fill in your own credentials before it will work.

The script will also attempt to update IRC with reconfig status. To do so, it uses a minimal bash IRC client called ii. You can download ii from their website, or install it via a package manager (e.g. port install ii). Updating irc is non-fatal, but make sure ii is in your PATH if you want in to work.

Help, my reconfig is stuck!

Reconfigs can take as little as 30 minutes to run, but can take up to 2 hours depending on how busy the systems are.

In general, linux test masters are the slowest to reconfig. You can tail the manage_masters-##########.log to keep up with progress. By default, this log is created in /tmp/reconfig.

If the reconfig gets stuck, see How To/Unstick a Stuck Slave From A Master.