Confirmed users
502
edits
CIDuty/Reconfigs: Difference between revisions
m
Jlund moved page Buildduty/Reconfigs to CIDuty/Reconfigs: changing team name
No edit summary |
m (Jlund moved page Buildduty/Reconfigs to CIDuty/Reconfigs: changing team name) |
||
| (10 intermediate revisions by 6 users not shown) | |||
| Line 1: | Line 1: | ||
__TOC__ | __TOC__ | ||
{{Release Engineering How To|Reconfigs}} | |||
A reconfig (short for "reconfigure the buildbot masters") is how changes to buildbot configurations make it into production. | A reconfig (short for "reconfigure the buildbot masters") is how changes to buildbot configurations make it into production. CiDuty is responsible for running reconfigs. | ||
In a nutshell, a reconfig consists of: | In a nutshell, a reconfig consists of: | ||
* moving the ''production'' tag for the [https://hg.mozilla.org/build/buildbot-configs buildbot-configs] | * moving the ''production'' tag for the [https://hg.mozilla.org/build/buildbot-configs buildbot-configs] repository, and the ''production-0.8'' tag for the [https://hg.mozilla.org/build/buildbotcustom buildbotcustom] to the current tip (or chosen revision) | ||
* updating the source checkout of both repositories on all the buildbot masters | * updating the source checkout of both repositories on all the buildbot masters | ||
* updating the [https://hg.mozilla.org/build/tools tools] checkout on each master to the current tip | * updating the [https://hg.mozilla.org/build/tools tools] checkout on each master to the current tip | ||
* executing a buildbot reconfig command on each buildbot master | * executing a buildbot reconfig command on each buildbot master | ||
Please see the [ | Please see the [[ReleaseEngineering/How_To/Land_Buildbot_Master_Changes|instructions for how to land buildbot changes]] for more information. | ||
It is polite to ask in #releng if anyone has further changes to land before starting the reconfig process. | It is polite to ask in #releng if anyone has further changes to land before starting the reconfig process. | ||
= How to reconfig = | = How to reconfig = | ||
The current state of the art is to use the [https://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/end_to_end_reconfig.sh end_to_end_reconfig.sh] script. | The current state of the art is to use the [https://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/end_to_end_reconfig.sh end_to_end_reconfig.sh] script. To see all the available options the script provides: | ||
<pre> | |||
bash ./end_to_end_reconfig.sh -h | |||
</pre> | |||
The end_to_end_reconfig.sh script uses the original [https://wiki.mozilla.org/ReleaseEngineering/Managing_Buildbot_with_Fabric fabric scripts] as it's core, but also takes care of updating the wiki, updating bugs affected by the merge, and updating the tools checkouts on foopies as well. As such, the script has a few, indirect python module dependencies: | The end_to_end_reconfig.sh script uses the original [https://wiki.mozilla.org/ReleaseEngineering/Managing_Buildbot_with_Fabric fabric scripts] as it's core, but also takes care of updating the wiki, updating bugs affected by the merge, and updating the tools checkouts on foopies as well. As such, the script has a few, indirect python module dependencies: | ||
| Line 38: | Line 41: | ||
export LDAP_PASSWORD='XXXXXXXX' | export LDAP_PASSWORD='XXXXXXXX' | ||
The script will also: | |||
* create a template config file (default is ''~/.reconfig/config'') if it does not exist, but you'll need to fill in your own credentials before it will work. | |||
* attempt to update IRC with reconfig status. To do so, it uses a minimal bash IRC client called [http://tools.suckless.org/ii/ ii]. You can [http://tools.suckless.org/ii/ download ii from their website], or install it via a package manager (e.g. port install ii). Updating irc is non-fatal, but make sure ''ii'' is in your PATH if you want it to work. | |||
* create a temporary folder (''/tmp/reconfig'') to store the reconfig files. When starting a new reconfig, you should delete the reconfig folder corresponding to an older run of the script before attempting to run it again. | |||
= Updating the pinned version of mozharness on mozilla-central = | = Updating the pinned version of mozharness on mozilla-central = | ||
The revision of [https://hg.mozilla.org/build/mozharness mozharness] used by a particular branch of mozilla code is now tracked in-tree. As a courtesy to developers and sheriffs, CiDuty is expected to update the pinned revision in the mozilla-central integration branch when they move the <code>production</code> tag. The change will be merged from mozilla-central to other branches by sheriffs as part of their normal duties. | |||
The revision of [https://hg.mozilla.org/build/mozharness mozharness] used by a particular branch of mozilla code is now tracked in-tree. As a courtesy to developers and sheriffs, | |||
The pinned revision is tracked in this file: http://hg.mozilla.org/mozilla-central/file/920ded6a1f77/testing/mozharness/mozharness.json | The pinned revision is tracked in this file: http://hg.mozilla.org/mozilla-central/file/920ded6a1f77/testing/mozharness/mozharness.json | ||
| Line 51: | Line 54: | ||
NOTE: once all of mozharness moves in-tree, this step will be unnecessary. | NOTE: once all of mozharness moves in-tree, this step will be unnecessary. | ||
= Updating master/master_config.json = | |||
If you have added/removed a platform that will change the content of master/master_config.json in tools/buildfarm/maintenance/production_masters.json, you'll need to manually update the masters that this change impacts because the end_to_end_reconfig.sh script does not do this step. {{bug|1215294}} opened to enable this in the script. | |||
Example | |||
<pre> | |||
cd /builds/buildbot/tests1-linux64/ | |||
export PRODUCTION_MASTERS=tools/buildfarm/maintenance/production-masters.json | |||
python buildbot-configs/update-master-json.py $PRODUCTION_MASTERS master/master_config.json | |||
make checkconfig | |||
make reconfig | |||
</pre> | |||
= Help, my reconfig failed! = | = Help, my reconfig failed! = | ||
Assuming you're using the end_to_end_reconfig.sh script, you can resume after fixing the error. Errors and exit state can be found in the manage_masters-##########.log which is created in /tmp/reconfig by default. | Assuming you're using the end_to_end_reconfig.sh script, you can resume after fixing the error. Errors and exit state can be found in the manage_masters-##########.log which is created in /tmp/reconfig by default. The hourly auto reconfig logs are in the bb dir as <tt>reconfig.{log,lock}</tt>. | ||
Running the script again will yield the following menu: | Running the script again will yield the following menu: | ||
| Line 67: | Line 83: | ||
Reconfigs can take as little as 30 minutes to run, but can take up to 2 hours depending on how busy the systems are. | Reconfigs can take as little as 30 minutes to run, but can take up to 2 hours depending on how busy the systems are. | ||
In general, linux test masters are the slowest to reconfig. You can tail the manage_masters-##########.log to keep up with progress. By default, this log is created in /tmp/reconfig. | In general, linux test masters are the slowest to reconfig. You can tail the manage_masters-##########.log to keep up with progress. By default, this log is created in /tmp/reconfig. The hourly auto reconfig logs are in the bb dir as <tt>reconfig.{log,lock}</tt>. | ||
If the reconfig gets stuck, see [ | If the reconfig gets stuck, see [[ReleaseEngineering/How_To/Unstick_a_Stuck_Slave_From_A_Master|How To/Unstick a Stuck Slave From A Master]]. | ||