CIDuty/Reconfigs: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(→‎How to reconfig: Update script link to use default, not an old rev)
No edit summary
Line 1: Line 1:
Buildduty is responsible for reconfig-ing the Buildbot masters to get release engineering code changes into production.
A reconfig (short for "reconfigure the buildbot masters") is how changes to buildbot configurations make it into production. Buildduty is responsible for running reconfigs.


The person doing reconfigs should also update the [https://wiki.mozilla.org/ReleaseEngineering:Maintenance#Reconfigs_.2F_Deployments reconfig deployments page].
In a nutshell, a reconfig consists of:
* moving the ''production'' tag for the [https://hg.mozilla.org/build/buildbot-configs buildbot-configs] repository and the ''production-0.8'' tag for the [https://hg.mozilla.org/build/buildbotcustom buildbotcustom] to the current tip (or chosen revision)
* updating the source checkout of both repositories on all the buildbot masters
* updating the [https://hg.mozilla.org/build/tools tools] checkout on each master to the current tip
* executing a buildbot reconfig command on each buildbot master
We also still move the production tag for the [https://hg.mozilla.org/build/mozharness mozharness] repository until mozharness is moved fully in-tree.  


= Scheduled reconfigs =
Please see the [https://wiki.mozilla.org/ReleaseEngineering/How_To/Land_Buildbot_Master_Changes instructions for how to land buildbot changes] for more information.
Scheduled reconfigs are *supposed* to happen <b>every Monday and Thursday</b>. During this, buildduty needs to merge default -> production branches and reconfig the affected masters. [[ReleaseEngineering/Landing_Buildbot_Master_Changes|The Landing Buildbot Master Changes wiki page]] has step by step instructions.  


It is also valid to do other additional reconfigs anytime you want. Other release engineers may have important changes to land that don't coincide with the scheduled reconfigs.
It is polite to ask in #releng if anyone has further changes to land before starting the reconfig process.


It is polite to ask in #mozbuild if anyone has further changes to land before starting the reconfig process.
= How to reconfig =
The current state of the art is to use the [https://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/end_to_end_reconfig.sh end_to_end_reconfig.sh] script.


While merging buildbot-configs default -> production and buildbotcustom default -> production-0.8, merge mozharness default -> production as well.
The end_to_end_reconfig.sh script uses the original [https://wiki.mozilla.org/ReleaseEngineering/Managing_Buildbot_with_Fabric fabric scripts] as it's core, but also takes care of updating the wiki, updating bugs affected by the merge, and updating the tools checkouts on foopies as well. As such, the script has a few, indirect python module dependencies:
* fabric
* requests
However, the script will try to install the dependencies automatically if they are missing.


If there are changes to tools that impact the foopies, you should update to the [[ReleaseEngineering/How_To/Android_Tegras#Update_software_on_the_foopies|latest version of code on the foopies]].
= Help, my reconfig is stuck! =
Reconfigs can take as little as 30 minutes to run, but can take up to 2 hours depending on how busy the systems are.  


= How to reconfig =
In general, linux test masters are the slowest to reconfig. You can tail the manage_masters-##########.log to keep up with progress. By default, this log is created in /tmp/reconfig.
You should [https://wiki.mozilla.org/ReleaseEngineering/Managing_Buildbot_with_Fabric use Fabric to do the reconfig].  That's kind of old school and error prone.  Even better - use the script pmoore wrote. It's full of awesome. Unicorns too. {{bug|1018248}} See https://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/end_to_end_reconfig.sh end_to_end_reconfig.sh


= Help, my reconfig is stuck! =
If the reconfig gets stuck, see [https://wiki.mozilla.org/ReleaseEngineering/How_To/Unstick_a_Stuck_Slave_From_A_Master How To/Unstick a Stuck Slave From A Master].
If the reconfig gets stuck, see [https://wiki.mozilla.org/ReleaseEngineering/How_To/Unstick_a_Stuck_Slave_From_A_Master How To/Unstick a Stuck Slave From A Master].

Revision as of 18:31, 31 March 2015

A reconfig (short for "reconfigure the buildbot masters") is how changes to buildbot configurations make it into production. Buildduty is responsible for running reconfigs.

In a nutshell, a reconfig consists of:

  • moving the production tag for the buildbot-configs repository and the production-0.8 tag for the buildbotcustom to the current tip (or chosen revision)
  • updating the source checkout of both repositories on all the buildbot masters
  • updating the tools checkout on each master to the current tip
  • executing a buildbot reconfig command on each buildbot master

We also still move the production tag for the mozharness repository until mozharness is moved fully in-tree.

Please see the instructions for how to land buildbot changes for more information.

It is polite to ask in #releng if anyone has further changes to land before starting the reconfig process.

How to reconfig

The current state of the art is to use the end_to_end_reconfig.sh script.

The end_to_end_reconfig.sh script uses the original fabric scripts as it's core, but also takes care of updating the wiki, updating bugs affected by the merge, and updating the tools checkouts on foopies as well. As such, the script has a few, indirect python module dependencies:

  • fabric
  • requests

However, the script will try to install the dependencies automatically if they are missing.

Help, my reconfig is stuck!

Reconfigs can take as little as 30 minutes to run, but can take up to 2 hours depending on how busy the systems are.

In general, linux test masters are the slowest to reconfig. You can tail the manage_masters-##########.log to keep up with progress. By default, this log is created in /tmp/reconfig.

If the reconfig gets stuck, see How To/Unstick a Stuck Slave From A Master.

Retrieved from "https://wiki.mozilla.org/index.php?title=CIDuty/Reconfigs&oldid=1065626"