ReleaseEngineering/Maintenance

From MozillaWiki
Jump to: navigation, search

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our master buildbots.

Relevant repositories

Non-releng repositories involved in the CI

Treeherder

  • Chief self-deploy recent history (needs VPN; more readable than raw logs, but is cleared if the Chief service is restarted)
  • [treeherderadm.private.scl3.mozilla.com/chief/treeherder.prod/logs/ Chief self-deploy logs archive] (needs VPN; use if the history page is empty)

Mozharness deployments

Merges to the production branch of the mozharness repository are live immediately. Please see that repo for push history.

Puppet deployments

Merges to the production branch of the puppet repository are automatically deployed by Puppet within a 1/2 hour. Please see that repo for push history.

Please note if you are applying a puppet change, unless you have good reason not to do so please merge it to production branch immediately. The reason for this is that we do not want puppet changes to "back up" on default, and then land en masse in production. It can be very difficult to untangle multiple puppet changes that arrive together.

The reason we have a production branch at all, is to allow contributions on default branch from a larger group of people, than those responsible for maintaining the infrastructure controlled by the puppet repo - in other words, a smaller group of people can merge to production, than the group of people that can land on default.

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as mozharness changes.)

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Older changes are archived in: 2016 | 2015 | 2014 | 2013 | 2012 | 2011 | 2010 | 2009

Outcome When Bug #(s) - Description(s)
in production 2017-09-21 07:51 PT

buildbot-configs

buildbotcustom

  • bug 1333792 - Try push confirmation emails for Taskcluster jobs should link to the Task inspector rather than the upload directory on archive.m.o - r=mtabara (994ad65b2abd)
in production 2017-09-14 03:55 PT

buildbot-configs

buildbotcustom

in production 2017-09-06 10:14 PT

buildbot-configs

buildbotcustom

in production 2017-08-31 06:50 PT

buildbotcustom

in production 2017-08-06 22:46 PT

buildbot-configs

in production 2017-08-04 03:42 PT

buildbot-configs

  • No bug - Update release config for Thunderbird-56.0b1-build1 (663ad30ff27d)
  • No bug - Relpro mac and win builds index changes required for 56.0b1 - r=mtabara (3c7a94990487)
in production 2017-08-02 00:27 PT

buildbot-configs

buildbotcustom

in production 2017-07-31 04:12 PT

buildbot-configs

  • bug 1369535 - m-r gecko version bump bbconfigs - r=jlorenzo (acbb8cf0729e)
  • No bug - No Bug - Staging release: Point to a disposable balrog instance - r=rail (c700658593c4)
  • No bug - No Bug - Staging release: Point to rail's disposable repo - r=jlorenzo (749272eda177)
in production 2017-06-12 03:31 PT

buildbot-configs

in production 2017-06-05 06:32 PT

buildbot-configs

in production 2017-06-05 02:12 PT

buildbot-configs

in production 2017-05-11 06:12 PT

buildbot-configs

in production 2017-03-16 07:25 PT

buildbot-configs

in production 2017-03-10 15:02 PT

buildbot-configs

in production 2017-03-10 10:11 PT

buildbot-configs

in production 2017-03-09 13:18 PT

buildbot-configs

in production 2017-03-06 11:57 PT

buildbot-configs

in production 2017-03-06 11:49 PT

buildbot-configs

in production 2017-03-03 13:50 PT

buildbot-configs

in production 2017-03-03 12:53 PT

buildbot-configs

in production 2017-03-01 11:50 PT

buildbot-configs

in production 2017-02-27 19:34 PT

buildbotcustom

in production 2017-02-27 18:50 PT

buildbot-configs

  • No bug - Update release config for Fennec-52.0b10-build1 (edf97f3ab377)

buildbotcustom

in production 2017-02-27 13:40 PT

buildbot-configs

in production 2017-02-27 11:56 PT

buildbot-configs

in production 2017-02-24 13:02 PT

buildbotcustom

in production 2017-02-23 06:27 PT

buildbot-configs

in production 2017-02-17 09:50 PT

buildbot-configs

in production 2017-02-16 13:27 PT

buildbot-configs

  • bug 1338610 - Split up mochitest-other into separate mochitest-a11y and mochitest-chrome jobs - r=jlund (4f814a83ba4f)
  • bug 1339151 - buildbot configs to use different tasks for signed and unsigned bits in release promotion - r=jlund (fb545d14c97f)
in production 2017-02-15 12:50 PT

buildbot-configs

  • No bug - backouts Bug No Bug - rev: c9b1f1f58974 (fbcd05687641)
  • No bug - reverts 51.0.4 release fennec patch (7d7513aadda6)
in production 2017-02-08 13:10 PT

buildbot-configs

in production 2017-02-08 12:19 PT

buildbot-configs

buildbotcustom

in production 2017-01-23 03:51 PT

buildbot-configs

in production 2017-01-05 04:33 PT

buildbot-configs