ReleaseEngineering/Maintenance

From MozillaWiki
Jump to: navigation, search

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our master buildbots.

Relevant repositories

Non-releng repositories involved in the CI

Treeherder

  • Chief self-deploy recent history (needs VPN; more readable than raw logs, but is cleared if the Chief service is restarted)
  • [treeherderadm.private.scl3.mozilla.com/chief/treeherder.prod/logs/ Chief self-deploy logs archive] (needs VPN; use if the history page is empty)

Mozharness deployments

Merges to the production branch of the mozharness repository are live immediately. Please see that repo for push history.

Puppet deployments

Merges to the production branch of the puppet repository are automatically deployed by Puppet within a 1/2 hour. Please see that repo for push history.

Please note if you are applying a puppet change, unless you have good reason not to do so please merge it to production branch immediately. The reason for this is that we do not want puppet changes to "back up" on default, and then land en masse in production. It can be very difficult to untangle multiple puppet changes that arrive together.

The reason we have a production branch at all, is to allow contributions on default branch from a larger group of people, than those responsible for maintaining the infrastructure controlled by the puppet repo - in other words, a smaller group of people can merge to production, than the group of people that can land on default.

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as mozharness changes.)

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Older changes are archived in: 2015 | 2014 | 2013 | 2012 | 2011 | 2010 | 2009

Outcome When Bug #(s) - Description(s)
in production 2015-05-21 09:46 PT

buildbot-configs

mozharness

in production 2015-05-14 16:55 PT

buildbot-configs

mozharness

in production 2015-05-14 07:26 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-05-11 11:32 PT

buildbot-configs

mozharness

in production 2015-05-08 07:31 PT

buildbotcustom

mozharness

  • bug 1156816 - Fix scopes for aws-provisioner. Switch nightly builds to production balrog - r=catlee (f2287384b419)
  • bug 1156816 - Add docker proxy balrog config. Use nightly channel for aries - r=catlee (3df294cbf9d8)
  • bug 1156816 - Add docker proxy balrog config. Use nightly channel for aries - r=catlee (834502a2011b)
  • bug 1161794 - Do not pass nonexistent --authentication-file - r=armenzg (598059deb80a)
  • No bug - Backed out changeset f2287384b419 - wrong commit message (a8278e87b7fc)
  • No bug - Backed out changeset 3df294cbf9d8 - *sigh* forgot to add the new file (9da8de3341ef)
in production 2015-05-07 14:44 PT

buildbot-configs

  • No bug - Update release config for Thunderbird-31.7.0-build1 (ba1c65aebb67)
  • No bug - Backout f895d6aaa604 (bug 1155349) for hosing up android tests (55d2f384d857)
in production 2015-05-07 09:41 PT

buildbot-configs

mozharness

  • bug 1149670 - Accept manifest arguments to try syntax in mozharness and filter master test manifests to include only those tests - r=ahal (aea5cb0e9630)
  • bug 1162171 - Split out source checkouts into its own action for fx_desktop_build - r=mshal (a8140ebac3b5)
in production 2015-05-06 11:02 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-05-04 09:40 PT

buildbot-configs

  • bug 1131269 - use SETA data to disable unneeded tests - r=jmaher (ca64766fa43c)
  • No bug - Update release config for Thunderbird-38.0b4-build3 (f059fe3da5e3)
  • No bug - Update release config for Thunderbird-38.0b4-build4 (c4894dd36300)
  • No bug - Update release config for Thunderbird-38.0b4-build5 (5c9bb97ab860)
  • No bug - Update release config for Fennec-38.0b10-build1 (3254fb581c55)
  • No bug - Update release config for Firefox-38.0-build1 (34917081c940)
  • No bug - Update release config for Fennec-38.0b10-build2 (e1cdac351d94)
  • No bug - Update release config for Firefox-38.0-build2 (9c6cb3e2c77c)