: New feature! MozillaWiki is now mobile-friendly. Visit from a mobile device to see new mobile theme + try editing. Release details.

ReleaseEngineering/Maintenance

From MozillaWiki
Jump to: navigation, search

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our master buildbots.

Relevant repositories

Non-releng repositories involved in the CI

Treeherder

  • Chief self-deploy recent history (needs VPN; more readable than raw logs, but is cleared if the Chief service is restarted)
  • [treeherderadm.private.scl3.mozilla.com/chief/treeherder.prod/logs/ Chief self-deploy logs archive] (needs VPN; use if the history page is empty)

Mozharness deployments

Merges to the production branch of the mozharness repository are live immediately. Please see that repo for push history.

Puppet deployments

Merges to the production branch of the puppet repository are automatically deployed by Puppet within a 1/2 hour. Please see that repo for push history.

Please note if you are applying a puppet change, unless you have good reason not to do so please merge it to production branch immediately. The reason for this is that we do not want puppet changes to "back up" on default, and then land en masse in production. It can be very difficult to untangle multiple puppet changes that arrive together.

The reason we have a production branch at all, is to allow contributions on default branch from a larger group of people, than those responsible for maintaining the infrastructure controlled by the puppet repo - in other words, a smaller group of people can merge to production, than the group of people that can land on default.

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as mozharness changes.)

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Older changes are archived in: 2015 | 2014 | 2013 | 2012 | 2011 | 2010 | 2009

Outcome When Bug #(s) - Description(s)
in production 2015-08-03 21:20 PT

buildbot-configs

in production 2015-08-03 05:43 PT

buildbot-configs

in production 2015-08-02 16:00 PT

buildbot-configs

in production 2015-07-31 07:02 PT

buildbot-configs

buildbotcustom

in production 2015-07-30 12:54 PT

buildbot-configs

  • bug 1184117 - Make Android 4.0 opt-in by default on Try - r=Callek (e7176ef993ef)
  • bug 1189273 - Drop twice-daily nightlies for v2.2/v2.2r and stop running emulator nightly builds on the B2G release branches - r=catlee (fbd3abad6d6b)

buildbotcustom

  • bug 1187966 - Set the env for periodic file update jobs so we can use TOOLTOOL_CACHE = - r=nthomas (dc0535892866)

mozharness

  • bug 1184594 - git 'repo' is not deployed correctly on build machines managed by buildbot - r=hwine (57dde819edcb)
  • bug 1188648 - Use HOST_LOG env var to log gecko output when running Gij on _all_ gecko builds - r=jgriffin (79fffa3a18ae)
  • bug 1188648 - Always log gecko output during Gaia JS Integration Tests - r=jgriffin (d72df953784c)
  • bug 1188698 - Do not fail when succeed count is 0 (672af8a8da3e)
  • No bug - Backed out changeset 79fffa3a18ae -- Unfortunately try run reveals this isn't working as expected (faf4013a7fc6)
  • No bug - Backed out changeset 57dde819edcb (3ae81f4458ef)
in production 2015-07-24 12:15 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-07-23 11:11 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-07-22 13:13 PT

buildbot-configs

  • No bug - Backout Bug 1181255 followup due to travis errors (f70c6a09f071)
in production 2015-07-22 13:03 PT

buildbot-configs

  • bug 1181255 - Follow up to fix slave platform in mozilla/config.py - r=jlund (84310ed35dc0)
  • bug 1186411 - Drop OSX/Windows Mulet builds on b2g37 and remove old Mulet test cruft from the buildbot configs - r=catlee (565bb899aa71)
  • bug 1186425 - set-up new signing format for sha2 signing cert - r=catlee (cb409abbe898)
  • No bug - Backout tooltool deprecation per dustin in IRC, to fix android bustage (7157300a6277)

mozharness

in production 2015-07-22 07:58 PT

buildbot-configs

buildbotcustom

in production 2015-07-21 07:06 PT

buildbot-configs

mozharness

in production 2015-07-17 10:43 PT

buildbot-configs

buildbotcustom

in production 2015-07-10 13:36 PT

buildbotcustom

  • No bug - backouts Bug 1179476 - rev: 5eb51c675b71 (4cfac21f6697)
  • No bug - merges backout 5eb51c675b71 -> default (f211996776f8)
  • No bug - backouts Bug 1179476 - rev: c5ab8b5a5491 (fdbf521e941d)
  • No bug - merges backout c5ab8b5a5491 -> default (62bb822bbe5e)
  • No bug - backouts Bug 1179476 - rev: 26593153083a (684eee5dd7e8)
  • No bug - merges backout 26593153083a -> default (f5821f98830e)
in production 2015-07-09 18:06 PT

buildbot-configs

  • No bug - Update release config for Firefox-40.0b3-build1 (a61dc8f679f7)

buildbotcustom

  • bug 1179476 - use in gecko tree mozharness across all buildbot jobs, ReleaseBuildFactory fix - r=nthomas (5eb51c675b71)
in production 2015-07-06 13:12 PT

buildbot-configs

  • No bug - Update release config for Fennec-40.0b1-build1 (89d2441d4fac)
  • No bug - Update release config for Firefox-40.0b1-build1 (8ce8ab12e902)
  • No bug - Update release config for Fennec-40.0b1-build2 (fc1925593fbc)
  • No bug - Update release config for Firefox-40.0b1-build2 (b06a458eed9b)
  • No bug - Update release config for Fennec-39.0.1-build1 (3fcb4a2a79cd)

buildbotcustom

  • bug 1154801 - add support in ScriptFactory for archiver mozharness, use relative destination - r=catlee (e86f26add3c8)
  • bug 1176360 - Do not use update_tests config and use --firefox-ui-branch & --update-verify-config - r=bhearsum (53b1c2b6a694)

mozharness

in production 2015-07-01 15:25 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-06-29 11:35 PT

buildbot-configs

  • bug 1170252 - Enable web-platform-tests for debug builds on Try - r=jlund (4956570a7555)
  • bug 1175533 - Tracking bug for June-29-2015 migration work, gecko version bumps - r=rail (daecf0003289)
  • bug 1176283 - L10N nightly repacks should use more chunks - r=bhearsum (cf5581631f65)
  • No bug - Update release config for Fennec-39.0b8-build1 (474a7a584661)
  • No bug - Update release config for Firefox-39.0-build1 (2cc43c63ecdf)
  • No bug - Update release config for Fennec-39.0b8-build2 (6c5f894ecede)
  • No bug - Update release config for Firefox-39.0-build2 (96ddea59a52c)
  • No bug - Update release config for Fennec-39.0b8-build3 (b58283299eed)
  • No bug - Update release config for Firefox-39.0-build3 (fbf9ffa38c73)
  • No bug - Update release config for Fennec-39.0b8-build4 (6589f4cc0f1b)
  • No bug - Update release config for Fennec-39.0b8-build5 (f979b7de5656)
  • No bug - Update release config for Fennec-39.0b8-build6 (42da0a76babd)
  • No bug - Update release config for Firefox-31.8.0esr-build1 (ec5808359f51)
  • No bug - Update release config for Firefox-38.1.0esr-build1 (3f261851fb33)
  • No bug - Update release config for Firefox-39.0-build4 (745d52adeb86)
  • No bug - Update release config for Fennec-39.0-build1 (00a178363ff9)
  • No bug - Update release config for Firefox-39.0-build4 (b1375f94fb5e)
  • No bug - Update release config for Fennec-39.0-build2 (f82bb55282ca)
  • No bug - Update release config for Firefox-39.0-build5 (52c54a6e006c)

mozharness

  • bug 1176358 - Log gecko output to stdout and use a different port for each locale when running firefox update tests - r=armenzg (243340ccbf2a)
  • bug 917999 - Accommodate directory rename for cppunittests -> cppunittest - r=jlund (94dbc2337e5f)
in production 2015-06-16 08:00 PT

puppet

  • bug 1174202 - disable fseventd logging on OSX
  • bug 1174173 - use runner to do shared checkouts, clobbers on OSX
in production 2015-06-16 08:26 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-06-05 14:55 PT

buildbot-configs

buildbotcustom

in production 2015-06-02 16:00 PT

buildbot-configs

in production 2015-05-21 09:46 PT

buildbot-configs

mozharness

in production 2015-05-14 16:55 PT

buildbot-configs

mozharness

in production 2015-05-14 07:26 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-05-11 11:32 PT

buildbot-configs

mozharness

in production 2015-05-08 07:31 PT

buildbotcustom

mozharness

  • bug 1156816 - Fix scopes for aws-provisioner. Switch nightly builds to production balrog - r=catlee (f2287384b419)
  • bug 1156816 - Add docker proxy balrog config. Use nightly channel for aries - r=catlee (3df294cbf9d8)
  • bug 1156816 - Add docker proxy balrog config. Use nightly channel for aries - r=catlee (834502a2011b)
  • bug 1161794 - Do not pass nonexistent --authentication-file - r=armenzg (598059deb80a)
  • No bug - Backed out changeset f2287384b419 - wrong commit message (a8278e87b7fc)
  • No bug - Backed out changeset 3df294cbf9d8 - *sigh* forgot to add the new file (9da8de3341ef)
in production 2015-05-07 14:44 PT

buildbot-configs

  • No bug - Update release config for Thunderbird-31.7.0-build1 (ba1c65aebb67)
  • No bug - Backout f895d6aaa604 (bug 1155349) for hosing up android tests (55d2f384d857)
in production 2015-05-07 09:41 PT

buildbot-configs

mozharness

  • bug 1149670 - Accept manifest arguments to try syntax in mozharness and filter master test manifests to include only those tests - r=ahal (aea5cb0e9630)
  • bug 1162171 - Split out source checkouts into its own action for fx_desktop_build - r=mshal (a8140ebac3b5)
in production 2015-05-06 11:02 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-05-04 09:40 PT

buildbot-configs

  • bug 1131269 - use SETA data to disable unneeded tests - r=jmaher (ca64766fa43c)
  • No bug - Update release config for Thunderbird-38.0b4-build3 (f059fe3da5e3)
  • No bug - Update release config for Thunderbird-38.0b4-build4 (c4894dd36300)
  • No bug - Update release config for Thunderbird-38.0b4-build5 (5c9bb97ab860)
  • No bug - Update release config for Fennec-38.0b10-build1 (3254fb581c55)
  • No bug - Update release config for Firefox-38.0-build1 (34917081c940)
  • No bug - Update release config for Fennec-38.0b10-build2 (e1cdac351d94)
  • No bug - Update release config for Firefox-38.0-build2 (9c6cb3e2c77c)