: Etherpad users! We are developing an extension that will allow you to create pages from etherpads quickly and easily. Please visit our sandbox and help us test it.

ReleaseEngineering/Maintenance

From MozillaWiki
Jump to: navigation, search

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our masters.

Relevant repositories

Non-releng repositories involved in the CI

Treeherder

  • Chief self-deploy recent history (needs VPN; more readable than raw logs, but is cleared if the Chief service is restarted)
  • [treeherderadm.private.scl3.mozilla.com/chief/treeherder.prod/logs/ Chief self-deploy logs archive] (needs VPN; use if the history page is empty)

Mozharness deployments

Merges to the production branch of the mozharness repository are live immediately. Please see that repo for push history.

Puppet deployments

Merges to the production branch of the puppet repository are automatically deployed by Puppet within a 1/2 hour. Please see that repo for push history.

Please note if you are applying a puppet change, unless you have good reason not to do so please merge it to production branch immediately. The reason for this is that we do not want puppet changes to "back up" on default, and then land en masse in production. It can be very difficult to untangle multiple puppet changes that arrive together.

The reason we have a production branch at all, is to allow contributions on default branch from a larger group of people, than those responsible for maintaining the infrastructure controlled by the puppet repo - in other words, a smaller group of people can merge to production, than the group of people that can land on default.

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as mozharness changes.)

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Older changes are archived in: 2014 | 2013 | 2012 | 2011 | 2010 | 2009

Outcome When Bug #(s) - Description(s)
in production 2015-03-27 11:53 PT

buildbot-configs

buildbotcustom

  • bug 1105485 - Fix up submission of release blobs so they can be used for beta builds - fix string formatting - r=rail (3d4d36c4d439)
in production 2015-03-25 12:58 PT

buildbot-configs

  • No bug - Turn off the arm-sim-osx builds (leaving them on try, but off by default). They require some bbc work - r=myself (2f15edda43e5)

mozharness

in production 2015-03-25 09:19 PT

buildbot-configs

  • bug 1146520 - Add ARM simulator spidermonkey builds on OSX - r=bhearsum (07b5232169d9)
  • bug 1146604 - Remove support for gaia-try, linux32_gecko on all branches, and linux64_gecko on trunk - r=rail (8cb8713bf219)
  • No bug - Update release config for Firefox-31.6.0esr-build1 (3e530cd9855b)
  • No bug - Update release config for Fennec-37.0b8-build1 (1a67866762a9)
  • No bug - Update release config for Firefox-37.0-build1 (94aa6fd0baad)

buildbotcustom

in production 2015-03-24 10:13 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-03-23 14:39 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-03-17 13:28 PT

buildbot-configs

buildbotcustom

  • bug 1105485 - Fix up submission of release blobs so they can be used for beta builds - fix up ready for cdntest scheduler name in trigger step - r=rail (a9d619d67436)
  • bug 1143683 - shorten builder names in a predictable way - r=rail (8931029f8825)

mozharness

in production 2015-03-16 08:19 PT

buildbot-configs

  • bug 1105485 - Fix up submission of release blobs so they can be used for beta builds - r=nthomas (6366a5b105af)
  • No bug - Update release config for Fennec-36.0.2-build2 (f2648de01e1b)

buildbotcustom

  • bug 1105485 - Fix up submission of release blobs so they can be used for beta builds - r=nthomas (e032c77d3ca8)
  • bug 1105485 - Fix up submission of release blobs so they can be used for beta builds - use a dummy factory for trigger bouncer check if no triggers are needed - r=rail (590502fb43f5)
  • No bug - Fix some ordering issues with name normalization + overlapping keys (e0b6a1bf0ccb)
  • No bug - Shorten more harder (faeae98891d4)

mozharness

in production 2015-03-12 12:42 PT

buildbot-configs

buildbotcustom

  • bug 1142967 - release automation mail getting eaten again - don't send mail directly to mail.build.mozilla.org - r=rail (0d505afd8fd8)
  • No bug - Fix relayhost in more place (1dd64e576971)

mozharness

in production 2015-03-12 09:32 PT

buildbot-configs

mozharness

in production 2015-03-11 12:18 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-03-09 08:51 PT

buildbot-configs

mozharness

in production 2015-03-05 15:28 PT

buildbot-configs

  • bug 1139403 - enable xulrunner jobs for win64. r=rail
  • bug 1139943 - disable everything except graphene on larch. r=jlund
  • bug 1137047 - Reallocate 10 buildpool Mac slaves to trybuildpool to balance load, r=coop
  • bug 1100386 - Repurpose servo-lion-r5-001 and servo-lion-r5-002, r=coop
in production 2015-03-04 15:28 PT

buildbot-configs

mozharness

in production 2015-03-03 16:57 PT

buildbot-configs

in production 2015-03-02 14:48 PT

buildbot-configs

buildbotcustom

in production 2015-02-27 10:10 PT

buildbot-configs

  • bug 1130090 - Set up new builds on larch - r=jlund (c96cca546f8a)
  • bug 1137243 - Enable try for macosx64-mulet and win32-mulet - r=bhearsum (02ddc1f242e3)
  • bug 1137243 - Attach try slaves for macosx64-mulet and win32-mulet - r=jlund (69f6a08162de)
  • No bug - (Bug 1126493 - rollout 10.10 tests in a way that doesn't impact wait times - r=coop (d3bcd9a37500)
  • No bug - (Bug 1126493 - rollout 10.10 tests in a way that doesn't impact wait times - r=jwatkins (c91ce36665aa)
  • No bug - revert Bug 1126493 - rollout 10.10 tests in a way that doesn't impact wait times - r=coop (b92baf1e36c1)

buildbotcustom

mozharness

in production 2015-02-27 07:43 PT

buildbot-configs

  • bug 1130090 - Set up new builds on larch - r=jlund (c96cca546f8a)
  • bug 1137243 - Enable try for macosx64-mulet and win32-mulet - r=bhearsum (02ddc1f242e3)
  • bug 1137243 - Attach try slaves for macosx64-mulet and win32-mulet - r=jlund (69f6a08162de)
  • No bug - (Bug 1126493 - rollout 10.10 tests in a way that doesn't impact wait times - r=coop (d3bcd9a37500)
  • No bug - (Bug 1126493 - rollout 10.10 tests in a way that doesn't impact wait times - r=jwatkins (c91ce36665aa)
  • No bug - revert Bug 1126493 - rollout 10.10 tests in a way that doesn't impact wait times - r=coop (b92baf1e36c1)

buildbotcustom

mozharness

in production 2015-02-26 10:27 PT

buildbot-configs

buildbotcustom

in production 2015-02-25 12:22 PT

buildbot-configs

buildbotcustom

in production 2015-02-24 13:05 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-02-18 12:31 PT

buildbot-configs

in production 2015-02-17 17:34 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-02-11 07:06 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-02-05 09:26 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-02-05 06:02 PT

buildbot-configs

buildbotcustom

in production 2015-02-04 12:16 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-02-02 15:38 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-01-30 12:25 PT

buildbot-configs

mozharness

in production 2015-01-27 13:24 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-01-26 14:30 PT

buildbot-configs

  • bug 1110286 - Disable L10n mozharness repacks for Ash. r=jlund
  • bug 1125888 - turn on win64 l10n repacks on central and aurora. r=rail
  • bug 1125353 - turn off android builds on esr31 r=rail
  • bug 1112966 - Discontinue EUballot builds, r=rail
  • bug 1125191 - disable yosemite tests on cedar, try is more useful r=coop
  • bug 1110286 - Enable pgo, nightly and repacks for Ash. r=rail
  • bug 1055919 - desktop mozharness mach builders should ride trains - stop aurora from using mh mach, r=hwine
  • bug 1118796 set up date for build promotion work, r=rail
  • bug 1123965 - Disable HPKP update on b2g30 - r=Callek
  • bug 1107747 - Add build config for v2.1s branch. r=hwine

mozharness

  • bug 1107747 - Add dolphin-512 and v2.1s b2g bumper configs. r=nthomas
  • bug 1116187 - Add mochitest-chrome to B2G emulator config
  • bug 1116187 - Add mochitest-chrome to abs dirs
  • bug 1123443 - Allow use of adb devicemanager in android emulator unittests; r=armenzg
  • bug 1117849 - Create .properties file for symlinks containing the full URL. r=catlee
  • bug 1124799 - Make Hazard job TinderboxPrints play nicely with Treeherder
  • bug 1112966 - Discontinue EUballot builds, r=rail
  • bug 1125392 - Make Mozharness' tox to run tests against hg2.6 instead of the users hg. r=rail
  • bug 1120969 - Merge automation should support new Android targets. r=~bhearsum
  • bug 984888 - bouncer submitter should run for every build attempt of a release. r=bhearsum
  • bug 1124913 - Use EmulatorMixin support to install emulator for android 4.4; r=armenzg
in production 2015-01-22 07:38 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-01-20 13:40 PT

mozharness

in production 2015-01-20 10:06 PT

buildbot-configs

  • bug 1073772 - Releng work for split apk - fixes buildernames to be 11 instead of 10 - r=coop (b9c34177b915)
  • bug 986990 - Modify test channel names to support releases on Balrog - switch release channel test channel names + rule ids - r=nthomas (70a8291cfe60)
in production 2015-01-19 14:33 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-01-14 09:19 PT

buildbot-configs

mozharness

in production 2015-01-13 07:03 PT

buildbot-configs

  • bug 1015018 - Remove "dep_signing_servers" and "nightly_signing_servers" from the macosx64-st-an-debug config (3673b25e6429)
  • bug 1118329 - only run a subset of talos tests on osx 10.6 - r=catlee (f45f97f226be)
  • No bug - Update release config for Fennec-36.0b1-build1 (4729be7705c3)
  • No bug - Update release config for Firefox-36.0b1-build1 (1ddca438b8fe)

mozharness

in production 2015-01-12 11:30 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-01-09 09:00 PT

buildbot-configs

buildbotcustom

mozharness

in production 2015-01-08 21:00 PT

puppet

  • bug 1013634 - All emulators in automation report "Failed to load libGL.so"
in production 2015-01-06 10:23 PT

buildbot-configs

  • bug 1076810 - RelEng CI tests should be available over the internet, not just from VPN - r=Callek (222b655abc77)
  • bug 1076810 - RelEng CI tests should be available over the internet, not just from VPN - allow pep8 failures not to break tests and remove coveralls from README.md - r=pmoore (a599d98c55de)
  • No bug - Update release config for Fennec-35.0b10-build1 (f7434f09a850)
  • No bug - Update release config for Firefox-31.4.0esr-build1 (09bf96107ce0)
  • No bug - Update release config for Firefox-35.0-build1 (93ac796aeca8)

buildbotcustom

  • bug 1072163 - Followup to correct string substitution (5668caef58e4)
  • bug 1076810 - RelEng CI tests should be available over the internet, not just from VPN - include commit message in irc notification for buildbotcustom repo - r=mgerva (4a94dc845f16)

mozharness

in production 2015-01-02 10:53 PT

buildbot-configs

  • bug 1015018 - Buildbot config for OS X static analysis builds - r=jlund (548d99344d7f)
  • bug 1045194 - Hazard builds need to reboot (95205c5c1956)
  • No bug - Update release config for Fennec-35.0b6-build1 (aa3c01123573)
  • No bug - Update release config for Firefox-35.0b6-build1 (1c9fdcd2e6fb)
  • No bug - No Bug - Remove dead code (host utils is no longer defined through buildbot) - r=no-one (f63c4306e5aa)
  • No bug - Update release config for Fennec-35.0b8-build1 (4bb66e9e7150)
  • No bug - Update release config for Firefox-35.0b8-build1 (e674ed26831a)

buildbotcustom

in production 2015-01-02 10:30 PT

mozharness

  • bug 1113081 - Remove references to metro mode from talos.py script. r=jmaher
  • bug 1078619 - Allow to run talos jobs as a developer. r=armenzg
  • bug 1076810 - RelEng CI tests should be available over the internet, not just from VPN: updated README.md for github viewing,r=mgerva
  • bug 1084066 - plugins and extensions moved to wrong directory by mozharness. r=armenzg
  • bug 1015018 - Update mozharness to support OSX static analysis builds; r=jlund
  • bug 1113460 - test jobs with minidump stackwalk should get use tooltool not tools repo, r=jgriffin
  • bug 1117071 - Mozharness is failing due to non-determinism in worst status. r=rail