ReleaseEngineering/Maintenance

From MozillaWiki
Jump to: navigation, search

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our master buildbots.

Relevant repositories

Non-releng repositories involved in the CI

Treeherder

  • Chief self-deploy recent history (needs VPN; more readable than raw logs, but is cleared if the Chief service is restarted)
  • [treeherderadm.private.scl3.mozilla.com/chief/treeherder.prod/logs/ Chief self-deploy logs archive] (needs VPN; use if the history page is empty)

Mozharness deployments

Merges to the production branch of the mozharness repository are live immediately. Please see that repo for push history.

Puppet deployments

Merges to the production branch of the puppet repository are automatically deployed by Puppet within a 1/2 hour. Please see that repo for push history.

Please note if you are applying a puppet change, unless you have good reason not to do so please merge it to production branch immediately. The reason for this is that we do not want puppet changes to "back up" on default, and then land en masse in production. It can be very difficult to untangle multiple puppet changes that arrive together.

The reason we have a production branch at all, is to allow contributions on default branch from a larger group of people, than those responsible for maintaining the infrastructure controlled by the puppet repo - in other words, a smaller group of people can merge to production, than the group of people that can land on default.

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as mozharness changes.)

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Older changes are archived in: 2015 | 2014 | 2013 | 2012 | 2011 | 2010 | 2009

Outcome When Bug #(s) - Description(s)
in production 2016-05-24 10:56 PT

buildbot-configs

in production 2016-05-23 09:45 PT

buildbot-configs

in production 2016-05-20 08:37 PT

buildbot-configs

in production 2016-05-19 10:47 PT

buildbot-configs

in production 2016-05-17 09:46 PT

buildbot-configs

in production 2016-05-16 11:22 PT

buildbot-configs

buildbotcustom

in production 2016-05-16 08:15 PT

buildbot-configs

buildbotcustom

in production 2016-05-12 08:43 PT

buildbot-configs

buildbotcustom

in production 2016-05-12 02:03 PT

buildbot-configs

  • bug 1253341 - Run duplicate Talos jobs in AWS for Linux (reverted) - r=Callek (2302dc47cb21)
  • bug 1268542 - Disable Linux64 Debug builds and tests in buildbot - r=kmoir (cfe4c07f8161)
  • bug 1271550 - Add new 10.10.5 machines to releng configs (750ab5df5a06)
  • No bug - Update release config for Thunderbird-45.1.0-build2 (48e0db6d4d22)
  • No bug - Update release config for Thunderbird-45.1.0-build3 (c199b13b84a3)
  • No bug - Backed out changeset c199b13b84a3 to let TB 45.1.0 build3 to restart (a55a7e24bba1)
  • No bug - Update release config for Thunderbird-45.1.0-build3 (3a7c1d3dbace)
  • No bug - Update release config for Fennec-47.0b4-build1 (f7b8a43eed4f)

buildbotcustom

in production 2016-05-06 09:58 PT

buildbot-configs

buildbotcustom

in production 2016-05-02 13:55 PT

buildbotcustom

in production 2016-05-02 13:00 PT

buildbot-configs

buildbotcustom

in production 2016-04-27 10:34 PT

buildbot-configs

in production 2016-04-19 06:35 PT

buildbot-configs

buildbotcustom

in production 2016-04-19 05:32 PT

buildbotcustom

in production 2016-04-13 06:07 PT

buildbot-configs

buildbotcustom

in production 2016-04-04 13:40 PT

buildbot-configs

buildbotcustom

in production 2016-03-29 13:32 PT

buildbot-configs

buildbotcustom

in production 2016-03-21 17:41 PT

buildbotcustom

in production 2016-03-21 14:15 PT

buildbot-configs

buildbotcustom

in production 2016-03-18 12:37 PT

buildbotcustom

in production 2016-03-16 10:35 PT

buildbot-configs

  • bug 1252292 - Don't run Ru-e10s on debug builds - r=me (04c8fe62d3a4)
  • bug 1253341 - Enable talos on linux64 AWS machines on try - r=kmoir (2d6158d681e0)
  • bug 1257230 - Incorrect branding for 45.0.1esr build - r=jlund (b5163b4a7814)
  • No bug - Update release config for Fennec-46.0b2-build1 (216ac6505867)
  • No bug - Update release config for Firefox-38.7.1esr-build1 (2b0b8272dda2)
  • No bug - backouts release change for 38.7.1esr - rev: 2b0b8272dda2 (ea85259cf436)
  • No bug - merges backout 2b0b8272dda2 -> default (7b0276c224b4)
  • No bug - Update release config for Firefox-38.7.1esr-build1 (b735b52c0d5a)
  • No bug - Update release config for Firefox-45.0.1-build1 (b3267390eb7d)
  • No bug - Update release config for Firefox-45.0.1esr-build1 (1f7801999788)
  • No bug - Update release config for Fennec-45.0.1-build1 (e759c66d311c)
  • No bug - Backed out 2d6158d681e0 for suspicion of causing intermittent checkconfig (7c463958098d)

buildbotcustom

  • bug 1253341 - Allow running talos on AWS machines - r=kmoir (42184dbdf8f3)
  • No bug - Backed out changeset 42184dbdf8f3 for suspicion of causing intermittent checkconfig failureis (046c46eb099d)
in production 2016-03-15 07:43 PT

buildbot-configs

in production 2016-03-11 09:50 PT

buildbot-configs

buildbotcustom

in production 2016-03-10 06:17 PT

buildbot-configs

in production 2016-03-09 06:14 PT

buildbot-configs

in production 2016-03-07 19:31 PT

buildbot-configs

in production 2016-03-07 16:12 PT

buildbot-configs

in production 2016-03-07 05:36 PT

buildbot-configs

in production 2016-03-04 13:23 PT

buildbot-configs

in production 2016-03-04 12:26 PT

buildbot-configs

in production 2016-03-04 09:03 PT

buildbot-configs

in production 2016-03-04 07:06 PT

buildbot-configs

in production 2016-03-03 16:54 PT

buildbot-configs

in production 2016-03-02 13:51 PT

buildbot-configs

buildbotcustom

  • No bug - No Bug - shorten travis results notification to be less spammy - r=catlee (4b96f5cb0d0f)
in production 2016-03-02 06:25 PT

buildbot-configs

in production 2016-03-01 07:30 PT

buildbot-configs

in production 2016-02-29 08:10 PT

buildbotcustom

in production 2016-02-25 09:50 PT

buildbot-configs

buildbotcustom

in production 2016-02-24 13:45 PT

buildbot-configs

  • bug 1250585 - adds push to mirrors automatic date, buildbot-configs - r=rail (8ff751a38618)
  • No bug - Update release config for Fennec-45.0b9-build2 (fc63a5db2c0c)
  • No bug - Update release config for Firefox-45.0b9-build2 (884497cab040)
in production 2016-02-23 13:53 PT

buildbot-configs

in production 2016-02-22 13:17 PT

buildbot-configs

  • No bug - Update release config for Firefox-45.0b8-build1 (5f9a37958eaf)
  • No bug - Update release config for Firefox-45.0b8-build1 (5f9a37958eaf)

buildbotcustom

in production 2016-02-19 13:00 PT

buildbotcustom

in production 2016-02-19 07:03 PT

buildbot-configs

buildbotcustom

in production 2016-02-17 08:46 PT

buildbot-configs

  • bug 1246627 - Schedule e10s jsreftests on win7 patch by aselagea - r=jgriffin (cc37bdb15492)
  • No bug - revert Bug 1246627 - Schedule e10s jsreftests on win7 (71b9ccb8abf1)
  • No bug - Update release config for Fennec-45.0b6-build1 (69f84fb1f35b)
  • No bug - Update release config for Firefox-45.0b6-build1 (df4ed20df968)

buildbotcustom

in production 2016-02-14 08:43 PT

buildbot-configs

buildbotcustom

  • bug 1244510 - Add a try_parser alias for mochitest-e10s-(bc|browser-chrome) patch by aselagea - r=kmoir (c994ec64ba03)
in production 2016-02-12 07:43 PT

buildbot-configs

  • bug 1218589 - Nightly mozilla-central builds of graphene patch by aselagea - r=kmoir (f88d45c3fbe6)
  • bug 1229532 - remove hg bundle builder config - r=bhearsum (f0c9c5b5a7b8)
  • bug 1246627 - Schedule e10s jsreftests on win7 patch by aselagea - r=kmoir (66bc120b896c)
  • bug 787208 - In support of merging comm-central and mozilla-central together, configure cypress as a Thunderbird branch. r=rail - r=Fallen f=jcranmer (b92aaf935173)
  • No bug - Update release config for Firefox-38.6.1esr-build1 (33fe33d65478)
  • No bug - Update release config for Firefox-44.0.2-build3 (09cd8415adb9)
  • No bug - Update release config for Thunderbird-38.6.0-build1 (91cf6aa50268)
  • No bug - Update release config for Firefox-45.0b5-build1 (6376fd9567c4)

buildbotcustom

in production 2016-02-02 14:53 PT

buildbot-configs

  • bug 1242103 - Add browser-chrome-screenshots to m-c and other try - r=armenzg (63614b6564ee)
  • No bug - Update release config for Fennec-45.0b2-build1 (7a121e3abe60)
  • No bug - Update release config for Firefox-45.0b2-build1 (a70a9918a489)
  • No bug - Update release config for Thunderbird-45.0b1-build1 (39642e4c3ba4)
in production 2016-01-26 20:05 PT

buildbotcustom

buildbot-configs

in production 2016-01-26 18:05 PT

buildbotcustom

in production 2016-01-26 16:00 PT

buildbotcustom

buildbot-configs

in production 2016-01-25 20:47 PT

buildbotcustom

in production 2016-01-25 14:54 PT

buildbotcustom

in production 2016-01-25 05:50 PT

buildbot-configs

in production 2016-01-21 12:12 PT

buildbot-configs

in production 2016-01-21 10:56 PT

buildbot-configs

buildbotcustom

in production 2016-01-20 06:16 PT

buildbot-configs

  • No bug - Update release config for Fennec-44.0b11-build1 (370703910927)

buildbotcustom

in production 2016-01-19 07:36 PT

buildbot-configs

buildbotcustom

in production 2016-01-18 06:19 PT

buildbot-configs

mozharness

in production 2016-01-15 08:55 PT

buildbot-configs

  • No bug - Update release config for Fennec-44.0b9-build1 (15dc357fc5bd)
  • No bug - Update release config for Firefox-44.0b9-build1 (df3ee52930d6)

buildbotcustom

in production 2016-01-13 15:03 PT

buildbot-configs

buildbotcustom

in production 2016-01-12 09:04 PT

buildbot-configs

buildbotcustom

mozharness

in production 2016-01-11 08:10 PT

buildbot-configs

buildbotcustom

in production 2016-01-11 07:41 PT

buildbot-configs

buildbotcustom

mozharness

in production 2016-01-07 06:39 PT

buildbot-configs

buildbotcustom