ReleaseEngineering/Maintenance

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our masters.

Relevant repositories

• buildbot
• buildbot-configs
• buildbotcustom
• tools
• mozharness
• puppet

Non-releng repositories involved in the CI

TBPL

• Chief self-deploy recent history (needs VPN; more readable than raw logs, but is cleared if the Chief service is restarted)
• Chief self-deploy logs archive (needs VPN; use if the history page is empty)
• TBPL production push bugs (Approximation, summaries may occasionally vary)

= Mozharness deployments - Merges to the production branch of the mozharness repository are live immediately. Please see that repo for push history.

Puppet deployments

Merges to the production branch of the puppet repository are automatically deployed by Puppet within a 1/2 hour. Please see that repo for push history.

Mobile talos webhost

When	Bug #(s) - Description(s)
20/11/2013	Bug 940968 - archive talos data on bm-remote web hosts before move to scl3 <- updated talos rev to ad927d1aa867

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as mozharness changes.)

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Older changes are archived in: 2013 | 2012 | 2011 | 2010 | 2009

Outcome	When	Bug #(s) - Description(s)
in production	2013-11-15 10:00 PT	bug 938266 - Reimage staging w64-ix-slaves to rev2 bug 937771 - Run linux64-br-haz builder on b2g-inbound/mozilla-inbound/fx-team/mozilla-central and try-by-default bug 910092 - Testing Android (and B2G) in a VM
in production	2013-11-13 10:20 PT	Backout of bug 931926
in production	2013-11-13 06:52 PT	No Bug, add more slaves to staging. r=armenzg No Bug, remove production slaves from staging. r=test_slave_allocation.py bug 931874 - Schedule jit-tests on try, r=callek bug 931926 - Schedule cpp tests on try, r=callek bug 936582 - delete profiling branch. r=armenzg Update release config for Fennec-26.0b4-build1 Update release config for Firefox-26.0b4-build1 Update release config for Fennec-25.0.1-build1 Update release config for Firefox-24.1.1esr-build1 Update release config for Firefox-25.0.1-build1 bug 931874 *Back out* 071cfc93d1c7 (bug 931874) to avoid having infinite retries on Try bug 937680 - Win64 debug builds on rev2 slaves on try busted cloning, "WindowsError: [Error 3] The system cannot find the path specified: 'e:\\'". r=rail bug 937755 - Make try and try-comm-central branches win64-rev2 only. r=rail bug 937823 - Enabled mixed win64 rev1/rev2 pool on mozilla-central/comm-central. r=rail bug 936222 - clean up old, unused nextSlave functions. r=rail bug 936582 - delete profiling branch. r=armenzg bug 935576 - Use the freshly built b2g desktop in gaia_integration.py instead of downloading a nightly. r=aki bug 937784 - more readable vcs-sync emails. r=hwine bug 936185 - Include desktop shims for gaia profile
in production	2013-11-11 10:10 PT	bug 936207 - browser-chrome chunks are failing to start on cedar bug 936222 - nextSlave should take into account retries and spot instances bug 936548 - Add 'xh' to the Firefox build bug 932504 - Schedule gaia-ui-tests on b2g desktop builds on osx on cedar, pine bug 936518 - browser-chrome chunks aren't passing --browser-chrome bug 933290 - Firefox 28: about pages not localized in multi-locale build bug 933918 - Replace sdk tools directory on Android x86 emulator test slaves bug 918932 - slaveapi doesn't escalate reboots properly when exceptions are thrown bug 922858 - Add method to slaveapi to dump basic slave data
in production	2013-11-08 16:00 PT	bug 934976 - Migrate 80 win64-rev1 machines to win64-rev2: day 4 batch 2. r=hwine
in production	2013-11-08 07:18 PT	nobug - Update release config for Firefox-26.0b3-build1 bug 934976 - Migrate 80 win64-rev1 machines to win64-rev2: day 3 batch 2 and day 4 batch 1. r=rail nobug: Use sets for slavenames, buildernames, etc. to speed up checkconfig. r=rail,jlin *buildbot upgrade*
in production	2013-11-07 09:34 PT	bug 935533 - Prepare infra to handle spot instances. r=catlee bug 933778 - Disable last-good-rev nightlies Firefox builds on the B2G release branches. r=catlee bug 934976 - Migrate 80 win64-rev1 machines to win64-rev2: day 2 batch 2. r=rail bug 923880 - Shut off gaia-unit everywhere except b2g-inbound. r=rail bug 749421 - blobber uploads for marionette. r=aki bug 935214 - Consolidate Android x86 test sets; r=armenzg bug 819963 - Split up mochitest-bc on desktop into 30minute chunks. r=rail / enable bc chunking on cedar bug 936003 - Re-enable b2g desktop tests on mozilla-b2g26, r=jgriffin bug 934976 - Migrate 80 win64-rev1 machines to win64-rev2: day 3 batch 1. r=catlee bug 935645 - Disable nightlies on cedar branch; r=jhopkins bug 819963 - Split up mochitest-bc on desktop into 30minute chunks - patch 1. r=rail bug 916677 - Upload analysis results before checking expectations, r=aki bug 916677 - Turn the build orange on hazard expectation failure, r=aki bug 931838 - Enable kvm for Android x86 emulators; r=armenzg bug 749421 - add blobber gear to marionette script. r=aki bug 935214 - Consolidate Android x86 chunks post-kvm; r=armenzg bug 935406 - vcs-sync: "can't clone" email not followed by a success email. r=hwine
in production	2013-11-06 13:28 PT	bug 934976 - Migrate 80 win64-rev1 machines to win64-rev2: day 2 batch 2. r=rail
in production	2013-11-06 07:31 PT	bug 897420 - adds metro talos suites and enables them on cedar bug 934976 - Migrate 80 win64-rev1 machines to win64-rev2: day 1 batch 2. r=catlee bug 927512 - Disable Windows and OSX talos on b2g26. r=aki bug 934976 - Migrate 80 win64-rev1 machines to win64-rev2: day 2 batch 1. r=rail bug 916677 - unit tests for read_from_file(). r=sfink
in production	2013-11-05 15:20 PT	bug 927512 - Disable Windows and OSX talos on b2g26. bug 932504 - fix botched merge per bug 932504 comment 16 and 17.
in production	2013-11-05 13:15 PT	bug 934976 - Migrate 80 win64-rev1 machines to win64-rev2: day 1 batch 2. r=catlee
in production	2013-11-05 08:05 PT	bug 932962 - Disable PGO of browser builds on the b2g release branches. r=catlee bug 932504 - *backout* Schedule gaia-ui-tests on b2g desktop builds on osx on cedar, pine bug 933778 - Disable last-good-rev nightlies and enable per-product builds on Aurora. r=catlee bug 933938 - *Backout* ab36e83d7fcb for Bug 933938 - Enable aurora nightlies for 27.0a2 bug 913197 followup, update active thunderbird release branches for dev masters bug 892691 - Add HD Panda chassis to android production pandas;r=kmoir bug 932504 - fix b2g osx sendchanges. r=armenzg bug 931809 - Schedule b2g desktop mochitests on all m-c based branches, r=aki nobug - Update release config for Fennec-26.0b2-build1 nobug - Update release config for Firefox-26.0b2-build1 bug 934976 - Migrate 80 win64-rev1 machines to win64-rev2: day 1 batch 1. r=catlee bug 897420 - enables mozprocess with talos + configurable timeout, default 3600s, r=aki bug 929093 - vcs-sync: previous repo status. r=hwine
in production	2013-11-01 11:30 PT	mozharness merge -- should mostly be noop bug 917433 - fix 1.2 updates bug 749421 - quick blobber fixes due to merge & unused imports bug 921596 - pep8-ify mozbase.py. bug 932821 - Android panda mozharness script calls _install_app() unnecessarily for cpp unittests and jit-tests
Backout	2013-11-01 10:20 PT	bug 932504 - set unittest_masters for macosx64_gecko builds, r=aki
in production	2013-11-01 07:32 PT	bug 933411 - designate bm66 as rev2 master, enable mixed pool on m-i and b2g-i. r=rail bug 932988 - Move marionette-webapi tests to EC2. r=aki bug 932504 - set unittest_masters for macosx64_gecko builds, r=aki