ReleaseEngineering/Maintenance

From MozillaWiki
Jump to navigation Jump to search

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our masters.

Relevant repositories

Non-releng repositories involved in the CI

TBPL

Mozharness deployments

Merges to the production branch of the mozharness repository are live immediately. Please see that repo for push history.

Puppet deployments

Merges to the production branch of the puppet repository are automatically deployed by Puppet within a 1/2 hour. Please see that repo for push history.

Mobile talos webhost

When Bug #(s) - Description(s)
20/11/2013 Bug 940968 - archive talos data on bm-remote web hosts before move to scl3 <- updated talos rev to ad927d1aa867

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as mozharness changes.)

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Older changes are archived in: 2013 | 2012 | 2011 | 2010 | 2009

Outcome When Bug #(s) - Description(s)
in production 2014-03-03 10:20 PT
  • bug 977579 - Please schedule linux jit-tests in two chunks; r=jlund <- mozharness
  • bug 978233 - emulator-jb should use --manifest instead of --test-manifest, r=jgriffin <- mozharness
  • bug 972835 - use tbirdbld account to submit thunderbird data to balrog. r=nthomas
  • bug 977702 - Remove --no-update, r=ahal
  • bug 977420 - remove nightlies and b2g builds from birch. r=rail
  • bug 966257 - Enable periodic b2g device builds on mozilla-inbound r=bhearsum
  • nobug - Update release config for Firefox-28.0b7-build1
  • bug 977420 - remove tests from birch. r=rail
  • bug 969988 - ASan is the only Linux platform running the separate jittest suite on try; r=bhearsum
  • bug 972835 - use tbirdbld account to submit thunderbird data to balrog. r=nthomas <- buildbotcustom
  • bug 748800 - replace TuxedoEntrySubmitterFactory with a mozharness script. r=nthomas <- buildbotcustom
in production 2014-02-27 08:08 PT
  • bug 942167 - Enable non-unified android{-debug} builds
  • bug 961314 - Switch to in-tree mozconfig and switch to tooltool.
  • bug 964438 - Update mozharness scripts with new marionette errors,
  • bug 965588 - Remove --no-update from B2G mozharness scripts,
  • bug 976158 - Tweak buildername for Android l10n nightlies to make consistent with other multi-part jobs;
  • bug 976822 - Schedule gaia-integration on OSX on cedar,
  • bug 977298 - Use -u when calling Talos runtests.py;
  • bug 977420 - Change birch config and add jacuzzis.
  • bug 924476 - s/qa-drivers@mozilla.org/qa-drivers@mozilla.com/g' per QA request.
in production 2014-02-26 0654 PT
  • bug 973063 - Use proper manifest path in emulator-jb mochitests, r=armenzg
  • bug 965588 - Remove --no-update from B2G script invocation, r=ahal
  • nobug - Update release config for Fennec-28.0b6-build1
  • nobug - Update release config for Firefox-28.0b6-build1
  • bug 947426 - Remove LINUX_VMS - r=armenzg
  • bug 976629 - Remove linux-opt arm-sim builder, r=bhearsum
  • bug 748800 - replace TuxedoEntrySubmitterFactory with a mozharness script. r=nthomas
  • bug 970738 - URL encode builder names. r=bhearsum
  • bug 748800 - replace TuxedoEntrySubmitterFactory with a mozharness script. r=nthomas
  • bug 976251 - Use 'make node_modules', r=gaye
  • bug 910689 - Add comment for original fetch url. r=rail
  • bug 973545 - Add -timeout 10000 to gaia-ui-tests, r=zac
  • bug 972659 - Add git dict to b2g_bumper, r=catlee
  • bug 973545 - s/append/extend
  • bug 967704 - Enable blobber for Android 2.3; r=armenzg
  • bug 966441 - Set -b option for mozbase unit tests; r=jlund
in production 2014-02-25 0820 PT
  • bug 973004 - deploy newer startTalos.bat for Windows machines
in production 2014-02-24 1113 PT
  • bug 975511 - Replace dev-master01 for dev-master1. r=bhearsum
  • bug 975496 - Use MozillaPulse from pypi. r=bhearsum
  • bug 974166 - Allow more flexible control of project try_by_default setting, r=bhearsum
  • bug 974166 - Allow the possibility of spidermonkey builds on more platforms, r=bhearsum
  • bug 965447 - Run arm-sim build by default on try, r=bhearsum
  • bug 970738 Integrate jacuzzi allocator with prioritizeBuilders r=bhearsum
  • bug 973063 - Schedule multimedia b2g mochitests for emulator-jb on Cedar. r=Callek
  • bug 974166 - Make spidermonkey try_by_default into a dict, r=bhearsum
  • bug 970738 - Jacuzzi implementation for nextSlave r=bhearsum
  • bug 973491 - port MockReset to mozharness, r=rail
  • bug 973492 - allow for multiple basedirs to be passed to PurgeMixin.purge_builds(), r=armenzg
  • bug 973493 - allow mozharness's tbox_print_summary() to be used outside of unittests, r=armenzg
  • bug 974777 - add --dump-config and --dump-config-hierarchy to mozharness, r=rail
  • bug 974777 - add --dump-config and --dump-config-hierarchy to mozharness -- nosetests, r=rail
in production 2014-02-21 0715 PT
  • bug 974622 - Configure holly like Aurora properly
  • bug 803764 - Run fuzzers on newer CentOS 6.x slaves instead of CentOS 5 ones
  • bug 973063 - Schedule multimedia b2g mochitests for emulator-jb on Cedar
  • bug 972415 - Intermittent Linux spot builder "command timed out: 2700 seconds without output, attempting to kill" while trying to install mock
  • bug 970432 - Do nightly builds with profiling disabled
  • bug 899969 - mozilla-central should contain a pointer to the revision of all external B2G repos, not just gaia
  • bug 974093 - Save qemu.log to blobber
  • bug 974082 - Don't reset mock environments if we don't have to
in production 2014-02-19 0640 PT
  • bug 795793 - decomm w64-ix-slave30
  • bug 960569 - Please decomm talos-r3-fed64-058
  • bug 753367 - decomm talos-r3-fed-057
  • bug 851270 - Don't clobber the source checkout
  • bug 974140 - release builds not using space requirement of platform config
  • bug 972415 - Intermittent Linux spot builder "command timed out: 1800 seconds without output, attempting to kill" while trying to install mock
  • bug 966441 - Run mozbase unit tests from test package
  • bug 974093 - Save qemu.log to blobber
in production 2014-02-14 07:45 PT
  • bug 971687 - Add config file specifications for mozbase tests; r=bhearsum
  • bug 971841 - [mq]: ant-apache-regex
  • bug 967704 - Chunk more the Android 2.3 test jobs. r=armenzg
in production 2014-02-13 14:00 PT
in production 2014-02-13 07:30 PT
  • bug 965447 - Add a new "arm-sim" spidermonkey variant, r=bhearsum
  • bug 969360 - Disable Firefox nightlies on b2g26_v1_2. r=aki
  • bug 971687 - Please schedule mozbase unit tests on Cedar; r=bhearsum
  • bug 971841 - install ant on try builders. r=bhearsum
  • bug 970552 - Do not use spot instances for some builders. r=catlee
in production 2014-02-11 07:30 PT
  • bug 967337 - Only run non-unified builds on trunk. r=catlee
  • bug 968545 - Remove nanojit support r=bhearsum
  • bug 969930 - Disable PGO builds and Talos on UX, and reduce priority to 5. r=aki
  • bug 969542 - Change b2g device builds to run periodically. r=rail
  • bug 970491 - Reset Fig and Elm. r=hwine
  • bug 818968 - Configure Elm to run b2g emulator jobs on Fedora and Ubuntu EC2 machines. r=rail
  • bug 968545 - remove nanojit support r=bhearsum
  • bug 969542 - Add periodic build support by platform r=rail
in production 2014-02-10 06:21 PT
  • no bug - Update release config for Firefox-28.0b1-build2
  • bug 967704 - (1) Run more mochitest and crashtest chunks on Android 2.3; r=armenzg
  • bug 967084 - Make Pine use mozharness production & switch off non-B2G jobs; r=aki
  • bug 967030 - override mozharness_python for win8_64
  • bug 968247 - Use spot instances for regular builds. r=catlee
  • bug 967976 - Stop setting CCACHE_BASEDIR. r=catlee
  • bug 967030 - allow mozharness_python to be overridden. r=catlee
  • bug 961314 - Pass through tooltool_url_list to mozharness, r=aki
  • bug 920153 - bump default mozharness test timeout to 1800. r=catlee
  • bug 965588 - Remove update_files logic from B2G scripts, r=ahal
  • bug 910092 - split patch-aosp step out, merge two make invocation in android AVD builder, r=gbrown.
  • bug 962863 - update vcs-sync l10n configs to match merge day settings. r=hwine
  • bug 960783 - Add browser.tabs.remote.autostart option for reftest-ipc tests (r=aki)
  • bug 920316, bug 963686 - update gecko.git configs. r=hwine
  • bug 962863 - Add the gaia 1.3 repos as well. r=hwine
  • bug 968797 - Flags passed to jit-test from mozharness should match flags in make check; r=jlund
  • bug 968199 - Add GALLIUM_DRIVER=1 to b2g emulator jobs when running on EC2 test machines. r=aki
  • bug 961048 - add Automation Error: to vcs failed retries. r=catlee
  • bug 920153 - try to fix the test buildbot timeouts. r=jgriffin
  • No bug - Code cleanup. r=me (aki)
  • No bug - Operation PEP8 part 2. r=me (aki)
  • No bug - Moar PEP8. Avoiding external_tools and moz{process,file,info} for easier future upgrades. r=me (aki)
in production 2014-02-07 10:20 PT
in production 2014-02-05 13:00 PT
in production 2014-02-05 11:00 PT
  • bug 968232 - android nightlies failing to submit to balrog because of forbidden domain. r=aki <- bbot-cfgs
in production 2014-02-04 11:00 PT
  • bug 964799 - Migrate mozilla-release and comm-release to win64-rev2. p=jhopkins r=aki <- buildbot-configs
  • nobug - Update release config for Fennec-28.0b1-build1 <- buildbot-configs
  • nobug - Only warn on builder limit if threshold exceeded. r=catlee <- buildbot-configs
  • bug 967030 - run Windows 8 unit tests against 64-bit builds on Date. r=catlee <- buildbot-configs
  • bug 940690 - enable metro talos on release/proj branches - backout dromaeojs from m-c, r=backout <- buildbot-configs
in production 2014-02-03 14:30 PT
in production 2014-01-30 14:45 PT
  • bug 920316 - add b2g28_v1_3 (without removing aurora, yet).
  • bug 965604 - Schedule gaia-unit everywhere.
  • bug 958202 - fota updates for hamachi on 1.1.
  • bug 950784 - Disable automatic HSTS updates on b2g18 because they don't work.
  • bug 962363 - multi nightlies for b2g26, not b2g18.
  • bug 946649 - no_update -> update_files,
  • bug 964527 - Run npm install as a separate step,
  • bug 965608 - lock repo down to the 1.12.9 tag.
  • bug 965445 - Set the tbpl status correctly in web-platform-test script,
  • bug 965519 - add mozprocess v0.15 to mozharness.
in production 2014-01-30 05:53 PT
  • bug 964411 - Create hg share directory for gaia-central for test slaves
in production 2014-01-28 14:55 PT
  • nobug - Update release config for various releases -- buildbot-configs
  • bug 958202 - hamachi and hamachi-eng fota updates. r=rail -- buildbot-configs
  • bug 916120 - Increase b2g desktop reftest chunks to 10, r=aki -- buildbot-configs tip
  • bug 964071 - Make getLatestRev return a long rev, so it compares better with prev revs r=bhearsum -- buildbotcustom tip
  • bug 946649 - Add tooltool_servers to gaia-ui emu config, r=aki -- mozharness
  • bug 953309 - Dump npm-debug.log if it exists, r=ahal -- mozharness
  • bug 958202 - fota updates for hamachi. r=rail -- mozharness tip
in production 2014-01-27 11:30 PT
  • bug 942167 - Disable check tests on nonunified builds. r=aki -- builbotcustom tip
  • bug 962363 - b2g nightlies 2x/day. r=catlee --buildbot-configs
  • nobug - Update release config for Fennec-27.0b9-build1 (2 changesets, I think this was backedout) --buildbot-configs
  • bug 963375 - backout FFx/fennec 27.0b9 build 1 updates; r=bustage --buildbot-configs
  • nobug - Update release config for Fennec-27.0b9-build1 (re-added fixed) --buildbot-configs
  • bug 959709 - Add web-platform-tests to cedar. r=aki --buildbot-configs
  • bug 924476 - add QA leads to release failure notifications; r=rail --buildbot-configs
  • bug 963572 - Schedule b2g desktop reftest-sanity on all trunk branches, r=aki--buildbot-configs
  • bug 943080 - bump release version. r=catlee --buildbot-configs tip
  • bug 961138 - Add support for web-platform-tests to mozharness; r=jgriffin --mozharness
  • bug 946065 - 946065 Move content/ to dom/. r=aki --mozharness
  • nobug - back out d8f601297cb7 --mozharness
  • bug 957768 - Pass the hide subtests option into b2g desktop mochitests, r=jgriffin --mozharness
  • bug 961138 - fix config file. r=configtest.py --mozharness
  • nobug - back out a7182b58a7b3 --mozharness
  • nobug - back out 1c3cd27cbbb5 --mozharness
  • nobug - back out 9e6b874c256b --mozharness
  • bug 958202 - back out ef0cb3ea6ab0 (b2g fota) r=configtest.py --mozharness tip
in production 2014-01-24 18:30 PT
in production 2014-01-23 07:23 PT
  • bug 946649 - Schedule gaia-ui-tests on emulators on cedar, r=aki
  • nobug - Backed out changeset e5abee747f33
  • bug 959521 - Update text for av-vendor email. r=nthomas
  • bug 950784 - Enable automatic HSTS updates for B2G branches based on Gecko 18 and later. r=catlee, f=keeler
  • bug 961205 - Enable blobber for Android x86 tests; r=armenzg
  • bug 933768 - Remove re-purposed machines. r=coop
  • bug 942167 - Enable non-unified builds for win32, macosx opt and debug. r=jhopkins
  • bug 664857 - Have 'plain-reftest' match the reftest alias. r=sfink
  • bug 961207 - Collect full logcats for Android x86 test jobs and save to blobber; r=armenzg
in production 2014-01-21 07:28 PT
  • bug 960674 - Schedule Android 4.2 x86 Opt set S4 (only) on all trunk trees and make them ride the trains. r=Callek
  • bug 960814 - Allow 1800s with no output for Gi jobs, r=aki
  • bug 961575 - No Android reftests running on Aurora/Beta; r=bhearsum
  • bug 933768 - Add b-2008-ix machines to the pool. r=jhopkins
  • nobug - Update release config for Fennec-27.0b8-build1
  • nobug - Update release config for Firefox-27.0b8-build1
  • bug 859162 - Remove Mountain Lion machines that were added incorrectly. r=backout
  • bug 739959 - submit release builds to balrog automatically - don't try to submit android release builds to balrog. r=nthomas
  • bug 961012 - trychooser's -e and -f emails about finished jobs only work for MoCo employees who use their MoCo address, r=catlee
  • bug 739959 - submit release builds to balrog automatically - don't try to submit android release builds to balrog. r=nthomas
  • bug 899969 - Fix b2g bumper configs to really remove darwin projects r=aki
  • Backout bug 957768 for bustage.
  • bug 946649 - Add emulator support to gaiatest mozharness script, r=ahal
  • bug 957768 - Pass the hide subtests option into b2g desktop mochitests, r=jgriffin
  • bug 899969 - Make sure manifests have trailing newline r=rail
  • bug 910092 - support patching AOSP build, forcing qemu to R17, and forcing 2.3.x to "gingerbread" tag
in production 2014-01-17 10:05 PT
  • bug 960323 - Some cypress jobs are run with mozharness default, some production
  • bug 957161 - Schedule b2g desktop Gu on all branches for OSX
  • bug 948744 - b2g wasabi build for 1.3
  • bug 942167 - Please add non-unified builds to mozilla-central
  • nobug - Decomm a bunch of tegras and pandas
  • bug 960594 - Turn off test coalescing on Date branch
  • bug 960326 - Disable debug B2G mochitests on mozilla-aurora
  • nobug - fix some bad usage of nomergeBuilders
  • bug 957768 - Add an option to hide subtest results in Mochitest
in production 2014-01-15 11:45 PT
  • bug 942167 - Please add non-unified builds to mozilla-central
  • bug 950206 - Enable 32-bit Windows builds on Date, test those builds on tst-w64-ec2-XXXX
  • bug 946065 - Flatten content subdirectories
  • bug 960065 - tegra and panda decomm
in production 2014-01-15 08:15 PT
  • bug 929825 - mozharness sendchange hits unicode errors
  • bug 957768 - Add an option to hide subtest results in Mochitest
  • bug 959747 - mozilla-aurora android nightly l10n repacks need fixed branch, revision to show up on tbpl
  • bug 950681 - generate "buri" builds with limited memory for B2G
  • bug 948744 - b2g wasabi build for 1.3
  • bug 929288 - Reconfigure alder builds
  • bug 942167 - Please add non-unified builds to mozilla-central
  • bug 929447 - Please schedule Android 4 reftests on all trunk trees and make them ride the trains
in production 2014-01-14 07:00 PT
  • bug 950681 - generate "buri" builds with limited memory for B2G
  • bug 950206 - Enable 32-bit Windows builds on Date, test those builds on tst-w64-ec2-XXXX
in production (build masters only) 2014-01-13 18:20 PT
  • bug 739959 - submit release builds to balrog automatically - fix up android builds
in production 2014-01-13 09:15 PT
  • Backout bug 950681 - generate "buri" builds with limited memory for B2G
  • bug 739959 - submit release builds to balrog automatically
in production 2014-01-13 06:30 PT
  • bug 950681 - generate "buri" builds with limited memory for B2G
  • bug 957348 - Use python -u flag when running Android unittests
  • bug 940768 - helix-eng builds
  • bug 940768 - fix helix_eng test-masters.sh
  • bug 956672 - Add blobber upload support to gaia-ui-test jobs
  • bug 940161 - Schedule debug B2G mochitests on trunk branches
  • bug 957161 - Schedule gaia-ui tests across all trunk branches for osx b2g desktop
  • bug 958697 - Turn off Firefox desktop builds for select locales
in production 2014-01-09 08:50 PT
  • bug 930897 - Bump mochitest-browser-chrome timeout by 15 minutes. r=RyanVM
  • bug 957629 - Make b2g desktop reftests run reftest-sanity for now, r=aki
backout 2014-01-07 12:15 PT
  • bug 956842 - Add second chunk to b2g desktop mochitests due to log limit reached, r=aki
in production 2014-01-07 07:30 PT
  • bug 951911 - disable gaia-integration on aurora and b2g26. r=rail
  • bug 955447 - Don't consider Win64 debug while looking for a good rev for nightlies, since it's been burning since 2011, r=catlee
  • bug 956035 - Enable blobber on all branches. r=aki
  • bug 787080 - use ffxbld/tbirdbld instead of auspush to upload snippets. r=nthomas
  • bug 942200 - Ensure jit-tests do not run on rev3 minis. r=armenzg
  • bug 956842 - Add second chunk to b2g desktop mochitests due to log limit reached, r=aki
  • bug 956624 - Try valgrind jobs coalesce when they shouldn't, r=catlee
Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/Maintenance&oldid=945040"