ReleaseEngineering/Maintenance

From MozillaWiki
Jump to navigation Jump to search

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our masters.

Relevant repositories

Non-releng repositories involved in the CI

TBPL

Mozharness deployments

Merges to the production branch of the mozharness repository are live immediately. Please see that repo for push history.

Puppet deployments

Merges to the production branch of the puppet repository are automatically deployed by Puppet within a 1/2 hour. Please see that repo for push history.

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as mozharness changes.)

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Older changes are archived in: 2014 | 2013 | 2012 | 2011 | 2010 | 2009

Outcome When Bug #(s) - Description(s)
in production 2014-03-26 12:00 PT
  • bug 916181 - Update firefox-latest, firefox-stub, firefox-latest-euballot bouncer aliases as a part of post-release builder (and same for beta)
  • bug 956631 - [tarako][build]create "tarako" build
  • bug 967704 - "Green up" test failures for Android 2.3 emulator
  • bug 981035 - Enable ASAN tests on the Jetpack tree
  • bug 984930 - Create mochitest-dt and move all mochitests under browser/devtools into that suite
  • bug 985837 - emulator-kk builds crash make half the time (which it describes as "failed to build")
  • bug 986213 - Schedule (mostly) B2G tests on Mozilla-B2g28-v1.3t
  • bug 987759 - Android 2.3 tests not starting on Ash
  • bug 988147 - Increase maxtime for browser-chrome-3
in production 2014-03-25 over the course of the day
  • bug 980120 - Add tooltool support to Windows builds
  • bug 987712 - generate signing token before doing build
  • bug 969988 - ASan is the only Linux platform running the separate jittest suite on try
  • bug 967704 - "Green up" test failures for Android 2.3 emulator
  • bug 986213 - Schedule (mostly) B2G tests on Mozilla-B2g28-v1.3t
  • bug 986904 - Increase free space requirement for Thunderbird l10n builds on central & aurora
  • bug 982225 - Run additional hidden 3 debug mochitest-browser-chrome chunks and b2g reftests
  • bug 787449 - Stop running builds and tests on pushes which only include changes to the b2g or mobile or browser directories
  • bug 981035 - Enable ASAN tests on the Jetpack tree
in production 2014-03-21 08:00 PT
  • bug 739959 - submit release builds to balrog automatically - run balrog release pusher as part of updates step. r=nthomas
  • bug 916181 - Update firefox-latest, firefox-stub, firefox-latest-euballot bouncer aliases as a part of post-release builder
  • bug 985556 - Bump MAX_BROKER_REFS to 4096. r=aki
  • bug 818968 - Enable b2g reftests on EC2 across the board. r=rail
  • bug 739959 - submit release builds to balrog automatically - add testChannels/testChannelRuleIds to beta release configs. r=nthomas
in production 2014-03-20 09:56 PT
  • bug 984923 - allow prioritizeBuilders to process more jobs
  • bug 985600 - Stop setting JAVA_HOME in buildbot and mozharness to a dir that does not exist
  • bug 971926 - Requesting OMTC Reftest (Ro) variant for Windows
  • bug 985710 - more js/src/config/milestone.txt
  • bug 985625 - Releng Config/Automation support of staging run of scl1->scl3 move.
  • bug 978307 - support desktop builders being created from Mozharness and through Buildbot - optimize builder_tracker placement
  • bug 985600 - Stop setting JAVA_HOME in buildbot and mozharness to a dir that does not exist
  • bug 985403 - Desktop marionette tests should install minidump stackwalk
in production 2014-03-19 29:18 PT
  • bug 970918 Fix sendchanges for emulator builds, r=aki
in production 2014-03-19 07:20 PT
  • bug 984705 - turn off inari updates. r=rail
  • bug 978307 - support desktop build builders being created from SigningScriptFactory or Buildbot
  • bug 983262 - Don't pass in json manifest to B2G mochitests if it doesn't exist, r=jmaher
  • bug 920153 - download_file output_timeout. r=catlee
  • bug 983417 - Download minidump_stackwalk for Android 2.3; r=kmoir
  • bug 970918 - b2g build improvements r=aki
in production 2014-03-18 08:55 PT
  • bug 962187 - start up new l10n conversions; r=aki
  • bug 892122 - support limiting slave platforms in test masters r=bhearsum
  • bug 982225 - Increase maxtime for devtools-chrome/bc2 to 100 minutes, r=armenzg
  • bug 984480 - Enable EC2 b2g reftests and Ubuntu EC2 debug mochitest-browser-chrome. r=rail
in production 2014-03-17 11:10 PT
  • bug 983314 - Pass tooltool_script in misc.py
  • bug 983939 - Allow to run retried jobs on spot instances.
  • bug 984392: submissions to balrog shouldn't silently fail
  • bug 962187 - Tracking bug for 17-mar-2014 migration work
  • bug 983269 - turn off mochitest-metro
  • bug 967646 - nexus-4 eng
  • bug 969930: Turn off PGO, talos and debug builds on UX project branch
in production 2014-03-17 06:30 PT
  • bug 970552 - Do not use spot instances for some builders
  • bug 983939 - Allow to run retried jobs on spot instances
  • bug 983941 - Increase the maxtime for browser-chrome to 3 hours and 20 minutes
in production 2014-03-15 07:21 PT
in production 2014-03-14 11:16 PT
  • bug 956631 - v1.3t b2g bumper configs
  • bug 983731 - Change periodic build internal on m-c to 3hr and disable lastgood nightly builds.
  • bug 976415 - don't flunk on failure
  • bug 976415 - Make sure we can read the file before reading it
in production 2014-03-14 08:05 PT
  • bug 983662 - Fix erroneous manifest path. r=armenzg
  • bug 982225 - Bump bc2 10 more minutes. r=jmaher
  • bug 979564 - Bump number of spot nodes to 999
  • bug 977579 - Please schedule linux jit-tests in two chunks - add platform configs;
  • bug 980279 - Rename cppunittest to cppunit in mobile_config.py; r=bhearsum
  • bug 818968 - Enable b2g reftests on EC2 for m-i. r=bhearsum
  • bug 956631 - Support 'skip_gaia_json' in config for branches we don't need a gaia.json file for. r=nthomas
  • bug 967630 - Add an option to trychooser to select Talos profiling options, r=jlund
  • bug 920153 - add an hg pull timeout. r=jgriffin
in production 2014-03-12 14:09 PT
  • bug 962187 - Tracking bug for 17-mar-2014 migration work. r=jhopkins
  • bug 975131 - migrate remaining w64-ix-slaves to rev2. r=bhearsum
  • bug 956631 - enable tarako-eng build on mozilla-b2g28_v1_3t, r=catlee

Note: bug 980120 was landed and then backed out and should be a no-op.

in production 2014-03-12 10:12 PT
  • bug 967704 - Use android23.json for Android 2.3 mochitests; r=armenzg
  • bug 979962 - Make Gi tests get node modules from git mirror, r=gaye
  • bug 956631 - [tarako][build]create "tarako" build, r=nthomas
  • bug 981777 - Stop running mochitest-browser-chrome on B2G branches. r=bhearsum
  • bug 981777 - Simplify our code while we deprecate Fedora jobs merge day note. r=bhearsum
  • bug 981777 - Reduce mochitest browser-chrome timeouts. r=jmaher
  • bug 837017 - Disable debug mochitest browser chrome for Elm. r=rail
  • bug 982225 - Run side-by-side debug mochitest-browser-chrome on m-i on EC2 test machines. r=rail
  • bug 975131 - day8batch1 -> rev2. r=bhearsum
  • bug 977298 - Use python -u for Android 2.2 Talos tests; r=jmaher
  • bug 975131 - remove disabled_branches, mixed_branches logic for w64-ix-slave rev2's. r=bhearsum
in production 2014-03-10 13:30 PT
  • bug 979450 enable emulator-kk builds r=mgerva
  • bug 980976 - buildbot-master82 doesn't have rev2 machines enabled for a bunch of branches. r=rail
  • bug 962187 - Tracking bug for 17-mar-2014 migration work. r=nthomas
  • bug 837017 - Run debug-mochitest-browser-chrome on Elm and streamline Jetpack in our configs. r=rail
  • bug 752836 - Don't build Thunderbird for Instantbird pushes. r=catlee
  • bug 980890 disable skip-spot logic to get builds going again. r=bustage
  • nobug: don't classify slaves that don't have names (or connected slave objects).
  • bug 979440 - we can't call get_output_from_command() from inside setup_mock(), r=catlee
  • bug 978512 - prep buildbot.py for desktop builds in mozharness, r=armenzg
in production 2014-03-10 07:38 PT
  • bug 968200 - Clean up the download directory before running tests on Android
in production 2014-03-07 09:05 PT
  • bug 980976 - buildbot-master82 doesn't have rev2 machines enabled for a bunch of branches
in production 2014-03-06 12:10 PT
  • Backed out changeset 55fec886d16d
  • Update release config for Thunderbird-28.0b1-build1
  • backout 86027af6a98f: * bug - Bug 979440
  • merge backout 02c74aa31b51 commit -> defaul
  • bug 977579 - Please schedule linux jit-tests in two chunks
  • bug 979440 - we can't call get_output_from_command() from inside setup_mock()
  • bug 979450 - Add emulator-kk to bumper.
  • bug 979584 - Increase timeout for Gaia UI emulator builds on buildbot.
  • bug 979963 - Fix binary path for gaia integration tests
  • bug 979963 - Fix binary path for gaia unit tests
  • bug 979963 - Install B2G using mozinstall
  • bug 979963 - binary -> binary_path
  • bug 980091 - , Increase free space requirement for linux debug builds to 15G
in production 2014-03-04 14:40 PT
  • bug 793989 - Self-serve should be able to request arbitrary builds on a push
  • Backout bug 846104 for breaking new nightly requests
in production 2014-03-04 13:36 PT
  • bug 976822 - Define config file for gaia-integration tests to use on OSX,
  • bug 976822 - Update config to work on OSX,
  • bug 977420 - Change birch config and add jacuzzis.
  • bug 979076 - b2g bumper should check tree status.
  • bug 979080 - Clear cache of device manifests if we fail to complete the push loop.
  • Update release config for Fennec-28.0b8-build1
  • Update release config for Firefox-28.0b8-build1
in production 2014-03-03 10:20 PT
  • bug 977579 - Please schedule linux jit-tests in two chunks; r=jlund <- mozharness
  • bug 978233 - emulator-jb should use --manifest instead of --test-manifest, r=jgriffin <- mozharness
  • bug 972835 - use tbirdbld account to submit thunderbird data to balrog. r=nthomas
  • bug 977702 - Remove --no-update, r=ahal
  • bug 977420 - remove nightlies and b2g builds from birch. r=rail
  • bug 966257 - Enable periodic b2g device builds on mozilla-inbound r=bhearsum
  • nobug - Update release config for Firefox-28.0b7-build1
  • bug 977420 - remove tests from birch. r=rail
  • bug 969988 - ASan is the only Linux platform running the separate jittest suite on try; r=bhearsum
  • bug 972835 - use tbirdbld account to submit thunderbird data to balrog. r=nthomas <- buildbotcustom
  • bug 748800 - replace TuxedoEntrySubmitterFactory with a mozharness script. r=nthomas <- buildbotcustom
in production 2014-02-27 08:08 PT
  • bug 942167 - Enable non-unified android{-debug} builds
  • bug 961314 - Switch to in-tree mozconfig and switch to tooltool.
  • bug 964438 - Update mozharness scripts with new marionette errors,
  • bug 965588 - Remove --no-update from B2G mozharness scripts,
  • bug 976158 - Tweak buildername for Android l10n nightlies to make consistent with other multi-part jobs;
  • bug 976822 - Schedule gaia-integration on OSX on cedar,
  • bug 977298 - Use -u when calling Talos runtests.py;
  • bug 977420 - Change birch config and add jacuzzis.
  • bug 924476 - s/qa-drivers@mozilla.org/qa-drivers@mozilla.com/g' per QA request.
in production 2014-02-26 0654 PT
  • bug 973063 - Use proper manifest path in emulator-jb mochitests, r=armenzg
  • bug 965588 - Remove --no-update from B2G script invocation, r=ahal
  • nobug - Update release config for Fennec-28.0b6-build1
  • nobug - Update release config for Firefox-28.0b6-build1
  • bug 947426 - Remove LINUX_VMS - r=armenzg
  • bug 976629 - Remove linux-opt arm-sim builder, r=bhearsum
  • bug 748800 - replace TuxedoEntrySubmitterFactory with a mozharness script. r=nthomas
  • bug 970738 - URL encode builder names. r=bhearsum
  • bug 748800 - replace TuxedoEntrySubmitterFactory with a mozharness script. r=nthomas
  • bug 976251 - Use 'make node_modules', r=gaye
  • bug 910689 - Add comment for original fetch url. r=rail
  • bug 973545 - Add -timeout 10000 to gaia-ui-tests, r=zac
  • bug 972659 - Add git dict to b2g_bumper, r=catlee
  • bug 973545 - s/append/extend
  • bug 967704 - Enable blobber for Android 2.3; r=armenzg
  • bug 966441 - Set -b option for mozbase unit tests; r=jlund
in production 2014-02-25 0820 PT
  • bug 973004 - deploy newer startTalos.bat for Windows machines
in production 2014-02-24 1113 PT
  • bug 975511 - Replace dev-master01 for dev-master1. r=bhearsum
  • bug 975496 - Use MozillaPulse from pypi. r=bhearsum
  • bug 974166 - Allow more flexible control of project try_by_default setting, r=bhearsum
  • bug 974166 - Allow the possibility of spidermonkey builds on more platforms, r=bhearsum
  • bug 965447 - Run arm-sim build by default on try, r=bhearsum
  • bug 970738 Integrate jacuzzi allocator with prioritizeBuilders r=bhearsum
  • bug 973063 - Schedule multimedia b2g mochitests for emulator-jb on Cedar. r=Callek
  • bug 974166 - Make spidermonkey try_by_default into a dict, r=bhearsum
  • bug 970738 - Jacuzzi implementation for nextSlave r=bhearsum
  • bug 973491 - port MockReset to mozharness, r=rail
  • bug 973492 - allow for multiple basedirs to be passed to PurgeMixin.purge_builds(), r=armenzg
  • bug 973493 - allow mozharness's tbox_print_summary() to be used outside of unittests, r=armenzg
  • bug 974777 - add --dump-config and --dump-config-hierarchy to mozharness, r=rail
  • bug 974777 - add --dump-config and --dump-config-hierarchy to mozharness -- nosetests, r=rail
in production 2014-02-21 0715 PT
  • bug 974622 - Configure holly like Aurora properly
  • bug 803764 - Run fuzzers on newer CentOS 6.x slaves instead of CentOS 5 ones
  • bug 973063 - Schedule multimedia b2g mochitests for emulator-jb on Cedar
  • bug 972415 - Intermittent Linux spot builder "command timed out: 2700 seconds without output, attempting to kill" while trying to install mock
  • bug 970432 - Do nightly builds with profiling disabled
  • bug 899969 - mozilla-central should contain a pointer to the revision of all external B2G repos, not just gaia
  • bug 974093 - Save qemu.log to blobber
  • bug 974082 - Don't reset mock environments if we don't have to
in production 2014-02-19 0640 PT
  • bug 795793 - decomm w64-ix-slave30
  • bug 960569 - Please decomm talos-r3-fed64-058
  • bug 753367 - decomm talos-r3-fed-057
  • bug 851270 - Don't clobber the source checkout
  • bug 974140 - release builds not using space requirement of platform config
  • bug 972415 - Intermittent Linux spot builder "command timed out: 1800 seconds without output, attempting to kill" while trying to install mock
  • bug 966441 - Run mozbase unit tests from test package
  • bug 974093 - Save qemu.log to blobber
in production 2014-02-14 07:45 PT
  • bug 971687 - Add config file specifications for mozbase tests; r=bhearsum
  • bug 971841 - [mq]: ant-apache-regex
  • bug 967704 - Chunk more the Android 2.3 test jobs. r=armenzg
in production 2014-02-13 14:00 PT
in production 2014-02-13 07:30 PT
  • bug 965447 - Add a new "arm-sim" spidermonkey variant, r=bhearsum
  • bug 969360 - Disable Firefox nightlies on b2g26_v1_2. r=aki
  • bug 971687 - Please schedule mozbase unit tests on Cedar; r=bhearsum
  • bug 971841 - install ant on try builders. r=bhearsum
  • bug 970552 - Do not use spot instances for some builders. r=catlee
in production 2014-02-11 07:30 PT
  • bug 967337 - Only run non-unified builds on trunk. r=catlee
  • bug 968545 - Remove nanojit support r=bhearsum
  • bug 969930 - Disable PGO builds and Talos on UX, and reduce priority to 5. r=aki
  • bug 969542 - Change b2g device builds to run periodically. r=rail
  • bug 970491 - Reset Fig and Elm. r=hwine
  • bug 818968 - Configure Elm to run b2g emulator jobs on Fedora and Ubuntu EC2 machines. r=rail
  • bug 968545 - remove nanojit support r=bhearsum
  • bug 969542 - Add periodic build support by platform r=rail
in production 2014-02-10 06:21 PT
  • no bug - Update release config for Firefox-28.0b1-build2
  • bug 967704 - (1) Run more mochitest and crashtest chunks on Android 2.3; r=armenzg
  • bug 967084 - Make Pine use mozharness production
Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/Maintenance&oldid=957654"