User:Callek/Maintenance Test

From MozillaWiki
Jump to: navigation, search

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our masters.

Relevant repositories

Are you changing the tool chain on a master? If so, let auto-tools know so they can update their masters

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems.

Did you change the tool chain on a master? If so, let auto-tools know so they can update their masters

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Older changes are archived in ReleaseEngineering:BuildbotMasterChanges:Archive

 Tools
Callek/Maintenance Test/Reconf/2012-12-20 10:00f58c08e73bb4
Outcome When Bug #(s) - Description(s)
in production 2012-12-20 10:00 pt
  • bug 821401 - turn off make check on b2g18 b2g desktop builds. r=bhearsum
  • bug 823114 - updates not available for android x86 builds r=armenzg
  • bug 822426- Use SpecificNightly scheduler with lastRevFunc for nightly builds where we don't want lastGoodRev r=aki
  • bug 787449 - Exclude 'CLOBBER' from triggering all-product builds. r=catlee
in production 2012-12-19 07:00 pt
  • bug 818191 - Create buildbot master instance on AWS. r=catlee
  • bug 822651: Desktop B2G localiser builds perma-red on fx-team - set gecko_l10n_root for project branches. r=aki
  • bug 822732 - try android x86 builds having ssh key problems r=hwine
  • bug 821137 - add 4 more twigs; r=kmoir
  • bug 822783 - enable builds for projects/date; r=bhearsum
in production 2012-12-17 11:30 pt
  • bug 820592 - buildbot config changes to support Android x86 builds r=armenzg
  • bug 821401 - fix dep b2g multilocale desktop. r=bhearsum
  • bug 821837 - Distribute pandas between B2G and Android. r=kmoir
in production 2012-12-14 16:30 pt
  • bug 821837 - Distribute pandas between B2G and Android.
  • bug 820210 - turn off unagi_stable dep builds; disable lastgood functionality for mozilla-b2g18; enable per-product builds for mozilla-b2g18
  • bug 755781 - Always download symbols for debug tests.
in production 2012-12-13 17:30 PT
in production 2012-12-13 09:51 PT
  • bug 812836 - Set up nightly multilocale B2G desktop builds for dev+l10n testing - re-enable gecko multilocale for desktop builds
in production 2012 12-13 08:20 PT
  • bug 811723 - change android reftests to run with --ignore-window-size for panda only
  • bug 811444 - panda boards magically reboot in the middle of the test
  • bug 812833 - Set up nightly multilocale B2G unagi builds for dev testing
  • bug 818378 - releng changes needed for mozilla-b2g18 branch
  • nobug - Turn off otoro everywhere except mozilla-b2g18
  • bug 812233 - Make use of w64-ix-slave03
in production 2012-12-12 12:35 PT
  • bug 811723 - change android reftests to run with --ignore-window-size for panda only
  • bug 811444 - panda boards magically reboot in the middle of the test.
  • bug 820165 - fix panda-gaia-central target.
  • bug 820958 - increase b2g emulator reftest chunks to 10.
  • bug 820949 - turn off b2g and android-noion on mozilla-beta.
  • bug 815185 - activate mozilla-b2g18 branch;
  • bug 807125 - roll out b2g emulator xpcshell.
in production 2012-12-10 2:45 PT
  • bug 812836 - Set up nightly multilocale B2G desktop builds for dev+l10n testing - disable gecko multilocale for aurora/beta/b2g18
in production 2012-12-10 12:30 PT
  • bug 819326 - Intermittent "command timed out: 5400 seconds elapsed, attempting to kill" during mochitest-bc tests on Windows debug builds
  • bug 812836 - Set up nightly multilocale B2G desktop builds for dev+l10n testing
  • bug 800596 - Add 'an', 'ur' to the Firefox build
  • bug 819356 Elm branch should run builds/tests for all platforms
in production 2012-12-10 8:20 PT
  • bug 817811 - panda builds against gaia-central - add gaia central to the default list of platforms.
  • bug 812833 - Set up nightly multilocale B2G unagi builds for dev testing - enable multilocale device builds on try.
  • bug 818135 - Do b2g nightlies from mozilla-beta at 7am/11pm pacific.
  • bug 815680 - Elm branch should use the whole pool of Win64 builders rather than a subset
  • bug 819381 - Add dev ec2 slaves to configs.
  • bug 818378 - releng changes needed for mozilla-b2g18 branch
  • bug 819793 - Use 6 mochitest chunks for B2G
  • bug 819660 - Add tegra-239 back to slaves list.
in production 2012-12-06 9:00 PT
  • bug 811279 - Download buildstep should cause job to RETRY if wget fails
  • bug 800428 - Failing test: buildbotcustom.test.test_misc_scheduler_propfuncs.TestPropFuncs.test_buildIDSchedFunc
  • bug 812833 - Set up nightly multilocale B2G unagi builds for dev testing
  • bug 802317 - Need Gaia Smoketests in buildbot automation
in production 2012-12-04 6:00 PT
  • bug 817576 - win32 desktop localizer builds fail with "bad file error"
  • bug 812833 - Set up nightly multilocale B2G unagi builds for dev testing
  • bug 817604 - Schedule mozharness b2g xpcshell tests on cedar
  • bug 816237 - Set up staging mozpool of pandas
  • bug 816275 - Take over nightly+beta updates from jgriffin
in production 2012-12-03 10:35 PT
  • bug 812836 - Set up nightly multilocale B2G desktop builds for l10n testing
in production 2012-11-30 07:48 PT
  • backout bug 816793 because it broke hg cloning in mock
in production 2012-11-30 07:10 PT
in production 2012-11-30 06:30 PT
in production 2012-11-29 12:59 PT
  • bug 815124 - Use ftp-scl3.mozilla.com for test downloads instead of ftp.mozilla.org
backed out 2012-11-28 17:15 PT
  • bug 815124 - Use ftp-scl3.mozilla.com for test downloads instead of ftp.mozilla.org
in production 2012-11-28 16:18 PT
  • bug 815124 - Use ftp-scl3.mozilla.com for test downloads instead of ftp.mozilla.org
in production 2012-11-27 20:10 PT
  • bug 815556 - clobberer times out for mozilla-inbound
in production 2012-11-27 11:30 PT
in production 2012-11-26 12:30 PT
in production 2012-11-23 ~14:00 PT
  • bug 810402 - add templates for all release configs - add comments to configs
  • bug 793782 - Stop appending redundant "buildbot.slave.commands.TimeoutError" string
  • bug 804873 - Add python 2.7 ahead of the buildbotve version of python on win32 and win64
  • bug 812966 - Disable Windows 64-bit builds on the UX branch
  • bug 796995 - create mozilla-esr17 branch
in production 2012-11-20 ~14:00 PT
  • bug 812513 - Remove buildsteps that output the slavename, since log_uploader.py's header makes them redundant
  • Backout 625c327617d8 (bug 794467) to help us diagnose bug 811063
  • bug 813432 - b2g on mozilla-beta appears not to have any slaves to build on.
  • bug 813352 - Increase space required for Linux64 nightly builds.
  • no bug - enable b2g tests on mozilla-beta.
  • bug 813728 - Merge on Nov-19-2012 should have uplifted Android-Noion builds+tests to Beta.
in production 2012-11-19 07:55 PT
  • bug 813122 - 11/19: Please disable Aurora updates until merge to mozilla-aurora has stabilized
in production 2012-11-19
  • bug 786550 - Tracking bug for 19-nov-2012 migration work
in production 2012-11-19 12:14 PT
  • bug 798361 - Bug 798361 - Make release automation work with mock slaves
in production 2012-11-15 16:40 PT
  • bug 808536 - ScriptFactory MercurialCloneCommand.
  • bug 811499 - Pass repo_path as a property.
  • bug 793709 - Use SigningScriptFactory if signing servers are present.
  • bug 763929 - create a script/tool that knows how to start a release - create templates for esr10.
  • bug 811783 - change b2g emulator reftest manifest for cedar.
in production 2012-11-13 11:55 PT
  • bug 803664 - Remove screensaver package from talos build slaves
in production 2012-11-08 07:15 PT
  • bug 806799 - Run PGO on elm. r=coop default
  • bug 789335 - XP shell test setup changes due to bug 755724. r=armenzg
in production 2012-11-07 08:00 PT
  • bug 801607 - Make EC2 instances less susceptible to "abort: No space left on device"
  • bug 806799 - Switch custom elm builders over to vs2010 / 8.0 sdk
  • bug 808780 - Schedule mozharness b2g marionette/webapi on mozilla-central
  • bug 800364 - Set up b2g unagi builds in production
  • bug 807540 - Add 100 slots for EC2 machines for a new region
  • bug 801214 - Need Ash for a B2G Automation Staging area for aurora
  • bug 796995 - create mozilla-esr17 branch
  • bug 803530 - dxr-mozilla-central builds failing after Oct 10-12th with "command timed out: 3600 seconds without output"
  • bug 808951 - Panda (Android 4.0) jobs for reftest and jsreftest pending forever
in production 2012-11-06 1100 PT
in production 2012-11-05 1900 PT
  • bug 678504 schedule mozharness b2g marionette/webapi on m-c, project branches.
  • backout 2c4505a257bf since we may reconfig before we're ready
  • bug 803248 fix pandas.
in production 2012-11-05 1500 PT
  • bug 802252 - turn off tdhtml on nov 1.
  • bug 803736 - Remove codesighs.
  • bug 807376 - Give each spidermonkey object a branch name.
  • bug 808085 - schedule b2g emulator reftests on m-c, m-i, try, etc.
  • bug 803248 - buildbot config changes to support panda_android*.
  • bug 802930 - Add a test for the newly discovered -p ics_armv7a_gecko
  • bug 802932 - Refactor some pieces of try_parser.py.
  • bug 802932 - Handle the 'none' value at the outer scope
  • bug 807376 - Fill in the branch name for spidermonkey builder names, and fix the ordering of pieces.
  • bug 800364 - Support for mozharness nightly builds. r=bhearsum
  • bug 807115 - inherit project slaves from their platform.
  • bug 806158 - graph server posts shouldn't haltOnFailure.
in production 2012-11-01 1503 PT
  • bug 807771 - b2g emulator mochitests on m-c based branches; bug 807732 - enable b2g emulator tests on ash.
  • bug 758694 - Adjust PATH for b2g builds so /tools/python/bin is found before /tools/buildbot/bin
  • bug 795206 - Tegra crashes intermittently ending with "OSError: [Errno 8] Exec format error"
in production 2012-11-01
  • bug 807362 - rename emulator_prod_config.py to emulator_automation_config.py
  • bug 807662 - Add gecko version files to be bumped when releasing Thunderbird builds
  • bug 802589 - Reduce underutilized Leopard pool
  • bug 803954 - Install Python 2.7 on l10n slaves
  • bug 807050 - schedule mozharness b2g emulator reftest on cedar
  • bug 802445 - Update Marionette buildbot config to include all trunk trees that merge into m-c
  • bug 683725 - release mail should be threadable
in production 2012-10-30 0630 PT
  • bug 802392 - Fix TB test builder names in test_try_parser.py
  • bug 800364 - Set up b2g unagi builds in production
  • bug 801386 - Please enable PGO on Win64 as bug #795594 is fixed now
  • bug 775355 - Run rooting analysis builds on try server
  • bug 794339 - Linux SpiderMonkey builds failing to find an hg to clone with
  • bug 805914 - Make B2G buildernames more consistent
  • bug 795525 - fix bustage mozharness unittest issues
  • bug 805138 - Use "mochitest-N" across the board rather than "mochitest(s)?-N(\/Z)?"
in production 2012-10-25 1045 PT
  • bug 800925 - update verify config bumper doesn't handle duplicate version (and dropped locales properly).
  • bug 585286 - Screensaver activates sometimes on Rev3 Fedora mochitest machines.
  • bug 803132 - Remove shark build support;
  • bug 804330 - 64-bit Linux Desktop B2G builds
  • bug 803204 - b2g builds should be on mozilla-aurora
  • bug 804763 - decommission vmware slaves.
  • bug 805433 - Increase disk space requirements for b2g_oak_linux32_gecko builds.
  • bug 787449 - Enable per-product builds on mozilla-inbound.
in production 2012-10-24 1415 PT
in production 2012-10-23 1230 PT
  • bug 804026 - Don't use slashes in builders names?
  • bug 785965 - adjust push to mirrors process for CDN
  • bug 802005 - Scheduling of pending test jobs chooses some test suites ahead of others that were scheduled earlier
  • bug 787449 - Stop running builds and tests on pushes which only include changes to the b2g or mobile or browser directories
  • bug 656286 - virus scan e-mails from staging are confusing
in production 2012-10-22 1130 PT
  • bug 653512 - Don't send "updates available on betatest" email when skip_updates is True.
  • bug 799654 - symbol upload should retry upload upon name resolution failure
  • bug 683725 - Make release e-mails threaded.
  • bug 789652 - b2g emulator mochitests on cedar.
  • bug 803954 - Add python27 to PATH on win32
  • bug 803682 - adjust branch priorities for multimaster releases.
in production 2012-10-18 1330 PT
  • bug 800025 - enable emulator webapi tests on cedar.
  • bug 791389 - Split mochitest-browser-chrome out of mochitest-other;
  • bug 778969 - Switch off OS X 10.5 jetpack builds;
  • bug 791492 - Do automated blocklist updates on mozilla-release and mozilla-esr10,
  • bug 802953 - tagging started mail broken in multimaster.
  • bug 800025 - reenable b2g gecko sendchanges.
  • bug 791209 - Fix l10n polling so we build the correct revision for dep repacks. Use in-tree mozconfig for nightly/dep repacks.
  • bug 803080 - fix mock 'chown' step indentation bug.
  • bug 791389 - Split mochitest-browser-chrome out of mochitest-other;
  • bug 794895 - Use the prefix "talosError:" rather than "TinderboxPrint:FAIL:", so Talos failures appear in TBPL's annotated summary;
in production 2012-10-16 0800 PT
  • bug 710312 - enable releases to be run on more than one master
  • bug 737556 - extra email notification sent for fennec "all signed builds available"
  • bug 800429 - Stop trying to test __init__.py
in production (Tegra masters) 2012-10-15 4:00 PDT
  • All Bugs listed in block below
in production (not Tegra masters) 2012-10-15 9:25 PDT
  • bug 799599 - don't use mock slaves for releases until bug 798361 is done - back out release_slaves support now that we're not using it.
  • bug 714313 - Test jobs should happen on random slaves. Only consider successful builds for recent sorting. r=rail,Callek
  • bug 794431 - "ready for release" email can be sent before update verify completes. r=rail
in production (not test masters) 2012-10-13 9:00 PDT
  • bug 801291 - Try b2g builds are red during packaging
in production 2012-10-12 16:50 PDT
  • Reverted bug 799164 - 10/8: Please disable Aurora updates until merge to mozilla-aurora has stabilized
in production 2012-10-12 09:15 PDT
  • bug 799337 - Do talos and periodic PGO on the Profiling branch. r=catlee
  • bug 799400 - Undo previous customizations for Birch, add new customizations, r=catlee
  • bug 798609 - disable android armv6 on mozilla-release. r=callek
  • bug 790741 - Store Gecko builds. r=aki,coop
in production 2012-10-05 12:00 PDT
  • Backout of bug 790741 - Store gecko ICS b2g builds
  • bug 797679 - Force reboot.py for mobile jobs to never retry on failure
  • Disable mock on mozilla-release android
  • bug 787241 - enable android armv6 on mozilla-release (dep + releases) when fx 16 merges in
  • bug 798460 - Remove decommissioned slaves from configs
in production 2012-10-05 7:45 PDT
  • bug 798225 - Turn off codesighs on Linux32 and Linux64 on Try until we can fix it
  • bug 790741 - Store gecko ICS b2g builds. r=armenzg
  • Adjust staging_config.py to look similar to production_config.py. r=rail
in production 2012-10-05 7:00 PDT
  • bug 794378 - Migrate Thunderbird Linux builders to CentOS 6
  • bug 787241 - enable android armv6 on mozilla-release (dep + releases) when fx 16 merges in
in production 2012-10-04 13:30 PDT
  • bug 786516 - Make Thunderbird builders use Pymake by default on Windows
  • bug 786177 - Buildbot config changes to support Mountain Lion - Thunderbird
  • bug 795180 - Add b2g builds for oak twig for tbpl results
  • bug 795516 - Enable Marionette tests on m-c and inbound
in production 2012-10-04 8:30 PDT
  • bug 794431 - "ready for release" email can be sent before update verify completes
in production 2012-10-03 1200 PDT
  • bug 766996 - Initialize rawBytes to avoid UnboundLocalError. r=bhearsum
  • bug 783815 - Tegra Verification steps should not buffer stdout/stderror r=Callek
  • bug 794185 - Uplift mock changes to Firefox 17/Aurora. r=catlee
in production 2012-09-30 1825 PDT
  • bug 795748 - disable pgo on win64, except for try
in production 2012-09-28 1500 PDT
  • bug 795019 - use mozharness/external_scripts/count_and_reboot.py instead of having to pull tools.
  • bug 713017 - mozharness talos.
  • bug 795026 - Fix Android NoIon builds by properly copying over ssh credentials.

Ensure mock_target is set (when using mock) at checkconfig time.

  • bug 794760 - 'clobber build tools' step should haltOnFailure & RETRY;
  • bug 794491 - Try build without 'who' information fails on post_upload.py.
in production 2012-09-27 1840 PDT
in production 2012-09-26 0630 PDT
  • bug 794617 - deploy _dumbwin32proc.py to all XP slaves. Some did not have it.
in production 2012-09-24 1145 PDT
  • bug 779903 - Enable win32-metro nightly builds on Elm
  • bug 792862 - turn down ||ization of update verify. r=nthomas
in production 2012-09-20 0640 PDT
in production 2012-09-13 1640 PDT
in production 2012-09-13 1000 PDT
  • bug 782355 - Add buildbot config for Marionette tests on try only, r=catlee
  • bug 785636 - Set blocklist_update approval by default - r=Callek
  • bug 790510 - Add a fourth chunk to Android reftests, r=aki
  • bug 749151 - correct ThunderbirdTry to Thunderbird-Try. r=Callek
  • bug 785636 - Pass -a option when blocklist_update_set_approval is set - r=Callek
  • bug 790698 - Hack to prevent new batch tegras taking reftests until. r=armenzg Minor Tweak to patch by Callek
in production 2012-09-11 1800 PDT
  • bug 788883 - Release automation should announce signed android builds.
  • bug 772928 - Enable reftests for armv6 builds on tegras
  • Bug ??? - Thunderbird releases don't need to tag partner-repacks repo
  • bug 789474 - update sutagent to 1.13 for tegra automation
in production 2012-09-10 1500 PDT
  • bug 781290 - Remove talos v8 support now that it is included in the dromaeojs suite; r=catlee
in production 2012-09-07 0830 PDT
in production 2012-09-07 0700 PDT
  • bug 788354 - change the xperf test from ts ->tp5n. r=armenzg
in production 2012-09-06 0745 PDT
  • bug 780915 - Remove gingerbread gecko builds. It also fixes the b2g desktop nightly builds. r=armenzg
  • Follow-up to bug 786984 Thunderbird's MozMill builders don't obey trychooser options - fix try-comm-central's broken unit tests by taking account of the branch name in the trychooser itself. r=rail
  • bug 788912 - Bouncer submitter failed for Firefox 15.0 and 15.0.1, r=rail
in production 2012-09-05 0700 PDT
  • bug 781290 - Turn off v8 talos tests on mozilla-inbound. r=rail
  • bug 779192 - Linux builder fails with elfhack, requires CXXABI_1.3.2
  • Fix bug 779192 a bit differently
  • bug 781290 - turn off v8 tests on ionmonkey. r=rail
  • bug 788354 - change the xperf test from ts ->tp5n. r=armenzg
  • bug 787655 - Enables pymake on win64, and removes the exception for the build-system project branch - r=armenzg
  • bug 690672 - Disable coalescing for requests from self-serve; r=catlee

Android Testing

Web Server Cluster

Revision Bug # When Who
97cbf16e9846 bug 797392 20121004 1110 AM PDT armenzg
4c23320b480a bug 768992 20120629 1000 PM PDT armenzg
8a66d9d0c3da bug 768992 20120628 1245 PM PDT armenzg
09e7ed37236b backed out bug 768992 20120628 1000 AM PDT armenzg
8a66d9d0c3da bug 767040 20120622 0630 AM PDT armenzg
7d28f2533eb5
backed out
bug 767040 20120621 1020 AM PDT armenzg
488bc187a3ef bug 753822 20120510 1045 AM PDT armenzg

Find here instructions to see how to update this.

foopy servers

Production

Outcome When Tools Rev Restart CP? Bug #(s) - Description(s)
Pending Deploy Not Yet No
In Production 2012-11-08 ~11:00 PT 40939f2608c6 No bug 809012 - panda boards do not reboot when the sutagent is in a hung state.

bug 808474 - Restore the ability to write a watcher.ini to tegras after reimage.

In Production 2012-10-29 ~3:00 PT 40939f2608c6 No bug 781341 - Develop smoketest to burn in mobile devices

bug 797868 - add a pdu cycle to the sut_tools for reboots instead of dm.reboot. bug 802768 - Cleanup code, begin to add official panda support to sut_tools.

In Production 2012-10-23 ~3:00 PT f25f378dd283 Yes bug 804739 - Update SUTAgent on tegras to 1.14

bug 794743 - Move tegras.json to devices.json

In Production 2012-10-15 12:00 PT bc38e47c0372 Yes bug 799896 - Clientproxy is stopping tegra buildbot instances in some cases where it seems we shouldn't really be.

Clobberer

See ReleaseEngineering/How_To/Modify_clobberer

TryChooser

See Update the try Syntax.

Date Bug Changeset
20121106 0910 IRC/sfink http://hg.mozilla.org/build/tools/rev/0c46d57109fe
20121015 0700 bug 798975 http://hg.mozilla.org/build/tools/rev/795e48c44325
20120918 1315 bug 791426 http://hg.mozilla.org/build/tools/rev/1aee438e4348
20120917 ? bug 791426 http://hg.mozilla.org/build/tools/rev/464dcbdcbc2d