Bugs hit

• bug 855289 - something needs to retry network operations during buildbot master reconfigs
• bug 855291 - ship it should handle reconfig/sendchange failures differently than release sanity ones

Build 1

Checklist

• Ship the l10n milestone ^{DESKTOP MOBILE RELEASE BETA}
• Submit to Ship It ^{DESKTOP MOBILE RELEASE BETA ESR}
• Push to mirrors ^{DESKTOP RELEASE}
• Publish Fennec ^{MOBILE RELEASE BETA}
• Push Fennec to mirrors ^{MOBILE RELEASE BETA}
• Apply Throttling ^{DESKTOP RELEASE}
• Verify throttling ^{DESKTOP RELEASE}
• Run pushsnip ^{DESKTOP RELEASE BETA ESR}
• Update bouncer links ^{DESKTOP BETA RELEASE}
• Update XULRunner wiki pages ^{DESKTOP RELEASE}
• Post-release tasks ^{DESKTOP RELEASE BETA ESR}
• Get the Browser Choice website updated ^{DESKTOP RELEASE}

Notes

L10n milestone creation and ship it submission done by lsblakk. Review done by nthomas/bhearsum.

Recover from errors during reconfig

During the reconfig done by release runner a few masters hit ISE 500 errors, eg:

[buildbot-master13.build.scl1.mozilla.com] out: abort: HTTP Error 500: Internal Server Error 

This caused the releases to be marked as failed on ship it. Because this error happened after configs had been committed and pushed, I commented out those sections of release runner before marking the releases as ready again. ~cltbld/release-runner/tools/buildfarm/release/release-runner.py:

diff -r 8304eecca5d7 buildfarm/release/release-runner.py
--- a/buildfarm/release/release-runner.py	Fri Mar 22 13:08:59 2013 -0400
+++ b/buildfarm/release/release-runner.py	Wed Mar 27 07:17:01 2013 -0700
@@ -341,19 +341,19 @@
         # for all releases to minimize the chance of bad configs being
         # pushed. apply_and_push calls process_configs, and if it returns
         # successfully, it pushes all of the changes that it made.
-        apply_and_push(configs_workdir, configs_pushRepo, process_configs,
-                       ssh_username=hg_username, ssh_key=hg_ssh_key)
+        #apply_and_push(configs_workdir, configs_pushRepo, process_configs,
+        #               ssh_username=hg_username, ssh_key=hg_ssh_key)
 
         # Now that we know that all of the configs are good, we can tag
         # the other repositories
-        for release in rr.new_releases:
-            rr.update_status(release, 'Tagging other repositories')
-        tag_repo(workdir=custom_workdir, branch=buildbotcustom_branch,
-                 tags=tags, force=force_tag, pushRepo=custom_pushRepo,
-                 hg_username=hg_username, hg_ssh_key=hg_ssh_key)
-        tag_repo(workdir=tools_workdir, branch=tools_branch, tags=tags,
-                 force=force_tag, pushRepo=tools_pushRepo,
-                 hg_username=hg_username, hg_ssh_key=hg_ssh_key)
+        #for release in rr.new_releases:
+        #    rr.update_status(release, 'Tagging other repositories')
+        #tag_repo(workdir=custom_workdir, branch=buildbotcustom_branch,
+        #         tags=tags, force=force_tag, pushRepo=custom_pushRepo,
+        #         hg_username=hg_username, hg_ssh_key=hg_ssh_key)
+        #tag_repo(workdir=tools_workdir, branch=tools_branch, tags=tags,
+        #         force=force_tag, pushRepo=tools_pushRepo,
+        #         hg_username=hg_username, hg_ssh_key=hg_ssh_key)
         for release in rr.new_releases:
             rr.update_status(release, 'Reconfiging masters')

Then ran 'supervisorctl restart releaserunner' as root to pick up the changes.

That reconfig also failed, this time because of a code problem due to a bad backout of bug 813613. I fixed that by hand, repushed, and then marked the releases as ready again. This time things proceeded smoothly.

After the releases were started I reverted my local change to release runner and restarted it with supervisorctl once again.

Final verification errors

HTTP 500 responses returned for

http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/20.0/update/linux-x86_64/bn-BD/firefox-16.0.2-20.0.partial.mar
http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/20.0/update/linux-i686/sv-SE/firefox-16.0.2-20.0.partial.mar

Only 2 in all of final verification. Rebuilt those jobs, no errors.

Throttling

Landed throttling patch prior to pushing snippets. Verified by manual repeated requests to https://aus3.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Darwin_x86_64-gcc3-u-i386-x86_64/ta/release/update.xml, and noting that a small percentage of them got updates.