- bug 855289 - something needs to retry network operations during buildbot master reconfigs
- bug 855291 - ship it should handle reconfig/sendchange failures differently than release sanity ones
Ship the l10n milestone DESKTOP MOBILE RELEASE BETA
- Submit to Ship It DESKTOP MOBILE RELEASE BETA ESR
- Push to mirrors DESKTOP RELEASE
- Publish Fennec MOBILE RELEASE BETA
- Push Fennec to mirrors MOBILE RELEASE BETA
- Apply Throttling DESKTOP RELEASE
- Verify throttling DESKTOP RELEASE
- Run pushsnip DESKTOP RELEASE BETA ESR
- Update bouncer links DESKTOP BETA RELEASE
- Update XULRunner wiki pages DESKTOP RELEASE
- Post-release tasks DESKTOP RELEASE BETA ESR
- Get the Browser Choice website updated DESKTOP RELEASE
L10n milestone creation and ship it submission done by lsblakk. Review done by nthomas/bhearsum.
Recover from errors during reconfig
During the reconfig done by release runner a few masters hit ISE 500 errors, eg:
[buildbot-master13.build.scl1.mozilla.com] out: abort: HTTP Error 500: Internal Server Error
This caused the releases to be marked as failed on ship it. Because this error happened after configs had been committed and pushed, I commented out those sections of release runner before marking the releases as ready again. ~cltbld/release-runner/tools/buildfarm/release/release-runner.py:
diff -r 8304eecca5d7 buildfarm/release/release-runner.py --- a/buildfarm/release/release-runner.py Fri Mar 22 13:08:59 2013 -0400 +++ b/buildfarm/release/release-runner.py Wed Mar 27 07:17:01 2013 -0700 @@ -341,19 +341,19 @@ # for all releases to minimize the chance of bad configs being # pushed. apply_and_push calls process_configs, and if it returns # successfully, it pushes all of the changes that it made. - apply_and_push(configs_workdir, configs_pushRepo, process_configs, - ssh_username=hg_username, ssh_key=hg_ssh_key) + #apply_and_push(configs_workdir, configs_pushRepo, process_configs, + # ssh_username=hg_username, ssh_key=hg_ssh_key) # Now that we know that all of the configs are good, we can tag # the other repositories - for release in rr.new_releases: - rr.update_status(release, 'Tagging other repositories') - tag_repo(workdir=custom_workdir, branch=buildbotcustom_branch, - tags=tags, force=force_tag, pushRepo=custom_pushRepo, - hg_username=hg_username, hg_ssh_key=hg_ssh_key) - tag_repo(workdir=tools_workdir, branch=tools_branch, tags=tags, - force=force_tag, pushRepo=tools_pushRepo, - hg_username=hg_username, hg_ssh_key=hg_ssh_key) + #for release in rr.new_releases: + # rr.update_status(release, 'Tagging other repositories') + #tag_repo(workdir=custom_workdir, branch=buildbotcustom_branch, + # tags=tags, force=force_tag, pushRepo=custom_pushRepo, + # hg_username=hg_username, hg_ssh_key=hg_ssh_key) + #tag_repo(workdir=tools_workdir, branch=tools_branch, tags=tags, + # force=force_tag, pushRepo=tools_pushRepo, + # hg_username=hg_username, hg_ssh_key=hg_ssh_key) for release in rr.new_releases: rr.update_status(release, 'Reconfiging masters')
Then ran 'supervisorctl restart releaserunner' as root to pick up the changes.
That reconfig also failed, this time because of a code problem due to a bad backout of bug 813613. I fixed that by hand, repushed, and then marked the releases as ready again. This time things proceeded smoothly.
After the releases were started I reverted my local change to release runner and restarted it with supervisorctl once again.
Final verification errors
HTTP 500 responses returned for
Only 2 in all of final verification. Rebuilt those jobs, no errors.
Landed throttling patch prior to pushing snippets. Verified by manual repeated requests to https://aus3.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Darwin_x86_64-gcc3-u-i386-x86_64/ta/release/update.xml, and noting that a small percentage of them got updates.