Ship the l10n milestone DESKTOP MOBILE RELEASE BETA
- Submit to Ship It DESKTOP MOBILE RELEASE BETA ESR
- Publish Fennec MOBILE RELEASE BETA
- Push Fennec to mirrors MOBILE RELEASE BETA
- Run pushsnip DESKTOP RELEASE BETA ESR
- Post-release tasks DESKTOP RELEASE BETA ESR
RelMan submitted this release and tried to start it. It failed, multistep fix documented below.
Fix mozconfig comparison failures for new android-x86 platform
This platform wasn't present in the mozconfig whitelist at all. This was fixed by https://hg.mozilla.org/build/tools/rev/8f74231fbbae. bug 763903 will get this issue raised at checkin time, rather than release time.
Wrong tools dir used when running release sanity
Despite the above fix for the whitelist, release sanity failed with the same error. We discovered that it was running out of /home/cltbld/release-runner/tools/buildfarm/release, which is release _runners_ workdir. Release sanity should run out of /home/cltbld/release-runner/tools/buildfarm/release/tools/scripts/release. Filed bug 879754. Also filed bug 879755 about improving logging a bit.
Bustage fix for manage_masters.py
manage_masters.py was broken in cases where "-R" was specified after a bad patch I landed yesterday. https://hg.mozilla.org/build/tools/rev/38e4672de124 was the bustage fix landed to fix it.
Get things going again
After all the above were fixed, we needed to run release runner again, but without rebumping the files. To do this, applied this local patch:
diff -r 38e4672de124 buildfarm/release/release-runner.py --- a/buildfarm/release/release-runner.py Wed Jun 05 09:18:41 2013 -0400 +++ b/buildfarm/release/release-runner.py Wed Jun 05 06:29:56 2013 -0700 @@ -345,21 +345,21 @@ # for all releases to minimize the chance of bad configs being # pushed. apply_and_push calls process_configs, and if it returns # successfully, it pushes all of the changes that it made. - apply_and_push(configs_workdir, configs_pushRepo, process_configs, - ssh_username=hg_username, ssh_key=hg_ssh_key) - - # Now that we know that all of the configs are good, we can tag - # the other repositories - for release in rr.new_releases: - rr.update_status(release, 'Tagging other repositories') - tag_repo(workdir=custom_workdir, branch=buildbotcustom_branch, - tags=tags, force=force_tag, pushRepo=custom_pushRepo, - hg_username=hg_username, hg_ssh_key=hg_ssh_key) - tag_repo(workdir=tools_workdir, branch=tools_branch, tags=tags, - force=force_tag, pushRepo=tools_pushRepo, - hg_username=hg_username, hg_ssh_key=hg_ssh_key) - for release in rr.new_releases: - rr.update_status(release, 'Reconfiging masters') +# apply_and_push(configs_workdir, configs_pushRepo, process_configs, +# ssh_username=hg_username, ssh_key=hg_ssh_key) +# +# # Now that we know that all of the configs are good, we can tag +# # the other repositories +# for release in rr.new_releases: +# rr.update_status(release, 'Tagging other repositories') +# tag_repo(workdir=custom_workdir, branch=buildbotcustom_branch, +# tags=tags, force=force_tag, pushRepo=custom_pushRepo, +# hg_username=hg_username, hg_ssh_key=hg_ssh_key) +# tag_repo(workdir=tools_workdir, branch=tools_branch, tags=tags, +# force=force_tag, pushRepo=tools_pushRepo, +# hg_username=hg_username, hg_ssh_key=hg_ssh_key) +# for release in rr.new_releases: +# rr.update_status(release, 'Reconfiging masters') # Reconfig the masters and configure the warning callback, if present. callback = None
...restarted release runner with 'supervisorctl restart releaserunner', and then marked the release ready again. The releases successfully started this time.
Later on, I noticed that there was an extra "release-runner.py" process still running:
root 13268 0.0 0.0 150328 5204 ? Ss Jan30 2:44 /usr/bin/python26 /usr/bin/supervisord cltbld 6902 0.0 0.0 63864 1128 ? S 06:30 0:00 \_ /bin/bash /home/cltbld/release-runner/tools/buildfarm/release/release-runner.sh cltbld 6911 0.5 0.2 165284 13356 ? S 06:30 0:00 \_ python release-runner.py -c /home/cltbld/.release-runner.ini cltbld 6801 0.1 0.2 169132 16988 ? S 06:28 0:00 python release-runner.py -c /home/cltbld/.release-runner.ini
It looks like release runner doesn't die properly when supervisord kills it, filed bug 879763