Confirmed users
3,990
edits
m (remove incorrect mentions of beta 4) |
|||
(24 intermediate revisions by 3 users not shown) | |||
Line 4: | Line 4: | ||
= Bugs hit = | = Bugs hit = | ||
The following are the list of bugs that we expect to hit or that we did not expect to hit. For the next release some of this will be already fixed. | The following are the list of bugs that we expect to hit or that we did not expect to hit. For the next release some of this will be already fixed. | ||
* {{bug|662803}} - Major update automation should support multiple "from" | |||
* We forgot that beta4 was not shipped and update verify failed | |||
These two required the manual work for major updates: | |||
* {{bug|662803}} - Major update automation should support multiple "from" versions | |||
* {{bug|662298}} - major update automation should be able to be run independently, or as part of the "to" release | * {{bug|662298}} - major update automation should be able to be run independently, or as part of the "to" release | ||
Also check the "[[Release:Release_Automation_on_Mercurial/Known_Issues|known issues]]" page. | Also check the "[[Release:Release_Automation_on_Mercurial/Known_Issues|known issues]]" page. | ||
== For postmortem == | |||
The whole release was rushed up (go to build later on EDT) and expectations of releasing on next afternoon was not explicit. If it was known I could have started things earlier. | |||
=Build Engineers= | =Build Engineers= | ||
Line 24: | Line 32: | ||
|- | |- | ||
| rowspan="2" | 1 | | rowspan="2" | 1 | ||
| | | GECKO50_2011060814_RELBRANCH | ||
| | | [http://hg.mozilla.org/releases/mozilla-beta/rev/cf9c20531b17 cf9c20531b17] | ||
|- | |- | ||
| | | FIREFOX_5_0b5_RELEASE FIREFOX_5_0b5_BUILD1 | ||
| | | [http://hg.mozilla.org/releases/mozilla-beta/rev/cf9c20531b17 cf9c20531b17] | ||
|} | |} | ||
Line 41: | Line 49: | ||
| rowspan="4" | 1 | | rowspan="4" | 1 | ||
| Linux | | Linux | ||
| rowspan="4" | | | rowspan="4" | 20110608151458 | ||
| | | mv-moz2-linux-ix-slave13 | ||
| | | 59 mins, 35 secs | ||
|- | |- | ||
| Linux64 | | Linux64 | ||
| | | moz2-linux64-slave02 | ||
| | | 1 hrs, 7 mins, 45 secs | ||
|- | |- | ||
| Mac | | Mac | ||
| | | moz2-darwin10-slave48 | ||
| | | 4 hrs, 9 mins, 3 secs | ||
|- | |- | ||
| Windows | | Windows | ||
| | | mw32-ix-slave17 | ||
| | | 2 hrs, 42 mins, 39 secs | ||
|} | |} | ||
Line 68: | Line 76: | ||
[cltbld@production-master03 builder_master]$ echo 6 > reserved_slaves_pm03 | [cltbld@production-master03 builder_master]$ echo 6 > reserved_slaves_pm03 | ||
</pre> | </pre> | ||
* Hit "ship it" button on "Firefox 5 Beta Build | * Hit "ship it" button on "Firefox 5 Beta Build 5"</i> (https://l10n-stage-sj.mozilla.org/shipping/milestones) | ||
* Landed configs, merged configs & custom to production. | * Landed configs, merged configs & custom to production. | ||
* Tagged buildbot-configs, buildbotcustom, and tools with | * Tagged buildbot-configs, buildbotcustom, and tools with FIREFOX_5_0b5_RELEASE and FIREFOX_5_0b5_BUILD1 | ||
* Update pm03 checkouts & reconfigured | * Update pm03 checkouts & reconfigured | ||
* Ran release_sanity dry run with: | * Ran release_sanity dry run with: | ||
Line 78: | Line 86: | ||
=== Tag === | === Tag === | ||
Worked fine. | |||
=== Start Autosign === | === Start Autosign === | ||
Line 102: | Line 111: | ||
=== Bouncer Submitter === | === Bouncer Submitter === | ||
Worked fine. | |||
=== Source === | === Source === | ||
Worked fine. | |||
=== Build/Repack === | === Build/Repack === | ||
* Linux: No problems | |||
* Linux64: No problems | |||
* Windows: No problems | |||
* Mac: No problems | |||
=== Reset reserved_slaves === | === Reset reserved_slaves === | ||
[cltbld@production-master03 builder_master]$ pwd | |||
/builds/buildbot/builder_master | |||
[cltbld@production-master03 builder_master]$ echo 0 > reserved_slaves_pm03 | |||
=== Signing === | === Signing === | ||
No problems. Closed signcodepwd.exe after signing finished. Got "signature verification succeeded" mail a few hours later. | |||
=== L10n Verify === | === L10n Verify === | ||
A few legitimate looking string changes. | |||
=== Updates (5.0b[1-4] -> 5.0b5) === | === Updates (5.0b[1-4] -> 5.0b5) === | ||
No problems. | |||
==== Update verify ==== | |||
Beta4 had not been shipped and therefore update verify failed. | |||
Process stderr: | |||
http://stage-old.mozilla.org/pub/mozilla.org//firefox/releases/5.0b4/linux-i686/af/firefox-5.0b4.tar.bz2: | |||
22:10:02 ERROR 404: Not Found. | |||
= | * We landed a switch from [https://bugzilla.mozilla.org/attachment.cgi?id=538260 releases/5.0b4 to nightly/5.0b4-candidates/build1]. | ||
* Tagged with FIREFOX_5_0b5_{RELEASE,BUILD1}_RUNTIME. | |||
* Re-build each one of the failing jobs | |||
It all looks green now. | |||
=== Updates (4.0many -> 5.0b5) === | === Updates (4.0many -> 5.0b5) === | ||
Manually bumped [https://bugzilla.mozilla.org/attachment.cgi?id=538274 patcher config] and [https://bugzilla.mozilla.org/attachment.cgi?id=537843 update verify config]. | |||
Bumped FIREFOX_4_0_1_RELEASE tag on buildbot-configs, FIREFOX_4_0_1_{BUILD1,RELEASE}_RUNTIME on tools. | |||
Generation of updates was done on mv-moz2-linux-ix-slave13, using the UPDATE_PACKAGING_R14 tag, and re-using the updates directory from the 5.0b[12] -> 5.0b5 updates run, to save time & energy. | |||
Disabled the slave from slavealloc and gracefully shut it down. | |||
<pre> | |||
cd /builds/slave/rel-beta-updates/build/patcher-configs | |||
cvs up | |||
cd .. | |||
perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/moz20-branch-major-update-patcher2.cfg | |||
cd temp/firefox | |||
ln -s 5.0b5 4.0.1-5.0b5 | |||
cd ../.. | |||
perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/moz20-branch-major-update-patcher2.cfg | |||
# remove partials for 4.0.1 | |||
cd temp/firefox/4.0.1-5.0b5 | |||
find aus2{,.test} -name partial.txt -exec rm {} \; | |||
# check releasetest == beta | |||
bash /builds/slave/rel-beta-updates//tools/release/compare-channel-snippets.sh aus2 beta aus2.test releasetest | |||
# push up to aus | |||
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-5.0b5-build1-4.0.1-build1-test | |||
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-5.0b5-build1-4.0.1-build1 | |||
# push test snippets live | |||
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/backupsnip Firefox-5.0b5-build1-4.0.1-build1-test' | |||
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/pushsnip Firefox-5.0b5-build1-4.0.1-build1-test' | |||
</pre> | |||
==== Update Verify ==== | ==== Update Verify ==== | ||
To force all of the verify builders, used the following script (warning: not OS X compatible): | |||
<pre> | |||
for p in linux linux64 macosx64 win32; do | |||
for i in `seq 1 10`; do | |||
curl "http://production-master1.build.sjc1.mozilla.com:8010/builders/release-mozilla-2.0-${p}_major_update_verify_${i}%2F10/force" >/dev/null | |||
done | |||
done | |||
</pre> | |||
Differences in all platforms: | |||
<pre> | |||
diff -r source/firefox/defaults/pref/channel-prefs.js target/firefox/defaults/pref/channel-prefs.js | |||
1,2c1,2 | |||
< //@line 2 "/builds/slave/rel-2.0-lnx-bld/build/browser/app/profile/channel-prefs.js" | |||
< pref("app.update.channel", "release"); | |||
--- | |||
> //@line 2 "/builds/slave/rel-beta-lnx-bld/build/browser/app/profile/channel-prefs.js" | |||
> pref("app.update.channel", "beta"); | |||
WARN: non-binary files found in diff | |||
</pre> | |||
Reviewing this GREEN logs are so verbose. How could this be improved? | |||
=== Channel switching snippets === | === Channel switching snippets === | ||
We'll use the 4.0.1 snippets for this, which has all of the locales we care about, plus a (harmless) few extra. Like 5.0b2, we put them in a bogus buildID of 30000000000000 to keep the channel switching snippets separate from more specific ones. | |||
NOTE: For this to work you have to trigger the MU first. | |||
<pre> | |||
# cltbld@aus2-staging | |||
cd /opt/aus2/snippets/staging | |||
mkdir -p Firefox-5.0b5-build1-channel-switching/Firefox/ | |||
rsync -a Firefox-5.0b5-build1-4.0.1-build1/Firefox/4.0.1 Firefox-5.0b5-build1-channel-switching/Firefox/ | |||
cd Firefox-5.0b5-build1-channel-switching/Firefox/ | |||
mv 4.0.1 5.0 | |||
for d in 5.0/*; do | |||
mv -v $d/* $d/30000000000000 | |||
done | |||
</pre> | |||
=== Permissions check and A/V Scan === | === Permissions check and A/V Scan === | ||
No problems. | |||
=== A/V Notification === | === A/V Notification === | ||
No problems. | |||
=== Push Files === | === Push Files === | ||
As in 5.0b2, we'll be pushing to a few internal mirrors rather than the whole network. To set this up, added the following to stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude: | |||
- firefox/releases/5.0b5 | |||
...and waited a few minutes, to ensure that the rsync servers have the file. | |||
Then, forced the "release-mozilla-beta-push_to_mirrors" builder with the following properties: | |||
release_config: mozilla/release-firefox-mozilla-beta.py | |||
script_repo_revision: FIREFOX_5_0b5_RELEASE | |||
=== Final verify === | === Final verify === | ||
Triggered "release-mozilla-beta-$PLATFORM_final_verification" without any problems. | |||
Using internal mirrors was so fast!! | |||
=== Push snippets === | === Push snippets === | ||
"Firefox-5.0b5-build1" and "Firefox-5.0b5-build1-channel-switching" both only have a "5.0" version in them, so we only need one backupsnip between them. "Firefox-5.0b5-build1-4.0.1-build1" has 4.0 versions, and needs its own. As cltbld@aus2-staging, did the following for backups: | |||
<pre> | |||
cd /opt/aus2/snippets/staging | |||
~/bin/backupsnip Firefox-5.0b5-build1 | |||
~/bin/backupsnip Firefox-5.0b5-build1-4.0.1-build1 | |||
</pre> | |||
And for the pushes: | |||
<pre> | |||
~/bin/pushsnip Firefox-5.0b5-build1 | |||
~/bin/pushsnip Firefox-5.0b5-build1-channel-switching | |||
~/bin/pushsnip Firefox-5.0b5-build1-4.0.1-build1 | |||
</pre> |