Releases/Firefox 3.0.3:BuildNotes
Contents
- 1 Build Engineers
- 2 Bonsai queries
- 3 Tags
- 4 Build data
- 5 Notes
- 5.1 Build 1
- 5.1.1 Tag
- 5.1.2 Source
- 5.1.3 Build & Repack
- 5.1.4 Sign
- 5.1.5 L10nVerify
- 5.1.6 Generate Updates
- 5.1.7 Publish Updates to Test Channels (betatest & releasetest)
- 5.1.8 Update Verify
- 5.1.9 Stage
- 5.1.10 Push to beta channel
- 5.1.11 Sign Installers
- 5.1.12 Update Bouncer
- 5.1.13 Push to mirrors
- 5.1.14 Final Verification
- 5.1.15 Publish Updates to Release Channel
- 5.1.16 Release
- 5.1 Build 1
Build Engineers
joduinn
Tracking release bug
Bonsai queries
Updated CVS Tags devmo page.
Tags
Build 1:
Module | Branch | Tag | Pull date |
cvsroot/mozilla | HEAD | FIREFOX_3_0_3_BUILD1 | 2008-08-27 01:31 PDT |
l10n/l10n | HEAD | FIREFOX_3_0_3_BUILD1 | 2008-08-27 01:31 PDT |
Build data
Type | Build ID | SHA1 | Push date | Build machine |
[Windows installer/zip] | fx-win32-1.9-slave2 | |||
[Mac compressed] | fx-mac-1.9-slave2 | |||
[Linux compressed] | fx-linux-1.9-slave2 |
Notes
Build 1
Bootstrap Tag: RELEASE_AUTOMATION_M11
Setup before starting:
- Updated master.cfg, bootstrap.cfg. Retagged bootstrap.cfg.
- On the slaves:
- ran 'DISPLAY=:0 xhost +' to make sure linux AliveTest works.
- remove /builds/verify/firefox-3.0.1
- update tinderbox to RELEASE_AUTOMATION_M11
- For win32:
cd /e/builds/tinderbox/mozilla/tools/tinderbox cvs up -r RELEASE_AUTOMATION_M11 for dir in fx19rel fx19l10nrel xr19rel; do cd /e/$dir; for f in `ls *.pl *.html INSTALL README | grep -v tinder-config.pl`; do cp -pv /e/builds/tinderbox/mozilla/tools/tinderbox/$f .; done cp -pv post-mozilla-rel.pl post-mozilla.pl done
- On the linux slave, removed:
- /builds/updates/firefox-3.0.1 and /builds/updates/3.0.2-test
- /builds/source/*
- /data/cltbld/thunderbird-3.0a2*
- On the mac slave, removed:
- /builds/tinderbox/Tb-Mozilla1.9-Release/Darwin_8.8.4_Depend
- /builds/tinderbox/Tb-Mozilla1.9-l10n-Release/Darwin_8.8.4_Depend
- /builds/tinderbox/Xr-Mozilla1.9-Release/Darwin_8.8.4_Depend
- On the linux slave, removed:
- Space on slaves before starting:
- fx-linux-1.9-slave2: 11G on /builds
- fx-mac-1.9-slave2: 24G on /
- fx-win32-1.9-slave2: 11G on d:, 2G on e: (disk heavy work is done on d, eg update_verify. build/repack is done on e, but mostly just overwrites existing data)
- Gulped, and kicked off automation:
buildbot sendchange --username=joduinn --master=localhost:9989 -m"Firefox 3.0.3build1 release" go
Tag
- Automated, no problems.
Source
- Automated, no problems.
Build & Repack
- mac: Automated, no problems.
- linux: build went fine, but then "AliveTest" failed because I missed the DISPLAY setting instruction above. This has not been done since the colo power outage a couple of days ago. Fixed missing DISPLAY, restarted build and this time linux build passed.
- win32: buildbot master crashed out during l10n repack. Manually restarted, patched master config to restart at win32 repack and continue onwards. This worked fine.
Sign
Signing failed out on 'de' with:
Error: Invalid timestamp http address Error: TimeStampign Failed. Result = 80070001, (-2147024895)
To work-around, did the following (from https://bugzilla.mozilla.org/show_bug.cgi?id=415966#c25):
mkdir signed-build6-partial unsigned-build6-partial cp signed-build1/*en-US* signed-build1-partial/ cp unsigned-build1/*en-US* unsigned-build1-partial/ # copied everything from 'de' -> 'zh-TW' from unsigned-build1 to unsigned-build1-partial perl ./signing/sign-release.pl -o signed-build${BUILD}-partial -M dir -d unsigned-build${BUILD}-partial -a ${PRODUCT} --release ${TAG} 2>&1 | tee -a win32_signing_build${BUILD}.log # to merge the signed builds back together mv signed-build6-partial/ signed-build6/
...and again when failed on 'fy'.
# then push the builds back to stage (instructions on the signing doc)
Resigning of af->fr (excluding en-US) =
When restarting signing we should've been calling sign-release.pl with '-p' to make sure the en-US doesn't get signed again. Unfortunately, the 3.0.2 build notes (where we first had to do this) did not get updated properly. To correct this, we had to resign af->fr so all locales + en-US have the same bits inside. To do so, did the following:
# delete af -> fr from signed-build1 cd ~/signing-work/firefox-3.0.3 cp signed-build1/*en-US* signed-build1-partial/ cp unsigned-build1/*en-US* unsigned-build1-partial/ # copied everything from af -> fr from unsigned-build1 to unsigned-build1-partial/ # NOTE THE '-p' ON THIS COMMAND perl ./signing/sign-release.pl -o signed-build${BUILD}-partial -M dir -d unsigned-build${BUILD}-partial -a ${PRODUCT} --release ${TAG} -p 2>&1 | tee -a win32_signing_build${BUILD}.log # to merge back together mv signed-build1-partial/* signed-build1/ # to verify all of internals are signed the same, did: mkdir ~/tmp cd ~/tmp cp ~/signing-work/firefox-3.0.3/signed-build1/*.exe . for i in *.exe; do ~/signing-utils/7-zip/7z x $i && FOO=`md5sum nonlocalized/ firefox.exe | awk '{print $1}'` && echo $i: $FOO | tee -a sums.log && rm -rf nonlocalized localized optional removed-files.log setup.exe; done
NOTE: WE DID NOT SHIP THESE RESIGNED BUILDS
L10nVerify
- Automated, no problems.
Generate Updates
Need to bring people who already updated to 3.0.2 build5 up to build6, so do this (after the automated run)
# connect as cltbld@fx-linux-1.9-slave2 cd /builds/updates/firefox-3.0.2/patcher/temp/firefox/3.0.1-3.0.2 for dir in aus2.test aus2.beta aus2; do # copy complete snippets from 3.0.1 into a new dir for 3.0.2 mkdir -p b5-b6-${dir}/Firefox rsync -a --exclude 'partial.txt' ${dir}/Firefox/3.0.1/ b5-b6-${dir}/Firefox/3.0.2/ # use the buildid's from build5 mv b5-b6-${dir}/Firefox/3.0.2/Darwin_Universal-gcc3/2008070206 b5-b6-${dir}/Firefox/3.0.2/Darwin_Universal-gcc3/2008090512 mv b5-b6-${dir}/Firefox/3.0.2/Linux_x86-gcc3/2008070206 b5-b6-${dir}/Firefox/3.0.2/Linux_x86-gcc3/2008090512 mv b5-b6-${dir}/Firefox/3.0.2/WINNT_x86-msvc/2008070208 b5-b6-${dir}/Firefox/3.0.2/WINNT_x86-msvc/2008090514 done rsync -a -e 'ssh -i ~/.ssh/aus' b5-b6-aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080919-Firefox-3.0.2build5-build6-test rsync -a -e 'ssh -i ~/.ssh/aus' b5-b6-aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080919-Firefox-3.0.2build5-build6-beta rsync -a -e 'ssh -i ~/.ssh/aus' b5-b6-aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080919-Firefox-3.0.2build5-build6
Then go to aus2-staging and push 20080919-Firefox-3.0.2build5-build6-test live.
Publish Updates to Test Channels (betatest & releasetest)
- Automated, no problems.
Update Verify
- linux no problems
- win32 expected problems (known checksum dif bug)
- mac expected problems (added/dropped locales)
Stage
- Automated, no problems.
Push to beta channel
Backup on aus2-staging doing:
# put snippets on beta $ sudo su - cltbld # make sure using latest version of scripts in mozilla/tools/release/bin/ $ cd bin $ cvs update . $ cd /opt/aus2/snippets/staging # note the required parameter must match what will be used with pushsnip below. $ time ~/bin/backupsnip 20080925-Firefox-3.0.3-beta Running /bin/tar cfvj /opt/aus2/snippets/backup/20080925-3-pre-20080925-Firefox-3.0.3-beta.tar.bz2 . real 37m9.378s user 0m45.106s sys 1m41.298s $ $ time ~/bin/pushsnip 20080925-Firefox-3.0.3-beta real 2m15.445s user 0m0.341s sys 0m19.679s
Sign Installers
Done manually using these installer-signing-instructions here
- complete stage-merged:
# on stage cd /data/cltbld/firefox-3.0.3/ rsync -av batch1/mar/ stage-merged/ rsync -av batch1/stage-signed/ stage-merged/
- Create MD5 and SHA1 checksum files
# on stage cd /data/cltbld/firefox-3.0.3/stage-merged/ ~/bin/checksum-files .
- Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox . chmod 644 *SUMS
Update Bouncer
- Manually added entries to download.mozilla.org
Push to mirrors
- push the stage-merged directory to the releases area:
# on stage rsync -av /data/cltbld/firefox-3.0.3/stage-merged/ /home/ftp/pub/firefox/releases/3.0.3/
- edit the exclude file /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to add the new release.
- For this release, as FF3.0.2 was so recent, I've left FF3.0.2 in place as well.
Final Verification
- Verify that releasetest points to valid bouncer links:
# this can be run from anywhere cvs co mozilla/testing/release cd mozilla/testing/release/updates cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg ./verify.sh -t update.cfg 2>&1 | tee quickVerify.log
- Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
- First testrun hit:
- 85 occurrences of 404s
- 11 occurrences of "500 Internal Server" errors. Disabled two mirror sites (mozilla-west.superbhosting.net, mozilla.xenoncore.net) until we contact them and figure this out.
- Next testrun still had problems with mozilla-east.superbhosting.net. Disabled.
- Next testrun returned totally clean! :-)
- Before pushing final updates,verify that "release" and "releasetest" channel match:
# on aus2-staging $ cd 20080925-Firefox-3.0.3 $ find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080925-Firefox-3.0.3-test/$a");' $
Publish Updates to Release Channel
- In case you need to cancel/abort updates, the quickest way to do this is here
While waiting for formal "go", do the following on aus2-staging:
$ cd /opt/aus2/snippets/staging $ time ~/bin/backupsnip 20080925-Firefox-3.0.3 real 44m41.791s user 0m43.892s sys 1m33.340s $ $ time ~/bin/pushsnip 20080925-Firefox-3.0.3
(exact times lost with sudden need to block updates.)
- QA found problem with url redirect for firstrun page, see bug 457328. This meant that all non en-US builds would get a broken url redirect to the firstrun page.
- I turned off all updates using instructions above.
- When redirects were fixed, and I turned back on updates, QA continued to not see updates, until I touched the /opt/aus2/incoming/3 directory. (Caused by bug 430737.) Immediately QA were able to see updates again.
- I've updated "how to cancel/abort updates" instructions above with this extra step.
Release
- On stage.m.o, update "latest" symlinks
$ cd /pub/mozilla.org/firefox/releases/ $ mv latest-3.0 latest-3.0.old $ ln -s 3.0.3 latest-3.0 $ rm latest-3.0.old