Releases/Firefox 3.0.3:BuildNotes

Build Engineers

joduinn
Tracking release bug

Bonsai queries

Updated CVS Tags devmo page.

Tags

Build 1:

Module	Branch	Tag	Pull date
cvsroot/mozilla	HEAD	FIREFOX_3_0_3_BUILD1	2008-08-27 01:31 PDT
l10n/l10n	HEAD	FIREFOX_3_0_3_BUILD1	2008-08-27 01:31 PDT

Build data

Type	Build ID	SHA1	Push date	Build machine
[Windows installer/zip]				fx-win32-1.9-slave2
[Mac compressed]				fx-mac-1.9-slave2
[Linux compressed]				fx-linux-1.9-slave2

Notes

Build 1

Bootstrap Tag: RELEASE_AUTOMATION_M11

Setup before starting:

• Updated master.cfg, bootstrap.cfg. Retagged bootstrap.cfg.
• On the slaves:
  • ran 'DISPLAY=:0 xhost +' to make sure linux AliveTest works.
  • remove /builds/verify/firefox-3.0.1
  • update tinderbox to RELEASE_AUTOMATION_M11
    • For win32:

cd /e/builds/tinderbox/mozilla/tools/tinderbox
cvs up -r RELEASE_AUTOMATION_M11
for dir in fx19rel fx19l10nrel xr19rel; do 
  cd /e/$dir; 
  for f in `ls *.pl *.html INSTALL README | grep -v tinder-config.pl`; do 
    cp -pv /e/builds/tinderbox/mozilla/tools/tinderbox/$f .;
  done
  cp -pv post-mozilla-rel.pl post-mozilla.pl
done

• • On the linux slave, removed:
    • /builds/updates/firefox-3.0.1 and /builds/updates/3.0.2-test
    • /builds/source/*
    • /data/cltbld/thunderbird-3.0a2*
  • On the mac slave, removed:
    • /builds/tinderbox/Tb-Mozilla1.9-Release/Darwin_8.8.4_Depend
    • /builds/tinderbox/Tb-Mozilla1.9-l10n-Release/Darwin_8.8.4_Depend
    • /builds/tinderbox/Xr-Mozilla1.9-Release/Darwin_8.8.4_Depend

• Space on slaves before starting:
  • fx-linux-1.9-slave2: 11G on /builds
  • fx-mac-1.9-slave2: 24G on /
  • fx-win32-1.9-slave2: 11G on d:, 2G on e: (disk heavy work is done on d, eg update_verify. build/repack is done on e, but mostly just overwrites existing data)

• Gulped, and kicked off automation:

buildbot sendchange --username=joduinn --master=localhost:9989 -m"Firefox 3.0.3build1 release" go

Tag

• Automated, no problems.

Source

• Automated, no problems.

Build & Repack

• mac: Automated, no problems.
• linux: build went fine, but then "AliveTest" failed because I missed the DISPLAY setting instruction above. This has not been done since the colo power outage a couple of days ago. Fixed missing DISPLAY, restarted build and this time linux build passed.
• win32: buildbot master crashed out during l10n repack. Manually restarted, patched master config to restart at win32 repack and continue onwards. This worked fine.

Sign

instructions

Signing failed out on 'de' with:

 Error: Invalid timestamp http address
 Error: TimeStampign Failed. Result = 80070001, (-2147024895)

To work-around, did the following (from https://bugzilla.mozilla.org/show_bug.cgi?id=415966#c25):

mkdir signed-build6-partial unsigned-build6-partial
cp signed-build1/*en-US* signed-build1-partial/
cp unsigned-build1/*en-US* unsigned-build1-partial/
# copied everything from 'de' -> 'zh-TW' from unsigned-build1 to unsigned-build1-partial
perl ./signing/sign-release.pl -o signed-build${BUILD}-partial -M dir -d unsigned-build${BUILD}-partial -a ${PRODUCT} --release ${TAG} 2>&1 | tee -a win32_signing_build${BUILD}.log
# to merge the signed builds back together
mv signed-build6-partial/ signed-build6/

...and again when failed on 'fy'.

# then push the builds back to stage (instructions on the signing doc)

Resigning of af->fr (excluding en-US) =

When restarting signing we should've been calling sign-release.pl with '-p' to make sure the en-US doesn't get signed again. Unfortunately, the 3.0.2 build notes (where we first had to do this) did not get updated properly. To correct this, we had to resign af->fr so all locales + en-US have the same bits inside. To do so, did the following:

# delete af -> fr from signed-build1
cd ~/signing-work/firefox-3.0.3
cp signed-build1/*en-US* signed-build1-partial/
cp unsigned-build1/*en-US* unsigned-build1-partial/
# copied everything from af -> fr from unsigned-build1 to unsigned-build1-partial/
# NOTE THE '-p' ON THIS COMMAND
perl ./signing/sign-release.pl -o signed-build${BUILD}-partial -M dir -d unsigned-build${BUILD}-partial -a ${PRODUCT} --release ${TAG} -p 2>&1 | tee -a win32_signing_build${BUILD}.log
# to merge back together
mv signed-build1-partial/* signed-build1/
# to verify all of internals are signed the same, did:
mkdir ~/tmp
cd ~/tmp
cp ~/signing-work/firefox-3.0.3/signed-build1/*.exe .
for i in *.exe; do ~/signing-utils/7-zip/7z x $i && FOO=`md5sum nonlocalized/ firefox.exe | awk '{print $1}'` && echo $i: $FOO | tee -a sums.log && rm -rf nonlocalized localized optional removed-files.log setup.exe; done

NOTE: WE DID NOT SHIP THESE RESIGNED BUILDS

L10nVerify

• Automated, no problems.

Generate Updates

Need to bring people who already updated to 3.0.2 build5 up to build6, so do this (after the automated run)

# connect as cltbld@fx-linux-1.9-slave2
cd /builds/updates/firefox-3.0.2/patcher/temp/firefox/3.0.1-3.0.2
for dir in aus2.test aus2.beta aus2; do
  # copy complete snippets from 3.0.1 into a new dir for 3.0.2
  mkdir -p b5-b6-${dir}/Firefox
  rsync -a --exclude 'partial.txt' ${dir}/Firefox/3.0.1/ b5-b6-${dir}/Firefox/3.0.2/
  # use the buildid's from build5
  mv b5-b6-${dir}/Firefox/3.0.2/Darwin_Universal-gcc3/2008070206  b5-b6-${dir}/Firefox/3.0.2/Darwin_Universal-gcc3/2008090512
  mv b5-b6-${dir}/Firefox/3.0.2/Linux_x86-gcc3/2008070206         b5-b6-${dir}/Firefox/3.0.2/Linux_x86-gcc3/2008090512
  mv b5-b6-${dir}/Firefox/3.0.2/WINNT_x86-msvc/2008070208         b5-b6-${dir}/Firefox/3.0.2/WINNT_x86-msvc/2008090514
done
rsync -a -e 'ssh -i ~/.ssh/aus' b5-b6-aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080919-Firefox-3.0.2build5-build6-test
rsync -a -e 'ssh -i ~/.ssh/aus' b5-b6-aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080919-Firefox-3.0.2build5-build6-beta
rsync -a -e 'ssh -i ~/.ssh/aus' b5-b6-aus2/      cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080919-Firefox-3.0.2build5-build6

Then go to aus2-staging and push 20080919-Firefox-3.0.2build5-build6-test live.

Publish Updates to Test Channels (betatest & releasetest)

• Automated, no problems.

Update Verify

• linux no problems
• win32 expected problems (known checksum dif bug)
• mac expected problems (added/dropped locales)

Stage

• Automated, no problems.

Push to beta channel

Backup on aus2-staging doing:

 # put snippets on beta
 $ sudo su - cltbld
 # make sure using latest version of scripts in mozilla/tools/release/bin/
 $ cd bin
 $ cvs update . 
 $ cd /opt/aus2/snippets/staging
 # note the required parameter must match what will be used with pushsnip below. 
 $  time ~/bin/backupsnip 20080925-Firefox-3.0.3-beta
 Running /bin/tar cfvj /opt/aus2/snippets/backup/20080925-3-pre-20080925-Firefox-3.0.3-beta.tar.bz2 .
 real    37m9.378s
 user    0m45.106s
 sys     1m41.298s
 $
 $ time ~/bin/pushsnip 20080925-Firefox-3.0.3-beta
real    2m15.445s
user    0m0.341s
sys     0m19.679s

Sign Installers

Done manually using these installer-signing-instructions here

• complete stage-merged:

# on stage
cd /data/cltbld/firefox-3.0.3/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

• Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/firefox-3.0.3/stage-merged/
~/bin/checksum-files .

• Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

• Manually added entries to download.mozilla.org

Push to mirrors

• push the stage-merged directory to the releases area:

# on stage
rsync -av /data/cltbld/firefox-3.0.3/stage-merged/ /home/ftp/pub/firefox/releases/3.0.3/

• edit the exclude file /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to add the new release.
  • For this release, as FF3.0.2 was so recent, I've left FF3.0.2 in place as well.

Final Verification

• Verify that releasetest points to valid bouncer links:

# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log

• Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):

grep HTTP quickVerify.log | grep -v 200 | grep -v 302

• First testrun hit:
  • 85 occurrences of 404s
  • 11 occurrences of "500 Internal Server" errors. Disabled two mirror sites (mozilla-west.superbhosting.net, mozilla.xenoncore.net) until we contact them and figure this out.
• Next testrun still had problems with mozilla-east.superbhosting.net. Disabled.
• Next testrun returned totally clean! :-)

• Before pushing final updates,verify that "release" and "releasetest" channel match:

# on aus2-staging
$ cd 20080925-Firefox-3.0.3
$ find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080925-Firefox-3.0.3-test/$a");'
$

Publish Updates to Release Channel

• In case you need to cancel/abort updates, the quickest way to do this is here

While waiting for formal "go", do the following on aus2-staging:

$ cd /opt/aus2/snippets/staging
$ time ~/bin/backupsnip 20080925-Firefox-3.0.3
real    44m41.791s
user    0m43.892s
sys     1m33.340s
$
$ time ~/bin/pushsnip 20080925-Firefox-3.0.3

(exact times lost with sudden need to block updates.)

• QA found problem with url redirect for firstrun page, see bug 457328. This meant that all non en-US builds would get a broken url redirect to the firstrun page.
  • I turned off all updates using instructions above.
  • When redirects were fixed, and I turned back on updates, QA continued to not see updates, until I touched the /opt/aus2/incoming/3 directory. (Caused by bug 430737.) Immediately QA were able to see updates again.
  • I've updated "how to cancel/abort updates" instructions above with this extra step.

Release

• On stage.m.o, update "latest" symlinks

$ cd /pub/mozilla.org/firefox/releases/
$ mv latest-3.0 latest-3.0.old
$ ln -s 3.0.3 latest-3.0
$ rm latest-3.0.old