Releases/Firefox 3.5 Preview/BuildNotes

Build Engineers

nthomas/bhearsum
release tracking bug

Signed-off Revision(s)

Build1: We started from rev:010761f9d36b, at that time the tip. Then landed the version bumps and titlebar/installer tweaks to get rev:46a0538a7ade.

Tags

On release/mozilla-1.9.1:

Build #	Tag	Changeset
1	GECKO191b99_20090604_RELBRANCH	46a0538a7ade
	FIREFOX_3_5b99_BUILD1	0d4036dfff71
2	FIREFOX_3_5b99_BUILD2 FIREFOX_3_5b99_RELEASE	3f775c3bb24a

Build data

Type	Build ID	Build machine
[Windows installer/zip]	Build1:20090605010243 Build2:20090605162636
[Mac compressed]	Build1:20090604233133 Build2:20090605144508
[Linux compressed]	Build1:20090604233331 Build2:20090605144815

Notes

Build 1

• clean up directories on slaves, except {linux,macosx,win32}_repack (this would prove to be a bad idea)
• patch buildbot-configs/mozilla2/release_config.py, get review, land
• checkin latest copy of l10n-changesets from tracking bug
• reconfig production-master
• do sendchange

buildbot sendchange --username=nthomas --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5 Preview (b99) build1" goforit

Tag

No problems

Source

No problems

Build/Repack

No problems with en-US, then any slave which did a 3.5b4 locale failed to build this release. In short, they end up with a mozilla-1.9.1 repo on the GECKO191b4_20090423_RELBRANCH, then we hg up -r default, but then can't resolve the FIREFOX_3_5b99_RELEASE tag.

The *_repack dirs were then removed from every slave, the automation patched to make tag, source, en-US builds a no-op by doing a 5 second wait. FIXME: Need an efficient way to clean up the slaves, particularly windows

Get review, land, reconfig master, sendchange:

buildbot sendchange --username=nthomas --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5 Preview (b99) build1 - l10n restart" goforit

Removed .../build1/en-US.xpi, which is not handled properly until bug 485860 is fixed.'

7 locales failed out with ssh errors:

ssh_exchange_identification: Connection closed by remote host

Uploaded fy-NL, si, fi, zh-TW, de, is, gu-IN by hand with:

cd /e/builds/moz2_slave/win32_repack/build/mozilla-1.9.1/browser/locales
make upload AB_CD=$locale MOZ_MAKE_COMPLETE_MAR=1 MOZ_PKG_PRETTYNAMES=1 MOZ_PKG_VERSION=3.5b99 UPLOAD_HOST=stage.mozilla.org UPLOAD_SSH_KEY="~/.ssh/ffxbld_dsa" UPLOAD_TO_TEMP=1 UPLOAD_USER=ffxbld POST_UPLOAD_CMD="post_upload.py -p firefox -v 3.5b99 -n 1 --release-to-candidates-dir"

Sign

instructions While signing was running a blocker was found that caused a respin, bug 496515

Build 2

• Merged all the code changes to the relbranch (changeset)
• Cleaned up $platform_repack builddirs on all production slaves. Also cleaned up 'tag' dir on moz2-linux-slave23 and 'source' dir on slave22.
• Landed a config bump for build2 that also backed out the dummy factories that were landed when we had to kick the automation to get l10n to work.
• Kicked off automation with:

buildbot sendchange --username=bhearsum --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5 Preview (b99) build2" mfbt

• forgot to reconfig before doing the sendchange, had to kill every single l10n build, reclean the directories, and start again
• Kicked off automation again with:

buildbot sendchange --username=bhearsum --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5 Preview (b99) build2" mfbt

Tag

No problems

Source

No problems

Build/Repack

• No problems on Linux
• No problems on en-US Mac.
• One Mac locale failed to unpack the en-US DMG file, and then a subsequent one failed beacuse of my mucking around. To fix, had to build and upload them both manually, like so:

# moz2-darwin9-slave07
cd /builds/slave/macosx_repack/build/mozilla-1.9.1/browser/locales
make installers-el MOZ_MAKE_COMPLETE_MAR=1 MOZ_PKG_PRETTYNAMES=1 MOZ_PKG_VERSION=3.5b99 ZIP_IN=/builds/slave/macosx_repack/build/mozilla-1.9.1/firefox.dmg
make l10n-upload-el MOZ_MAKE_COMPLETE_MAR=1 MOZ_PKG_PRETTYNAMES=1 MOZ_PKG_VERSION=3.5b99 POST_UPLOAD_CMD="post_upload.py -p firefox -v 3.5b99 -n 2 --release-to-candidates-dir" UPLOAD_HOST=stage.mozilla.org UPLOAD_SSH_KEY=~/.ssh/ffxbld_dsa UPLOAD_TO_TEMP=1 UPLOAD_USER=ffxbld
make installers-sk MOZ_MAKE_COMPLETE_MAR=1 MOZ_PKG_PRETTYNAMES=1 MOZ_PKG_VERSION=3.5b99 ZIP_IN=/builds/slave/macosx_repack/build/mozilla-1.9.1/firefox.dmg
make l10n-upload-sk MOZ_MAKE_COMPLETE_MAR=1 MOZ_PKG_PRETTYNAMES=1 MOZ_PKG_VERSION=3.5b99 POST_UPLOAD_CMD="post_upload.py -p firefox -v 3.5b99 -n 2 --release-to-candidates-dir" UPLOAD_HOST=stage.mozilla.org UPLOAD_SSH_KEY=~/.ssh/ffxbld_dsa UPLOAD_TO_TEMP=1 UPLOAD_USER=ffxbld

• No problems with win32.

Sign

instructions

• Blat failed because of the office move. Had to s/mail.mozilla.com/smtp.mozilla.org/ in the registry. Restarted with:

make sign stage verify fake-upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} TAG=${TAG} REPO=${REPO} EMAIL=${EMAIL}

• Removed en-US.xpi, and the entry for that in the two SUMS files
• Uploaded

make upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} TAG=${TAG} REPO=${REPO} EMAIL=${EMAIL}

• Uploaded signing log by writing upload-log target (bug 496841) and doing

make upload-log PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} TAG=${TAG} REPO=${REPO} EMAIL=${EMAIL}

• bug 497041: FtpPoller is looking in the wrong place, from twistd.log:

2009-06-07 17:31:08-0700 [-] Polling dir http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/3.5b4-candidates/build1/

• Manually triggered off l10n_verify and updates

L10nVerify

Failed due to usage of 3.1b4 in config instead of 3.5b4, committed a fix and restarted.

Second attempt looks ok. 'Only in' output matches shipped-locales changes. The general classes of differences are

• bug 488936 moved the locale info from updater.ini to a new file called update.locale, eg

diff -r firefox-3.5b99-build2/diffs/linux-i686.af.diff firefox-3.5b4-build1/diffs/linux-i686.af.diff
193,197d192
< diff -r source/firefox/update.locale target/firefox/update.locale
< 1c1
< < en-US
< ---
< > af
204a200,203
> 9c9
> < Locale=en-US
> ---
> > Locale=af

• changes to updater, crashreporter.ini and so on show up directly in this log
• some locales changed the searchplugins they shipped
• Path to buildbot slaves differs across pool, recorded in js comment

< > //@line 36 "/builds/moz2_slave/linux_repack/build/releases/l10n-mozilla-1.9.1/oc/browser/firefox-l10n.js"
---
> > //@line 36 "/builds/slave/linux_repack/build/releases/l10n-mozilla-1.9.1/oc/browser/firefox-l10n.js"

Generate updates and push betatest snippets

Failed due to usage of 3.1b4 in config instead of 3.5b4, committed a fix and restarted. No problems with second run.

Update verify

Essentially a pass mark, with the following FAIL's detected

• mn and tr missed this release, so there is no update path from 3.5b4 and older
• ml missed 3.5b4, and patcher doesn't know how to generate a 3.1b3 -> 3.5b99 path
• win32 has the usual reports about freebl3.chk & softokn3.chk

Update Bouncer

Done

Create working 3.5b4 snippets

bug 496917 has the details on this. Basically, 'appv=3.5 Beta 99' in snippets broke in 3.5b4. To fix, we need to switch it to 'appv=3.5b99'. Here's how:

# cltbld@aus2-staging.mozilla.org
cd ~/
rsync -av /opt/aus2/snippets/staging/20090607-Firefox-3.5b99/ 20090607-Firefox-3.5b99-3.5b4-fix
cd 20090608-Firefox-3.5b99-3.5b4-fix/Firefox
rm -rf 3.1a1 3.1a2 3.1b1 3.1b2 3.1b3
find . -type f -exec sed -i -e 's/3.5 Beta 99/3.5b99/' {} \;
# this next command should print out a bunch of 'appv=3.5b99' lines
find . -type f -exec grep appv {} \;
cd ../../

# now the test channel
rsync -av /opt/aus2/snippets/staging/20090607-Firefox-3.5b99-test/ 20090608-Firefox-3.5b99-test-3.5b4-fix
cd 20090608-Firefox-3.5b99-test-3.5b4-fix/Firefox
rm -rf 3.1a1 3.1a2 3.1b1 3.1b2 3.1b3
find . -type f -exec sed -i -e 's/3.5 Beta 99/3.5b99/' {} \;
find . -type f -exec sed -i -e 's/3.5b99 (build 2)/3.5b99/' {} \;
# this next command should print out a bunch of 'appv=3.5b99' lines
find . -type f -exec grep appv {} \;

# now rsync these snippets over to the staging area
cd ~
rsync -av 20090608-Firefox-3.5b99-3.5b4-fix /opt/aus2/snippets/staging/
rsync -av 20090608-Firefox-3.5b99-test-3.5b4-fix /opt/aus2/snippets/staging/

# backupsnip before repushing the test snippets
~/bin/backupsnip 20090608-Firefox-3.5b99-test-3.5b4-fix
~/bin/pushsnip 20090608-Firefox-3.5b99-test-3.5b4-fix

Final checks before push

Each of the find calls should return no output.

cd /pub/mozilla.org/firefox/nightly/3.5b99-candidates/build2/
find . ! -user ffxbld
find . ! -group firefox
find . -type f ! -perm 644
find . -type d -mindepth 1 ! -perm 755 ! -name 'contrib*'
find . -type d -maxdepth 1 ! -perm 2775  -name 'contrib*' 

Got some Solaris builds in contrib/ for the user check, which are OK.

Push to mirrors

# root@stage
mkdir /home/ftp/pub/firefox/releases/3.5b99/
chown ffxbld /home/ftp/pub/firefox/releases/3.5b99/

FIXME: need to figure out a better solution than this directory creation.

# ffxbld@stage
rsync -av --exclude=*.log --exclude=*.txt --exclude=*unsigned* \
 /pub/mozilla.org/firefox/nightly/3.5b99-candidates/build2/ \
 /pub/mozilla.org/firefox/releases/3.5b99/

Update the mozilla-current rsync module and latest-3.5 symlink (created this time)

# cltbld@stage
nano /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude
# replace 3.5b5 with 3.5b99.

cd /pub/mozilla.org/firefox/releases/
ln -s 3.5b99 latest-3.5

Final verification

Use 'Force Build to start the final_verification builder, chase down any 503 or 404 errors to find bad mirrors. Get the same warnings as update verify (mn & tr missing 3.5b4 update, ml getting nothing from 3.1b3).

Publish updates to beta channel

~/bin/backupsnip 20090607-Firefox-3.5b99
~/bin/pushsnip 20090607-Firefox-3.5b99
~/bin/pushsnip 20090608-Firefox-3.5b99-3.5b4-fix

The additional pushsnip is for "Create working 3.5b4 snippets" above.