Releases/Firefox 3.6.2/BuildNotes

Build Engineers

Signed-off Revision(s)

Build data

Build #	Type	Build ID	Build machine
1	Linux		moz2-linux-slave20
	Mac		bm-xserve11
	Windows		win32-slave42
2	Linux		moz2-linux-slave21
	Mac		moz2-darwin9-slave13
	Windows		win32-slave34
3	Linux	20100316060223	moz2-linux-slave21
	Mac	20100316055951	moz2-darwin9-slave10
	Windows	20100316074819	win32-slave18

Notes

Build 1

Clobbered 1.9.2 on pm with the clobberer page
Landed automation patch
Tag buildbot-configs and buildbotcustom
- hg tag -r 7848b054e13a FIREFOX_3_6_2_BUILD1; hg tag -r 7848b054e13a FIREFOX_3_6_2_RELEASE (buildbot-configs)
- hg tag -r 5e5c12c1e83a FIREFOX_3_6_2_RELEASE; hg tag -r 5e5c12c1e83a FIREFOX_3_6_2_BUILD1 (buildbotcustom)
Triggered the builds

buildbot sendchange --username=armenzg --master=localhost:9010 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.2 build1" Go

I noticed that I used the wrong changeset before Mac and Linux builds were available so I stopped and started a build 2

Build 2

Clobbered 1.9.2 on pm with the clobberer page
Landed build 2 patch
Tagged buildbot-config
- hg tag -r 4b4fad2279f6 FIREFOX_3_6_2_BUILD2
Triggered the builds

buildbot sendchange --username=armenzg --master=localhost:9010 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.2 build2" Go

All builds are available but win32 repacks did not trigger:
- Specific instructions to resume l10n repacks
- python force_release_l10n.py -m http://production-master.build.mozilla.org:8010 -t FIREFOX_3_6_2_RELEASE -v -b releases/mozilla-1.9.2 -p win32 -n armenzg

Build 3

Clobbered 1.9.2 on pm with the clobberer page
bhearsum landed the two missing changesets that had landed after changeset on build1 (cd857b3b0e33) on the release branch
Landed automation patch
Triggered the builds

buildbot sendchange --username=armenzg --master=localhost:9010 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.2 build3" Go

Tag builder finished but did not trigger the rest of the builders
souce, linux_build, mac_build and win32_build were triggered manually
Tagging our repos:
- hg tag -r hg tag -f -r 5b24dd7ca36f FIREFOX_3_6_2_RELEASE; hg tag -r 5b24dd7ca36f FIREFOX_3_6_2_BUILD3 (buildbotcustom)
- hg tag -r 6e420bf20bec FIREFOX_3_6_2_BUILD3; hg tag -r 6e420bf20bec FIREFOX_3_6_2_RELEASE (tools)
- hg tag -r f0db7a70f0eb FIREFOX_3_6_2_BUILD3; hg tag -f -r f0db7a70f0eb FIREFOX_3_6_2_RELEASE (buildbot-configs)
We hit bug in "updates" generation
Updates have been generated
Snipptes have been pushed to beta

Tag

Good

Source

Good
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.6.2-candidates/build3/source/

Build/Repack

One of linux's sendchanges was not timing out so I had to stop it manually
- It failed the step but it did not fail the build
- The sendchange that wasn't timing out was for talos-staging-master02.build.mozilla.org:9010
- The same sendchange was affecting all platforms. I stopped the step in all 3 platforms
To trigger the locales do the following from the master:
- python force_release_l10n.py -m localhost:8010 -t FIREFOX_3_6_2_RELEASE -v -b releases/mozilla-1.9.2 -p linux -n armenzg
- python force_release_l10n.py -m localhost:8010 -t FIREFOX_3_6_2_RELEASE -v -b releases/mozilla-1.9.2 -p macosx -n armenzg
- python force_release_l10n.py -m localhost:8010 -t FIREFOX_3_6_2_RELEASE -v -b releases/mozilla-1.9.2 -p win32 -n armenzg
I retriggered the "id" locale which failed downloading the candidates zip
- http://production-master.build.mozilla.org:8010/builders/win32_repack/builds/2033

--12:11:13--  http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/3.6.2-candidates/build3/unsigned/win32/en-US/firefox-3.6.2.zip
           => `firefox.zip'
Resolving stage.mozilla.org... 10.2.74.116
Connecting to stage.mozilla.org|10.2.74.116|:80... connected.
HTTP request sent, awaiting response... 502 Proxy Error
12:11:21 ERROR 502: Proxy Error.

partner repack step failed to upload the generated builds. Given all the other network failures at the time, this isn't surprising. Re-ran the upload step by hand without issue.

Remove build1/en-US.xpi bug 485860

Unit tests

Make sure that you check to see that all unit tests were green

Sign

I tried to start signing before all-locales were ready (WRONG!)
Before started signing I got an rsync failure and after talking with bhearsum I figured that I need all locales for all platforms

Saving to: `app.tag'

100%[======================================>] 81          --.-K/s   in 0s

2010-03-16 10:40:17 (36.5 MB/s) - `app.tag' saved [81/81]

/home/cltsign/signing-work/firefox-3.6.2/upx/upx.exe --best -o checkouts/stubs/7z/7zSD.sfx.compressed checkouts/stubs/7z/7zSD.sfx
                      Ultimate Packer for eXecutables
   Copyright (C) 1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006
UPX 2.03w       Markus Oberhumer, Laszlo Molnar & John Reiser    Nov 7th 2006

       File size         Ratio      Format      Name
  --------------------   ------   -----------   -----------
   122368 ->     70144   57.32%    win32/pe     7zSD.sfx.compressed

Packed 1 file.
blat -to armenzg@mozilla.com \
            -subject "sign-release.py for firefox 3.6.2 starting" \
                -body "EOM"
Blat v2.5.0 w/GSS encryption (build : Sep 14 2005 22:46:29) 

Sending stdin.txt to armenzg@mozilla.com
Subject: sign-release.py for firefox 3.6.2 starting
Login name is cltsign@cm-keymaster01.mozilla.org
./sign-release.py --keydir d:/2009-keys -j4 --product firefox -o signed-build3 unsigned-build3 >> win32_signing_build3.log 2>&1
# And copy over the info file and XPIs.
cp -pv ./unsigned-build3/unsigned/win32_info.txt ./signed-build3/
`./unsigned-build3/unsigned/win32_info.txt' -> `./signed-build3/win32_info.txt'
rsync -av ./unsigned-build3/unsigned/win32/xpi ./signed-build3/win32
sending incremental file list
rsync: link_stat "/cygdrive/d/signing-work/firefox-3.6.2/unsigned-build3/unsigned/win32/xpi" failed: No such file or directory (2)

sent 18 bytes  received 12 bytes  60.00 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at /home/lapo/packaging/rsync-3.0.4-1/src/rsync-3.0.4/main.c(1040) [sender=3.0.4]
make: *** [sign-files] Error 23

I triggered the missing windows locales
I have to clobber everything on keymaster and start again once all repacks are done
This second time had no complains. Signed builds are currently being uploaded.

Partner Signing

NOTE: For the next release we will probably have bug 554321 fixed and signing repacks will be different than what we did here.

Lots of problems here:

The partner builds generated by the automation are not useful because they cannot be signed with the same internal bits as the rest of the builds.
Coop redid the partner builds against already signed builds. These builds refused to get signed because repacked NSIS installers do that.

Kev provided us with a tarball of internally signed, but not externally signed partner builds. Those were signed with the instructions here.

Resign MD5SUMS and SHA1SUMS

I had removed en-US.xpi after having signed. bug 554455
This is what I did to get this fixed:

# keymaster
mkdir -p ~/signing-work/firefox-3.6.2-remove_en-US.xpi/unsigned
cd ~/signing-work/firefox-3.6.2-remove_en-US.xpi/unsigned
scp -i ~/.ssh/ffxbld_dsa ffxbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.6.2-candidates/build3/{MD5SUMS,SHA1SUMS} .
# modified MD5SUMS and SHA1SUMS
gpg -v -u releases -sba MD5SUMS
gpg -v -u releases -sba SHA1SUMS

scp -i ~/.ssh/ffxbld_dsa * ffxbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.6.2-candidates/build3/{MD5SUMS,SHA1SUMS}

After Nick's approval

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/3.6.2-candidates/build3/
cp MD5SUMS* SHA1SUMS* ../../../releases/3.6.2/

L10nVerify

No problems

Generate updates and push betatest snippets

We hit a bug. Nick helped me with this.
bug 552831 - patcher-config-bump.pl fails when flipping useBeta to True

perl ../tools/release/patcher-config-bump.pl -p firefox -r Firefox -v 3.6.2 -a 3.6.2 -o 3.6rc2 -b 3 -c patcher-configs/moz192-branch-patcher2.cfg -t stage-old.mozilla.org -f ftp.mozilla.org -d download.mozilla.org -l shipped-locales -u
...
Use of uninitialized value in substitution (s///) at ../tools/release/patcher-config-bump.pl line 127.
Use of uninitialized value in pattern match (m//) at ../tools/release/patcher-config-bump.pl line 129.
Use of uninitialized value in pattern match (m//) at ../tools/release/patcher-config-bump.pl line 129.
Use of uninitialized value in concatenation (.) or string at ../tools/release/patcher-config-bump.pl line 140.
ASSERT: BumpFilePath() - Unknown file type for  at ../tools/release/patcher-config-bump.pl line 140.

Once this landed, I deleted moz2-linux-slave12:/builds/slave/updates/* and force the builder
updates generation is completed

Update verify

Waiting on mac_update_verify
mac_update_verify took forever because of the network issues but it finished

Publish updates to beta channel

THU. 18th of March - 6:30 PDT
Backing up snippets

# cltbld@aus2-staging.m.o
# make sure scripts are up to date
cd bin
cvs update
cd /opt/aus2/snippets/staging/
# NOTE: That that backupsnip only backs up the folder you specify and nothing older as it used to
~/bin/backupsnip 20100316-Firefox-3.6.2-beta
~/bin/pushsnip 20100316-Firefox-3.6.2-beta

Socorro product data updates

We can do this ourselves (a subset of relengs, ask nthomas or bhearsum). No need to file an IT bug for this.

Nick added for me the entries.

Added 3.6.2 and 3.6.3pre with the crash-stats admin page.

Update Bouncer

Added the following products:
- Firefox-3.6.2
- Firefox-3.6.2-Complete
- Firefox-3.6.2-Partial-3.6rc2
Added files to bouncer

Final checks before push

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/3.6.2-candidates/build3/
find . ! -user ffxbld #only the solaris contributed builds showed up
find . ! -group firefox
find . -type f ! -perm 644 # unsigned/partner-repacks have 664 permission
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*' # unsigned/partner-repacks have 664 permission
find . -maxdepth 1 -type d ! -perm 2775  -name 'contrib*'

Remove throttling on previous releases

No need to remove throttling from 3.6.0

Virus Scan

# stage
$ cd /pub/mozilla.org/firefox/nightly/3.6.2-candidates/build3
$ clamdscan -m
/mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.6.2-candidates/build3: OK

----------- SCAN SUMMARY -----------
Infected files: 0
Time: 2917.127 sec (48 m 37 s)

Push to mirrors

Check https://nagios.mozilla.org/sentry or bouncer's mirror uptake
We want to push partner repacks to candidates dirs but we want to make sure we don't send it to the mirrors. This will be the first time we do this and that is why I am adding a note beforehand in this section

After GO from driver:

# ffxbld@stage
rsync -av --exclude=*.log --exclude=*.txt --exclude=*unsigned* \
 --exclude=*.crashreporter-symbols.zip --exclude=*.tests.tar.bz2* \
 --exclude=*partner-repacks* \
 /pub/mozilla.org/firefox/nightly/3.6.2-candidates/build3/ \
 /pub/mozilla.org/firefox/releases/3.6.2/

Updated cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to include 3.6.2 instead of 3.6.

Final Verification

Got 9 failures like this

Using  https://aus2.mozilla.org/update/1/Firefox/3.6b5/20091204132459/Linux_x86-gcc3/nn-NO/releasetest/update.xml?force=1
Testing http://download.mozilla.org/?product=firefox-3.6.2-complete&os=linux&lang=nn-NO
HTTP/1.1 302 Found
Date: Tue, 23 Mar 2010 00:22:22 GMT
Server: Apache
Set-Cookie: dmo=10.2.84.4.1269303742243225; path=/; expires=Wed, 23-Mar-11 00:22:22 GMT
X-Powered-By: PHP/5.1.6
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, private
Pragma: no-cache
Location: http://3347-mozilla.voxcdn.com/pub/mozilla.org/firefox/releases/3.6.2/update/linux-i686/nn-NO/firefox-3.6.2.complete.mar
Content-Type: text/html; charset=UTF-8

FAIL: download_mars returned non-zero exit code: 7

Exit status of 7 comes from curl and means 'Failed to connect to host'. nthomas and reed have attempted to debug this in the past and reed has not been able to reproduce it on his end. We are assuming that it's something between the build network and voxcdn only.

Push updates to release channel

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging/
~/bin/backupsnip 20100316-Firefox-3.6.2
~/bin/pushsnip 20100316-Firefox-3.6.2

Release

# ffxbld@stage
rm latest-3.6
ln -s 3.6.2 latest-3.6

EU Ballot Build

This is a win32 partner build for the EU browser ballot. Ideally they would be pushed at the same time as the other builds, so this work should happen earlier for 3.6.3. The process documented here may be a bit different for 3.6.3, eg partner repacks coming out of the automation and getting signed automatically.

After the builds are created (this time by Kev), and passed QA, they were signed and moved into the candidates dir

# ffxbld@stage
# get copy of signed tarball to /tmp/
mkdir /tmp/ffxbld
cd /tmp/ffxbld
tar xfv /tmp/euballot-362-signed.tar
mv win32 win32-EUballot
mv -v win32-EUballot /pub/mozilla.org/firefox/nightly/3.6.2-candidates/build3/

cd /pub/mozilla.org/firefox/nightly/3.6.2-candidates/build3/win32-EUballot
find . ! -user ffxbld 
find . ! -group firefox
find . -type f ! -perm 644 
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*'
find . -maxdepth 1 -type d ! -perm 2775  -name 'contrib*'

rsync -av --exclude=*.log --exclude=*.txt --exclude=*unsigned* \
--exclude=*.crashreporter-symbols.zip --exclude=*.tests.tar.bz2* \
--exclude=*partner-repacks* \
/pub/mozilla.org/firefox/nightly/3.6.2-candidates/build3/ \
/pub/mozilla.org/firefox/releases/3.6.2/

In bouncer, add a Firefox-3.6.2-EUballot product and point it at /firefox/releases/3.6.2/win32-EUballot/en-US/Firefox%20Setup%203.6.2.exe. Strange as it may seem, using en-US is correct even though this build doesn't exist - bouncer requires it that way and will do s/en-US/sv-SE/ when checking for the file.

After mirror uptake more than 10k, filed bug 554261 to finish off release. We have to update our browser choice website and cut over to the Firefox-3.6.2-EUballot in bouncer.

XULRunner build

This step doesn't have to be the last one

Triggered the builds by going to the buildbot waterfall for mozilla-1.9.2 and forcing builds for linux (not x64), mac and windows xulrunner. Set the field "Revision to build" (not properties) to 'FIREFOX_3_6_2_RELEASE'. Email the builds to release-drivers.

# as xrbld@stage on a production slave
cd /pub/mozilla.org/xulrunner/nightly
mkdir -p 1.9.2.2-candidates/build1
mv 2010-03-23-07-mozilla-1.9.2/* 1.9.2.2-candidates/build1/
rm -rf 2010-03-23-07-mozilla-1.9.2/

1.9.2.2 candidates dir

Waiting on reply from mfinkle and/or dtwonsed to publish these builds

Build #	Tag	Changeset
1	GECKO1922_20100315_RELBRANCH	f14062c981ba
1	FIREFOX_3_6_2_BUILD1	f14062c981ba
2	FIREFOX_3_6_2_BUILD2	4b4fad2279f6
3	FIREFOX_3_6_2_BUILD3 FIREFOX_3_6_2_RELEASE	cd857b3b0e33