Releases/Firefox 3.6.2/BuildNotes

From MozillaWiki
Jump to: navigation, search

Build Engineers

armenzg - bug 551249

Signed-off Revision(s)

Build1: 827a6883442f


On releases/mozilla-1.9.2:

Build # Tag Changeset
1 GECKO1922_20100315_RELBRANCH f14062c981ba
FIREFOX_3_6_2_BUILD1 f14062c981ba
2 FIREFOX_3_6_2_BUILD2 4b4fad2279f6
3 FIREFOX_3_6_2_BUILD3 FIREFOX_3_6_2_RELEASE cd857b3b0e33

Build data

Build # Type Build ID Build machine
1 Linux
2 Linux
3 Linux 20100316060223 moz2-linux-slave21
Mac 20100316055951 moz2-darwin9-slave10
Windows 20100316074819 win32-slave18


Build 1

  • Clobbered 1.9.2 on pm with the clobberer page
  • Landed automation patch
  • Tag buildbot-configs and buildbotcustom
    • hg tag -r 7848b054e13a FIREFOX_3_6_2_BUILD1; hg tag -r 7848b054e13a FIREFOX_3_6_2_RELEASE (buildbot-configs)
    • hg tag -r 5e5c12c1e83a FIREFOX_3_6_2_RELEASE; hg tag -r 5e5c12c1e83a FIREFOX_3_6_2_BUILD1 (buildbotcustom)
  • Triggered the builds
buildbot sendchange --username=armenzg --master=localhost:9010 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.2 build1" Go
  • I noticed that I used the wrong changeset before Mac and Linux builds were available so I stopped and started a build 2

Build 2

  • Clobbered 1.9.2 on pm with the clobberer page
  • Landed build 2 patch
  • Tagged buildbot-config
    • hg tag -r 4b4fad2279f6 FIREFOX_3_6_2_BUILD2
  • Triggered the builds
buildbot sendchange --username=armenzg --master=localhost:9010 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.2 build2" Go

Build 3

  • Clobbered 1.9.2 on pm with the clobberer page
  • bhearsum landed the two missing changesets that had landed after changeset on build1 (cd857b3b0e33) on the release branch
  • Landed automation patch
  • Triggered the builds
buildbot sendchange --username=armenzg --master=localhost:9010 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.2 build3" Go
  • Tag builder finished but did not trigger the rest of the builders
  • souce, linux_build, mac_build and win32_build were triggered manually
  • Tagging our repos:
    • hg tag -r hg tag -f -r 5b24dd7ca36f FIREFOX_3_6_2_RELEASE; hg tag -r 5b24dd7ca36f FIREFOX_3_6_2_BUILD3 (buildbotcustom)
    • hg tag -r 6e420bf20bec FIREFOX_3_6_2_BUILD3; hg tag -r 6e420bf20bec FIREFOX_3_6_2_RELEASE (tools)
    • hg tag -r f0db7a70f0eb FIREFOX_3_6_2_BUILD3; hg tag -f -r f0db7a70f0eb FIREFOX_3_6_2_RELEASE (buildbot-configs)
  • We hit bug in "updates" generation
  • Updates have been generated
  • Snipptes have been pushed to beta






  • One of linux's sendchanges was not timing out so I had to stop it manually
    • It failed the step but it did not fail the build
    • The sendchange that wasn't timing out was for
    • The same sendchange was affecting all platforms. I stopped the step in all 3 platforms
  • To trigger the locales do the following from the master:
    • python -m localhost:8010 -t FIREFOX_3_6_2_RELEASE -v -b releases/mozilla-1.9.2 -p linux -n armenzg
    • python -m localhost:8010 -t FIREFOX_3_6_2_RELEASE -v -b releases/mozilla-1.9.2 -p macosx -n armenzg
    • python -m localhost:8010 -t FIREFOX_3_6_2_RELEASE -v -b releases/mozilla-1.9.2 -p win32 -n armenzg
  • I retriggered the "id" locale which failed downloading the candidates zip
           => `'
Connecting to||:80... connected.
HTTP request sent, awaiting response... 502 Proxy Error
12:11:21 ERROR 502: Proxy Error.
  • partner repack step failed to upload the generated builds. Given all the other network failures at the time, this isn't surprising. Re-ran the upload step by hand without issue.

Unit tests

  • Make sure that you check to see that all unit tests were green


  • I tried to start signing before all-locales were ready (WRONG!)
  • Before started signing I got an rsync failure and after talking with bhearsum I figured that I need all locales for all platforms
Saving to: `app.tag'

100%[======================================>] 81          --.-K/s   in 0s

2010-03-16 10:40:17 (36.5 MB/s) - `app.tag' saved [81/81]

/home/cltsign/signing-work/firefox-3.6.2/upx/upx.exe --best -o checkouts/stubs/7z/7zSD.sfx.compressed checkouts/stubs/7z/7zSD.sfx
                      Ultimate Packer for eXecutables
   Copyright (C) 1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006
UPX 2.03w       Markus Oberhumer, Laszlo Molnar & John Reiser    Nov 7th 2006

       File size         Ratio      Format      Name
  --------------------   ------   -----------   -----------
   122368 ->     70144   57.32%    win32/pe     7zSD.sfx.compressed

Packed 1 file.
blat -to \
            -subject " for firefox 3.6.2 starting" \
                -body "EOM"
Blat v2.5.0 w/GSS encryption (build : Sep 14 2005 22:46:29) 

Sending stdin.txt to
Subject: for firefox 3.6.2 starting
Login name is
./ --keydir d:/2009-keys -j4 --product firefox -o signed-build3 unsigned-build3 >> win32_signing_build3.log 2>&1
# And copy over the info file and XPIs.
cp -pv ./unsigned-build3/unsigned/win32_info.txt ./signed-build3/
`./unsigned-build3/unsigned/win32_info.txt' -> `./signed-build3/win32_info.txt'
rsync -av ./unsigned-build3/unsigned/win32/xpi ./signed-build3/win32
sending incremental file list
rsync: link_stat "/cygdrive/d/signing-work/firefox-3.6.2/unsigned-build3/unsigned/win32/xpi" failed: No such file or directory (2)

sent 18 bytes  received 12 bytes  60.00 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at /home/lapo/packaging/rsync-3.0.4-1/src/rsync-3.0.4/main.c(1040) [sender=3.0.4]
make: *** [sign-files] Error 23
  • I triggered the missing windows locales
  • I have to clobber everything on keymaster and start again once all repacks are done
  • This second time had no complains. Signed builds are currently being uploaded.
Partner Signing

NOTE: For the next release we will probably have bug 554321 fixed and signing repacks will be different than what we did here.

Lots of problems here:

  • The partner builds generated by the automation are not useful because they cannot be signed with the same internal bits as the rest of the builds.
  • Coop redid the partner builds against already signed builds. These builds refused to get signed because repacked NSIS installers do that.

Kev provided us with a tarball of internally signed, but not externally signed partner builds. Those were signed with the instructions here.

  • I had removed en-US.xpi after having signed. bug 554455
  • This is what I did to get this fixed:
# keymaster
mkdir -p ~/signing-work/firefox-3.6.2-remove_en-US.xpi/unsigned
cd ~/signing-work/firefox-3.6.2-remove_en-US.xpi/unsigned
scp -i ~/.ssh/ffxbld_dsa{MD5SUMS,SHA1SUMS} .
# modified MD5SUMS and SHA1SUMS
gpg -v -u releases -sba MD5SUMS
gpg -v -u releases -sba SHA1SUMS
scp -i ~/.ssh/ffxbld_dsa *{MD5SUMS,SHA1SUMS}
  • After Nick's approval
# ffxbld@stage
cd /pub/
cp MD5SUMS* SHA1SUMS* ../../../releases/3.6.2/


No problems

Generate updates and push betatest snippets

  • We hit a bug. Nick helped me with this.
  • bug 552831 - fails when flipping useBeta to True
perl ../tools/release/ -p firefox -r Firefox -v 3.6.2 -a 3.6.2 -o 3.6rc2 -b 3 -c patcher-configs/moz192-branch-patcher2.cfg -t -f -d -l shipped-locales -u
Use of uninitialized value in substitution (s///) at ../tools/release/ line 127.
Use of uninitialized value in pattern match (m//) at ../tools/release/ line 129.
Use of uninitialized value in pattern match (m//) at ../tools/release/ line 129.
Use of uninitialized value in concatenation (.) or string at ../tools/release/ line 140.
ASSERT: BumpFilePath() - Unknown file type for  at ../tools/release/ line 140.
  • Once this landed, I deleted moz2-linux-slave12:/builds/slave/updates/* and force the builder
  • updates generation is completed

Update verify

  • Waiting on mac_update_verify
  • mac_update_verify took forever because of the network issues but it finished

Publish updates to beta channel

  • THU. 18th of March - 6:30 PDT
  • Backing up snippets
# cltbld@aus2-staging.m.o
# make sure scripts are up to date
cd bin
cvs update
cd /opt/aus2/snippets/staging/
# NOTE: That that backupsnip only backs up the folder you specify and nothing older as it used to
~/bin/backupsnip 20100316-Firefox-3.6.2-beta
~/bin/pushsnip 20100316-Firefox-3.6.2-beta

Socorro product data updates

We can do this ourselves (a subset of relengs, ask nthomas or bhearsum). No need to file an IT bug for this.

Nick added for me the entries.

Added 3.6.2 and 3.6.3pre with the crash-stats admin page.

Update Bouncer

  • Added the following products:
    • Firefox-3.6.2
    • Firefox-3.6.2-Complete
    • Firefox-3.6.2-Partial-3.6rc2
  • Added files to bouncer

Final checks before push

# ffxbld@stage
cd /pub/
find . ! -user ffxbld #only the solaris contributed builds showed up
find . ! -group firefox
find . -type f ! -perm 644 # unsigned/partner-repacks have 664 permission
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*' # unsigned/partner-repacks have 664 permission
find . -maxdepth 1 -type d ! -perm 2775  -name 'contrib*'

Remove throttling on previous releases

  • No need to remove throttling from 3.6.0

Virus Scan

# stage
$ cd /pub/
$ clamdscan -m
/mnt/netapp/stage/ OK

----------- SCAN SUMMARY -----------
Infected files: 0
Time: 2917.127 sec (48 m 37 s)

Push to mirrors

  • Check or bouncer's mirror uptake
  • We want to push partner repacks to candidates dirs but we want to make sure we don't send it to the mirrors. This will be the first time we do this and that is why I am adding a note beforehand in this section

After GO from driver:

# ffxbld@stage
rsync -av --exclude=*.log --exclude=*.txt --exclude=*unsigned* \
 --exclude=* --exclude=*.tests.tar.bz2* \
 --exclude=*partner-repacks* \
 /pub/ \

Updated cltbld@stage:/pub/ to include 3.6.2 instead of 3.6.

Final Verification

Got 9 failures like this

HTTP/1.1 302 Found
Date: Tue, 23 Mar 2010 00:22:22 GMT
Server: Apache
Set-Cookie: dmo=; path=/; expires=Wed, 23-Mar-11 00:22:22 GMT
X-Powered-By: PHP/5.1.6
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, private
Pragma: no-cache
Content-Type: text/html; charset=UTF-8

FAIL: download_mars returned non-zero exit code: 7

Exit status of 7 comes from curl and means 'Failed to connect to host'. nthomas and reed have attempted to debug this in the past and reed has not been able to reproduce it on his end. We are assuming that it's something between the build network and voxcdn only.

Push updates to release channel

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging/
~/bin/backupsnip 20100316-Firefox-3.6.2
~/bin/pushsnip 20100316-Firefox-3.6.2


# ffxbld@stage
rm latest-3.6
ln -s 3.6.2 latest-3.6

EU Ballot Build

This is a win32 partner build for the EU browser ballot. Ideally they would be pushed at the same time as the other builds, so this work should happen earlier for 3.6.3. The process documented here may be a bit different for 3.6.3, eg partner repacks coming out of the automation and getting signed automatically.

After the builds are created (this time by Kev), and passed QA, they were signed and moved into the candidates dir

# ffxbld@stage
# get copy of signed tarball to /tmp/
mkdir /tmp/ffxbld
cd /tmp/ffxbld
tar xfv /tmp/euballot-362-signed.tar
mv win32 win32-EUballot
mv -v win32-EUballot /pub/

cd /pub/
find . ! -user ffxbld 
find . ! -group firefox
find . -type f ! -perm 644 
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*'
find . -maxdepth 1 -type d ! -perm 2775  -name 'contrib*'

rsync -av --exclude=*.log --exclude=*.txt --exclude=*unsigned* \
--exclude=* --exclude=*.tests.tar.bz2* \
--exclude=*partner-repacks* \
/pub/ \

In bouncer, add a Firefox-3.6.2-EUballot product and point it at /firefox/releases/3.6.2/win32-EUballot/en-US/Firefox%20Setup%203.6.2.exe. Strange as it may seem, using en-US is correct even though this build doesn't exist - bouncer requires it that way and will do s/en-US/sv-SE/ when checking for the file.

After mirror uptake more than 10k, filed bug 554261 to finish off release. We have to update our browser choice website and cut over to the Firefox-3.6.2-EUballot in bouncer.

XULRunner build

  • This step doesn't have to be the last one

Triggered the builds by going to the buildbot waterfall for mozilla-1.9.2 and forcing builds for linux (not x64), mac and windows xulrunner. Set the field "Revision to build" (not properties) to 'FIREFOX_3_6_2_RELEASE'. Email the builds to release-drivers.

# as xrbld@stage on a production slave
cd /pub/
mkdir -p
mv 2010-03-23-07-mozilla-1.9.2/*
rm -rf 2010-03-23-07-mozilla-1.9.2/
  • Waiting on reply from mfinkle and/or dtwonsed to publish these builds