Releases/Firefox 4.0b7/BuildNotes

From MozillaWiki
Jump to: navigation, search

Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Build Engineers

armenzg - bug 596259

Signed-off Revision(s)

L10N changesets


Build # Tag Changeset
1 GECKO20b7_20101104_RELBRANCH f7016571b472
FIREFOX_4_0b7_BUILD1/FIREFOX_4_0b7_RELEASE 297086a0fb61

Build data

Build # Type Build ID Build machine Time to build
1 Linux [compressed .tar.bz2] 20101104125541 mv-moz2-linux-ix-slave17 55 mins, 38 secs
Linux64 [compressed .tar.bz2] 20101104131204 moz2-linux64-slave09 1hr 35 mins
Macosx64 [.dmg] 20101104131842 moz2-darwin10-slave19 4hrs 26mins
Win32 [.exe/.zip] 20101104142426 mw32-ix-slave03 2hrs 46mins

Using pm:8010.

Bugs hit

During this release we have hit the following bugs:

  • [NEW] bug 589914 - tag step should not fail on APPROVAL REQUIRED
  • [BLOCKED ON SCHEDULERDB] bug 553300 - we have to move 10.5 slaves to run l10n verification. The day we share pools with production we won't hit this. I added a note on the wiki note to remember to bring them over for the next release.
  • [BLOCKED ON SCHEDULERDB] bug 609887 - we currently run repacks in a subset of slaves. I moved slaves that were not in the subset for needed for macosx64_repack. I added a note on the wiki note to remember to bring them over for the next release.
  • [FIXED] bug 609932 - L10n Verification failed for Linux because we now have checksums
  • [ENHANCEMENT] bug 609981 - Create a TBPL page for Firefox-Release Tinderbox page. This will ease finding release unit test and performance test jobs going orange.
  • [FIXED][BLOCKER] bug 610708 - cannot add new locales through the tuxedo API because of HTTP 400 error
  • We (nthomas did) had to rename "mac64" to "mac", regenerate the checksums and new snippets. Very important for next release to watch for this. See section "Naming fixup" and following to learn all the details.
  • ja-JP-mac snippets were not generated. See section "Add ja-JP-mac"
    • We went live without partial updates for ja-JP-mac

Build 1


  • marked "any master" for "any release" for "any builder" to clobber (not connected to Build-VPN to force LDAP auth - - check the date for one of release builders to confirm it worked)
  • moving slaves
    • from pm01:
      • mv-moz2-linux-ix-slave{08,10,17}, linux-ix-slave13
      • bm-xserve{15,18,19,21} We don't have 10.5 for this beta
      • moz2-darwin10-slave{18,19,40-43,45-46}
      • mw32-ix-slave{03,04,09,10,12}
    • from pm03:
      • moz2-linux64-slave0{1,2,8,9,12}

  • moved more slaves (see "Bugs hit" section - I did this after build jobs started)
    • from pm01:
      • moz2-darwin9-slave{05,06,07,09,11,12} - so we can do L10n verification
      • moz2-darwin10-slave{06,07,08,09,11,12} - so we can do L10n Macos64 repacks

  • received GO
  • closed TREE
  • submitted patch for review with new l10n-changeset and driver's changeset
  • Tagged buildbot-configs, buildbotcustom and tools
# buildbot-configs
hg tag -r 23deb7d0556b FIREFOX_4_0b7_BUILD1
hg tag -r 23deb7d0556b FIREFOX_4_0b7_RELEASE
# tools
hg tag -r cd019fbfb429 FIREFOX_4_0b7_BUILD1
hg tag -r cd019fbfb429 FIREFOX_4_0b7_RELEASE
hg tag -r ed7c08dbed63 FIREFOX_4_0b7_BUILD1
hg tag -r ed7c08dbed63 FIREFOX_4_0b7_RELEASE
  • Landed config changes
  • removed *.pyc from moz2-master dir in pm:8010
  • reconfig'd
  • Trigger the automation
buildbot sendchange --username=armenzg --master=localhost:9010 --branch=mozilla-central -m "Firefox 4.0b7 build1" Go


slave: linux-ix-slave13

  • tagging failed on the last step because I used HTML comment on the tinderbox admin page. Failed with:
hg push -e "ssh -l ffxbld -i ~cltbld/.ssh/ffxbld_dsa" -f ssh://
pushing to ssh://
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 3 changesets with 5 changes to 4 files
remote: Pushing to an APPROVAL REQUIRED tree requires your top changeset comment to include: a=... (or, more accurately, a\S*=...)
remote: transaction abort!
remote: rollback completed
remote: abort: pretxnchangegroup.a_treeclosure hook failed
abort: unexpected response: empty string
  • I should have remove the following instead of commenting it out:
<!--<span id="treestatus" style="color: red;">APPROVAL REQUIRED</span> for Gecko 2.0b7 blockers or Fennec 2.0b2 blockers - see the <a href="">Tree Rules</a>. )-->
  • After fixing tree rule I logged into linux-ix-slave13 and run the step manually:
cd /builds/slave/tag/mozilla-central
hg push -e "ssh -l ffxbld -i ~cltbld/.ssh/ffxbld_dsa" -f ssh://
  • reconfigured master with Dummy factory and did another sendchange
  • Re-opened the tree
  • Filed bug 609717


slave: linux-ix-slave13

  • No problems


  • Linux: green, no problems
  • Linux64: green, no problems
  • Mac64: green, no problems (took too long)
  • Win32: green, no problems



Looked at Firefox-Release tbox page to find orange jobs.

NOTE: Next time use the buildapi and call it with buildapi/revision/mozilla-central/297086a0fb61 (substitute the revision appropriately).



  • No en-US.xpi to remove from stage, bug 485860 is FIXED. May need reopening as no en-US.xpi in $platform/xpi either
  • Used instructions
  • Hit an error on the verify-asc target. Filed bug 609932Did I really file anything?. Fallout from the three new files
    • I think we removed the checksums files
  • ran the fake-upload target manually, then upload
  • We had to respin l10n repacks and sign the files again
    • removed signed files from stage
    • rerun the signing step w/o any problem

L10N Verify

  • Linux L10n and Linux L64 verification failed with:
Unknown package type for file: firefox-4.0b7-build1/linux-i686/en-US/firefox-4.0b7.checksums
Unknown package type for file: firefox-4.0b7-build1/linux-x86_64/en-US/firefox-4.0b7.checksums
  • bug 609932 - L10n Verification failed for Linux because we now have checksums
  • land fix
  • clobber
  • force build both builders that failed
  • All green


No problems

Special case for old mac universal builds

In this beta we are not generating the old ppc+386 universal builds for mac. Therefore we have to play a little bit with snippets to make them update to the new 32-bit and 64-bit updates.

To do so we are doing "Darwin_Universal-gcc3" and "Darwin_Universal-gcc3" directories with the contents of "Darwin_x86_64-gcc3" which point to beta7. We are also not giving partial updates from the old universal to the new universal since we don't have a partial mar for that case and it was not easy to generate.

Here is an overview of how we did it and in the following section the mkdir & rsync dance that we have to do.

cd /opt/aus2/snippets/staging/
cp -r 20101104-Firefox-4.0b7-test 20101104-Firefox-4.0b7-test-fixed
cd 20101104-Firefox-4.0b7-test-fixed/Firefox
# mkdir & rsync dance for each 3.7* and 4.0* directory
~/bin/pushsnip 20101104-Firefox-4.0b7-test-fixed
cd -

cp -r 20101104-Firefox-4.0b7 20101104-Firefox-4.0b7-fixed
cd 20101104-Firefox-4.0b7-fixed/Firefox
# mkdir & rsync dance for each 3.7* and 4.0* directory
# we push the fixed snippets in a latter section
# after QA & rel-driver grants us to do it.

The mkdir & rsync dance mentioned in the previous blockquote:

mkdir -p 3.7a1/Darwin_Universal-gcc3/20100208064157
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 3.7a1/Darwin_Universal-gcc3/20100208064157

mkdir -p 3.7a2/Darwin_Universal-gcc3/20100228155422
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 3.7a2/Darwin_Universal-gcc3/20100228155422/

mkdir -p 3.7a3/Darwin_Universal-gcc3/20100315065900
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 3.7a3/Darwin_Universal-gcc3/20100315065900/

mkdir -p 3.7a4/Darwin_Universal-gcc3/20100407105356
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 3.7a4/Darwin_Universal-gcc3/20100407105356/

mkdir -p 3.7a5/Darwin_Universal-gcc3/20100610053455
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 3.7a5/Darwin_Universal-gcc3/20100610053455

mkdir -p 4.0b1/Darwin_Universal-gcc3/20100630131607
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 4.0b1/Darwin_Universal-gcc3/20100630131607/

mkdir -p 4.0b2/Darwin_Universal-gcc3/20100720175703
rsync -av --exclude partial.txt 4.0b2/Darwin_x86_64-gcc3/20100720180051/ 4.0b2/Darwin_Universal-gcc3/20100720175703/

mkdir -p 4.0b3/Darwin_Universal-gcc3/20100805181943/
rsync -av --exclude partial.txt 4.0b3/Darwin_x86_64-gcc3/20100805182338/ 4.0b3/Darwin_Universal-gcc3/20100805181943/

mkdir -p 4.0b4/Darwin_Universal-gcc3/20100818121614
rsync -av --exclude partial.txt 4.0b4/Darwin_x86_64-gcc3/20100818121922/ 4.0b4/Darwin_Universal-gcc3/20100818121614/

mkdir -p 4.0b5/Darwin_x86-gcc3-u-ppc-i386/20100831070010
rsync -av --exclude partial.txt 4.0b5/Darwin_x86_64-gcc3/20100831070403/ 4.0b5/Darwin_x86-gcc3-u-ppc-i386/20100831070010/

mkdir -p 4.0b6/Darwin_x86-gcc3-u-ppc-i386/20100914072643
rsync -av --exclude partial.txt 4.0b6/Darwin_x86_64-gcc3/20100914073111/ 4.0b6/Darwin_x86-gcc3-u-ppc-i386/20100914072643/

Add ja-JP-mac and swap to mac in dir names

We don't have any snippets for ja-JP-mac, probably because shipped locales says 'ja-JP-mac osx' and we don't handle that properly for macosx64 when creating the patcher config. We don't have a partial or snippets. FILE BUG?.

And we're swapping the ftp dir to mac from mac64, and bouncer from os=osx64 to os=osx (for continuity with previous releases).

Manual fix:

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
rsync -a 20101104-Firefox-4.0b7-test-fixed/ 20101109-Firefox-4.0b7-test-fixed2/
cd 20101109-Firefox-4.0b7-test-fixed2/Firefox
mkdir -p 4.0b6/Darwin_x86_64-gcc3/20100914073111/ja-JP-mac/{beta,release}test
# setup snippet
cat > 4.0b6/Darwin_x86_64-gcc3/20100914073111/ja-JP-mac/releasetest/complete.txt << EOF
appv=4.0 Beta 7

sed -e 's!^url=.*!url=!' \
   4.0b6/Darwin_x86_64-gcc3/20100914073111/ja-JP-mac/releasetest/complete.txt \
 > 4.0b6/Darwin_x86_64-gcc3/20100914073111/ja-JP-mac/betatest/complete.txt

# copy to other locations, ja was added at b5
dirs=`find 4.0b{4,5,6} -maxdepth 1 -mindepth 1 -type d -name '*Darwin*' ! -wholename '4.0b6/Darwin_x86_64-gcc3' | sort`
for d in $dirs; do
  rsync -av 4.0b6/Darwin_x86_64-gcc3/20100914073111/ja-JP-mac $d/*/

# fix paths/os
find */Darwin* -type f -exec sed -i -e 's!update/mac64/!update/mac/!' -e 's!os=osx64!os=osx!' {} \;
~/bin/backupsnip 20101109-Firefox-4.0b7-test-fixed2
~/bin/pushsnip   20101109-Firefox-4.0b7-test-fixed2

And the beta snippets

cd /opt/aus2/snippets/staging
rsync -a 20101109-Firefox-4.0b7-test-fixed2/ 20101109-Firefox-4.0b7-fixed2/
cd 20101109-Firefox-4.0b7-fixed2/Firefox
find . -depth -name 'betatest' -type d -exec rm -rf {} \;
find . -mindepth 4 -maxdepth 4 -type d -exec mv {}/{releasetest,beta} \;

Update Verify

All green but I can see the following:

# for linux
remove failed: -1,2 (searchplugins/bing.xml.moz-backup)
# for osx
remove failed: -1,2 (Contents/MacOS/searchplugins/bing.xml.moz-backup)
remove failed: -1,2 (force_plist_reload.moz-backup)
# for win32
remove failed: -1,2 (mozjs.dll.moz-backup)
remove failed: -1,2 (searchplugins/bing.xml.moz-backup)
  • bing.xml is new for all platforms.
  • bhearsum mentions that force_plist_reload is known to be new
  • khuey mentions that mozjs.dll is new on Windows for beta7

Return slaves

Put back

  • moz2-linux64-slave{01,02,08,09,12}, since that's half our linux64 build pool
  • mv-moz2-linux-ix-slave{08,10,17}, linux-ix-slave13
  • moz2-darwin10-slave{06-09,11-12,18,19,40-43,45-46}
  • mw32-ix-slave{03,04,09,10,12}
  • moz2-darwin9-slave{05,06,07,09,11,12}

Final checks before push

cd /pub/
find . ! -user ffxbld
find . ! -group firefox
find . -type f ! -perm 644
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*'
find . -maxdepth 1 -type d ! -perm 2775 -name 'contrib*'

No Solaris builds yet, no output from any of these.

Update Bouncer

  • We hit "HTTPError: HTTP Error 400: BAD REQUEST" because in this release we have new locales that were not part of "b6" or 3.6.12
  • It seems that this will be fixed once bug 590493 is fixed
  • [BLOCKER] bug 610708 - cannot add new locales through the tuxedo API because of HTTP 400 error
  • wenzel updated the json files with the new list of locales that have never existed for neither the betas nor the stable releases.
  • we also discovered that a restart of the apacha server was needed to pick them up
  • wenzel also filed bug 610893 for avoiding the PHP library and the json to don't fall out of sync
  • manually updated the three locations for mac to use OS of 'osx' instead of 'osx64', and replace 'mac64' with 'mac' in the path (part of the continuity with previous releases effort)

Socorro product data updates

Everything looks good. I have checked for "4.0b7" and "4.0b8pre" on the "Products"-"Firefox" table (4th one below Fennec as of writing).

Please leave the following note (or improve it) for following betas:

If you don't see $version and $version-pre on the "Products-Firefox" table
then you should add it. Ignore it if you see those entries on the "Missing Entries" table.

Virus check

# ffxbld@stage
cd /pub/
clamdscan .
/mnt/netapp/stage/ OK

----------- SCAN SUMMARY -----------
Infected files: 0
Time: 4089.461 sec (68 m 9 s)

Naming fixup

The buildbot setup leaves us with mac64 directories on the ftp server, and an expectation to use os=osx64 in bouncer. The maintainers would really rather not change from os=osx, so we're making everything match what we did for 4.0b6.

Rename on ftp server

Fix the file locations, but leave a symlink so that betatest keeps working for QA

# ffxbld@stage
cd /pub/
mv mac64 mac
cd update
mv mac64 mac && ln -s mac mac64

The mac64 symlink was removed after the modified betatest snippets were pushed live (see here).

--Rail 08:47, 6 December 2010 (PST): Removing mac64 symlink causes a failure of updates builder for the next version, which tries to use mac64 directory to download the previous release. I restored the symlink. See bug 616726 for more details.

Fix up *SUMS

On signing box

cd ~/signing-work/firefox-4.0b7/
mkdir signed-build1-redo-SUMS
cp signed-build1/*SUMS signed-build1-redo-SUMS/
cd signed-build1-redo-SUMS
sed -i -e 's/mac64/mac/' *SUMS
../signing/sign-files .
rsync -av -e "ssh -i $HOME/.ssh/ffxbld_dsa" * \

Push to mirrors

  • Wait for email from driver before continuing
rsync -av --exclude=*tests* --exclude=*crashreporter* --exclude=*.log \
  --exclude=*.txt --exclude=*.checksums --exclude=*unsigned* \
  /pub/ \
# ffbld@stage
cd /pub/
curl -sL | sed 's/Beta 1/Beta 7/g' > index.html
for dir in `find . -mindepth 1 -type d`; do cp -pv index.html $dir/; done
  • Send email to r-d when rsync/push to mirrors is done.
  • Send email to r-d when uptake has reached 10-20k (since it is a beta) for binaries, complete and partial mars.

Final verify

Patch to add ja-JP-mac to the list of locales for 4.0b4, b5, b6. That should test all the partial and complete update paths, but we should also write some lines to make sure the old universals have working update links too.

The only failure is that we don't have ja-JP-mac partials:

FAIL: no partial update found for
FAIL: download_mars returned non-zero exit code: 1
FAIL: no partial update found for
FAIL: download_mars returned non-zero exit code: 1

Push snippets to beta channel

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20101109-Firefox-4.0b7-fixed2
  • Upon GO from release-drivers
~/bin/pushsnip 20101109-Firefox-4.0b7-fixed2
  • Announce to release-drivers

Remove index.html files

# ffxbld@stage
cd /pub/
find . -name index.html
find . -name index.html | xargs rm -v


Sent mail to metrics with the following contents: