Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Build Engineers

Signed-off Revision(s)

Build 1: f7016571b472

L10N changesets

checked with Pike about two pending reviews (not taken)
wget https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=fx4.0b7
hit "ship it" from FF4 beta7 page

Build data

Build #	Type	Build ID	Build machine	Time to build
1	Linux [compressed .tar.bz2]	20101104125541	mv-moz2-linux-ix-slave17	55 mins, 38 secs
	Linux64 [compressed .tar.bz2]	20101104131204	moz2-linux64-slave09	1hr 35 mins
	Macosx64 [.dmg]	20101104131842	moz2-darwin10-slave19	4hrs 26mins
	Win32 [.exe/.zip]	20101104142426	mw32-ix-slave03	2hrs 46mins

Using pm:8010.

Bugs hit

During this release we have hit the following bugs:

~~[NEW] bug 589914 - tag step should not fail on APPROVAL REQUIRED~~
[BLOCKED ON SCHEDULERDB] bug 553300 - we have to move 10.5 slaves to run l10n verification. The day we share pools with production we won't hit this. I added a note on the wiki note to remember to bring them over for the next release.
[BLOCKED ON SCHEDULERDB] bug 609887 - we currently run repacks in a subset of slaves. I moved slaves that were not in the subset for needed for macosx64_repack. I added a note on the wiki note to remember to bring them over for the next release.
~~[FIXED] bug 609932 - L10n Verification failed for Linux because we now have checksums~~
[ENHANCEMENT] bug 609981 - Create a TBPL page for Firefox-Release Tinderbox page. This will ease finding release unit test and performance test jobs going orange.
~~[FIXED][BLOCKER] bug 610708 - cannot add new locales through the tuxedo API because of HTTP 400 error~~
We (nthomas did) had to rename "mac64" to "mac", regenerate the checksums and new snippets. Very important for next release to watch for this. See section "Naming fixup" and following to learn all the details.
ja-JP-mac snippets were not generated. See section "Add ja-JP-mac"
- We went live without partial updates for ja-JP-mac

Build 1

Setup

marked "any master" for "any release" for "any builder" to clobber (not connected to Build-VPN to force LDAP auth - https://build.mozilla.org/clobberer - check the date for one of release builders to confirm it worked)
moving slaves
- from pm01:
  - mv-moz2-linux-ix-slave{08,10,17}, linux-ix-slave13
  - ~~bm-xserve{15,18,19,21}~~ We don't have 10.5 for this beta
  - moz2-darwin10-slave{18,19,40-43,45-46}
  - mw32-ix-slave{03,04,09,10,12}
- from pm03:
  - moz2-linux64-slave0{1,2,8,9,12}

moved more slaves (see "Bugs hit" section - I did this after build jobs started)
- from pm01:
  - moz2-darwin9-slave{05,06,07,09,11,12} - so we can do L10n verification
  - moz2-darwin10-slave{06,07,08,09,11,12} - so we can do L10n Macos64 repacks

received GO
closed TREE
submitted patch for review with new l10n-changeset and driver's changeset
Tagged buildbot-configs, buildbotcustom and tools

# buildbot-configs
hg tag -r 23deb7d0556b FIREFOX_4_0b7_BUILD1
hg tag -r 23deb7d0556b FIREFOX_4_0b7_RELEASE
# tools
hg tag -r cd019fbfb429 FIREFOX_4_0b7_BUILD1
hg tag -r cd019fbfb429 FIREFOX_4_0b7_RELEASE
#buildbotcustom
hg tag -r ed7c08dbed63 FIREFOX_4_0b7_BUILD1
hg tag -r ed7c08dbed63 FIREFOX_4_0b7_RELEASE

Landed config changes
removed *.pyc from moz2-master dir in pm:8010
reconfig'd
Trigger the automation

buildbot sendchange --username=armenzg --master=localhost:9010 --branch=mozilla-central -m "Firefox 4.0b7 build1" Go

Tag

slave: linux-ix-slave13

tagging failed on the last step because I used HTML comment on the tinderbox admin page. Failed with:

hg push -e "ssh -l ffxbld -i ~cltbld/.ssh/ffxbld_dsa" -f ssh://hg.mozilla.org/mozilla-central
...
pushing to ssh://hg.mozilla.org/mozilla-central
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 3 changesets with 5 changes to 4 files
remote: Pushing to an APPROVAL REQUIRED tree requires your top changeset comment to include: a=... (or, more accurately, a\S*=...)
remote: transaction abort!
remote: rollback completed
remote: abort: pretxnchangegroup.a_treeclosure hook failed
abort: unexpected response: empty string

I should have remove the following instead of commenting it out:

<!--<span id="treestatus" style="color: red;">APPROVAL REQUIRED</span> for Gecko 2.0b7 blockers or Fennec 2.0b2 blockers - see the <a href="https://wiki.mozilla.org/Tree_Rules#mozilla-central_-_Trunk_.28Firefox_4.0.2C_Gecko_2.0_work.29">Tree Rules</a>. )-->

After fixing tree rule I logged into linux-ix-slave13 and run the step manually:

cd /builds/slave/tag/mozilla-central
hg push -e "ssh -l ffxbld -i ~cltbld/.ssh/ffxbld_dsa" -f ssh://hg.mozilla.org/mozilla-central

reconfigured master with Dummy factory and did another sendchange
Re-opened the tree
Filed bug 609717

Source

slave: linux-ix-slave13

No problems

Build/Repack

Linux: green, no problems
Linux64: green, no problems
Mac64: green, no problems (took too long)
Win32: green, no problems

Unittests

Results

Looked at Firefox-Release tbox page to find orange jobs.

NOTE: Next time use the buildapi and call it with buildapi/revision/mozilla-central/297086a0fb61 (substitute the revision appropriately).

Oranges:

Signing

No en-US.xpi to remove from stage, bug 485860 is FIXED. May need reopening as no en-US.xpi in $platform/xpi either
Used instructions
Hit an error on the verify-asc target. ~~Filed bug 609932~~Did I really file anything?. Fallout from the three new files
- I think we removed the checksums files

linux-i686/en-US/firefox-4.0b7.checksums

ran the fake-upload target manually, then upload

We had to respin l10n repacks and sign the files again
- removed signed files from stage
- rerun the signing step w/o any problem

L10N Verify

Linux L10n and Linux L64 verification failed with:

Unknown package type for file: firefox-4.0b7-build1/linux-i686/en-US/firefox-4.0b7.checksums
Unknown package type for file: firefox-4.0b7-build1/linux-x86_64/en-US/firefox-4.0b7.checksums

bug 609932 - L10n Verification failed for Linux because we now have checksums
land fix
clobber
force build both builders that failed
All green

Updates

No problems

Special case for old mac universal builds

In this beta we are not generating the old ppc+386 universal builds for mac. Therefore we have to play a little bit with snippets to make them update to the new 32-bit and 64-bit updates.

To do so we are doing "Darwin_Universal-gcc3" and "Darwin_Universal-gcc3" directories with the contents of "Darwin_x86_64-gcc3" which point to beta7. We are also not giving partial updates from the old universal to the new universal since we don't have a partial mar for that case and it was not easy to generate.

Here is an overview of how we did it and in the following section the mkdir & rsync dance that we have to do.

#cltbld@aus-staging
cd /opt/aus2/snippets/staging/
cp -r 20101104-Firefox-4.0b7-test 20101104-Firefox-4.0b7-test-fixed
cd 20101104-Firefox-4.0b7-test-fixed/Firefox
# mkdir & rsync dance for each 3.7* and 4.0* directory
~/bin/pushsnip 20101104-Firefox-4.0b7-test-fixed
cd -

cp -r 20101104-Firefox-4.0b7 20101104-Firefox-4.0b7-fixed
cd 20101104-Firefox-4.0b7-fixed/Firefox
# mkdir & rsync dance for each 3.7* and 4.0* directory
# we push the fixed snippets in a latter section
# after QA & rel-driver grants us to do it.

The mkdir & rsync dance mentioned in the previous blockquote:

mkdir -p 3.7a1/Darwin_Universal-gcc3/20100208064157
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 3.7a1/Darwin_Universal-gcc3/20100208064157

mkdir -p 3.7a2/Darwin_Universal-gcc3/20100228155422
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 3.7a2/Darwin_Universal-gcc3/20100228155422/

mkdir -p 3.7a3/Darwin_Universal-gcc3/20100315065900
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 3.7a3/Darwin_Universal-gcc3/20100315065900/

mkdir -p 3.7a4/Darwin_Universal-gcc3/20100407105356
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 3.7a4/Darwin_Universal-gcc3/20100407105356/

mkdir -p 3.7a5/Darwin_Universal-gcc3/20100610053455
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 3.7a5/Darwin_Universal-gcc3/20100610053455

mkdir -p 4.0b1/Darwin_Universal-gcc3/20100630131607
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 4.0b1/Darwin_Universal-gcc3/20100630131607/

mkdir -p 4.0b2/Darwin_Universal-gcc3/20100720175703
rsync -av --exclude partial.txt 4.0b2/Darwin_x86_64-gcc3/20100720180051/ 4.0b2/Darwin_Universal-gcc3/20100720175703/

mkdir -p 4.0b3/Darwin_Universal-gcc3/20100805181943/
rsync -av --exclude partial.txt 4.0b3/Darwin_x86_64-gcc3/20100805182338/ 4.0b3/Darwin_Universal-gcc3/20100805181943/

mkdir -p 4.0b4/Darwin_Universal-gcc3/20100818121614
rsync -av --exclude partial.txt 4.0b4/Darwin_x86_64-gcc3/20100818121922/ 4.0b4/Darwin_Universal-gcc3/20100818121614/

mkdir -p 4.0b5/Darwin_x86-gcc3-u-ppc-i386/20100831070010
rsync -av --exclude partial.txt 4.0b5/Darwin_x86_64-gcc3/20100831070403/ 4.0b5/Darwin_x86-gcc3-u-ppc-i386/20100831070010/

mkdir -p 4.0b6/Darwin_x86-gcc3-u-ppc-i386/20100914072643
rsync -av --exclude partial.txt 4.0b6/Darwin_x86_64-gcc3/20100914073111/ 4.0b6/Darwin_x86-gcc3-u-ppc-i386/20100914072643/

Add ja-JP-mac and swap to mac in dir names

We don't have any snippets for ja-JP-mac, probably because shipped locales says 'ja-JP-mac osx' and we don't handle that properly for macosx64 when creating the patcher config. We don't have a partial or snippets. FILE BUG?.

And we're swapping the ftp dir to mac from mac64, and bouncer from os=osx64 to os=osx (for continuity with previous releases).

Manual fix:

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
rsync -a 20101104-Firefox-4.0b7-test-fixed/ 20101109-Firefox-4.0b7-test-fixed2/
cd 20101109-Firefox-4.0b7-test-fixed2/Firefox
mkdir -p 4.0b6/Darwin_x86_64-gcc3/20100914073111/ja-JP-mac/{beta,release}test

# setup snippet
cat > 4.0b6/Darwin_x86_64-gcc3/20100914073111/ja-JP-mac/releasetest/complete.txt << EOF
version=1
type=complete
url=url=http://download.mozilla.org/?product=firefox-4.0b7-complete&os=osx64&lang=ja-JP-mac
hashFunction=SHA512
hashValue=6d08a0d03abc2fb7dbe601ecc3dadad2238f9c1f292ce60e3c4f9696b34252110fc50d6442a1ed733edb99390978a1090becb9b812aad5dbfe19b7811c81462b
size=26974569
build=20101104131842
appv=4.0 Beta 7
extv=4.0b7
detailsUrl=http://www.mozilla.com/ja-JP-mac/firefox/4.0b7/releasenotes/
EOF

sed -e 's!^url=.*!url=http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b7-candidates/build1/update/mac64/ja-JP-mac/firefox-4.0b7.complete.mar!' \
   4.0b6/Darwin_x86_64-gcc3/20100914073111/ja-JP-mac/releasetest/complete.txt \
 > 4.0b6/Darwin_x86_64-gcc3/20100914073111/ja-JP-mac/betatest/complete.txt

# copy to other locations, ja was added at b5
dirs=`find 4.0b{4,5,6} -maxdepth 1 -mindepth 1 -type d -name '*Darwin*' ! -wholename '4.0b6/Darwin_x86_64-gcc3' | sort`
for d in $dirs; do
  rsync -av 4.0b6/Darwin_x86_64-gcc3/20100914073111/ja-JP-mac $d/*/
done

# fix paths/os
find */Darwin* -type f -exec sed -i -e 's!update/mac64/!update/mac/!' -e 's!os=osx64!os=osx!' {} \;

~/bin/backupsnip 20101109-Firefox-4.0b7-test-fixed2
~/bin/pushsnip   20101109-Firefox-4.0b7-test-fixed2

And the beta snippets

cd /opt/aus2/snippets/staging
rsync -a 20101109-Firefox-4.0b7-test-fixed2/ 20101109-Firefox-4.0b7-fixed2/
cd 20101109-Firefox-4.0b7-fixed2/Firefox
find . -depth -name 'betatest' -type d -exec rm -rf {} \;
find . -mindepth 4 -maxdepth 4 -type d -exec mv {}/{releasetest,beta} \;

Update Verify

All green but I can see the following:

# for linux
remove failed: -1,2 (searchplugins/bing.xml.moz-backup)
# for osx
remove failed: -1,2 (Contents/MacOS/searchplugins/bing.xml.moz-backup)
remove failed: -1,2 (force_plist_reload.moz-backup)
# for win32
remove failed: -1,2 (mozjs.dll.moz-backup)
remove failed: -1,2 (searchplugins/bing.xml.moz-backup)

bing.xml is new for all platforms.
bhearsum mentions that force_plist_reload is known to be new
khuey mentions that mozjs.dll is new on Windows for beta7

Return slaves

Put back

moz2-linux64-slave{01,02,08,09,12}, since that's half our linux64 build pool
mv-moz2-linux-ix-slave{08,10,17}, linux-ix-slave13
moz2-darwin10-slave{06-09,11-12,18,19,40-43,45-46}
mw32-ix-slave{03,04,09,10,12}
moz2-darwin9-slave{05,06,07,09,11,12}

Final checks before push

cd /pub/mozilla.org/firefox/nightly/4.0b7-candidates/build1/
find . ! -user ffxbld
find . ! -group firefox
find . -type f ! -perm 644
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*'
find . -maxdepth 1 -type d ! -perm 2775 -name 'contrib*'

No Solaris builds yet, no output from any of these.

Update Bouncer

We hit "HTTPError: HTTP Error 400: BAD REQUEST" because in this release we have new locales that were not part of "b6" or 3.6.12
It seems that this will be fixed once bug 590493 is fixed
~~[BLOCKER] bug 610708 - cannot add new locales through the tuxedo API because of HTTP 400 error~~
wenzel updated the json files with the new list of locales that have never existed for neither the betas nor the stable releases.

+ak
+br
+en-ZA
+hy-AM
+lg
+nso
+son

we also discovered that a restart of the apacha server was needed to pick them up
wenzel also filed bug 610893 for avoiding the PHP library and the json to don't fall out of sync

manually updated the three locations for mac to use OS of 'osx' instead of 'osx64', and replace 'mac64' with 'mac' in the path (part of the continuity with previous releases effort)

Socorro product data updates

Everything looks good. I have checked for "4.0b7" and "4.0b8pre" on the "Products"-"Firefox" table (4th one below Fennec as of writing).

Please leave the following note (or improve it) for following betas:

If you don't see $version and $version-pre on the "Products-Firefox" table
then you should add it. Ignore it if you see those entries on the "Missing Entries" table.

Virus check

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/4.0b7-candidates/build1/
clamdscan .
/mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/4.0b7-candidates/build1/.: OK

----------- SCAN SUMMARY -----------
Infected files: 0
Time: 4089.461 sec (68 m 9 s)

Naming fixup

The buildbot setup leaves us with mac64 directories on the ftp server, and an expectation to use os=osx64 in bouncer. The mozilla.com maintainers would really rather not change from os=osx, so we're making everything match what we did for 4.0b6.

Rename on ftp server

Fix the file locations, but leave a symlink so that betatest keeps working for QA

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/4.0b7-candidates/build1
mv mac64 mac
cd update
mv mac64 mac && ln -s mac mac64

The mac64 symlink was removed after the modified betatest snippets were pushed live (see here).

--Rail 08:47, 6 December 2010 (PST): Removing mac64 symlink causes a failure of updates builder for the next version, which tries to use mac64 directory to download the previous release. I restored the symlink. See bug 616726 for more details.

Fix up *SUMS

On signing box

cd ~/signing-work/firefox-4.0b7/
mkdir signed-build1-redo-SUMS
cp signed-build1/*SUMS signed-build1-redo-SUMS/
cd signed-build1-redo-SUMS
sed -i -e 's/mac64/mac/' *SUMS
../signing/sign-files .
rsync -av -e "ssh -i $HOME/.ssh/ffxbld_dsa" * \
 ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/4.0b7-candidates/build1/

Push to mirrors

Wait for email from driver before continuing

rsync -av --exclude=*tests* --exclude=*crashreporter* --exclude=*.log \
  --exclude=*.txt --exclude=*.checksums --exclude=*unsigned* \
  /pub/mozilla.org/firefox/nightly/4.0b7-candidates/build1/ \
  /pub/mozilla.org/firefox/releases/4.0b7/

# ffbld@stage
cd /pub/mozilla.org/firefox/releases/4.0b7
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=456169 | sed 's/Beta 1/Beta 7/g' > index.html
for dir in `find . -mindepth 1 -type d`; do cp -pv index.html $dir/; done

Send email to r-d when rsync/push to mirrors is done.
Send email to r-d when uptake has reached 10-20k (since it is a beta) for binaries, complete and partial mars.

Final verify

Patch to add ja-JP-mac to the list of locales for 4.0b4, b5, b6. That should test all the partial and complete update paths, but we should also write some lines to make sure the old universals have working update links too.

The only failure is that we don't have ja-JP-mac partials:

FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/4.0b6/20100914073111/Darwin_x86_64-gcc3/ja-JP-mac/releasetest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/4.0b6/20100914073111/Darwin_x86_64-gcc3/ja-JP-mac/releasetest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1

Push snippets to beta channel

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20101109-Firefox-4.0b7-fixed2

Upon GO from release-drivers

~/bin/pushsnip 20101109-Firefox-4.0b7-fixed2

Announce to release-drivers

Remove index.html files

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases/4.0b7/
find . -name index.html
find . -name index.html | xargs rm -v

Release

Sent mail to metrics with the following contents:

Firefox,firefox,firefox,4.0b7,4.0
Firefox,firefox,firefox,4.0b8pre,4.0

Build #	Tag	Changeset
1	GECKO20b7_20101104_RELBRANCH	f7016571b472
1	FIREFOX_4_0b7_BUILD1/FIREFOX_4_0b7_RELEASE	297086a0fb61

Releases/Firefox 4.0b7/BuildNotes

Contents