Releases/Firefox 3.5.10/BuildNotes

Build Engineers

Signed-off Revision(s)

http://hg.mozilla.org/releases/mozilla-1.9.1/rev/6d296f4b8ad9

Build data

Build #	Type	Build ID	Build machine
1	Linux	20100504085637	mv-moz2-linux-ix-slave16
1	Mac	20100504085753	bm-xserve21
1	Windows	20100504093643	mw32-ix-slave22

Notes

Build 1

Used clobberer to clean up previous release
Landed automation version bumps, reconfigured production-master02
- Deleted all the .pyc files in the master dir before reconfig just in case
Kicked off automation

buildbot sendchange --username=armenzg --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5.10 build1" Go

Tagged build repos (as soon as tagging and branching finishes)

# buildbot-configs
hg tag -r 2a7d23c0b20a FIREFOX_3_5_10_RELEASE; hg tag -r 2a7d23c0b20a FIREFOX_3_5_10_BUILD1
hg push
# buildbotcustom
hg tag -r 6490ead4317f FIREFOX_3_5_10_RELEASE; hg tag -r 6490ead4317f FIREFOX_3_5_10_BUILD1
hg push
# tools
hg tag -r 4911ca459d50 FIREFOX_3_5_10_RELEASE; hg tag -r 4911ca459d50 FIREFOX_3_5_10_BUILD1
hg push

Tag

mv-moz2-linux-ix-slave24

First run failed #19 log

remote: ssh_exchange_identification: Connection closed by remote host
abort: no suitable response from remote hg!

Read Tagging_failed_out_part_way_through
I removed the already tagged locales locally from l10n-changeset
I reconfigured
I clobbered again
re-kicked automation

diff -r 2a7d23c0b20a mozilla2/l10n-changesets
--- a/mozilla2/l10n-changesets  Tue May 04 11:14:05 2010 -0400
+++ b/mozilla2/l10n-changesets  Tue May 04 08:35:45 2010 -0700
@@ -1,21 +1,3 @@
-af 13e01380d0ae
-ar b5139ff0f18a
-as cf8a573950c5
-be 33f4b80fbc7e
-bg 47fb5af14c40
-bn-BD 406e0431af75
-bn-IN f1653cf6cf20
-ca 30f4322ce9d1
-cs 06e8039a3a88
-cy 1ec72cb0a4f5
-da 987d06dc1013
-de 75a62aa26f22
-el 4359d2ea676f
-en-GB 5cbd966ebcc0
-eo ce59fce0702c
-es-AR 26ab5157d62c
-es-CL ff0d83ed9733
-es-ES e860ef7ebcf9
 es-MX d8f55a9f91ed
 et 6f16a54f0986
 eu 1c17d7ee65b4

Source

mv-moz2-linux-ix-slave13

No problems

Build/Repack

mv-moz2-linux-ix-slave16, bm-xserve21 and mw32-ix-slave22

Linux: No problems
Win32: No problems
Mac: No problems. 2 red repacks
- KaiRo pushed around that time

abort: data/seamonkey/l10n-changesets.i@fddfb9ec8dd4: no match found!

Removed manually en-US.xpi from candidates dir

Sign

Signing Instructions

I had to trigger this twice since I had typed the wrong signcodepwd
You don't get any error until it is actually going to sign (few minutes after having typed the wrong password)
I checked the signature verification log and everything was OK

L10nVerify

bm-xserve21

No problems

Unthrottle beta channel

Unthrottle previous dot release on beta channel
- This will bring beta users to the current dot release. The day of the release we have to do the same for the "release" users.
- bug 567040 - Unthrottle 3.5.9 -> 3.5.10build1 on beta channel (it was forgotten on beta day)

Generate updates and push betatest snippets

mv-moz2-linux-ix-slave17

No problems

Update verify

mv-moz2-linux-ix-slave17, bm-xserve22 and mw32-ix-slave22

Linux: PASS
MacOSX: PASS
Windows: PASS

Major update generation and verification

3.5.10 -> 3.6.4 build1

mv-moz2-linux-ix-slave24

NOTE for next release: generate the MU the day before going live

I forced "major_update"
Send email to release drivers once the updates are generated (do not wait for major_update_verify to finish). updates available for 3.5.10 -> 3.6.4 build3 on betatest"
Known red for "major_update" builder bug 563935.
Read Major Update & Verification

Verification

Check the last 2 or 3 Build Notes if this is your first time verifying a major update generated in this new way. Carrying forward the known issues might be overkill.

Do this to check for the test failures:

Open each of the logs and search for "https://" (use Apple + G to repeat search) and you will find errors like this just two or three lines before the matching string:

Affects all 3 platforms:
- failed to get partial/complete mar because we don't have "mn" for 3.6.x

Using  https://aus2.mozilla.org/update/1/Firefox/3.5.10/20100504085637/Linux_x86-gcc3/mn/betatest/update.xml?force=1
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.5.10/20100504085637/Linux_x86-gcc3/mn/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1
Using  https://aus2.mozilla.org/update/1/Firefox/3.5.10/20100504085637/Linux_x86-gcc3/mn/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.5.10/20100504085637/Linux_x86-gcc3/mn/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1

- For 'sl' we get these warnings "These were removed between 3.6 and 3.6.2 which is why they're no. No issues here." (confirm this quote from previous notes):

Only in source/firefox/searchplugins: iskalnik-gov-si.xml
Only in source/firefox/searchplugins: najdi-si.xml
Only in source/firefox/searchplugins: yahoo.xml
WARN: non-binary files found in diff
WARN: check_updates returned warning for Linux_x86-gcc3 downloads/firefox-3.5.10.tar.bz2 vs. downloads/firefox-3.6.4.tar.bz2: 2

Linux:
- No test failure that was only Linux specific

MacOSX:
- For all locales:

Contents of source/Firefox.app/Contents/Plug-Ins dir only in source or target 379277599        0 drwxr-xr-x    3 cltbld   staff         102 May  4 10:31 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin
379277600        0 drwxr-xr-x    4 cltbld   staff         136 May  5 08:05 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents
379277604        0 drwxr-xr-x    2 cltbld   staff          68 May  5 08:05 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/MacOS
379277605        0 drwxr-xr-x    3 cltbld   staff         102 May  4 10:31 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources
379277607        0 drwxr-xr-x    3 cltbld   staff         102 May  5 08:05 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj
379277611        0 drwxr-xr-x    2 cltbld   staff          68 May  5 08:05 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj/PrintPDE.nib
WARN: non-binary files found in diff
WARN: check_updates returned warning for Darwin_Universal-gcc3 downloads/Firefox 3.5.10.dmg vs. downloads/Firefox 3.6.4.dmg: 2

Windows:
- The chk files will go away with bug 489961
- For all locales:

Only in source/bin: .autoreg
Files source/bin/freebl3.chk and target/bin/freebl3.chk differ
Files source/bin/nssdbm3.chk and target/bin/nssdbm3.chk differ
Only in source/bin: removed-files
Files source/bin/softokn3.chk and target/bin/softokn3.chk differ
WARN: non-binary files found in diff
WARN: check_updates returned warning for WINNT_x86-msvc downloads/Firefox Setup 3.5.10.exe vs. downloads/Firefox Setup 3.6.4.exe: 2

3.5.10 -> 3.6.3

mv-moz2-linux-ix-slave24

It seems that after all we are waiting for 3.6.4 to release. Nevertheless, I generated the MU for 3.5.10->3.6.3 just in case the plans change again
I had to trigger it a couple of times due to:
- clobber does not clobber "major_update" so it required manual clobbering
- Downloading failed
I have not spent time on checking the verification logs

3.5.10 -> 3.6.4 build7

NOTE for post-mortem:

pm02 was pointing to release-mozilla-central.py instead of release-mozilla-1.9.1.py. Therefore I did not see any "major_update" builder. After a while I realized about this.
pm was turned off so I turned it on. I got a slave and started doing "final_verification".
I landed a change for changing release_config1.py to point at release-mozilla-1.9.1.py. Unfortunately I had landed it on the buildbot-0.8.0 branch and therefore I was not able to see any "major_update" builder.
I started preparing the patches for manual generation of the MU until catlee noticed that the "major_update" builder was there (There was too much going and I probably missed it).
I made sure that now my patches were correctly landed on default and reconfigured.
I have now "force build" the builder

First run failed on the download step:
- bug 573828 - "major update" failed to download file for 3.5.10 -> 3.6.4 MU
I clobbered and re-triggered
I had to trigger for a third time. This time it worked
It seems that downloading from stage-old.mozilla.org is not 100% safe since it is a proxy.

Verification

All platforms
- no update for mn since that doesn't exist for 3.6.4
- sl leaves these searchplugins behind (they were removed between 3.6 and 3.6.2)

iskalnik-gov-si.xml
najdi-si.xml
yahoo.xml

Linux - PASS, no extra differences found.
Windows - PASS, with the usual issues differences for .autoreg, freebl3.chk, nssdbm3.chk, removed-files, softokn3.chk
Mac - PASS, with some leftover directories

Only in source/Firefox.app/Contents: Plug-Ins
Contents of source/Firefox.app/Contents/Plug-Ins dir only in source or target
229503469        0 drwxr-xr-x    3 cltbld   staff         102 May  4 10:31 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin
229503470        0 drwxr-xr-x    4 cltbld   staff         136 Jun 22 18:30 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents
229503474        0 drwxr-xr-x    2 cltbld   staff          68 Jun 22 18:30 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/MacOS
229503475        0 drwxr-xr-x    3 cltbld   staff         102 May  4 10:31 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources
229503477        0 drwxr-xr-x    3 cltbld   staff         102 Jun 22 18:30 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj
229503481        0 drwxr-xr-x    2 cltbld   staff          68 Jun 22 18:30 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj/PrintPDE.nib

Publish updates to beta channel

backupsnip took 18 minutes

# cltbld @ aus-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20100504-Firefox-3.5.10-beta
~/bin/pushsnip 20100504-Firefox-3.5.10-beta

Socorro product data updates

Use crash-stats admin page
"Branch Data Sources"
Added Firefox 3.5.10 and 3.5.11pre
- note that there has to be crashes to "add" - this needs the "bump version" patch to have landed first.
- Adjust the "branch" drop down list to be 1.9.1 and that's it

Update Bouncer

Used bouncer admin page
Added product "Firefox-3.5.10", "Firefox-3.5.10-Complete" and "Firefox-3.5.10-Partial" with "Check Now" checked
Add the file locations as well or you won't see product uptake (I had missed this step)

Final checks before push

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/3.5.10-candidates/build1/
find . ! -user ffxbld
find . ! -group firefox
find . -type f ! -perm 644
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*'
find . -maxdepth 1 -type d ! -perm 2775  -name 'contrib*'

The first check returned Solaris builds, all others returned nothing, as expected.

Virus Scan

This is what I got at first:

[ffxbld@surf build1]$ clamdscan -m /pub/mozilla.org/firefox/nightly/3.5.10-candidates/build1
ERROR: Can't access file /pub/mozilla.org/firefox/nightyl/3.5.10-candidates/build1
/pub/mozilla.org/firefox/nightyl/3.5.10-candidates/build1: No files scanned

Ask IT to login as root or clamav user and run this:

/etc/init.d/clamd start

and then:

[ffxbld@surf build1]$ clamdscan -m /pub/mozilla.org/firefox/nightly/3.5.10-candidates/build1
/pub/mozilla.org/firefox/nightly/3.5.10-candidates/build1: OK

----------- SCAN SUMMARY -----------
Infected files: 0
Time: 1933.489 sec (32 m 13 s)

Push to mirrors

I decided to wait for 3.6.4's uptake to go up first
3.6.4 was pushed to mirrors at 8:13 PDT
I pushed to mirrors at 9:15 PDT.
NOTE: according to reed from mirrors' perspective it is better to push all releases at the same time.

# ffxbld@stage
rsync -av --exclude="*reporter*" --exclude=*.log --exclude=*.txt --exclude=*unsigned* /pub/mozilla.org/firefox/nightly/3.5.10-candidates/build1/ /pub/mozilla.org/firefox/releases/3.5.10/

# cltbld@stage
vim /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude
# replace 3.5.9 with 3.5.10

Final Verification

After all products/files had some uptake the "force build" button was used to start the "final_verification" builder.

We have a bunch of 404 message for cacheboy.net.
nthomas suggested me to chat with adrian on #mirrors
nthomas is verifying the log as well
nthomas suggested me for next time to use something like this:

while [[ 2 > 1 ]]; do curl -sI  http://mozilla.cdn.cacheboy.net/firefox/releases/3.5.10/update/win32/en-US/firefox-3.5.10.complete.mar | grep ^HTTP; done

NOTE: If there had been neither 404s nor 503s in the log I should have marked this step as a PASS.

Push updates to release channel

Push 3.5.x snippets

# cltbld @ aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20100504-Firefox-3.5.10
~/bin/pushsnip 20100504-Firefox-3.5.10

push major update snippets (unadvertised)

~/bin/backupsnip 20100622-Firefox-3.5.10-3.6.4-MU
~/bin/pushsnip   20100622-Firefox-3.5.10-3.6.4-MU

Unthrottle release channel

We have to remove all throttling for the previous dotrelease since only the "release" users are excepted.
- Unthrottle 3.5.9 - patch, tag, and push.

Unthrottle MU

bug 578450 do unthrottled 3.5.10 -> 3.6.6 major update
- [1] remove 3.5.10 throttling, add 3.5.11 throttling (NOTE: I forgot to add "releasetest" in the patch)

Release

# ffxbld@stage
cd /home/ftp/pub/firefox/releases 
rm latest-3.5 && ln -s 3.5.10 latest-3.5

XULRunner build

Discontinued ?

Build #	Tag	Changeset
1	GECKO19110_20100504_RELBRANCH	6d296f4b8ad9
1	FIREFOX_3_5_10_BUILD1 FIREFOX_3_5_10_RELEASE	1e0223b57404

Releases/Firefox 3.5.10/BuildNotes

Contents

Build Engineers

Signed-off Revision(s)

Tags

Build data

Notes

Build 1

Tag

Source

Build/Repack

Sign

L10nVerify

Unthrottle beta channel

Generate updates and push betatest snippets

Update verify

Major update generation and verification

3.5.10 -> 3.6.4 build1

Verification

3.5.10 -> 3.6.3

3.5.10 -> 3.6.4 build7

Verification

Publish updates to beta channel

Socorro product data updates

Update Bouncer

Final checks before push

Virus Scan

Push to mirrors

Final Verification

Push updates to release channel

Unthrottle release channel

Unthrottle MU

Release

XULRunner build

Navigation menu

Search