Releases/Firefox 3.5.10/BuildNotes

From MozillaWiki
Jump to: navigation, search

Build Engineers

armenzg bug 563338

Signed-off Revision(s)


On releases/mozilla-1.9.1:

Build # Tag Changeset
1 GECKO19110_20100504_RELBRANCH
FIREFOX_3_5_10_BUILD1 FIREFOX_3_5_10_RELEASE 1e0223b57404

Build data

Build # Type Build ID Build machine
1 Linux 20100504085637 mv-moz2-linux-ix-slave16
1 Mac 20100504085753 bm-xserve21
1 Windows 20100504093643 mw32-ix-slave22


Build 1

  • Used clobberer to clean up previous release
  • Landed automation version bumps, reconfigured production-master02
    • Deleted all the .pyc files in the master dir before reconfig just in case
  • Kicked off automation
buildbot sendchange --username=armenzg --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5.10 build1" Go
  • Tagged build repos (as soon as tagging and branching finishes)
# buildbot-configs
hg tag -r 2a7d23c0b20a FIREFOX_3_5_10_RELEASE; hg tag -r 2a7d23c0b20a FIREFOX_3_5_10_BUILD1
hg push
# buildbotcustom
hg tag -r 6490ead4317f FIREFOX_3_5_10_RELEASE; hg tag -r 6490ead4317f FIREFOX_3_5_10_BUILD1
hg push
# tools
hg tag -r 4911ca459d50 FIREFOX_3_5_10_RELEASE; hg tag -r 4911ca459d50 FIREFOX_3_5_10_BUILD1
hg push



remote: ssh_exchange_identification: Connection closed by remote host
abort: no suitable response from remote hg!
diff -r 2a7d23c0b20a mozilla2/l10n-changesets
--- a/mozilla2/l10n-changesets  Tue May 04 11:14:05 2010 -0400
+++ b/mozilla2/l10n-changesets  Tue May 04 08:35:45 2010 -0700
@@ -1,21 +1,3 @@
-af 13e01380d0ae
-ar b5139ff0f18a
-as cf8a573950c5
-be 33f4b80fbc7e
-bg 47fb5af14c40
-bn-BD 406e0431af75
-bn-IN f1653cf6cf20
-ca 30f4322ce9d1
-cs 06e8039a3a88
-cy 1ec72cb0a4f5
-da 987d06dc1013
-de 75a62aa26f22
-el 4359d2ea676f
-en-GB 5cbd966ebcc0
-eo ce59fce0702c
-es-AR 26ab5157d62c
-es-CL ff0d83ed9733
-es-ES e860ef7ebcf9
 es-MX d8f55a9f91ed
 et 6f16a54f0986
 eu 1c17d7ee65b4



  • No problems


mv-moz2-linux-ix-slave16, bm-xserve21 and mw32-ix-slave22

  • Linux: No problems
  • Win32: No problems
  • Mac: No problems. 2 red repacks
    • KaiRo pushed around that time
abort: data/seamonkey/l10n-changesets.i@fddfb9ec8dd4: no match found!
  • Removed manually en-US.xpi from candidates dir


Signing Instructions

  • I had to trigger this twice since I had typed the wrong signcodepwd
  • You don't get any error until it is actually going to sign (few minutes after having typed the wrong password)
  • I checked the signature verification log and everything was OK



  • No problems

Unthrottle beta channel

  • Unthrottle previous dot release on beta channel
    • This will bring beta users to the current dot release. The day of the release we have to do the same for the "release" users.
    • bug 567040 - Unthrottle 3.5.9 -> 3.5.10build1 on beta channel (it was forgotten on beta day)

Generate updates and push betatest snippets


  • No problems

Update verify

mv-moz2-linux-ix-slave17, bm-xserve22 and mw32-ix-slave22

  • Linux: PASS
  • MacOSX: PASS
  • Windows: PASS

Major update generation and verification

3.5.10 -> 3.6.4 build1


NOTE for next release: generate the MU the day before going live

  • I forced "major_update"
  • Send email to release drivers once the updates are generated (do not wait for major_update_verify to finish). updates available for 3.5.10 -> 3.6.4 build3 on betatest"
  • Known red for "major_update" builder bug 563935.
  • Read Major Update & Verification
  • Check the last 2 or 3 Build Notes if this is your first time verifying a major update generated in this new way. Carrying forward the known issues might be overkill.
  • Do this to check for the test failures:

Open each of the logs and search for "https://" (use Apple + G to repeat search) and you will find errors like this just two or three lines before the matching string:

  • Affects all 3 platforms:
    • failed to get partial/complete mar because we don't have "mn" for 3.6.x
FAIL: no partial update found for
FAIL: download_mars returned non-zero exit code: 1
FAIL: no complete update found for
FAIL: download_mars returned non-zero exit code: 1
    • For 'sl' we get these warnings "These were removed between 3.6 and 3.6.2 which is why they're no. No issues here." (confirm this quote from previous notes):
Only in source/firefox/searchplugins: iskalnik-gov-si.xml
Only in source/firefox/searchplugins: najdi-si.xml
Only in source/firefox/searchplugins: yahoo.xml
WARN: non-binary files found in diff
WARN: check_updates returned warning for Linux_x86-gcc3 downloads/firefox-3.5.10.tar.bz2 vs. downloads/firefox-3.6.4.tar.bz2: 2
  • Linux:
    • No test failure that was only Linux specific
  • MacOSX:
    • For all locales:
Contents of source/ dir only in source or target 379277599        0 drwxr-xr-x    3 cltbld   staff         102 May  4 10:31 source/
379277600        0 drwxr-xr-x    4 cltbld   staff         136 May  5 08:05 source/
379277604        0 drwxr-xr-x    2 cltbld   staff          68 May  5 08:05 source/
379277605        0 drwxr-xr-x    3 cltbld   staff         102 May  4 10:31 source/
379277607        0 drwxr-xr-x    3 cltbld   staff         102 May  5 08:05 source/
379277611        0 drwxr-xr-x    2 cltbld   staff          68 May  5 08:05 source/
WARN: non-binary files found in diff
WARN: check_updates returned warning for Darwin_Universal-gcc3 downloads/Firefox 3.5.10.dmg vs. downloads/Firefox 3.6.4.dmg: 2
  • Windows:
    • The chk files will go away with bug 489961
    • For all locales:
Only in source/bin: .autoreg
Files source/bin/freebl3.chk and target/bin/freebl3.chk differ
Files source/bin/nssdbm3.chk and target/bin/nssdbm3.chk differ
Only in source/bin: removed-files
Files source/bin/softokn3.chk and target/bin/softokn3.chk differ
WARN: non-binary files found in diff
WARN: check_updates returned warning for WINNT_x86-msvc downloads/Firefox Setup 3.5.10.exe vs. downloads/Firefox Setup 3.6.4.exe: 2
3.5.10 -> 3.6.3


  • It seems that after all we are waiting for 3.6.4 to release. Nevertheless, I generated the MU for 3.5.10->3.6.3 just in case the plans change again
  • I had to trigger it a couple of times due to:
    • clobber does not clobber "major_update" so it required manual clobbering
    • Downloading failed
  • I have not spent time on checking the verification logs
3.5.10 -> 3.6.4 build7

NOTE for post-mortem:

  • pm02 was pointing to instead of Therefore I did not see any "major_update" builder. After a while I realized about this.
  • pm was turned off so I turned it on. I got a slave and started doing "final_verification".
  • I landed a change for changing to point at Unfortunately I had landed it on the buildbot-0.8.0 branch and therefore I was not able to see any "major_update" builder.
  • I started preparing the patches for manual generation of the MU until catlee noticed that the "major_update" builder was there (There was too much going and I probably missed it).
  • I made sure that now my patches were correctly landed on default and reconfigured.
  • I have now "force build" the builder

  • First run failed on the download step:
    • bug 573828 - "major update" failed to download file for 3.5.10 -> 3.6.4 MU
  • I clobbered and re-triggered
  • I had to trigger for a third time. This time it worked
  • It seems that downloading from is not 100% safe since it is a proxy.
  • All platforms
    • no update for mn since that doesn't exist for 3.6.4
    • sl leaves these searchplugins behind (they were removed between 3.6 and 3.6.2)
  • Linux - PASS, no extra differences found.
  • Windows - PASS, with the usual issues differences for .autoreg, freebl3.chk, nssdbm3.chk, removed-files, softokn3.chk
  • Mac - PASS, with some leftover directories
Only in source/ Plug-Ins
Contents of source/ dir only in source or target
229503469        0 drwxr-xr-x    3 cltbld   staff         102 May  4 10:31 source/
229503470        0 drwxr-xr-x    4 cltbld   staff         136 Jun 22 18:30 source/
229503474        0 drwxr-xr-x    2 cltbld   staff          68 Jun 22 18:30 source/
229503475        0 drwxr-xr-x    3 cltbld   staff         102 May  4 10:31 source/
229503477        0 drwxr-xr-x    3 cltbld   staff         102 Jun 22 18:30 source/
229503481        0 drwxr-xr-x    2 cltbld   staff          68 Jun 22 18:30 source/

Publish updates to beta channel

  • backupsnip took 18 minutes
# cltbld @ aus-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20100504-Firefox-3.5.10-beta
~/bin/pushsnip 20100504-Firefox-3.5.10-beta

Socorro product data updates

  • Use crash-stats admin page
  • "Branch Data Sources"
  • Added Firefox 3.5.10 and 3.5.11pre
    • note that there has to be crashes to "add" - this needs the "bump version" patch to have landed first.
    • Adjust the "branch" drop down list to be 1.9.1 and that's it

Update Bouncer

  • Used bouncer admin page
  • Added product "Firefox-3.5.10", "Firefox-3.5.10-Complete" and "Firefox-3.5.10-Partial" with "Check Now" checked
  • Add the file locations as well or you won't see product uptake (I had missed this step)

Final checks before push

# ffxbld@stage
cd /pub/
find . ! -user ffxbld
find . ! -group firefox
find . -type f ! -perm 644
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*'
find . -maxdepth 1 -type d ! -perm 2775  -name 'contrib*'

The first check returned Solaris builds, all others returned nothing, as expected.

Virus Scan

This is what I got at first:

[ffxbld@surf build1]$ clamdscan -m /pub/
ERROR: Can't access file /pub/
/pub/ No files scanned

Ask IT to login as root or clamav user and run this:

/etc/init.d/clamd start

and then:

[ffxbld@surf build1]$ clamdscan -m /pub/
/pub/ OK

----------- SCAN SUMMARY -----------
Infected files: 0
Time: 1933.489 sec (32 m 13 s)

Push to mirrors

  • I decided to wait for 3.6.4's uptake to go up first
  • 3.6.4 was pushed to mirrors at 8:13 PDT
  • I pushed to mirrors at 9:15 PDT.
  • NOTE: according to reed from mirrors' perspective it is better to push all releases at the same time.
# ffxbld@stage
rsync -av --exclude="*reporter*" --exclude=*.log --exclude=*.txt --exclude=*unsigned* /pub/ /pub/
# cltbld@stage
vim /pub/
# replace 3.5.9 with 3.5.10

Final Verification

After all products/files had some uptake the "force build" button was used to start the "final_verification" builder.

  • We have a bunch of 404 message for
  • nthomas suggested me to chat with adrian on #mirrors
  • nthomas is verifying the log as well
  • nthomas suggested me for next time to use something like this:
while [[ 2 > 1 ]]; do curl -sI | grep ^HTTP; done

NOTE: If there had been neither 404s nor 503s in the log I should have marked this step as a PASS.

Push updates to release channel

  • Push 3.5.x snippets
# cltbld @ aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20100504-Firefox-3.5.10
~/bin/pushsnip 20100504-Firefox-3.5.10
  • push major update snippets (unadvertised)
~/bin/backupsnip 20100622-Firefox-3.5.10-3.6.4-MU
~/bin/pushsnip   20100622-Firefox-3.5.10-3.6.4-MU

Unthrottle release channel

  • We have to remove all throttling for the previous dotrelease since only the "release" users are excepted.

Unthrottle MU

  • bug 578450 do unthrottled 3.5.10 -> 3.6.6 major update
    • [1] remove 3.5.10 throttling, add 3.5.11 throttling (NOTE: I forgot to add "releasetest" in the patch)


# ffxbld@stage
cd /home/ftp/pub/firefox/releases 
rm latest-3.5 && ln -s 3.5.10 latest-3.5

XULRunner build

Discontinued ?