- 1 Build Engineers
- 2 Signed-off Revision(s)
- 3 Tags
- 4 Build data
- 5 Notes
- 5.1 Build 1
- 5.1.1 Tag
- 5.1.2 Source
- 5.1.3 Build/Repack
- 5.1.4 Sign
- 5.1.5 L10nVerify
- 5.1.6 Unthrottle beta channel
- 5.1.7 Generate updates and push betatest snippets
- 5.1.8 Update verify
- 5.1.9 Major update generation and verification
- 5.1.10 Publish updates to beta channel
- 5.1.11 Socorro product data updates
- 5.1.12 Update Bouncer
- 5.1.13 Final checks before push
- 5.1.14 Virus Scan
- 5.1.15 Push to mirrors
- 5.1.16 Final Verification
- 5.1.17 Push updates to release channel
- 5.1.18 Unthrottle release channel
- 5.1.19 Unthrottle MU
- 5.1.20 Release
- 5.1.21 XULRunner build
- 5.1 Build 1
armenzg bug 563338
|FIREFOX_3_5_10_BUILD1 FIREFOX_3_5_10_RELEASE|| 1e0223b57404|
|Build #||Type||Build ID||Build machine|
- Used clobberer to clean up previous release
- Landed automation version bumps, reconfigured production-master02
- Deleted all the .pyc files in the master dir before reconfig just in case
- Kicked off automation
buildbot sendchange --username=armenzg --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5.10 build1" Go
- Tagged build repos (as soon as tagging and branching finishes)
# buildbot-configs hg tag -r 2a7d23c0b20a FIREFOX_3_5_10_RELEASE; hg tag -r 2a7d23c0b20a FIREFOX_3_5_10_BUILD1 hg push # buildbotcustom hg tag -r 6490ead4317f FIREFOX_3_5_10_RELEASE; hg tag -r 6490ead4317f FIREFOX_3_5_10_BUILD1 hg push # tools hg tag -r 4911ca459d50 FIREFOX_3_5_10_RELEASE; hg tag -r 4911ca459d50 FIREFOX_3_5_10_BUILD1 hg push
- First run failed #19 log
remote: ssh_exchange_identification: Connection closed by remote host abort: no suitable response from remote hg!
- Read Tagging_failed_out_part_way_through
- I removed the already tagged locales locally from l10n-changeset
- I reconfigured
- I clobbered again
- re-kicked automation
diff -r 2a7d23c0b20a mozilla2/l10n-changesets --- a/mozilla2/l10n-changesets Tue May 04 11:14:05 2010 -0400 +++ b/mozilla2/l10n-changesets Tue May 04 08:35:45 2010 -0700 @@ -1,21 +1,3 @@ -af 13e01380d0ae -ar b5139ff0f18a -as cf8a573950c5 -be 33f4b80fbc7e -bg 47fb5af14c40 -bn-BD 406e0431af75 -bn-IN f1653cf6cf20 -ca 30f4322ce9d1 -cs 06e8039a3a88 -cy 1ec72cb0a4f5 -da 987d06dc1013 -de 75a62aa26f22 -el 4359d2ea676f -en-GB 5cbd966ebcc0 -eo ce59fce0702c -es-AR 26ab5157d62c -es-CL ff0d83ed9733 -es-ES e860ef7ebcf9 es-MX d8f55a9f91ed et 6f16a54f0986 eu 1c17d7ee65b4
- No problems
mv-moz2-linux-ix-slave16, bm-xserve21 and mw32-ix-slave22
- Linux: No problems
- Win32: No problems
- Mac: No problems. 2 red repacks
- KaiRo pushed around that time
abort: data/seamonkey/l10n-changesets.i@fddfb9ec8dd4: no match found!
- Removed manually en-US.xpi from candidates dir
- I had to trigger this twice since I had typed the wrong signcodepwd
- You don't get any error until it is actually going to sign (few minutes after having typed the wrong password)
- I checked the signature verification log and everything was OK
- No problems
Unthrottle beta channel
- Unthrottle previous dot release on beta channel
- This will bring beta users to the current dot release. The day of the release we have to do the same for the "release" users.
- bug 567040 - Unthrottle 3.5.9 -> 3.5.10build1 on beta channel (it was forgotten on beta day)
Generate updates and push betatest snippets
- No problems
mv-moz2-linux-ix-slave17, bm-xserve22 and mw32-ix-slave22
- Linux: PASS
- MacOSX: PASS
- Windows: PASS
Major update generation and verification
3.5.10 -> 3.6.4 build1
NOTE for next release: generate the MU the day before going live
- I forced "major_update"
- Send email to release drivers once the updates are generated (do not wait for major_update_verify to finish). updates available for 3.5.10 -> 3.6.4 build3 on betatest"
- Known red for "major_update" builder bug 563935.
- Read Major Update & Verification
- Check the last 2 or 3 Build Notes if this is your first time verifying a major update generated in this new way. Carrying forward the known issues might be overkill.
- Do this to check for the test failures:
Open each of the logs and search for "https://" (use Apple + G to repeat search) and you will find errors like this just two or three lines before the matching string:
- Affects all 3 platforms:
- failed to get partial/complete mar because we don't have "mn" for 3.6.x
Using https://aus2.mozilla.org/update/1/Firefox/3.5.10/20100504085637/Linux_x86-gcc3/mn/betatest/update.xml?force=1 FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.5.10/20100504085637/Linux_x86-gcc3/mn/betatest/update.xml?force=1 FAIL: download_mars returned non-zero exit code: 1 Using https://aus2.mozilla.org/update/1/Firefox/3.5.10/20100504085637/Linux_x86-gcc3/mn/betatest/update.xml?force=1 FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.5.10/20100504085637/Linux_x86-gcc3/mn/betatest/update.xml?force=1 FAIL: download_mars returned non-zero exit code: 1
- For 'sl' we get these warnings "These were removed between 3.6 and 3.6.2 which is why they're no. No issues here." (confirm this quote from previous notes):
Only in source/firefox/searchplugins: iskalnik-gov-si.xml Only in source/firefox/searchplugins: najdi-si.xml Only in source/firefox/searchplugins: yahoo.xml WARN: non-binary files found in diff WARN: check_updates returned warning for Linux_x86-gcc3 downloads/firefox-3.5.10.tar.bz2 vs. downloads/firefox-3.6.4.tar.bz2: 2
- No test failure that was only Linux specific
- For all locales:
Contents of source/Firefox.app/Contents/Plug-Ins dir only in source or target 379277599 0 drwxr-xr-x 3 cltbld staff 102 May 4 10:31 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin 379277600 0 drwxr-xr-x 4 cltbld staff 136 May 5 08:05 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents 379277604 0 drwxr-xr-x 2 cltbld staff 68 May 5 08:05 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/MacOS 379277605 0 drwxr-xr-x 3 cltbld staff 102 May 4 10:31 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources 379277607 0 drwxr-xr-x 3 cltbld staff 102 May 5 08:05 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj 379277611 0 drwxr-xr-x 2 cltbld staff 68 May 5 08:05 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj/PrintPDE.nib WARN: non-binary files found in diff WARN: check_updates returned warning for Darwin_Universal-gcc3 downloads/Firefox 3.5.10.dmg vs. downloads/Firefox 3.6.4.dmg: 2
- The chk files will go away with bug 489961
- For all locales:
Only in source/bin: .autoreg Files source/bin/freebl3.chk and target/bin/freebl3.chk differ Files source/bin/nssdbm3.chk and target/bin/nssdbm3.chk differ Only in source/bin: removed-files Files source/bin/softokn3.chk and target/bin/softokn3.chk differ WARN: non-binary files found in diff WARN: check_updates returned warning for WINNT_x86-msvc downloads/Firefox Setup 3.5.10.exe vs. downloads/Firefox Setup 3.6.4.exe: 2
3.5.10 -> 3.6.3
- It seems that after all we are waiting for 3.6.4 to release. Nevertheless, I generated the MU for 3.5.10->3.6.3 just in case the plans change again
- I had to trigger it a couple of times due to:
- clobber does not clobber "major_update" so it required manual clobbering
- Downloading failed
- I have not spent time on checking the verification logs
3.5.10 -> 3.6.4 build7
NOTE for post-mortem:
- pm02 was pointing to release-mozilla-central.py instead of release-mozilla-1.9.1.py. Therefore I did not see any "major_update" builder. After a while I realized about this.
- pm was turned off so I turned it on. I got a slave and started doing "final_verification".
- I landed a change for changing release_config1.py to point at release-mozilla-1.9.1.py. Unfortunately I had landed it on the buildbot-0.8.0 branch and therefore I was not able to see any "major_update" builder.
- I started preparing the patches for manual generation of the MU until catlee noticed that the "major_update" builder was there (There was too much going and I probably missed it).
- I made sure that now my patches were correctly landed on default and reconfigured.
- I have now "force build" the builder
- First run failed on the download step:
- bug 573828 - "major update" failed to download file for 3.5.10 -> 3.6.4 MU
- I clobbered and re-triggered
- I had to trigger for a third time. This time it worked
- It seems that downloading from stage-old.mozilla.org is not 100% safe since it is a proxy.
- All platforms
- no update for mn since that doesn't exist for 3.6.4
- sl leaves these searchplugins behind (they were removed between 3.6 and 3.6.2)
iskalnik-gov-si.xml najdi-si.xml yahoo.xml
- Linux - PASS, no extra differences found.
- Windows - PASS, with the usual issues differences for .autoreg, freebl3.chk, nssdbm3.chk, removed-files, softokn3.chk
- Mac - PASS, with some leftover directories
Only in source/Firefox.app/Contents: Plug-Ins Contents of source/Firefox.app/Contents/Plug-Ins dir only in source or target 229503469 0 drwxr-xr-x 3 cltbld staff 102 May 4 10:31 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin 229503470 0 drwxr-xr-x 4 cltbld staff 136 Jun 22 18:30 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents 229503474 0 drwxr-xr-x 2 cltbld staff 68 Jun 22 18:30 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/MacOS 229503475 0 drwxr-xr-x 3 cltbld staff 102 May 4 10:31 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources 229503477 0 drwxr-xr-x 3 cltbld staff 102 Jun 22 18:30 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj 229503481 0 drwxr-xr-x 2 cltbld staff 68 Jun 22 18:30 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj/PrintPDE.nib
Publish updates to beta channel
- backupsnip took 18 minutes
# cltbld @ aus-staging cd /opt/aus2/snippets/staging ~/bin/backupsnip 20100504-Firefox-3.5.10-beta ~/bin/pushsnip 20100504-Firefox-3.5.10-beta
Socorro product data updates
- Use crash-stats admin page
- "Branch Data Sources"
- Added Firefox 3.5.10 and 3.5.11pre
- note that there has to be crashes to "add" - this needs the "bump version" patch to have landed first.
- Adjust the "branch" drop down list to be 1.9.1 and that's it
- Used bouncer admin page
- Added product "Firefox-3.5.10", "Firefox-3.5.10-Complete" and "Firefox-3.5.10-Partial" with "Check Now" checked
- Add the file locations as well or you won't see product uptake (I had missed this step)
Final checks before push
# ffxbld@stage cd /pub/mozilla.org/firefox/nightly/3.5.10-candidates/build1/ find . ! -user ffxbld find . ! -group firefox find . -type f ! -perm 644 find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*' find . -maxdepth 1 -type d ! -perm 2775 -name 'contrib*'
The first check returned Solaris builds, all others returned nothing, as expected.
This is what I got at first:
[ffxbld@surf build1]$ clamdscan -m /pub/mozilla.org/firefox/nightly/3.5.10-candidates/build1 ERROR: Can't access file /pub/mozilla.org/firefox/nightyl/3.5.10-candidates/build1 /pub/mozilla.org/firefox/nightyl/3.5.10-candidates/build1: No files scanned
Ask IT to login as root or clamav user and run this:
[ffxbld@surf build1]$ clamdscan -m /pub/mozilla.org/firefox/nightly/3.5.10-candidates/build1 /pub/mozilla.org/firefox/nightly/3.5.10-candidates/build1: OK ----------- SCAN SUMMARY ----------- Infected files: 0 Time: 1933.489 sec (32 m 13 s)
Push to mirrors
- I decided to wait for 3.6.4's uptake to go up first
- 3.6.4 was pushed to mirrors at 8:13 PDT
- I pushed to mirrors at 9:15 PDT.
- NOTE: according to reed from mirrors' perspective it is better to push all releases at the same time.
# ffxbld@stage rsync -av --exclude="*reporter*" --exclude=*.log --exclude=*.txt --exclude=*unsigned* /pub/mozilla.org/firefox/nightly/3.5.10-candidates/build1/ /pub/mozilla.org/firefox/releases/3.5.10/
# cltbld@stage vim /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude # replace 3.5.9 with 3.5.10
After all products/files had some uptake the "force build" button was used to start the "final_verification" builder.
- We have a bunch of 404 message for cacheboy.net.
- nthomas suggested me to chat with adrian on #mirrors
- nthomas is verifying the log as well
- nthomas suggested me for next time to use something like this:
while [[ 2 > 1 ]]; do curl -sI http://mozilla.cdn.cacheboy.net/firefox/releases/3.5.10/update/win32/en-US/firefox-3.5.10.complete.mar | grep ^HTTP; done
NOTE: If there had been neither 404s nor 503s in the log I should have marked this step as a PASS.
Push updates to release channel
- Push 3.5.x snippets
# cltbld @ aus2-staging cd /opt/aus2/snippets/staging ~/bin/backupsnip 20100504-Firefox-3.5.10 ~/bin/pushsnip 20100504-Firefox-3.5.10
- push major update snippets (unadvertised)
~/bin/backupsnip 20100622-Firefox-3.5.10-3.6.4-MU ~/bin/pushsnip 20100622-Firefox-3.5.10-3.6.4-MU
Unthrottle release channel
- We have to remove all throttling for the previous dotrelease since only the "release" users are excepted.
- bug 578450 do unthrottled 3.5.10 -> 3.6.6 major update
-  remove 3.5.10 throttling, add 3.5.11 throttling (NOTE: I forgot to add "releasetest" in the patch)
# ffxbld@stage cd /home/ftp/pub/firefox/releases rm latest-3.5 && ln -s 3.5.10 latest-3.5