Releases/Firefox 3.5.4/BuildNotes

From MozillaWiki
Jump to navigation Jump to search

Build Engineers

coop https://bugzilla.mozilla.org/show_bug.cgi?id=519365

Signed-off Revision(s)

Build1: 09522dd1776e
Build2: e178dbd2a389

Tags

On releases/mozilla-1.9.1:

Build # Tag Changeset
1 GECKO1914_20091006_RELBRANCH 09522dd1776e
FIREFOX_3_5_4_BUILD1 FIREFOX_3_5_4_RELEASE f2eb72702e5b
2 GECKO1914_20091006_RELBRANCH 4a35b0f000c7
FIREFOX_3_5_4_BUILD2 FIREFOX_3_5_4_RELEASE e178dbd2a389
3 GECKO1914_20091006_RELBRANCH ???
FIREFOX_3_5_4_BUILD3 FIREFOX_3_5_4_RELEASE ???

Build data

Build # Type Build ID Build machine
1 [Windows installer/zip] 20091007001339 moz2-win32-slave44
[Mac compressed] 20091006224018 moz2-darwin9-slave13
[Linux compressed] 20091006225203 moz2-linux-slave38
2 [Windows installer/zip] 20091014223740 moz2-win32-slave47
[Mac compressed] 20091014210210 moz2-darwin9-slave14
[Linux compressed] 20091014211016 moz2-linux-slave35
3 [Windows installer/zip] moz2-win32-slave52
[Mac compressed] moz2-darwin9-slave16
[Linux compressed] moz2-linux-slave36

Notes

  • Useful docs and FAQs here.
  • Done on production-master02

Build 1

  • Cleaned up previous releases' build dirs
    • Went through logs on the master to gather hostnames for linux/win32. Just cleared all possible buildslaves on mac. Used csshX on linux/mac, RDP as Administrator on win32.
      • Linux (moz2-linux-slave[18-23,25])
        • /builds/slave/{tag,source,linux_build,linux_repack,updates,linux_update_verify,final_verification}/*
      • Mac (bm-xserve22, moz2-darwin9-slave[13-18])
        • /builds/slave/{macosx_build,macosx_repack,l10n_verification,macosx_update_verify}/*
      • Win32 (moz2-win32-slave[32-39]
        • /e/builds/moz2_slave/{win32_build,win32_repack,win32_update_verify}/*
  • l10n-changesets updated in http://hg.mozilla.org/build/buildbot-configs/rev/61377630502c
  • Land version bump, reconfig production-master02
  • Kicked-off automation:
buildbot sendchange --username=coop --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5.4 build1" DOOOOEEEET

Tag

  • (slave: moz2-linux-slave35) No problems

Source

  • (slave: moz2-linux-slave35) No problems

Build/Repack

  • Linux: No problems
  • Mac: No problems
  • Win32: No problems

Sign

  • Cleaned up Firefox 3.0.12 signing dirs for a little more space (22GB total)
  • First attempt failed, exiting with a make error.
    • Removed the signed-build1/ dir, and tried again successfully using -j1
  • instructions

L10nVerify

  • (slave: moz2-darwin9-slave16) No problems

Generate updates and push betatest snippets

  • 1st attempt: (slave: moz2-linux-slave36) patcher_create_patches failed: failure log
Exception: cmd failed mar -C /builds/slave/updates/build/tmpsVvBD6-fastmode/223 ...
  • 2nd attempt: (slave: moz2-linux-slave30) diff_patcher_config failed (didn't remove bump step) and patcher_create_patches failed: failure log
Exception: cmd failed mar -x /builds/slave/updates/build/temp/firefox/3.5.4/ftp/firefox-3.5.4.sv-SE.win32.complete.mar
  • 3rd attempt: (slave: moz2-linux-slave49) checkout_patcher failed. This slave had never connected to the CVS server before. Updated the linux slave migration doc to include the CVS server.
  • 4th attempt: (slave: moz2-linux-slave47) backupsnip timed out (timeout is 7200s)
  • at this point, I gave up with automation, took moz2-linux-slave47 out of rotation, checked the updates steps for the 3.5.3 run, and ran the remaining commands (backupsnip/pushsnip/wait/upload_snippets) by hand (updated for 3.5.4, of course)
[cltbld@moz2-linux-slave47 build]$ ssh -l cltbld aus2-staging.mozilla.org ~/bin/backupsnip 20091007-Firefox-3.5.4-test
  • OK, that failed too. We hit a hard ssh timeout of (2 hours). joduinn filed bug 521139 to track the slowdown on aus2-staging. The next step was running the command directly on aus2-staging in a screen session:
bash-3.2$ time ~/bin/backupsnip 20091007-Firefox3.5.4-test
Running /bin/tar cfvj /opt/aus2/snippets/backup/20091007-4-pre-20091007-Firefox-3.5.4-test.tar.bz2 .
bash-3.2$ time ~/bin/pushsnip 20091007-Firefox-3.5.4-test
  • The above commands succeeded. backupsnip took 60 minutes, pushsnip took almost 3 minutes.
  • then re-ran the rsync command (upload_snippets) on the original build slave (moz2-linux-slave47):
[cltbld@moz2-linux-slave47 ~]$ cd /builds/slave/updates/build/temp/firefox/3.5.3-3.5.4
[cltbld@moz2-linux-slave47 3.5.3-3.5.4]$ rsync -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091007-Firefox-3.5.4-beta

Notes: I'm unsure as to why the early mar failures occurred and why the patch generation magically worked on the 4th attempt. I *think* the slaves in question might have been running out of space, but the slaves were being re-used by buildbot so quickly after the failed step that it's hard to tell. The clean_old_builds step passed in all cases, so maybe we need to up the required space?

Update verify

  • Linux: (slave: moz2-linux-slave46) No problems
  • Mac: (slave: bm-xserve22) No problems
  • Win32: (slave: moz2-win32-slave34) No problems

Publish updates to beta channel

# cltbld @ aus2-staging
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20091007-Firefox-3.5.4-beta
Running /bin/tar cfvj /opt/aus2/snippets/backup/20091009-1-pre-20091007-Firefox-3.5.4-beta.tar.bz2 .
real    67m23.979s
user    0m55.489s
sys     2m2.785s
~/bin/pushsnip 20091007-Firefox-3.5.4-beta

XULRunner build

Create the builds by going to the buildbot waterfall for mozilla-1.9.1 and forcing builds for linux (not x64), mac and windows xulrunner. Set the branch to 'releases/mozilla-1.9.1' and revision to 'FIREFOX_3_5_4_RELEASE'.

When they finish, as xrbld@stage move the dir from /pub/mozilla.org/xulrunner/nightly/2009/10/2009-10-xx-xx-mozilla-1.9.1/ to ..../nightly/1.9.1.4-candidates/build1. Remove the broken symlink at xulrunner/nightly/2009-10-xx-xx-mozilla-1.9.1. Contact mfinkle and dtownsend for smoketest QA.

RESPIN

Build 2

  • Cleaned up build dirs from build 1
    • Went through builders on the master to gather hostnames for linux/win32/mac. Used cssh on linux/mac, RDP as Administrator on win32.
      • Linux (moz2-linux-slave[30,35,36,38,40,43-50])
        • /builds/slave/{tag,source,linux_build,linux_repack,updates,linux_update_verify,final_verification}/*
      • Mac (bm-xserve22, moz2-darwin9-slave[13-18])
        • /builds/slave/{macosx_build,macosx_repack,l10n_verification,macosx_update_verify}/*
      • Win32 (moz2-win32-slave[34,44,52-59])
        • /e/builds/moz2_slave/{win32_build,win32_repack,win32_update_verify}/*

Tag

  • attempt 1 (slave: moz2-linux-slave44): initial reconfig didn't take due to a local typo I introduced while checking that the config was updated. I didn't notice because the reconfig usually times out anyway :/
  • attempt 2 (slave: moz2-linux-slave44): failed because I hadn't cleaned out the release dirs after the previous attempt.
  • attempt 3 (slave: moz2-linux-slave33): third time's the charm.

Source

  • (slave: moz2-linux-slave37) No problems

Build/Repack

  • Linux: No problems
  • Mac: No problems
  • Win32: No problems

Sign

L10nVerify

  • (slave: bm-xserve22) No problems

Generate updates and push betatest snippets

  • first backupsnip completed properly before respin was called, which is better than can be said for build 1.

RESPIN

Build 3

  • Cleaned up build dirs from build 2
    • Went through builders on the master to gather hostnames for linux/win32/mac. Used cssh on linux/mac, RDP as Administrator on win32.
      • Linux (moz2-linux-slave[20,22,33,35,37,44])
        • /builds/slave/{tag,source,linux_build,linux_repack,updates,linux_update_verify,final_verification}/*
      • Mac (bm-xserve[11,19,22], moz2-darwin9-slave14)
        • /builds/slave/{macosx_build,macosx_repack,l10n_verification,macosx_update_verify}/*
      • Win32 (moz2-win32-slave[32,33,39,47])
        • /e/builds/moz2_slave/{win32_build,win32_repack,win32_update_verify}/*

Tag

  • (slave: moz2-linux-slave39) KaiRo re-tagged mozilla-1.9.1 in the middle of our tagging step, so the tagging step failed on mozilla-1.9.1. All locales were tagged fine though
  • to recover, I logged into the slave (moz2-linux-slave39), enabled the hg rebase extension in the .hgrc file, and then ran the following commands:
cd /builds/slave/tag/mozilla-1.9.1
hg pull --rebase
hg push -e 'ssh -l ffxbld -i ~cltbld/.ssh/ffxbld_dsa' -f ssh://hg.mozilla.org/releases/mozilla-1.9.1
  • I then disabled the rebase extension in the .hgrc file again

Source

  • (slave: moz2-linux-slave37) forced from the builder page, no problems

Build/Repack

  • Linux: en-US build forced from the builder page
  • Mac: en-US build forced from the builder page
  • Win32: en-US build forced from the builder page
  • to force the l10n repacks, I had to use the same process as bhearsum used for 3.6b1:
# on coop's machine:
hg clone http://hg.mozilla.org/users/bhearsum_mozilla.com/tools tools-bhearsum
cd tools-bhearsum/buildbot-helpers
wget http://people.mozilla.org/~coop/files/3.5.4/shipped-locales-linux
wget http://people.mozilla.org/~coop/files/3.5.4/shipped-locales-macosx
wget http://people.mozilla.org/~coop/files/3.5.4/shipped-locales-win32
# after linux en-US finished:
time python force_release_l10n.py -m http://production-master02.build.mozilla.org:8010 -v -t FIREFOX_3_5_4_RELEASE -s shipped-locales-linux -n coop 2>&1 | tee linux_repack_trigger.log
# after mac en-US finished
time python force_release_l10n.py -m http://production-master02.build.mozilla.org:8010 -v -t FIREFOX_3_5_4_RELEASE -s shipped-locales-macosx -n coop 2>&1 | tee linux_repack_trigger.log
# after win32 en-US finished
time python force_release_l10n.py -m http://production-master02.build.mozilla.org:8010 -v -t FIREFOX_3_5_4_RELEASE -s shipped-locales-win32 -n coop 2>&1 | tee linux_repack_trigger.log

Sign

L10nVerify

  • (slave: )

Generate updates and push betatest snippets

Update verify

  • Linux:
  • Mac:
  • Win32:

Publish updates to beta channel

Update Bouncer

Final checks before push

Push to mirrors

Final Verification

Push updates to release channel

Release

XULRunner build

Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_3.5.4/BuildNotes&oldid=176151"