Releases/Firefox 3.5.4/BuildNotes

From MozillaWiki
Jump to navigation Jump to search

Build Engineers

coop https://bugzilla.mozilla.org/show_bug.cgi?id=519365

Signed-off Revision(s)

Build1: 09522dd1776e
Build2: e178dbd2a389

Tags

On releases/mozilla-1.9.1:

Build # Tag Changeset
1 GECKO1914_20091006_RELBRANCH 09522dd1776e
FIREFOX_3_5_4_BUILD1 FIREFOX_3_5_4_RELEASE f2eb72702e5b
2 GECKO1914_20091006_RELBRANCH 4a35b0f000c7
FIREFOX_3_5_4_BUILD2 FIREFOX_3_5_4_RELEASE e178dbd2a389
3 GECKO1914_20091006_RELBRANCH ???
FIREFOX_3_5_4_BUILD3 FIREFOX_3_5_4_RELEASE ???

Build data

Build # Type Build ID Build machine
1 [Windows installer/zip] 20091007001339 moz2-win32-slave44
[Mac compressed] 20091006224018 moz2-darwin9-slave13
[Linux compressed] 20091006225203 moz2-linux-slave38
2 [Windows installer/zip] 20091014223740 moz2-win32-slave47
[Mac compressed] 20091014210210 moz2-darwin9-slave14
[Linux compressed] 20091014211016 moz2-linux-slave35
3 [Windows installer/zip] 20091016092926 moz2-win32-slave52
[Mac compressed] 20091016081620 moz2-darwin9-slave16
[Linux compressed] 20091016081727 moz2-linux-slave36

Notes

  • Useful docs and FAQs here.
  • Done on production-master02

Build 1

  • Cleaned up previous releases' build dirs
    • Went through logs on the master to gather hostnames for linux/win32. Just cleared all possible buildslaves on mac. Used csshX on linux/mac, RDP as Administrator on win32.
      • Linux (moz2-linux-slave[18-23,25])
        • /builds/slave/{tag,source,linux_build,linux_repack,updates,linux_update_verify,final_verification}/*
      • Mac (bm-xserve22, moz2-darwin9-slave[13-18])
        • /builds/slave/{macosx_build,macosx_repack,l10n_verification,macosx_update_verify}/*
      • Win32 (moz2-win32-slave[32-39]
        • /e/builds/moz2_slave/{win32_build,win32_repack,win32_update_verify}/*
  • l10n-changesets updated in http://hg.mozilla.org/build/buildbot-configs/rev/61377630502c
  • Land version bump, reconfig production-master02
  • Kicked-off automation:
buildbot sendchange --username=coop --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5.4 build1" DOOOOEEEET

Tag

  • (slave: moz2-linux-slave35) No problems

Source

  • (slave: moz2-linux-slave35) No problems

Build/Repack

  • Linux: No problems
  • Mac: No problems
  • Win32: No problems

Sign

  • Cleaned up Firefox 3.0.12 signing dirs for a little more space (22GB total)
  • First attempt failed, exiting with a make error.
    • Removed the signed-build1/ dir, and tried again successfully using -j1
  • instructions

L10nVerify

  • (slave: moz2-darwin9-slave16) No problems

Generate updates and push betatest snippets

  • 1st attempt: (slave: moz2-linux-slave36) patcher_create_patches failed: failure log
Exception: cmd failed mar -C /builds/slave/updates/build/tmpsVvBD6-fastmode/223 ...
  • 2nd attempt: (slave: moz2-linux-slave30) diff_patcher_config failed (didn't remove bump step) and patcher_create_patches failed: failure log
Exception: cmd failed mar -x /builds/slave/updates/build/temp/firefox/3.5.4/ftp/firefox-3.5.4.sv-SE.win32.complete.mar
  • 3rd attempt: (slave: moz2-linux-slave49) checkout_patcher failed. This slave had never connected to the CVS server before. Updated the linux slave migration doc to include the CVS server.
  • 4th attempt: (slave: moz2-linux-slave47) backupsnip timed out (timeout is 7200s)
  • at this point, I gave up with automation, took moz2-linux-slave47 out of rotation, checked the updates steps for the 3.5.3 run, and ran the remaining commands (backupsnip/pushsnip/wait/upload_snippets) by hand (updated for 3.5.4, of course)
[cltbld@moz2-linux-slave47 build]$ ssh -l cltbld aus2-staging.mozilla.org ~/bin/backupsnip 20091007-Firefox-3.5.4-test
  • OK, that failed too. We hit a hard ssh timeout of (2 hours). joduinn filed bug 521139 to track the slowdown on aus2-staging. The next step was running the command directly on aus2-staging in a screen session:
bash-3.2$ time ~/bin/backupsnip 20091007-Firefox3.5.4-test
Running /bin/tar cfvj /opt/aus2/snippets/backup/20091007-4-pre-20091007-Firefox-3.5.4-test.tar.bz2 .
bash-3.2$ time ~/bin/pushsnip 20091007-Firefox-3.5.4-test
  • The above commands succeeded. backupsnip took 60 minutes, pushsnip took almost 3 minutes.
  • then re-ran the rsync command (upload_snippets) on the original build slave (moz2-linux-slave47):
[cltbld@moz2-linux-slave47 ~]$ cd /builds/slave/updates/build/temp/firefox/3.5.3-3.5.4
[cltbld@moz2-linux-slave47 3.5.3-3.5.4]$ rsync -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091007-Firefox-3.5.4-beta

Notes: I'm unsure as to why the early mar failures occurred and why the patch generation magically worked on the 4th attempt. I *think* the slaves in question might have been running out of space, but the slaves were being re-used by buildbot so quickly after the failed step that it's hard to tell. The clean_old_builds step passed in all cases, so maybe we need to up the required space?

Update verify

  • Linux: (slave: moz2-linux-slave46) No problems
  • Mac: (slave: bm-xserve22) No problems
  • Win32: (slave: moz2-win32-slave34) No problems

Publish updates to beta channel

# cltbld @ aus2-staging
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20091007-Firefox-3.5.4-beta
Running /bin/tar cfvj /opt/aus2/snippets/backup/20091009-1-pre-20091007-Firefox-3.5.4-beta.tar.bz2 .
real    67m23.979s
user    0m55.489s
sys     2m2.785s
~/bin/pushsnip 20091007-Firefox-3.5.4-beta

XULRunner build

Create the builds by going to the buildbot waterfall for mozilla-1.9.1 and forcing builds for linux (not x64), mac and windows xulrunner. Set the branch to 'releases/mozilla-1.9.1' and revision to 'FIREFOX_3_5_4_RELEASE'.

When they finish, as xrbld@stage move the dir from /pub/mozilla.org/xulrunner/nightly/2009/10/2009-10-xx-xx-mozilla-1.9.1/ to ..../nightly/1.9.1.4-candidates/build1. Remove the broken symlink at xulrunner/nightly/2009-10-xx-xx-mozilla-1.9.1. Contact mfinkle and dtownsend for smoketest QA.

RESPIN

Build 2

  • Cleaned up build dirs from build 1
    • Went through builders on the master to gather hostnames for linux/win32/mac. Used cssh on linux/mac, RDP as Administrator on win32.
      • Linux (moz2-linux-slave[30,35,36,38,40,43-50])
        • /builds/slave/{tag,source,linux_build,linux_repack,updates,linux_update_verify,final_verification}/*
      • Mac (bm-xserve22, moz2-darwin9-slave[13-18])
        • /builds/slave/{macosx_build,macosx_repack,l10n_verification,macosx_update_verify}/*
      • Win32 (moz2-win32-slave[34,44,52-59])
        • /e/builds/moz2_slave/{win32_build,win32_repack,win32_update_verify}/*

Tag

  • attempt 1 (slave: moz2-linux-slave44): initial reconfig didn't take due to a local typo I introduced while checking that the config was updated. I didn't notice because the reconfig usually times out anyway :/
  • attempt 2 (slave: moz2-linux-slave44): failed because I hadn't cleaned out the release dirs after the previous attempt.
  • attempt 3 (slave: moz2-linux-slave33): third time's the charm.

Source

  • (slave: moz2-linux-slave37) No problems

Build/Repack

  • Linux: No problems
  • Mac: No problems
  • Win32: No problems

Sign

L10nVerify

  • (slave: bm-xserve22) No problems

Generate updates and push betatest snippets

  • first backupsnip completed properly before respin was called, which is better than can be said for build 1.

RESPIN

Build 3

  • Cleaned up build dirs from build 2
    • Went through builders on the master to gather hostnames for linux/win32/mac. Used cssh on linux/mac, RDP as Administrator on win32.
      • Linux (moz2-linux-slave[20,22,33,35,37,44])
        • /builds/slave/{tag,source,linux_build,linux_repack,updates,linux_update_verify,final_verification}/*
      • Mac (bm-xserve[11,19,22], moz2-darwin9-slave14)
        • /builds/slave/{macosx_build,macosx_repack,l10n_verification,macosx_update_verify}/*
      • Win32 (moz2-win32-slave[32,33,39,47])
        • /e/builds/moz2_slave/{win32_build,win32_repack,win32_update_verify}/*

Tag

  • (slave: moz2-linux-slave39) KaiRo re-tagged mozilla-1.9.1 in the middle of our tagging step, so the tagging step failed on mozilla-1.9.1. All locales were tagged fine though
  • to recover, I logged into the slave (moz2-linux-slave39), enabled the hg rebase extension in the .hgrc file, and then ran the following commands:
cd /builds/slave/tag/mozilla-1.9.1
hg pull --rebase
hg push -e 'ssh -l ffxbld -i ~cltbld/.ssh/ffxbld_dsa' -f ssh://hg.mozilla.org/releases/mozilla-1.9.1
  • I then disabled the rebase extension in the .hgrc file again

Source

  • (slave: moz2-linux-slave37) forced from the builder page, no problems

Build/Repack

  • Linux: en-US build forced from the builder page
  • Mac: en-US build forced from the builder page
  • Win32: en-US build forced from the builder page
  • to force the l10n repacks, I had to use the same process as bhearsum used for 3.6b1:
# on coop's machine:
hg clone http://hg.mozilla.org/users/bhearsum_mozilla.com/tools tools-bhearsum
cd tools-bhearsum/buildbot-helpers
wget http://people.mozilla.org/~coop/files/3.5.4/shipped-locales-linux
wget http://people.mozilla.org/~coop/files/3.5.4/shipped-locales-macosx
wget http://people.mozilla.org/~coop/files/3.5.4/shipped-locales-win32
# after linux en-US finished:
time python force_release_l10n.py -m http://production-master02.build.mozilla.org:8010 -v -t FIREFOX_3_5_4_RELEASE -s shipped-locales-linux -n coop 2>&1 | tee linux_repack_trigger.log
real	3m11.262s
user	0m0.606s
sys	0m0.274s
# after mac en-US finished
time python force_release_l10n.py -m http://production-master02.build.mozilla.org:8010 -v -t FIREFOX_3_5_4_RELEASE -s shipped-locales-macosx -n coop 2>&1 | tee macosx_repack_trigger.log
real	3m9.023s
user	0m0.609s
sys	0m0.293s
# after win32 en-US finished
time python force_release_l10n.py -m http://production-master02.build.mozilla.org:8010 -v -t FIREFOX_3_5_4_RELEASE -s shipped-locales-win32 -n coop 2>&1 | tee win32_repack_trigger.log
real	3m34.634s
user	0m0.612s
sys	0m0.300s
  • NOTE: I sent the initial batch of linux l10n re-triggers to the wrong master (production-master), so I had to kill off a bunch of linux_repack builds there.

Sign

  • ran with CONCURRENCY=1 for bug 521080
  • removed signed-builds[1-2] and unsigned-builds[1-2] for 3.5.4 to free up space (19G)
  • instructions

L10nVerify

  • (slave: bm-xserve22) No problems

Generate updates and push betatest snippets

  • attempt 1: (slave: moz2-linux-slave30) failed while extracting a mar (log link):
Exception: cmd failed mar -x /builds/slave/updates/build/temp/firefox/3.5.4/ftp/firefox-3.5.4.ta.win32.complete.mar
  • cleaned up slave and forced the step again from the builder page
  • attempt 2: (slave: moz2-linux-slave32) No problems
# connect as cltbld@moz2-linux-slave32
cd /builds/slave/updates/build/temp/firefox/3.5.3-3.5.4
for dir in aus2.test aus2.beta; do
  # copy complete snippets from 3.5.3 into a new dir for 3.5.4
  mkdir -p b1-b3-${dir}/Firefox
  rsync -a --exclude 'partial.txt' ${dir}/Firefox/3.5.3/ b1-b3-${dir}/Firefox/3.5.4/
  # use the buildid's from build1
  mv b1-b3-${dir}/Firefox/3.5.4/Darwin_Universal-gcc3/20091006224018  b1-b2-${dir}/Firefox/3.5.4/Darwin_Universal-gcc3/20091016081620 
  mv b1-b3-${dir}/Firefox/3.5.4/Linux_x86-gcc3/20091006225203         b1-b2-${dir}/Firefox/3.5.4/Linux_x86-gcc3/20091016081727
  mv b1-b3-${dir}/Firefox/3.5.4/WINNT_x86-msvc/20091007001339         b1-b2-${dir}/Firefox/3.5.4/WINNT_x86-msvc/20091016092926
done
rsync -a -e 'ssh -i ~/.ssh/aus' b1-b3-aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091019-Firefox-3.5.4build1-build3-test
rsync -a -e 'ssh -i ~/.ssh/aus' b1-b3-aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091019-Firefox-3.5.4build1-build3-beta

Update verify

  • Linux: (slave: moz2-linux-slave37) No problems.
  • Mac: (slave: moz2-darwin9-slave13) No problems.
  • Win32: (slave: moz2-win32-slave45) No problems.
  • some discussion about "respinning" the tr locale for bug 522989

Publish updates to beta channel

Update Bouncer

Final checks before push

Push to mirrors

Final Verification

Push updates to release channel

Release

XULRunner build

Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_3.5.4/BuildNotes&oldid=176622"