Releases/Firefox 21.0/BuildNotes

Summary

• Who: nthomas
• Tracking: bug 854372
• Bugs encountered:
  • bug 869401 - disk full on signing3

Build 1

Checklist

• Ship the l10n milestone ^{DESKTOP MOBILE RELEASE BETA}
• Submit to Ship It ^{DESKTOP MOBILE RELEASE BETA ESR}
• --> ABANDONED

Notes

• merge from mozilla-beta, l10n milestone creation, and ship it submission done by bajaj. Review done by nthomas.
• xulrunner checksums and push_to_mirrors jobs hit hg server errors when cloning the tools repo. The automation retried them successfully

Recover from full signing server disk

(bhearsum)
signing3's disk filled up (bug 869401) and failed the win32 build and many linux repacks. After the server was fixed, I did the following to recover:

• Rebuilt the win32 build (failed to upload symbols, bug 869522, second rebuild was green)
• Used the standalone builder to run the failed repacks for Linux (br bs ca cs csb et eu fa ff mai mk ml mr nb-NO si sk sl vi zh-CN zu) and Linux64 (bn-BD bn-IN bs ca cs es-ES es-MX et eu fa ff hr hu hy-AM id is it lt lv mai mk ml nb-NO rm ro ru si sk sl)
• Once the standalone builders completed, ran "repack complete" builders with force build to get the automation unstuck.

Build 2 (Desktop)

This is a straight rebuild of the same revision as build1 for the AMD crasher problem. We may not ship it, and builds are hidden from the general public to avoid confusion & diluting feedback on build1.

Checklist

• Ship the l10n milestone ^{DESKTOP MOBILE RELEASE BETA}
• Submit to Ship It ^{DESKTOP MOBILE RELEASE BETA ESR}
• --> ABANDONED

Notes

• ship it submission done by bajaj
• review done by nthomas: moved from same revision as build1 to using the same relbranch (GECKO210_2013050704_RELBRANCH)
• xulrunner push_to_mirrors failed because build1 had already been pushed, this is no problem

Things to undo

Due to the 'run build2 while build1 not actually dead' situation there are some things we had to undo:

• repush 21.0build1 snippets after build2 are generated (QA had some issues with this until we repushed build1)

If we end up shipping build1:

• undo release config change, move tags and ensure reconfig before running any builders on build1
• move tags on tools repo back
• move FIREFOX_21_0_RELEASE tag back on to revision 3402fc52312c

Build 2 (Android) - SHIPPED THIS ONE

Code change resolving regression from bug 853522

Checklist

• Ship the l10n milestone ^{DESKTOP MOBILE RELEASE BETA}
• Submit to Ship It ^{DESKTOP MOBILE RELEASE BETA ESR}
• Publish Fennec ^{MOBILE RELEASE BETA}
• Push Fennec to mirrors ^{MOBILE RELEASE BETA}

Notes

• bajaj submitted, nthomas review, no issues

Publish Fennec

Published to Yandex + Google Play.

Build 3 (Desktop) SHIPPED THIS ONE

This is a new build to address several bugs.

Checklist

• Ship the l10n milestone ^{DESKTOP MOBILE RELEASE BETA}
• Submit to Ship It ^{DESKTOP MOBILE RELEASE BETA ESR}
• Push to mirrors ^{DESKTOP RELEASE}
• Apply Throttling ^{DESKTOP RELEASE}
• Verify throttling ^{DESKTOP RELEASE}
• Run pushsnip ^{DESKTOP RELEASE BETA ESR}
• Update bouncer links ^{DESKTOP BETA RELEASE}
• Update XULRunner wiki pages ^{DESKTOP RELEASE}
• Post-release tasks ^{DESKTOP RELEASE BETA ESR}
• Get the Browser Choice website updated ^{DESKTOP RELEASE}

Notes

• ship it submission done by bajaj
• review done by hwine: unchecked "l10n dashboard check"
• requires undo of directory changes to partner repacks:
  • done in https://hg.mozilla.org/build/partner-repacks/rev/3ca4cf4b87ac

• xulrunner push_to_mirrors failed because build1 had already been pushed
  • removed /pub/mozilla.org/xulrunner/releases/21.0 as xrbld@stage, then rebuilt
• QA automation expects there to be N directories if you're up to buildN, but we have hidden build2. Created empty dir at .../21.0-candidates/build2-for-qa-automation/ to work around this.
• separate out 21.0 build1 and build2 --> build3 paths so we can push that early

 # aus3-staging
 cd /opt/aus2/snippets/staging/
 rsync -a --include='Firefox' --include='Firefox/21.0' --include='Firefox/21.0/**' --exclude='*' \
    ../pushed/Firefox-21.0-build3-test/ Firefox-21.0-build3-early-update/
 cd Firefox-21.0-build3-early-update
 # get rid of unwanted releasetest
 find . -maxdepth 6 -type d -name 'releasetest' -exec rm -rf {} \;
 # move betatest to release
 find . -mindepth 5 -maxdepth 5 -type d -exec mv {}/betatest {}/release \;
 # swap from stage.m.o to ftp.m.o
 for f in `find . -type f`; do  
   sed -i -e 's/stage\.mozilla\.org/ftp.mozilla.org/' $f
 done

• push 21.0 build1 and build2 --> build3 updates

 # ffxbld@aus3-staging
 pushsnip Firefox-21.0-build3-early-update

Reset automation to build3

To ship build3 after the automation ran for build4 we need to reset the automation.

• undo build4's release config change in buildbot-config, move FIREFOX_21_0_RELEASE tag back to FIREFOX_21_0_BUILD3 (pushlog)
• in tools backout out build4 changes, move FIREFOX_21_0_RELEASE tag back to FIREFOX_21_0_BUILD3, and FIREFOX_21_0_RELEASE_RUNTIME tag back FIREFOX_21_0_BUILD3_RUNTIME (pushlog)
• move FIREFOX_21_0_RELEASE tag back to FIREFOX_21_0_BUILD3 on mozilla-release, plus buildbotcustom, partner-repacks (mozharness would be a noop)
• reconfig build masters and schedulers

Also repushed the 21.0build3 test snippets

# aus3-staging
cd /opt/aus2/snippets/pushed
mv Firefox-21.0-build3-test ../staging/Firefox-21.0-build3-test-repush
pushsnip Firefox-21.0-build3-test-repush

and obsolete the production snippets we're not going to use

mkdir /opt/aus2/snippets/obsolete
mv /opt/aus2/snippets/staging/Firefox-21.0-build{1,2,4} /opt/aus2/snippets/obsolete/

Later fix releasetest which was going old --> 21.0 build3 --> 21.0 build4 (bug 871851). build3 is 20130511120803:

$ cd /opt/aus2/incoming/3/Firefox/21.0

# verify everything in */20130511120803 is betatest or releasetest
$ find */20130511120803 -type f | wc -l
890
$ find */20130511120803 -type f | egrep '/(releasetest|betatest)/'  | wc -l
890
 
$ rm -rf */20130511120803

Apply/Verify Throttling

Throttling was requested and applied in bug 870883. Watched https://aus3.mozilla.org/update/1/Firefox/10.0.2/20120215223356/WINNT_x86-msvc/en-US/release/update.xml before and after the push to verify it.

Push snippets

As ffxbld@aus3-staging:

cd /opt/aus2/snippets/staging
~/bin/pushsnip Firefox-21.0-build3

Build 4 (Desktop) NOT SHIPPED

This is build 3 rebuilt for AMD crash issue.

Checklist

• Ship the l10n milestone ^{DESKTOP MOBILE RELEASE BETA}
• Submit to Ship It ^{DESKTOP MOBILE RELEASE BETA ESR}
• We decided to ship build3, abandon build4 unless there's a crash spike

Notes

• ship it submission done by bajaj
• reviewed by callek

• xulrunner push_to_mirrors failed because build3 had already been pushed, this is no problem
• repush 21.0build3 snippets after build4 are generated QA completed build3 testing before build4 got this far