Releases/Fennec 4.0b2/BuildNotes

From MozillaWiki
Jump to: navigation, search

Build Engineers

lsblakk, aki - bug 607238

Bugs encountered

Signed-off Revision(s)

Build # mozilla-central mobile-browser
build 1 b0132c16f574 3f6795993406
build 2 57bc64a3949f 9218ed68347b
build 3 ec7521865660 a73da861034a

Tags

On mozilla-central:

Build # Tag Moz Changeset Mobile Changeset
build 1 GECKO20b7pre_20101029_RELBRANCH 35b148bd6ba0 ae7a1d4dab7d
FENNEC_4_0b2_RELEASE, FENNEC_4_0b2_BUILD1 def2833dd2ac 6b7cbad98c1b
build 2 FENNEC_4_0b2_RELEASE, FENNEC_4_0b2_BUILD2 57bc64a3949f 9218ed68347b
build 3 FENNEC_4_0b2_RELEASE, FENNEC_4_0b2_BUILD3 ec7521865660 a73da861034a
build 4 FENNEC_4_0b2_RELEASE, FENNEC_4_0b2_BUILD4 ec7521865660 a73da861034a

Build data

Build # Type Build ID Build machine
build 1 Android 20101029154619 mv-moz2-linux-ix-slave15
Maemo5-GTK 20101029170031 moz2-linux-slave42
Linux 20101029161724 moz2-linux-slave43
Mac 20101029165620 moz2-darwin9-slave35
Windows failed compile (3x) mw32-ix-slave07, mw32-ix-slave20, mw32-ix-slave07
build 2 Android 20101101113708 moz2-linux-slave36
Maemo5-GTK 20101101122918 mv-moz2-linux-ix-slave18
Linux 20101101104511 mv-moz2-linux-ix-slave13
Mac 20101101122921 moz2-darwin9-slave35
Windows 20101101162647 mw32-ix-slave07
build 3 Android 20101102172540 mv-moz2-linux-ix-slave06
Maemo5-GTK 20101102163940 mv-moz2-linux-ix-slave18
Linux 20101102172628 moz2-linux-slave40
Mac 20101102183947 moz2-darwin9-slave35
Windows 20101102184204 mw32-ix-slave07
build 4 Android 20101103123628 mv-moz2-linux-ix-slave16
Maemo5-GTK 20101103114710 mv-moz2-linux-ix-slave18
Linux 20101103132008 moz2-linux-slave42
Mac 20101103134910 moz2-darwin9-slave35
Windows 20101103122720 mw32-ix-slave07

Notes

Build 1

  • Landed debsign bumps - this has to happen before tagging starts
  • Landed configs
  • Reconfig master
  • Set pm02:8010 to clobber
  • removed mobile_source from slaves that had done this step in October
  • Notify #developers and #mobile that tree is closing
  • Sent release-drivers email that tree is closing for tagging
  • Closed tree
  • Sendchange:
buildbot sendchange --username lsblakk --master=localhost:9010 --branch=mobile-browser "Fennec 4.0b2 build1" go

Then I tagged buildbot-configs, buildbotcustom, tools, and mozharness with FENNEC_4_0b2_RELEASE, FENNEC_4_0b2_BUILD1

Tag

Completed without problem.

Bumped the version in mobile-browser to 4.0b3 on pm02 as ffxbld:

hg push -e 'ssh -l ffxbld -i ~/.ssh/ffxbld_dsa' -f ssh://hg.mozilla.org/mobile-browser

Source

Source failed in create_bundle (timeout)

  • cleaned out mobile_source from moz2-linux-slave45
  • kicked off a rebuild from builder page

Build/Repack

mobile_win32-i686_desktop_build
  • failed hg pull due to checkout fail from hg.mozilla.org on mw32-ix-slave07
  • hit rebuild on builder page (don't need to clobber on slave because that is part of this builder's steps), got mw32-ix-slave20
  • second build failed compile
  • no obvious reasons why this failed (there is space on slave, it had recently done a nightly successfully)
  • force rebuild a third time, got mw32-ix-slave07, still failed compile so filed bug bug 608456
mobile_linux-i686_desktop_build
  • finished no problems
mobile_macosx-i686_desktop_build
  • no problems
android-r7
  • Android-r7 finished with warnings because upload complete snippet failed - this is part of bug bug 608430 along with the moving of unsigned builds (see Signing step)
maemo5-gtk
  • build complete no problems (just slow on a vm)

Sign Android

21G available on keymaster d:

Moved the android builds to unsigned (this will be addressed in bug 608430)

# ffxbld@staging
cd /pub/mozilla.org/mobile/candidates/4.0b2-candidates/build1
mkdir unsigned
mv android-r7 unsigned/
mkdir android-r7

Signing improvement bug filed bug 608432

 # on keymaster in ~/signing-work
 mkdir fennec-4.0b2
 cd fennec-4.0b2/

 # we signed both en-US and multi since no one was sure in #mobile which to do
 # need sep dirs since they are both named gecko-unsigned-unaligned.apk
 mkdir -p build1/en-US
 mkdir build1/multi
 
 # get en-US apk
 cd build1/en-US/
 wget http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build1/unsigned/android-r7/en-US/gecko-unsigned-unaligned.apk
 
 # get multi apk 
 cd ../multi/
 wget http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build1/unsigned/android-r7/gecko-unsigned-unaligned.apk
 
 # put the signing script in the ~/signing-work/fennec-4.0b2/ dir
 cp ~/hg-tools/release/signing/* .
 
 # copy the en-US apk up to pwd for signing -- there is a bug on this (608432
), it's because sign_android.sh relies on mozpass.py and that has to be in the same dir as .apk to work
 cp build1/en-US/gecko-unsigned-unaligned.apk .
 ./sign_android.sh 

 * if any problems with signing, remove gecko-unaligned.apk and start again

 # signing successful?  copy back to the en-US
 mv gecko-unaligned.apk build1/en-US/

 # rename the fennec.apk, then move
 mv fennec.apk fennec-4.0b2.en-US.eabi-arm.apk
 mv fennec-4.0b2.en-US.eabi-arm.apk build1/en-US/
 
 # clean up any left overs  
 rm *.apk

 # repeat for multi
 cp build1/multi/gecko-unsigned-unaligned.apk .
 ./sign_android.sh 
 mv gecko-unaligned.apk build1/multi/
 mv fennec.apk fennec-4.0b2.multi.eabi-arm.apk
 mv fennec-4.0b2.multi.eabi-arm.apk build1/multi/
 rm *.apk

 # upload to stage
 cd build1/ 
 scp -i ~/.ssh/ffxbld_dsa -r * ffxbld@stage.mozilla.org:/home/ftp/pub/mobile/candidates/4.0b2-candidates/build1/android-r7

Update Android Snippets

update 4.0b1 snippets, look for the newest files

 # as cltbld@aus2-staging
 cd /opt/aus2/incoming/3/Fennec

 find . -mtime -1
 .
 ./4.0b1/Android_arm-eabi-gcc3/20101006203925/en-US
 ./4.0b1/Android_arm-eabi-gcc3/20101006203925/en-US/beta-cck-test
 ./4.0b1/Android_arm-eabi-gcc3/20101006203925/en-US/beta-cck-test/complete.txt
 ./4.0b2
 ./4.0b2/Android_arm-eabi-gcc3
 ./4.0b2/Android_arm-eabi-gcc3/20101029154619
 ./4.0b2/Android_arm-eabi-gcc3/20101029154619/en-US
 ./4.0b2/Android_arm-eabi-gcc3/20101029154619/en-US/beta-cck-test
 ./4.0b2/Android_arm-eabi-gcc3/20101029154619/en-US/beta-cck-test/complete.txt
 ./4.0b2/Android_arm-eabi-gcc3/20101029154619/en-US/beta-cck-test/partial.txt
 # edited 4.0b1 complete.txt to point at multi by changing
 vim 4.0b1/Android_arm-eabi-gcc3/20101006203925/en-US/beta-cck-test/complete.txt 
 
 url=http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build1/android-r7/multi/fennec-4.0b2.multi.eabi-arm.apk
 hashValue=fbdbd0668c36dd202b7ae2187acd67e6177831f1e25e64066e85d191ee9fbc138d68185066fbfb39e93c602483111024ba80d6e6b312bff6f1bfda97b3414b36
 size=14488447

 # if the en-US is preferred, the details are:
 url=http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build1/android-r7/en-US/fennec-4.0b2.en-US.eabi-arm.apk
 hashValue=0a32c7e7faf709480b9b278954df2731d268dbae19ca5f148323dc64df77b788aed5b52ca344f478ddf8967c72e4f831365554aa2a79214af4facb557fea7ae4
 size=12239733

Deb Repos

On production-mobile-master force build on deb sign release builder

First run of deb sign release on production-mobile-master failed because of config dir name conflict between mozharness/configs and buildbot-configs being cloned as configs. Aki put up a patch and bumped up the mozharness PRODUCTION tag, and then another force build could be triggered.

Second run, all passed.

Send email to r-d about signed builds being available, hand off to QA. (except win32 build because of bug 608456

At this point a respin was called via Release-Drivers email

Build 2

  • Set pm02:8010 clobber
  • rm -rf mobile_source from mv-moz2-linux-ix-slave12, moz2-linux-slave45
  • rm -rf *release from /scratchbox/users/cltbld/home/cltbld/build on slaves that did l10n repacks
  • landed config bumps
  • landed debsign bumps and moved PRODUCTION tag
  • notify #developers and #mobile that tree is closing for tagging
  • email to r-d that tree is closing for tagging
  • update pm02:8010 configs
  • reconfig master
  • Start automation with:
buildbot sendchange --username lsblakk --master=localhost:9010 --branch=mobile-browser "Fennec 4.0b2 build2" doit

Tag

mv-moz2-linux-ix-slave18

No problems, re-opened tree.

Tagged buildbotcustom, buildbot-configs, tools, mozharness with FENNEC_4_0b2_BUILD2, and bumped up FENNEC_4_0b2_RELEASE

Source

moz2-linux-slave45

no problems.

Build/Repack

mobile_win32-i686_desktop_build

mw32-ix-slave07

  • First run through failed with same tests linkage error in compile as build1 so landed a change to mozconfig to explicitly disable-tests and re-build from buildbot page.

mw32-ix-slave07

  • Failed package-tests because tests were disabled!
  • landed a dir name shortening and re-enabled test patch, reconfig'd pm02:8010, and kicked it off again

mw32-ix-slave07

  • third time's a charm. successful build and upload - closed bug 608456
mobile_linux-i686_desktop_build

mv-moz2-linux-ix-slave13

no problems

mobile_macosx-i686_desktop_build

moz2-darwin9-slave35

no problems.

android-r7

moz2-linux-slave36

  • Android-r7 finished with warnings because upload complete snippet failed - this is part of bug bug 608430 along with the moving of unsigned builds (see Signing step)
maemo5-gtk

moz2-linux-slave28

  • Failed to upload en-US to stage because of DNS bustage, forced rebuild in the hopes that it could make it through without multi-locale issues.

mv-moz2-linux-ix-slave18

  • successful
repack

had to manually trigger all the locales with en_revision, l10n_revision, and locale property settings which were all successful as well. checked on stage to be sure that the count of locale dirs matched build1 (both 41)

Sign Android

20G available on keymaster d:

Moved the android builds to unsigned (this will be addressed in bug 608430)

# ffxbld@staging
cd /pub/mozilla.org/mobile/candidates/4.0b2-candidates/build2
mkdir unsigned
mv android-r7 unsigned/
mkdir android-r7
 # on keymaster in ~/signing-work
 cd fennec-4.0b2/

 # make dirs for both en-US and multi
 mkdir -p build2/en-US
 mkdir build2/multi
 
 # get en-US apk
 cd build1/en-US/
 wget http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build2/unsigned/android-r7/en-US/gecko-unsigned-unaligned.apk
 
 # get multi apk 
 cd ../multi/
 wget http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build2/unsigned/android-r7/gecko-unsigned-unaligned.apk
 
 # copy the en-US apk up to pwd for signing
 cp build1/en-US/gecko-unsigned-unaligned.apk .
 ./sign_android.sh 

 # copy back to the en-US
 mv *.apk build2/en-US/

 # repeat for multi
 cp build1/multi/gecko-unsigned-unaligned.apk .
 ./sign_android.sh 
 mv *.apk build2/multi/

 # upload to stage
 cd build1/ 
 scp -i ~/.ssh/ffxbld_dsa -r * ffxbld@stage.mozilla.org:/home/ftp/pub/mobile/candidates/4.0b2-candidates/build2/android-r7
# ffxbld@stage
# rename en-US and multi fennec apk:
mv fennec.apk fennec-4.0b2.en-US.eabi-arm.apk
mv fennec.apk fennec-4.0b2.multi.eabi-arm.apk

Update Android Snippets

Get the size and hash for the apk

 # ffxbld@stage
 cd /pub/mozilla.org/mobile/candidates/4.0b2-candidates/build2/android-r7
 ls -la ./*   # gets file size
 # sha512 hash
 openssl dgst -sha512 {path to each fennec apk}
 
 en-US:
 url=http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build2/android-r7/en-US/fennec-4.0b2.en-US.eabi-arm.apk
 size=12239829
 hashValue=b51c3828d840518832d86bf74551007a55a72b52f8a52a2a890683c13b49d5b52b91900c3fa39e8b553b6d15425cf75442fa14db28bab1406f0809fbaf23008f

 multi:
 url=http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build2/android-r7/multi/fennec-4.0b2.multi.eabi-arm.apk
 size=14488519
 hashValue=aea2895f48c3c44ac119a07b9e82f46d34163a140c28364b156be7a4daaa86ded55ada5452138f2bc2c56e5928953f440c3b47c97a31748bf3950ef716d36b49

 # edited 4.0b1 snippet to add /multi to stage path, correct size and hashValue (using multi because that's what we are shipping for 4.0b2)
 vim 4.0b1/Android_arm-eabi-gcc3/20101006203925/en-US/beta-cck-test/complete.txt 

Deb Repos

Forced build on pmm, didn't reconfig since only changes were config bumps in mozharness.

No problems.

Update testing

  • did a symlink for 4.0b2 build1 -> 4.0b2 build2 on aus2-staging:
# in /opt/aus2/incoming/3/Fennec/4.0b2/Android_arm-eabi-gcc3/20101029154619/en-US/beta-cck-test
ln -s ../../../../../4.0b1/Android_arm-eabi-gcc3/20101006203925/en-US/beta-cck-test/complete.txt complete.txt
  • assigned bug 608837 to me, we'll need a better solution going forward like a latest-beta-cck-test or something

At this point I handed off to QA in an email to release-drivers.

Then we got the email that "We unfortunately found a crasher on a couple major sites in build 2, which has been fixed on trunk." and a respin was called.

Build 3

  • Set pm02:8010 to clobber
  • clobbered repacks from build2 slaves (all linux ix)
  • removed mobile_source from build2 slaves (moz2-linux-slave45)
  • Landed debsign bumps - this has to happen before tagging starts
  • Landed configs
  • Reconfig master
  • Notify #developers and #mobile that tree is closing
  • Sent release-drivers email that tree is closing for tagging
  • Closed tree
  • Sendchange:
buildbot sendchange --username lsblakk --master=localhost:9010 --branch=mobile-browser "Fennec 4.0b2 build3" go

Then I tagged buildbot-configs, buildbotcustom, tools, and mozharness with FENNEC_4_0b2_RELEASE, FENNEC_4_0b2_BUILD3

Tag

No problem, re-opened tree.

Source

No Problem

Build/Repack

mobile_win32-i686_desktop_build

No Problem

mobile_linux-i686_desktop_build

No Problem

mobile_macosx-i686_desktop_build

No Problem

android-r7

build completed with the expected snippet warning, same as previous builds.

  • aki moved build3/android-r7 to build3/unsigned/android-r7 and created an empty build3/android-r7 directory
maemo5-gtk

No Problem

repack

No Problem

At this point build4 respin was called because of l10n bustage in German locale, Axel provided an updated list of l10-changesets for build4.

Build 4

  • Set pm02:8010 to clobber
  • clobbered repacks from build2 slaves (all linux ix)
  • removed mobile_source from build2 slaves (mv-moz2-linux-ix-slave21)
  • Landed debsign bumps - this has to happen before tagging starts
  • Landed configs
  • Reconfig master
  • Notify #developers and #mobile that tree is closing
  • Sent release-drivers email that tree is closing for tagging
  • Closed tree
  • Sendchange:
buildbot sendchange --username lsblakk --master=localhost:9010 --branch=mobile-browser "Fennec 4.0b2 build4" go

Note, did this after the fact but for future builds - use https://l10n-stage-sj.mozilla.org/shipping/confirm-ship?ms=fennec4.0b2 to generate the l10n json and then be sure to 'ship it' (you must be logged in) so that Axel's records of what was actually used is up to date.

Tag

No problems.

I tagged buildbot-configs, buildbotcustom, tools, and mozharness with FENNEC_4_0b2_RELEASE, FENNEC_4_0b2_BUILD4

Source

First run on mv-mox2-ix-linux-slave21 failed because of a failed clobber (which is my fault from a bad autocomplete in csshx), I rm -rf /builds/slave/mobile_source and did a rebuild.

Second run, no problems.

Build/Repack

mobile_win32-i686_desktop_build

No problems

mobile_linux-i686_desktop_build

No problem

mobile_macosx-i686_desktop_build

No problems

android-r7

build completed with the expected snippet warning, same as previous builds.

  • moved build3/android-r7 to build4/unsigned/android-r7 and created an empty build4/android-r7 directory
maemo5-gtk

No problems

repack

No problems

Sign Android

20G available on keymaster d:

 # on keymaster in ~/signing-work
 cd fennec-4.0b2/

 # we signed both en-US and multi since no one was sure in #mobile which to do
 # need sep dirs since they are both named gecko-unsigned-unaligned.apk
 mkdir -p build4/en-US
 mkdir build4/multi
 
 # get en-US apk
 cd build4/en-US/
 wget http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build4/unsigned/android-r7/en-US/gecko-unsigned-unaligned.apk
 
 # get multi apk 
 cd ../multi/
 wget http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build4/unsigned/android-r7/gecko-unsigned-unaligned.apk
 
 # copy the en-US apk up to pwd for signing -- there is a bug on this (608432
), it's because sign_android.sh relies on mozpass.py and that has to be in the same dir as .apk to work
 cp build4/en-US/gecko-unsigned-unaligned.apk .
 ./sign_android.sh
 mv fennec.apk fennec-4.0b2.en-US.eabi-arm.apk 
 mv *apk build4/en-US/
 
 # repeat for multi
 cp build1/multi/gecko-unsigned-unaligned.apk .
 ./sign_android.sh 
 mv fennec.apk fennec-4.0b2.multi.eabi-arm.apk
 mv *.apk build4/multi

 # upload to stage
 cd build4/ 
 scp -i ~/.ssh/ffxbld_dsa -r * ffxbld@stage.mozilla.org:/home/ftp/pub/mobile/candidates/4.0b2-candidates/build4/android-r7

Update Android Snippets

Get the size and hash for the apk

 # ffxbld@stage
 cd /pub/mozilla.org/mobile/candidates/4.0b2-candidates/build4/android-r7
 ls -la ./*   # gets file size
 # sha512 hash
 openssl dgst -sha512 {path to each fennec apk}
 
 en-US:
 url=http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build4/android-r7/en-US/fennec-4.0b2.en-US.eabi-arm.apk
 size=12245366
 hashValue=cef4095145cfc478e13ef7e2b1c330343d2900116a2eb456bd4a990851ba4c444b07359d265dd2801378cc88f9dee4d469d08a7261ef4b08d39737f210df7298

 multi:
 url=http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build4/android-r7/multi/fennec-4.0b2.multi.eabi-arm.apk
 size=13922106
 hashValue=9f2380c46512d5fbde349e02e0b15c7bb1ac3bad9c0fc47988b0520106a77af769a5cb3c603ca6e50175a62e67d4c0c9d3ea40d42c1b998227f5364f8f0eae04

 # edited 4.0b1 snippet to add correct size and hashValue for multi (what we are shipping for 4.0b2)
 vim 4.0b1/Android_arm-eabi-gcc3/20101006203925/en-US/beta-cck-test/complete.txt 

 # symlinks for 4.0b2 build2,build3 -> 4.0b2 build 4
 cd /opt/aus2/incoming/3/Fennec/4.0b2/Android_arm-eabi-gcc3/20101101113708/en-US/beta-cck-test
 rm complete.txt && ln -s /opt/aus2/incoming/3/Fennec/latest-beta-cck-test complete.txt
 cd /opt/aus2/incoming/3/Fennec/4.0b2/Android_arm-eabi-gcc3/20101102172540/en-US/beta-cck-test
 rm complete.txt && ln -s /opt/aus2/incoming/3/Fennec/latest-beta-cck-test complete.txt

Deb Repos

Force build on deb signing builder on pmm, no reconfig since nothing but bumps on debsign dirs, completed without problems.


-- QA handoff

Update testing

Push Live

Copy bits to releases/

We didn't release single locales, so this is different than the ReleaseDocs

 cd /home/ftp/pub/mobile/releases
 mkdir 4.0b2
 cd 4.0b2
 cp -r ../../candidates/4.0b2-candidates/build4/*-i686 .
 cp -r ../../candidates/4.0b2-candidates/build4/source .
 cp -r ../../candidates/4.0b2-candidates/build4/android-r7/multi android-r7/multi
 cp -r ../../candidates/4.0b2-candidates/build4/maemo5-gtk/multi maemo5-gtk/multi

(all the cp instead of rsync brought over test, crashreporter, txt, unsigned, unaligned files that were not needed and I removed them from the various dirs as needed)

 # symlinks for en-US because we are shipping multi and want en-US users to update to it
 ln -s android-r7/multi android-r7/en-US
 ln -s maemo5-gtk/multi maemo5-gtk/en-US

 # and repos
 mkdir -p repos/multi
 cp -r ../../candidates/4.0b2-candidates/repos/multi/* repos/multi
 ln -s repos/multi repos/en-US
 
 # softlinks for maemo
 ln -s repos maemo
 cd ..

 # latest- softlinks:
 # latest-alpha is the alpha "channel", latest-beta is the beta "channel"
 # latest is the release "channel"
 #
 rm latest-alpha latest-beta
 ln -s 4.0b2 latest-alpha
 ln -s 4.0b2 latest-beta
 # hack chinook for now... long term fix later?
 cd 4.0b2/repos/en-US/dists
 ln -s fremantle chinook
Create new .install files

nothing to be done here for this release.

Create bouncer entries
# added product Firefox-Mobile-4.0b2
android
/mobile/releases/4.0b2/android-r7/multi/fennec-4.0b2.multi.eabi-arm.apk
maemo
/mobile/releases/maemo/multi/firefox-beta.install
Updates to beta channel
# cltbld on aus2-staging
cd /opt/aus2/incoming/3/Fennec
cp latest-beta-cck-test beta.tmp
# edit beta.tmp to point to 
url=http://download.mozilla.org/?product=Firefox-Mobile-4.0b2&os=android&lang=multi

# Put snippet in 4.0b1/Android_arm-eabi-gcc3/en-US/20101006203925/beta as complete.txt
ln -s /opt/aus2/incoming/3/Fennec/latest-beta complete.txt

We're live, sent email to release-drivers.