Releases/Firefox 17.0b1/BuildNotes

• Ship the l10n milestone ^{DESKTOP MOBILE RELEASE BETA}
• Set reserved_slaves ^{DESKTOP MOBILE RELEASE BETA ESR}
• Check to make sure you have all of the different types of slaves that you need ^{DESKTOP MOBILE RELEASE BETA ESR}
• Start the automation ^{DESKTOP MOBILE RELEASE BETA ESR}
• Sign Android builds ^{MOBILE RELEASE BETA}
• Reset reserved_slaves and unlock slaves ^{DESKTOP MOBILE RELEASE BETA ESR}
• Push to mirrors ^{DESKTOP RELEASE}
• Publish Fennec ^{MOBILE RELEASE BETA}
• Run pushsnip ^{DESKTOP RELEASE BETA ESR}

Bugs hit

• bug 799599 - don't use mock slaves for releases until bug 798361 is done
• bug 799806 - compile failures in libmar for repacks
• bug 799347 - Investigate packet loss within SCL3
• bug 799954 - linux repacks failed in firefox 17.0b1
• bug 799988 - Install android-sdk-r16 RPM to IX machines
• [no bug] - Add --enable-stdcxx-compat to l10n release mozconfigs for Linux mozilla-beta, to fix compiler errors.
• bug 800329 - multiple locales donn't build properly on win32, empty values break

Notes

• Tracking bug: bug 796970
• Release engineer: jhopkins
• Buildbot master: bm13

Build 1

Starting the automation

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u jhopkins \
    -V 17.0b1 --branch mozilla-beta --build-number 1 \
    --release-config release-firefox-mozilla-beta.py \
    --release-config release-fennec-mozilla-beta.py --products firefox,fennec  \
    --dryrun localhost:9001

The initial tagging build and Linux builds failed because mozilla-beta was configured to use mock (release automation is not using mock yet). Rail wrote a patch to allow release configs to override "use_mock" setting of the platform config (eg. pf['linux']['use_mock']). We also needed to restore the "disable mock" block for mozilla-beta in config.py.

Fix repacks

First we hit bug 799806. nthomas wrote a patch to resolve the iseue.

After that, Windows and Mac repacks worked fine but Linux hit:

===
=== If you get failures below, please file a bug describing the error
=== and your environment (compiler and linker versions), and use
=== --disable-elf-hack until this is fixed.
===
# Fail if the library doesn't have INIT_ARRAY .dynamic info
readelf -d test-array.so | grep '(INIT_ARRAY)'
 0x00000019 (INIT_ARRAY)                 0x54a8
/builds/slave/rel-m-beta-lnx-rpk-2/mozilla-beta/obj-l10n/build/unix/elfhack/elfhack -b -f test-array.so
/builds/slave/rel-m-beta-lnx-rpk-2/mozilla-beta/obj-l10n/build/unix/elfhack/elfhack: /tools/gcc-4.3.3/installed/lib/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /builds/slave/rel-m-beta-lnx-rpk-2/mozilla-beta/obj-l10n/build/unix/elfhack/elfhack)
make[4]: *** [test-array.so] Error 1

Catlee suggested that adding --enable-stdcxx-compat to the mozconfig would fix it. I (bhearsum) tested this by hand to verify, and then landed a fix in buildbot-configs: https://hg.mozilla.org/build/buildbot-configs/rev/e0994e97bf3a. After retagging and testing one repack chunk, I triggered the rest.

Respin declared for the same issue as the 16.0 chemspill

Build 2

Starting the automation

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u catlee \
    -V 17.0b1 --branch mozilla-beta --build-number 2 \
    --release-config release-firefox-mozilla-beta.py \
    --products firefox \
    --dryrun localhost:9001

Fennec beta tagging delayed due to SCL3 network issues. See bug 799347. Once the workaround was done:

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas \
    -V 17.0b1 --branch mozilla-beta --build-number 2 \
    --release-config release-fennec-mozilla-beta.py \
    --products fennec \
    --dryrun localhost:9001

which was clean, ran without the --dryrun.

Sign Android builds

Ran the following as cltsign@signing1:

cd ~/signing-work/mozharness
hg pull -u && hg up -C
cd ..
python2.6 mozharness/scripts/sign_android.py --config-file signing/android_mozilla-beta.py

Update generation failed because of missing 'km' build

Update generation failed with:

2012-10-11 08:22:03,160 - DEBUG - requests.packages.urllib3.connectionpool._make_request#267: "GET /pub/mozilla.org/firefox/nightly/17.0b1-candidates/build2/win32/km/firefox-17.0b1.checksums HTTP/1.1" 404 288
Traceback (most recent call last):
  File "tools/scripts/updates/create-snippets.py", line 90, in <module>
    info = getChecksum(platform, locale, checksumsFile)
  File "tools/scripts/updates/create-snippets.py", line 72, in getChecksum
    contents = requests.get(checksumsUrl, config={'danger_mode': True}).content
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/api.py", line 51, in get
    return request('get', url, **kwargs)
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/api.py", line 39, in request
    return s.request(method=method, url=url, **kwargs)
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/sessions.py", line 203, in request
    r.send(prefetch=prefetch)
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/models.py", line 585, in send
    self.response.raise_for_status()
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/models.py", line 810, in raise_for_status
    raise http_error
requests.exceptions.HTTPError: 404 Client Error

To fix, had to manually change the patcher config to exclude 'km' from win32 for 17.0b1: https://hg.mozilla.org/build/tools/rev/f85e15d49942. Retagged tools with FIREFOX_17_0b1_{RELEASE,BUILD2} and used "force build" to start updates again.

Failed again because "updates" regenerated the patcher config from shipped locales. Manually changed shipped-locales this time, on the relbranch: https://hg.mozilla.org/releases/mozilla-beta/rev/a3cbcfe120bd. Retagged tools with FIREFOX_17_0b1_{RELEASE,BUILD2} and used "force build" to start updates again.

Failed again because it still tried to download checksums file. FIXME there's a bug in the new patcher config code that means we don't support removed locales. Next around attempt: faking out the checksums and removing the snippets later:

# ffxbld@stage
cd /pub/mozilla.org/firefox/candidates/17.0b1-candidates/build2/win32
mkdir km
cd km
cp ../ach/*.checksums .

Then forced the builder again.

After it completed, had to do some cleanup on stage:

# ffxbld@stage
cd /pub/mozilla.org/firefox/candidates/17.0b1-candidates/build2/win32
rm -rf km

And in the snippets:

# ffxbld@aus3-staging
cd /opt/aus2/snippets/staging
find Firefox-17.0b1-build2 -maxdepth 5 -type d -wholename '*WINNT*km*' -exec rm -rf {} \;

Didn't bother deleting km test snippets from the test snippets. They'll get overwritten in 17.0b2.