Releases/Firefox 17.0b1/BuildNotes

From MozillaWiki
< Releases
Revision as of 19:01, 11 October 2012 by Jhopkins (talk | contribs)
Jump to navigation Jump to search

Bugs hit

  • bug 799599 - don't use mock slaves for releases until bug 798361 is done
  • bug 799806 - compile failures in libmar for repacks
  • bug 799347 - Investigate packet loss within SCL3
  • bug 799954 - linux repacks failed in firefox 17.0b1
  • bug 799988 - Install android-sdk-r16 RPM to IX machines
  • [no bug] - Add --enable-stdcxx-compat to l10n release mozconfigs for Linux mozilla-beta, to fix compiler errors.
  • bug 800329 - multiple locales donn't build properly on win32, empty values break
  • bug 800457 - 401 error hitting bouncer

Notes

  • Tracking bug: bug 796970
  • Release engineer: jhopkins
  • Buildbot master: bm13

Build 1

Starting the automation

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u jhopkins \
    -V 17.0b1 --branch mozilla-beta --build-number 1 \
    --release-config release-firefox-mozilla-beta.py \
    --release-config release-fennec-mozilla-beta.py --products firefox,fennec  \
    --dryrun localhost:9001

The initial tagging build and Linux builds failed because mozilla-beta was configured to use mock (release automation is not using mock yet). Rail wrote a patch to allow release configs to override "use_mock" setting of the platform config (eg. pf['linux']['use_mock']). We also needed to restore the "disable mock" block for mozilla-beta in config.py.

Fix repacks

First we hit bug 799806. nthomas wrote a patch to resolve the issue.

After that, Windows and Mac repacks worked fine but Linux hit: 

===
=== If you get failures below, please file a bug describing the error
=== and your environment (compiler and linker versions), and use
=== --disable-elf-hack until this is fixed.
===
# Fail if the library doesn't have INIT_ARRAY .dynamic info
readelf -d test-array.so | grep '(INIT_ARRAY)'
 0x00000019 (INIT_ARRAY)                 0x54a8
/builds/slave/rel-m-beta-lnx-rpk-2/mozilla-beta/obj-l10n/build/unix/elfhack/elfhack -b -f test-array.so
/builds/slave/rel-m-beta-lnx-rpk-2/mozilla-beta/obj-l10n/build/unix/elfhack/elfhack: /tools/gcc-4.3.3/installed/lib/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /builds/slave/rel-m-beta-lnx-rpk-2/mozilla-beta/obj-l10n/build/unix/elfhack/elfhack)
make[4]: *** [test-array.so] Error 1

Catlee suggested that adding --enable-stdcxx-compat to the mozconfig would fix it. I (bhearsum) tested this by hand to verify, and then landed a fix in buildbot-configs: https://hg.mozilla.org/build/buildbot-configs/rev/e0994e97bf3a. After retagging and testing one repack chunk, I triggered the rest.

Respin declared for the same issue as the 16.0 chemspill

Build 2

Starting the automation

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u catlee \
    -V 17.0b1 --branch mozilla-beta --build-number 2 \
    --release-config release-firefox-mozilla-beta.py \
    --products firefox \
    --dryrun localhost:9001

Fennec beta tagging delayed due to SCL3 network issues. See bug 799347. Once the workaround was done: 

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas \
    -V 17.0b1 --branch mozilla-beta --build-number 2 \
    --release-config release-fennec-mozilla-beta.py \
    --products fennec \
    --dryrun localhost:9001

which was clean, ran without the --dryrun.

Sign Android builds

Ran the following as cltsign@signing1: 

cd ~/signing-work/mozharness
hg pull -u && hg up -C
cd ..
python2.6 mozharness/scripts/sign_android.py --config-file signing/android_mozilla-beta.py

Update generation failed because of missing 'km' build

Update generation failed with: 

2012-10-11 08:22:03,160 - DEBUG - requests.packages.urllib3.connectionpool._make_request#267: "GET /pub/mozilla.org/firefox/nightly/17.0b1-candidates/build2/win32/km/firefox-17.0b1.checksums HTTP/1.1" 404 288
Traceback (most recent call last):
  File "tools/scripts/updates/create-snippets.py", line 90, in <module>
    info = getChecksum(platform, locale, checksumsFile)
  File "tools/scripts/updates/create-snippets.py", line 72, in getChecksum
    contents = requests.get(checksumsUrl, config={'danger_mode': True}).content
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/api.py", line 51, in get
    return request('get', url, **kwargs)
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/api.py", line 39, in request
    return s.request(method=method, url=url, **kwargs)
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/sessions.py", line 203, in request
    r.send(prefetch=prefetch)
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/models.py", line 585, in send
    self.response.raise_for_status()
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/models.py", line 810, in raise_for_status
    raise http_error
requests.exceptions.HTTPError: 404 Client Error

To fix, had to manually change the patcher config to exclude 'km' from win32 for 17.0b1: https://hg.mozilla.org/build/tools/rev/f85e15d49942. Retagged tools with FIREFOX_17_0b1_{RELEASE,BUILD2} and used "force build" to start updates again.

Failed again because "updates" regenerated the patcher config from shipped locales. Manually changed shipped-locales this time, on the relbranch: https://hg.mozilla.org/releases/mozilla-beta/rev/a3cbcfe120bd. Retagged tools with FIREFOX_17_0b1_{RELEASE,BUILD2} and used "force build" to start updates again.

Failed again because it still tried to download checksums file. FIXME there's a bug in the new patcher config code that means we don't support removed locales. Next around attempt: faking out the checksums and removing the snippets later: 

# ffxbld@stage
cd /pub/mozilla.org/firefox/candidates/17.0b1-candidates/build2/win32
mkdir km
cd km
cp ../ach/*.checksums .

Then forced the builder again.

After it completed, had to do some cleanup on stage: 

# ffxbld@stage
cd /pub/mozilla.org/firefox/candidates/17.0b1-candidates/build2/win32
rm -rf km

And in the snippets: 

# ffxbld@aus3-staging
cd /opt/aus2/snippets/staging
find Firefox-17.0b1-build2 -maxdepth 5 -type d -wholename '*WINNT*km*' -exec rm -rf {} \;

Didn't bother deleting km test snippets from the test snippets. They'll get overwritten in 17.0b2.

Push to Mirrors

[2:37pm] bhearsum: jhopkins: because of the bug noted above bug 800457, you'll need
 to manually monitor 17.0b1's mirror uptake 
 (https://bounceradmin.mozilla.com/stats/uptake/?p=2038&p=2039&p=2041&p=2040), 
 and force build the final verify builders + "ready for releasetest" + "ready for release"
 when there's uptake
[2:41pm] jhopkins: bhearsum: i thought CDN uptake was instant?
[2:41pm] Callek: catlee-buildduty: bhearsum: just to say it -- bouncer in prod is working now -- at least for my seamonkey bouncer script
[2:42pm] bhearsum: jhopkins: near-instant
[2:42pm] bhearsum: sentry takes a few minutes to notice that it has uptake
[2:42pm] bhearsum: bounceradmin seems to be showing 50000002 availability now though, which means you have uptake
[2:43pm] jhopkins: so it's sentry we're waiting for, really
[2:43pm] bhearsum: kind of
[2:43pm] bhearsum: we're waiting for the bouncer system (which is bounceradmin + download.m.o + sentry) to be fully aware of the release
[2:43pm] bhearsum: if we don't, we'll serve 404s
[2:44pm] rail: plus sync from ftp.m.o to cdn?
[2:44pm] bhearsum: rail: that's near-instant, aiui
[2:46pm] jhopkins: i think how it works is the CDN only caches requested files.
 so it'll be cached on the CDN when the first user requests a file.  it will probably
 pull from us multiple times due to more than one distribution node plus when TTL expires
 for a file
[2:47pm] jhopkins: the other way to do it is to upload to the CDN manually and they store
 all the files but we're not operating that way

Kicked off final_verification builders

Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_17.0b1/BuildNotes&oldid=478927"