Releases/Firefox 17.0b1/BuildNotes

From MozillaWiki
Jump to: navigation, search

Bugs hit

  • bug 799599 - don't use mock slaves for releases until bug 798361 is done
  • bug 799806 - compile failures in libmar for repacks
  • bug 799347 - Investigate packet loss within SCL3
  • bug 799954 - linux repacks failed in firefox 17.0b1
  • bug 799988 - Install android-sdk-r16 RPM to IX machines
  • [no bug] - Add --enable-stdcxx-compat to l10n release mozconfigs for Linux mozilla-beta, to fix compiler errors.
  • bug 800329 - multiple locales donn't build properly on win32, empty values break
  • bug 800457 - 401 error hitting bouncer
  • bug 800925 - update verify config bumper doesn't handle dropped locales properly
  • bug 800924 - PatcherConfig class returns wrong update paths when a locale is dropped

Notes

  • Tracking bug: bug 796970
  • Release engineer: jhopkins
  • Buildbot master: bm13

Build 1

Starting the automation

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u jhopkins \
    -V 17.0b1 --branch mozilla-beta --build-number 1 \
    --release-config release-firefox-mozilla-beta.py \
    --release-config release-fennec-mozilla-beta.py --products firefox,fennec  \
    --dryrun localhost:9001

The initial tagging build and Linux builds failed because mozilla-beta was configured to use mock (release automation is not using mock yet). Rail wrote a patch to allow release configs to override "use_mock" setting of the platform config (eg. pf['linux']['use_mock']). We also needed to restore the "disable mock" block for mozilla-beta in config.py.

Fix repacks

First we hit bug 799806. nthomas wrote a patch to resolve the issue.

After that, Windows and Mac repacks worked fine but Linux hit: 

===
=== If you get failures below, please file a bug describing the error
=== and your environment (compiler and linker versions), and use
=== --disable-elf-hack until this is fixed.
===
# Fail if the library doesn't have INIT_ARRAY .dynamic info
readelf -d test-array.so | grep '(INIT_ARRAY)'
 0x00000019 (INIT_ARRAY)                 0x54a8
/builds/slave/rel-m-beta-lnx-rpk-2/mozilla-beta/obj-l10n/build/unix/elfhack/elfhack -b -f test-array.so
/builds/slave/rel-m-beta-lnx-rpk-2/mozilla-beta/obj-l10n/build/unix/elfhack/elfhack: /tools/gcc-4.3.3/installed/lib/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /builds/slave/rel-m-beta-lnx-rpk-2/mozilla-beta/obj-l10n/build/unix/elfhack/elfhack)
make[4]: *** [test-array.so] Error 1

Catlee suggested that adding --enable-stdcxx-compat to the mozconfig would fix it. I (bhearsum) tested this by hand to verify, and then landed a fix in buildbot-configs: https://hg.mozilla.org/build/buildbot-configs/rev/e0994e97bf3a. After retagging and testing one repack chunk, I triggered the rest.

Respin declared for the same issue as the 16.0 chemspill

Build 2

Starting the automation

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u catlee \
    -V 17.0b1 --branch mozilla-beta --build-number 2 \
    --release-config release-firefox-mozilla-beta.py \
    --products firefox \
    --dryrun localhost:9001

Fennec beta tagging delayed due to SCL3 network issues. See bug 799347. Once the workaround was done: 

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas \
    -V 17.0b1 --branch mozilla-beta --build-number 2 \
    --release-config release-fennec-mozilla-beta.py \
    --products fennec \
    --dryrun localhost:9001

which was clean, ran without the --dryrun.

Sign Android builds

Ran the following as cltsign@signing1: 

cd ~/signing-work/mozharness
hg pull -u && hg up -C
cd ..
python2.6 mozharness/scripts/sign_android.py --config-file signing/android_mozilla-beta.py

Update generation failed because of missing 'km' build

Update generation failed with: 

2012-10-11 08:22:03,160 - DEBUG - requests.packages.urllib3.connectionpool._make_request#267: "GET /pub/mozilla.org/firefox/nightly/17.0b1-candidates/build2/win32/km/firefox-17.0b1.checksums HTTP/1.1" 404 288
Traceback (most recent call last):
  File "tools/scripts/updates/create-snippets.py", line 90, in <module>
    info = getChecksum(platform, locale, checksumsFile)
  File "tools/scripts/updates/create-snippets.py", line 72, in getChecksum
    contents = requests.get(checksumsUrl, config={'danger_mode': True}).content
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/api.py", line 51, in get
    return request('get', url, **kwargs)
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/api.py", line 39, in request
    return s.request(method=method, url=url, **kwargs)
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/sessions.py", line 203, in request
    r.send(prefetch=prefetch)
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/models.py", line 585, in send
    self.response.raise_for_status()
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/models.py", line 810, in raise_for_status
    raise http_error
requests.exceptions.HTTPError: 404 Client Error

To fix, had to manually change the patcher config to exclude 'km' from win32 for 17.0b1: https://hg.mozilla.org/build/tools/rev/f85e15d49942. Retagged tools with FIREFOX_17_0b1_{RELEASE,BUILD2} and used "force build" to start updates again.

Failed again because "updates" regenerated the patcher config from shipped locales. Manually changed shipped-locales this time, on the relbranch: https://hg.mozilla.org/releases/mozilla-beta/rev/a3cbcfe120bd. Retagged tools with FIREFOX_17_0b1_{RELEASE,BUILD2} and used "force build" to start updates again.

Failed again because it still tried to download checksums file. FIXME there's a bug in the new patcher config code that means we don't support removed locales. Next around attempt: faking out the checksums and removing the snippets later: 

# ffxbld@stage
cd /pub/mozilla.org/firefox/candidates/17.0b1-candidates/build2/win32
mkdir km
cd km
cp ../ach/*.checksums .

Then forced the builder again.

After it completed, had to do some cleanup on stage: 

# ffxbld@stage
cd /pub/mozilla.org/firefox/candidates/17.0b1-candidates/build2/win32
rm -rf km

And in the snippets: 

# ffxbld@aus3-staging
cd /opt/aus2/snippets/staging
find Firefox-17.0b1-build2 -maxdepth 5 -type d -wholename '*WINNT*km*' -exec rm -rf {} \;

Didn't bother deleting km test snippets from the test snippets. They'll get overwritten in 17.0b2.

Push to Mirrors

[2:37pm] bhearsum: jhopkins: because of the bug noted above bug 800457, you'll need
 to manually monitor 17.0b1's mirror uptake 
 (https://bounceradmin.mozilla.com/stats/uptake/?p=2038&p=2039&p=2041&p=2040), 
 and force build the final verify builders + "ready for releasetest" + "ready for release"
 when there's uptake
[2:41pm] jhopkins: bhearsum: i thought CDN uptake was instant?
[2:41pm] Callek: catlee-buildduty: bhearsum: just to say it -- bouncer in prod is working now -- at least for my seamonkey bouncer script
[2:42pm] bhearsum: jhopkins: near-instant
[2:42pm] bhearsum: sentry takes a few minutes to notice that it has uptake
[2:42pm] bhearsum: bounceradmin seems to be showing 50000002 availability now though, which means you have uptake
[2:43pm] jhopkins: so it's sentry we're waiting for, really
[2:43pm] bhearsum: kind of
[2:43pm] bhearsum: we're waiting for the bouncer system (which is bounceradmin + download.m.o + sentry) to be fully aware of the release
[2:43pm] bhearsum: if we don't, we'll serve 404s
[2:44pm] rail: plus sync from ftp.m.o to cdn?
[2:44pm] bhearsum: rail: that's near-instant, aiui
[2:46pm] jhopkins: i think how it works is the CDN only caches requested files.
 so it'll be cached on the CDN when the first user requests a file.  it will probably
 pull from us multiple times due to more than one distribution node plus when TTL expires
 for a file
[2:47pm] jhopkins: the other way to do it is to upload to the CDN manually and they store
 all the files but we're not operating that way

Bouncer indicates that uptake is over 50000000 for all platforms.

Kicked off final_verification builders. Some time later, IT fixed the bouncer issue and buildbot started a duplicate set of final_verification builders.

Investigated win32_update_verify failures. All '404' errors were due to the missing 'km' locale.

win32_final_verification '404' errors also failed predictably on the 'km' locale.

[5:12pm EST] QA has given approval for 17.0b1 _desktop_. lsblakk talking to PR and getting ready to give a final release 'go' - we have not received one yet. No word on Fennec QA progress yet.

Publish Fennec

(Done by Hwine due to jhopkins wanting to have a life.)

Grabbed: 

   $ curl -O 'https://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/17.0b1-candidates/build2/android-armv6/mult
   i/fennec-17.0b1.multi.android-arm-armv6.apk'
     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
   100 20.8M  100 20.8M    0     0   140k      0  0:02:31  0:02:31 --:--:--  142k
   $ curl -O 'https://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/17.0b1-candidates/build2/android/multi/fennec-17.0b1.multi.android-arm.apk'
     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
   100 19.5M  100 19.5M    0     0   206k      0  0:01:36  0:01:36 --:--:--  568k
   0 [Hal@Hals-MacBook-Air Desktop]
   $ ll *apk
   -rw-r--r--  1 Hal  staff  21840767 Oct 11 14:59 fennec-17.0b1.multi.android-arm-armv6.apk
   -rw-r--r--  1 Hal  staff  20449419 Oct 11 15:25 fennec-17.0b1.multi.android-arm.apk
   0 [Hal@Hals-MacBook-Air Desktop]

Posted per instructions - neither the "recent changes" nor "description" content needed version specific updates. (Tried to clarify in notes.)

Pushed snippets w/o incident.

Publish Desktop

Pushsnips 

   [ffxbld@dp-ausstage01 ~]$ cd /opt/aus2/snippets/staging
   [ffxbld@dp-ausstage01 staging]$ ~/bin/pushsnip Firefox-17.0b1-build2/

Ran w/o incident

Bouncer Admin update worked fine: 

   $ curl -I 'http://download.mozilla.org/?product=firefox-beta-latest&os=win&lang=en-US'
   HTTP/1.1 302 Found
   Server: Apache
   X-Backend-Server: bouncer2.webapp.phx1.mozilla.com
   Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, private
   Content-Type: text/html; charset=UTF-8
   Date: Thu, 11 Oct 2012 23:46:21 GMT
   Location: http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/17.0b1/win32/en-US/Firefox%20Setup%2017.0b1.exe
   Pragma: no-cache
   Transfer-Encoding: chunked
   Connection: Keep-Alive
Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_17.0b1/BuildNotes&oldid=479271"