MozillaWiki

Releases/Firefox 17.0b1/BuildNotes: Difference between revisions

Page Discussion

Releases/Firefox 17.0b1/BuildNotes (view source)

Revision as of 14:54, 12 October 2012

11,164 bytes added ,  12 October 2012
no edit summary
No edit summary
No edit summary
 
(29 intermediate revisions by 5 users not shown)
Line 1: Line 1:
* <s>[[Release:Release_Automation_on_Mercurial:Preparation#L10N_Changesets | Ship the l10n milestone]] {{DesktopTag}} {{MobileTag}} {{ReleaseTag}} {{BetaTag}}</s>
* <s>[[Release:Release_Automation_on_Mercurial:Preparation#L10N_Changesets | Ship the l10n milestone]] {{DesktopTag}} {{MobileTag}} {{ReleaseTag}} {{BetaTag}}</s>
* <s>[[Release:Release_Automation_on_Mercurial:Starting_a_Release#Setting_reserved_slaves | Set reserved_slaves]] {{DesktopTag}} {{MobileTag}} {{AllTag}}</s>
* <s>[[Release:Release_Automation_on_Mercurial:Starting_a_Release#Setting_reserved_slaves | Set reserved_slaves]] {{DesktopTag}} {{MobileTag}} {{AllTag}}</s>
* [[Release:Release_Automation_on_Mercurial:Starting_a_Release#Locking_slaves | Check to make sure you have all of the different types of slaves that you need]] {{DesktopTag}} {{MobileTag}} {{AllTag}}
* <s>[[Release:Release_Automation_on_Mercurial:Starting_a_Release#Locking_slaves | Check to make sure you have all of the different types of slaves that you need]] {{DesktopTag}} {{MobileTag}} {{AllTag}}</s>
* [[Release:Release_Automation_on_Mercurial:Preparation#Starting_the_automation | Start the automation]] {{DesktopTag}} {{MobileTag}} {{AllTag}}
* <s>[[Release:Release_Automation_on_Mercurial:Preparation#Starting_the_automation | Start the automation]] {{DesktopTag}} {{MobileTag}} {{AllTag}}</s>
* [https://intranet.mozilla.org/RelEngWiki/index.php/Signing#Android_Signing Sign Android builds] {{MobileTag}} {{ReleaseTag}} {{BetaTag}}
* <s>[https://intranet.mozilla.org/RelEngWiki/index.php/Signing#Android_Signing Sign Android builds] {{MobileTag}} {{ReleaseTag}} {{BetaTag}}</s>
* [[Release:Release_Automation_on_Mercurial:Build#Reset_reserved_slaves | Reset reserved_slaves and unlock slaves]] {{DesktopTag}} {{MobileTag}} {{AllTag}}
* <s>[[Release:Release_Automation_on_Mercurial:Build#Reset_reserved_slaves | Reset reserved_slaves and unlock slaves]] {{DesktopTag}} {{MobileTag}} {{AllTag}}</s>
* [[Release:Release_Automation_on_Mercurial:Updates#Push_to_mirrors | Push to mirrors]] {{DesktopTag}} {{ReleaseTag}}
* <s>[[Release:Release_Automation_on_Mercurial:Updates#Push_to_mirrors | Push to mirrors]] {{DesktopTag}} {{ReleaseTag}}</s>
* [[Release:Release_Automation_on_Mercurial:Updates#Publish_Fennec | Publish Fennec]] {{MobileTag}} {{ReleaseTag}} {{BetaTag}}
* <s>[[Release:Release_Automation_on_Mercurial:Updates#Publish_Fennec | Publish Fennec]] {{MobileTag}} {{ReleaseTag}} {{BetaTag}}</s>
* [[Release:Release_Automation_on_Mercurial:Updates#Push_snippets | Run pushsnip]] {{DesktopTag}} {{AllTag}}
* <s>[[Release:Release_Automation_on_Mercurial:Updates#Push_snippets | Run pushsnip]] {{DesktopTag}} {{AllTag}}</s>
* <s>[[Release:Release_Automation_on_Mercurial:Updates#Update_Bouncer | Update bouncer links]] {{DesktopTag}} {{ReleaseTag}} {{BetaTag}}</s>


= Bugs hit =
= Bugs hit =
* {{bug|799599}} - don't use mock slaves for releases until bug 798361 is done
* {{bug|799806}} - compile failures in libmar for repacks
* {{bug|799347}} - Investigate packet loss within SCL3
* {{bug|799954}} - linux repacks failed in firefox 17.0b1
* {{bug|799988}} - Install android-sdk-r16 RPM to IX machines
* [no bug] - Add --enable-stdcxx-compat to l10n release mozconfigs for Linux mozilla-beta, to fix compiler errors.
* {{bug|800329}} - multiple locales donn't build properly on win32, empty values break
* {{bug|800457}} - 401 error hitting bouncer
* {{bug|800925}} - update verify config bumper doesn't handle dropped locales properly
* {{bug|800924}} - PatcherConfig class returns wrong update paths when a locale is dropped


= Notes =
= Notes =
Line 18: Line 30:
== Build 1 ==
== Build 1 ==
=== Starting the automation ===
=== Starting the automation ===
<pre>
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u jhopkins \
    -V 17.0b1 --branch mozilla-beta --build-number 1 \
    --release-config release-firefox-mozilla-beta.py \
    --release-config release-fennec-mozilla-beta.py --products firefox,fennec  \
    --dryrun localhost:9001
</pre>
The initial tagging build and Linux builds failed because mozilla-beta was configured to use mock (release automation is not using mock yet).
Rail wrote a patch to allow release configs to override "use_mock" setting of the platform config (eg. pf['linux']['use_mock']).  We also needed to restore the "disable mock" block for mozilla-beta in config.py.
=== Fix repacks ===
First we hit {{bug|799806}}.  nthomas wrote a patch to resolve the issue.
After that, Windows and Mac repacks worked fine but Linux hit:
<pre>
===
=== If you get failures below, please file a bug describing the error
=== and your environment (compiler and linker versions), and use
=== --disable-elf-hack until this is fixed.
===
# Fail if the library doesn't have INIT_ARRAY .dynamic info
readelf -d test-array.so | grep '(INIT_ARRAY)'
0x00000019 (INIT_ARRAY)                0x54a8
/builds/slave/rel-m-beta-lnx-rpk-2/mozilla-beta/obj-l10n/build/unix/elfhack/elfhack -b -f test-array.so
/builds/slave/rel-m-beta-lnx-rpk-2/mozilla-beta/obj-l10n/build/unix/elfhack/elfhack: /tools/gcc-4.3.3/installed/lib/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /builds/slave/rel-m-beta-lnx-rpk-2/mozilla-beta/obj-l10n/build/unix/elfhack/elfhack)
make[4]: *** [test-array.so] Error 1
</pre>
Catlee suggested that adding --enable-stdcxx-compat to the mozconfig would fix it. I (bhearsum) tested this by hand to verify, and then landed a fix in buildbot-configs: https://hg.mozilla.org/build/buildbot-configs/rev/e0994e97bf3a. After retagging and testing one repack chunk, I triggered the rest.
Respin declared for the same issue as the 16.0 chemspill
== Build 2 ==
=== Starting the automation ===
<pre>
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u catlee \
    -V 17.0b1 --branch mozilla-beta --build-number 2 \
    --release-config release-firefox-mozilla-beta.py \
    --products firefox \
    --dryrun localhost:9001
</pre>
Fennec beta tagging delayed due to SCL3 network issues.  See bug 799347. Once the workaround was done:
<pre>
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas \
    -V 17.0b1 --branch mozilla-beta --build-number 2 \
    --release-config release-fennec-mozilla-beta.py \
    --products fennec \
    --dryrun localhost:9001
</pre>
which was clean, ran without the --dryrun.


=== Sign Android builds ===
=== Sign Android builds ===
Ran the following as cltsign@signing1:
<pre>
cd ~/signing-work/mozharness
hg pull -u && hg up -C
cd ..
python2.6 mozharness/scripts/sign_android.py --config-file signing/android_mozilla-beta.py
</pre>
=== Update generation failed because of missing 'km' build ===
Update generation failed with:
<pre>2012-10-11 08:22:03,160 - DEBUG - requests.packages.urllib3.connectionpool._make_request#267: "GET /pub/mozilla.org/firefox/nightly/17.0b1-candidates/build2/win32/km/firefox-17.0b1.checksums HTTP/1.1" 404 288
Traceback (most recent call last):
  File "tools/scripts/updates/create-snippets.py", line 90, in <module>
    info = getChecksum(platform, locale, checksumsFile)
  File "tools/scripts/updates/create-snippets.py", line 72, in getChecksum
    contents = requests.get(checksumsUrl, config={'danger_mode': True}).content
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/api.py", line 51, in get
    return request('get', url, **kwargs)
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/api.py", line 39, in request
    return s.request(method=method, url=url, **kwargs)
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/sessions.py", line 203, in request
    r.send(prefetch=prefetch)
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/models.py", line 585, in send
    self.response.raise_for_status()
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/models.py", line 810, in raise_for_status
    raise http_error
requests.exceptions.HTTPError: 404 Client Error
</pre>
To fix, had to manually change the patcher config to exclude 'km' from win32 for 17.0b1: https://hg.mozilla.org/build/tools/rev/f85e15d49942. Retagged tools with FIREFOX_17_0b1_{RELEASE,BUILD2} and used "force build" to start updates again.
Failed again because "updates" regenerated the patcher config from shipped locales. Manually changed shipped-locales this time, on the relbranch: https://hg.mozilla.org/releases/mozilla-beta/rev/a3cbcfe120bd. Retagged tools with FIREFOX_17_0b1_{RELEASE,BUILD2} and used "force build" to start updates again.
Failed again because it still tried to download checksums file. FIXME there's a bug in the new patcher config code that means we don't support removed locales. Next around attempt: faking out the checksums and removing the snippets later:
# ffxbld@stage
cd /pub/mozilla.org/firefox/candidates/17.0b1-candidates/build2/win32
mkdir km
cd km
cp ../ach/*.checksums .
Then forced the builder again.
After it completed, had to do some cleanup on stage:
# ffxbld@stage
cd /pub/mozilla.org/firefox/candidates/17.0b1-candidates/build2/win32
rm -rf km
And in the snippets:
# ffxbld@aus3-staging
cd /opt/aus2/snippets/staging
find Firefox-17.0b1-build2 -maxdepth 5 -type d -wholename '*WINNT*km*' -exec rm -rf {} \;
Didn't bother deleting km test snippets from the test snippets. They'll get overwritten in 17.0b2.
=== Push to Mirrors ===
[2:37pm] bhearsum: jhopkins: because of the bug noted above {{bug|800457}}, you'll need
  to manually monitor 17.0b1's mirror uptake
  (https://bounceradmin.mozilla.com/stats/uptake/?p=2038&p=2039&p=2041&p=2040),
  and force build the final verify builders + "ready for releasetest" + "ready for release"
  when there's uptake
[2:41pm] jhopkins: bhearsum: i thought CDN uptake was instant?
[2:41pm] Callek: catlee-buildduty: bhearsum: just to say it -- bouncer in prod is working now -- at least for my seamonkey bouncer script
[2:42pm] bhearsum: jhopkins: near-instant
[2:42pm] bhearsum: sentry takes a few minutes to notice that it has uptake
[2:42pm] bhearsum: bounceradmin seems to be showing 50000002 availability now though, which means you have uptake
[2:43pm] jhopkins: so it's sentry we're waiting for, really
[2:43pm] bhearsum: kind of
[2:43pm] bhearsum: we're waiting for the bouncer system (which is bounceradmin + download.m.o + sentry) to be fully aware of the release
[2:43pm] bhearsum: if we don't, we'll serve 404s
[2:44pm] rail: plus sync from ftp.m.o to cdn?
[2:44pm] bhearsum: rail: that's near-instant, aiui
[2:46pm] jhopkins: i think how it works is the CDN only caches requested files.
  so it'll be cached on the CDN when the first user requests a file.  it will probably
  pull from us multiple times due to more than one distribution node plus when TTL expires
  for a file
[2:47pm] jhopkins: the other way to do it is to upload to the CDN manually and they store
  all the files but we're not operating that way


=== Final verification ===
Bouncer indicates that uptake is over 50000000 for all platforms.


=== Push beta (pushsnip) ===
Kicked off final_verification builders.  Some time later, IT fixed the bouncer issue and buildbot started a duplicate set of final_verification builders.
 
Investigated win32_update_verify failures.  All '404' errors were due to the missing 'km' locale.
 
win32_final_verification '404' errors also failed predictably on the 'km' locale.
 
[5:12pm EST] QA has given approval for 17.0b1 _desktop_.  lsblakk talking to PR and getting ready to give a final release 'go' - we have not received one yet.  No word on Fennec QA progress yet.


=== Publish Fennec ===
=== Publish Fennec ===
(Done by Hwine due to jhopkins wanting to have a life.)
Grabbed:
    $ curl -O 'https://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/17.0b1-candidates/build2/android-armv6/mult
    i/fennec-17.0b1.multi.android-arm-armv6.apk'
      % Total    % Received % Xferd  Average Speed  Time    Time    Time  Current
                                    Dload  Upload  Total  Spent    Left  Speed
    100 20.8M  100 20.8M    0    0  140k      0  0:02:31  0:02:31 --:--:--  142k
    $ curl -O 'https://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/17.0b1-candidates/build2/android/multi/fennec-17.0b1.multi.android-arm.apk'
      % Total    % Received % Xferd  Average Speed  Time    Time    Time  Current
                                    Dload  Upload  Total  Spent    Left  Speed
    100 19.5M  100 19.5M    0    0  206k      0  0:01:36  0:01:36 --:--:--  568k
    0 [Hal@Hals-MacBook-Air Desktop]
    $ ll *apk
    -rw-r--r--  1 Hal  staff  21840767 Oct 11 14:59 fennec-17.0b1.multi.android-arm-armv6.apk
    -rw-r--r--  1 Hal  staff  20449419 Oct 11 15:25 fennec-17.0b1.multi.android-arm.apk
    0 [Hal@Hals-MacBook-Air Desktop]
Posted per instructions - neither the "recent changes" nor "description" content needed version specific updates. (Tried to [https://wiki.mozilla.org/index.php?title=Release:Release_Automation_on_Mercurial:Updates_through_Shipping&diff=479050&oldid=478992 clarify] in notes.)
Pushed snippets w/o incident.
=== Publish Desktop ===
Pushsnips
    [ffxbld@dp-ausstage01 ~]$ cd /opt/aus2/snippets/staging
    [ffxbld@dp-ausstage01 staging]$ ~/bin/pushsnip Firefox-17.0b1-build2/
Ran w/o incident
Bouncer Admin update worked fine:
    $ curl -I 'http://download.mozilla.org/?product=firefox-beta-latest&os=win&lang=en-US'
    HTTP/1.1 302 Found
    Server: Apache
    X-Backend-Server: bouncer2.webapp.phx1.mozilla.com
    Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, private
    Content-Type: text/html; charset=UTF-8
    Date: Thu, 11 Oct 2012 23:46:21 GMT
    Location: http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/17.0b1/win32/en-US/Firefox%20Setup%2017.0b1.exe
    Pragma: no-cache
    Transfer-Encoding: chunked
    Connection: Keep-Alive
Bhearsum
canmove, Confirmed users
6,441

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/477823...479271"