Releases/Firefox 28.0/BuildNotes
From MozillaWiki
< Releases
bug 961831 - bhearsum
Contents
Build 1
Issues hit:
- 5x win32 update verify failures from network disruption, auto-retried
Updates for beta channel
# using bld-centos6-hp-016 with screen mkdir ~/bug961831 cd ~/bug961831 hg clone http://hg.mozilla.org/build/tools # get a signing token for release certs, 6 hour expiry export PYTHONPATH=tools/lib/python; export PATH=$PATH:`pwd` wget -O repack.py https://bugzilla.mozilla.org/attachment.cgi?id=8389037
20/20 Hindsight: should have been 55G.
# make sure enough disk space is free python ~/bug961831/tools/buildfarm/maintenance/purge_builds.py -s 40 -n 'rel-*:45d' -n 'tb-rel-*:45d' /builds/slave
# start mac repack time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P mac 2>&1 | tee mac.log # in separate screen windows, generate other platforms cd ~/bug961831 export PYTHONPATH=tools/lib/python; export PATH=$PATH:`pwd` time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-i686 --no-env 2>&1 | tee linux.log # repeat for these: time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-x86_64 --no-env 2>&1 | tee linux64.log time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P win32 --no-env 2>&1 | tee win32.log time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P mac --no-env 2>&1 | tee mac.log
Unfortunately, not enough disk space was free, and each process failed out because of it. Mac failed at "lij", linux-i686 failed at "tr", linux-x86_64 at "uk", and win32 at "zu". To get things going again, did the following:
- Cleared 14G of disk space by hand
- Modified repack.py to restart where it left off, with this patch:
--- repack.py.orig 2014-03-11 06:24:30.996377161 -0700
+++ repack.py 2014-03-11 06:23:59.384371739 -0700
@@ -329,6 +329,7 @@
# calculate locale overlap
locales = getLocales(from_build, to_build)
+ firstLocale = {"mac": "lij", "linux-i686": "tr", "linux-x86_64": "uk", "win32": "zu"}
if options.platforms:
platforms = []
@@ -344,6 +345,11 @@
print "%s locales to do, %s platforms" % (len(locales), len(platforms))
for p in platforms:
+ newLocales = []
+ for i in sorted(locales):
+ if i >= firstLocale[p]:
+ newLocales.append(i)
+ locales = newLocales
for l in locales:
if l == 'ja-JP-mac' and p != 'mac':
continue
- Tested the above by commenting out all actual work in the script and adding a print, output looked fine, eg:
[cltbld@bld-centos6-hp-016.build.scl1.mozilla.com bug961831]$ time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-x86_64 --no-env ERROR: Failed to open cert /etc/ssl/certs/Makefile: (-34). ERROR: Failed to open cert /etc/ssl/certs/ca-bundle.trust.crt: (-34). ERROR: Failed to open cert /etc/ssl/certs/make-dummy-cert: (-34). 2014-03-11 06:23:38 URL:https://hg.mozilla.org/releases/mozilla-beta/raw-file/FIREFOX_28_0b9_RELEASE/browser/locales/shipped-locales [365/365] -> "from_shipped-locales" [1] ERROR: Failed to open cert /etc/ssl/certs/Makefile: (-34). ERROR: Failed to open cert /etc/ssl/certs/ca-bundle.trust.crt: (-34). ERROR: Failed to open cert /etc/ssl/certs/make-dummy-cert: (-34). 2014-03-11 06:23:38 URL:https://hg.mozilla.org/releases/mozilla-release/raw-file/FIREFOX_28_0_RELEASE/browser/locales/shipped-locales [359/359] -> "to_shipped-locales" [1] 88 locales to do, 1 platforms ** Working on 28.0-build1 linux-x86_64 uk ------------------------------------------------------------------------------------------------------------------------ ** Working on 28.0-build1 linux-x86_64 vi ------------------------------------------------------------------------------------------------------------------------ ** Working on 28.0-build1 linux-x86_64 xh ------------------------------------------------------------------------------------------------------------------------ ** Working on 28.0-build1 linux-x86_64 zh-CN ------------------------------------------------------------------------------------------------------------------------ ** Working on 28.0-build1 linux-x86_64 zh-TW ------------------------------------------------------------------------------------------------------------------------ ** Working on 28.0-build1 linux-x86_64 zu ------------------------------------------------------------------------------------------------------------------------
- Removed comment and restart each platforms' MAR generation with these commands (note "tee -a" to append to the original logs):
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-i686 --no-env 2>&1 | tee -a linux.log time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-x86_64 --no-env 2>&1 | tee -a linux64.log time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P win32 --no-env 2>&1 | tee -a win32.log time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P mac --no-env 2>&1 | tee -a mac.log
Windows completed successfully, but mac and both linux platforms failed to sign most of their MARs due to an expired token. To fix:
- Generated a new signing token & deleted the "nonce" file
- Deleted the files generated for the locales that failed to sign (mai for mac, zh-TW for linux-i686, zu for linux-x86_64)
- Adjusted the firstLocale for mac & linux platforms in the modified repack.py
- Started the mac & linux platforms again (I didn't bother adjusting the locales because only 1 of each platform at most succeeded):
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-i686 --no-env 2>&1 | tee -a linux.log time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-x86_64 --no-env 2>&1 | tee -a linux64.log time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P mac --no-env 2>&1 | tee -a mac.log
# fix permissions and upload to stage
find repacked/ -type f -exec chmod 644 {} \;
find repacked/ -type d -exec chmod 755 {} \;
# rsync -n is your friend
rsync -av -e 'ssh -i /home/cltbld/.ssh/ffxbld_dsa' \
repacked/28.0-candidates/build1/ \
ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/candidates/28.0-candidates/build1/
# reran check_permissions builder to make sure all is correct.
# run av scan on partials, anyone on stage cd /pub/mozilla.org/firefox/candidates/28.0-candidates/build1/update nice ionice -c2 -n7 extract_and_run_command.py -j2 clamdscan -m --no-summary -- \ `find . -name firefox-28.0b9-28.0.partial.mar -o -name firefox-28.0-forbeta.complete.mar` 2>&1 | tee /tmp/av.log
# bump + modify patcher config - I did this on my laptop for 27.0 cd <parent_of_tools_checkout> export PERL5LIB="tools/lib/perl:$HOME/usr/perl" # need Config::General v2.44 wget http://hg.mozilla.org/releases/mozilla-release/raw-file/FIREFOX_28_0_RELEASE/browser/locales/shipped-locales perl tools/release/patcher-config-bump.pl -p firefox -r Firefox -v 28.0 -a 28.0 -o 28.0b9 -b 1 \ -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg -t stage.mozilla.org -f ftp.mozilla.org \ -d download.mozilla.org -l shipped-locales --partial-version 28.0b9 --platform linux --platform linux64 \ --platform macosx64 --platform win32 # then some manual modifications <28.0> block ——> <28.0-build1>, to -> 28.0-build1, add mar-channel-ids, pretty version # tweak paths for mar filenames + bouncer url. Update verify configs can be generated from this # upload patch to bug 961831 (update verify harness changes from previous repackings have already landed)
# generate snippets (on bld-centos6-hp-016): cd /home/cltbld/bug961831/tools hg pull hg up -r FIREFOX_28_0_RELEASE_RUNTIME wget -O- https://bugzilla.mozilla.org/attachment.cgi?id=8389205 | patch -p1 cd .. export PYTHONPATH="tools/lib/python:tools/lib/python/vendor" python tools/scripts/updates/create-snippets.py --config tools/release/patcher-configs/mozBeta-branch-patcher2.cfg \ --checksums-dir checksums --snippet-dir aus2 --test-snippet-dir aus2.test -v 2>&1 | tee snippets.log
# ditch the betatest snippets, we're going to use releasetest only before pushing to beta rm -rf aus2.test/Firefox/*/WIN*/*/*/betatest rm -rf aus2.test/Firefox/*/Lin*/*/*/betatest rm -rf aus2.test/Firefox/*/Dar*/*/*/betatest
# Sync to aus3-staging rsync -av -e 'ssh -i /home/cltbld/.ssh/ffxbld_dsa' aus2.test/ \ ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-28.0-build1-forbeta-test/ rsync -av -e 'ssh -i /home/cltbld/.ssh/ffxbld_dsa' aus2/ \ ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-28.0-build1-forbeta/
# push test snippets live, on aus3-staging ~/bin/pushsnip Firefox-28.0-build1-forbeta-test
# Bouncer setup (to match patcher config) # add Products: Firefox-28.0-build1-complete, Firefox-28.0-build1-Partial-28.0b9 # add Locations: 4 locations for each
# announce to r-d
# generate update verify configs, done on my laptop cd $parent_of_tools_repo python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg --platform linux64 --output tools/release/updates/mozBeta-firefox-linux64.cfg --release-config-file mozilla/release-firefox-mozilla-release.py -b https://hg.mozilla.org/build/buildbot-configs --channel releasetest -t FIREFOX_28_0_RELEASE python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg --platform linux --output tools/release/updates/mozBeta-firefox-linux.cfg --release-config-file mozilla/release-firefox-mozilla-release.py -b https://hg.mozilla.org/build/buildbot-configs --channel releasetest -t FIREFOX_28_0_RELEASE python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg --platform macosx64 --output tools/release/updates/mozBeta-firefox-mac64.cfg --release-config-file mozilla/release-firefox-mozilla-release.py -b https://hg.mozilla.org/build/buildbot-configs --channel releasetest -t FIREFOX_28_0_RELEASE python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg --platform win32 --output tools/release/updates/mozBeta-firefox-win32.cfg --release-config-file mozilla/release-firefox-mozilla-release.py -b https://hg.mozilla.org/build/buildbot-configs --channel releasetest -t FIREFOX_28_0_RELEASE # hand edit to make the following changes: # - remove everything except the last two betas (in this case, that was everything except the top 3 lines) # - fix "to" variable to point at valid candidates directory
# update verify, manual launch on appropriate slaves, eg # linux32 is special: need a slave that last did a 32bit build in the mock env (or set one up)
mock_mozilla -r mozilla-centos6-x86_64 --cwd /builds/slave --unpriv --shell 'PATH="/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin" bash'
# other platforms: no special action # all: mkdir bug961831; cd $_ hg clone http://hg.mozilla.org/build/tools cd tools hg up -r FIREFOX_28_0_RELEASE_RUNTIME wget -O- https://bugzilla.mozilla.org/attachment.cgi?id=8389239 | patch -p1 cd release/updates # use correct .cfg file for platform ./verify.sh -c mozBeta-firefox-linux64.cfg 2>&1 | tee log
For posterity, both linux update verifies were run on bld-centos6-hp-016 (64-bit outside of mock, 32-bit inside of mock). bld-lion-r5-005 was used for Mac. w64-ix-slave21 was used for Windows.
Update verify results:
- Linux/Linux64 - PASS
- Only expected diffs found in output:
78813 diff -r source/firefox/defaults/pref/channel-prefs.js target/firefox/defaults/pref/channel-prefs.js
78814 1c1
78815 < //@line 2 "/builds/slave/rel-m-beta-l64_bld-00000000000/build/browser/app/profile/channel-prefs.js"
78816 ---
78817 > //@line 2 "/builds/slave/rel-m-rel-l64_bld-000000000000/build/browser/app/profile/channel-prefs.js"
78818 6c6
78819 < pref("app.update.channel", "beta");
78820 ---
78821 > pref("app.update.channel", "release");
78822 diff -r source/firefox/update-settings.ini target/firefox/update-settings.ini
78823 5c5
78824 < ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-beta,firefox-mozilla-release
78825 ---
78826 > ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-release
- Mac - PASS
- Same diffs as above
- Win32 - PASS
- Same diffs as above
Fix up releasetest for beta
ashughes reported failures when testing from 27.0 using releasetest. On aus3, we have a partial.txt from release updates (27.0 -> 28.0), and a complete.txt from beta updates. Ideally we have separate channels for this, but in the meantime
# ffxbld@aus3-staging
for f in /opt/aus2/incoming/3/Firefox/27.0/*/20140127194636/*/releasetest/partial.txt; do
mv -v $f{,.for_release};
done
We'll need to undo this for release testing, and fix the complete.txt. Repushing the Firefox-28.0-build1-test snippets would one way to do that.
Build 2
No issues with build2 until push to mirrors. We decided not to ship it to Beta users, so none of the manual work that we did for build1 was needed.
Uptake check
Uptake check failed with 400 error from bouncer. More details to follow.
Final verification
This failed because there were no 25.0.1 partial mar bouncer entries. This happened because build1 didn't have 25.0.1 listed as partial, and build2 didn't run bouncer submitter. I added them by hand and re-ran final verify.
