Releases/Firefox 28.0/BuildNotes

bug 961831 - bhearsum

Build 1

Issues hit:

• 5x win32 update verify failures from network disruption, auto-retried

Updates for beta channel

# using bld-centos6-hp-016 with screen
mkdir ~/bug961831
cd ~/bug961831
hg clone http://hg.mozilla.org/build/tools
# get a signing token for release certs, 6 hour expiry
export PYTHONPATH=tools/lib/python; export PATH=$PATH:`pwd`
wget -O repack.py https://bugzilla.mozilla.org/attachment.cgi?id=8389037

20/20 Hindsight: should have been 55G.

# make sure enough disk space is free
python ~/bug961831/tools/buildfarm/maintenance/purge_builds.py -s 40 -n 'rel-*:45d' -n 'tb-rel-*:45d' /builds/slave

# start mac repack
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P mac 2>&1 | tee mac.log

# in separate screen windows, generate other platforms
cd ~/bug961831
export PYTHONPATH=tools/lib/python; export PATH=$PATH:`pwd`
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-i686 --no-env 2>&1 | tee linux.log
# repeat for these:
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-x86_64 --no-env 2>&1 | tee linux64.log
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P win32 --no-env 2>&1 | tee win32.log
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P mac --no-env 2>&1 | tee mac.log

Unfortunately, not enough disk space was free, and each process failed out because of it. Mac failed at "lij", linux-i686 failed at "tr", linux-x86_64 at "uk", and win32 at "zu". To get things going again, did the following:

• Cleared 14G of disk space by hand
• Modified repack.py to restart where it left off, with this patch:

--- repack.py.orig      2014-03-11 06:24:30.996377161 -0700
+++ repack.py   2014-03-11 06:23:59.384371739 -0700
@@ -329,6 +329,7 @@
 
     # calculate locale overlap
     locales = getLocales(from_build, to_build)
+    firstLocale = {"mac": "lij", "linux-i686": "tr", "linux-x86_64": "uk", "win32": "zu"}
 
     if options.platforms:
         platforms = []
@@ -344,6 +345,11 @@
 
     print "%s locales to do, %s platforms" % (len(locales), len(platforms))
     for p in platforms:
+        newLocales = []
+        for i in sorted(locales):
+            if i >= firstLocale[p]:
+                newLocales.append(i)
+        locales = newLocales
         for l in locales:
             if l == 'ja-JP-mac' and p != 'mac':
                 continue

• Tested the above by commenting out all actual work in the script and adding a print, output looked fine, eg:

[cltbld@bld-centos6-hp-016.build.scl1.mozilla.com bug961831]$  time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-x86_64 --no-env
ERROR: Failed to open cert /etc/ssl/certs/Makefile: (-34).
ERROR: Failed to open cert /etc/ssl/certs/ca-bundle.trust.crt: (-34).
ERROR: Failed to open cert /etc/ssl/certs/make-dummy-cert: (-34).
2014-03-11 06:23:38 URL:https://hg.mozilla.org/releases/mozilla-beta/raw-file/FIREFOX_28_0b9_RELEASE/browser/locales/shipped-locales [365/365] -> "from_shipped-locales" [1]

ERROR: Failed to open cert /etc/ssl/certs/Makefile: (-34).
ERROR: Failed to open cert /etc/ssl/certs/ca-bundle.trust.crt: (-34).
ERROR: Failed to open cert /etc/ssl/certs/make-dummy-cert: (-34).
2014-03-11 06:23:38 URL:https://hg.mozilla.org/releases/mozilla-release/raw-file/FIREFOX_28_0_RELEASE/browser/locales/shipped-locales [359/359] -> "to_shipped-locales" [1]

88 locales to do, 1 platforms
** Working on 28.0-build1 linux-x86_64 uk
------------------------------------------------------------------------------------------------------------------------
** Working on 28.0-build1 linux-x86_64 vi
------------------------------------------------------------------------------------------------------------------------
** Working on 28.0-build1 linux-x86_64 xh
------------------------------------------------------------------------------------------------------------------------
** Working on 28.0-build1 linux-x86_64 zh-CN
------------------------------------------------------------------------------------------------------------------------
** Working on 28.0-build1 linux-x86_64 zh-TW
------------------------------------------------------------------------------------------------------------------------
** Working on 28.0-build1 linux-x86_64 zu
------------------------------------------------------------------------------------------------------------------------

• Removed comment and restart each platforms' MAR generation with these commands (note "tee -a" to append to the original logs):

time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-i686 --no-env 2>&1 | tee -a linux.log
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-x86_64 --no-env 2>&1 | tee -a linux64.log
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P win32 --no-env 2>&1 | tee -a win32.log
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P mac --no-env 2>&1 | tee -a mac.log

Windows completed successfully, but mac and both linux platforms failed to sign most of their MARs due to an expired token. To fix:

• Generated a new signing token & deleted the "nonce" file
• Deleted the files generated for the locales that failed to sign (mai for mac, zh-TW for linux-i686, zu for linux-x86_64)
• Adjusted the firstLocale for mac & linux platforms in the modified repack.py
• Started the mac & linux platforms again (I didn't bother adjusting the locales because only 1 of each platform at most succeeded):

time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-i686 --no-env 2>&1 | tee -a linux.log
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-x86_64 --no-env 2>&1 | tee -a linux64.log
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P mac --no-env 2>&1 | tee -a mac.log

# fix permissions and upload to stage
find repacked/ -type f -exec chmod 644 {} \;
find repacked/ -type d -exec chmod 755 {} \;
# rsync -n is your friend
rsync -av -e 'ssh -i /home/cltbld/.ssh/ffxbld_dsa' \
  repacked/28.0-candidates/build1/ \
  ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/candidates/28.0-candidates/build1/
# reran check_permissions builder to make sure all is correct.

# run av scan on partials, anyone on stage
cd /pub/mozilla.org/firefox/candidates/28.0-candidates/build1/update
nice ionice -c2 -n7 extract_and_run_command.py -j2 clamdscan -m --no-summary -- \
  `find . -name firefox-28.0b9-28.0.partial.mar -o -name firefox-28.0-forbeta.complete.mar` 2>&1 | tee /tmp/av.log

# bump + modify patcher config - I did this on my laptop for 27.0
cd <parent_of_tools_checkout>
export PERL5LIB="tools/lib/perl:$HOME/usr/perl"  # need Config::General v2.44
wget http://hg.mozilla.org/releases/mozilla-release/raw-file/FIREFOX_28_0_RELEASE/browser/locales/shipped-locales
perl tools/release/patcher-config-bump.pl -p firefox -r Firefox -v 28.0 -a 28.0 -o 28.0b9 -b 1 \
  -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg -t stage.mozilla.org -f ftp.mozilla.org \
  -d download.mozilla.org -l shipped-locales --partial-version 28.0b9 --platform linux --platform linux64 \
  --platform macosx64 --platform win32
# then some manual modifications <28.0> block ——> <28.0-build1>, to -> 28.0-build1, add mar-channel-ids, pretty version
# tweak paths for mar filenames + bouncer url. Update verify configs can be generated from this
# upload patch to bug 961831  (update verify harness changes from previous repackings have already landed)

# generate snippets (on bld-centos6-hp-016):
cd /home/cltbld/bug961831/tools
hg pull
hg up -r FIREFOX_28_0_RELEASE_RUNTIME
wget -O- https://bugzilla.mozilla.org/attachment.cgi?id=8389205 | patch -p1 
cd ..
export PYTHONPATH="tools/lib/python:tools/lib/python/vendor"
python tools/scripts/updates/create-snippets.py --config tools/release/patcher-configs/mozBeta-branch-patcher2.cfg \
   --checksums-dir checksums --snippet-dir aus2 --test-snippet-dir aus2.test -v 2>&1 | tee snippets.log

# ditch the betatest snippets, we're going to use releasetest only before pushing to beta
rm -rf aus2.test/Firefox/*/WIN*/*/*/betatest
rm -rf aus2.test/Firefox/*/Lin*/*/*/betatest
rm -rf aus2.test/Firefox/*/Dar*/*/*/betatest

# Sync to aus3-staging 
rsync -av -e 'ssh -i /home/cltbld/.ssh/ffxbld_dsa' aus2.test/ \
  ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-28.0-build1-forbeta-test/
rsync -av -e 'ssh -i /home/cltbld/.ssh/ffxbld_dsa' aus2/ \
  ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-28.0-build1-forbeta/

# push test snippets live, on aus3-staging
~/bin/pushsnip Firefox-28.0-build1-forbeta-test

# Bouncer setup (to match patcher config)
# add Products: Firefox-28.0-build1-complete, Firefox-28.0-build1-Partial-28.0b9
# add Locations: 4 locations for each

# announce to r-d

# generate update verify configs, done on my laptop
cd $parent_of_tools_repo
python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg --platform linux64 --output tools/release/updates/mozBeta-firefox-linux64.cfg --release-config-file mozilla/release-firefox-mozilla-release.py -b https://hg.mozilla.org/build/buildbot-configs --channel releasetest -t FIREFOX_28_0_RELEASE
python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg --platform linux --output tools/release/updates/mozBeta-firefox-linux.cfg --release-config-file mozilla/release-firefox-mozilla-release.py -b https://hg.mozilla.org/build/buildbot-configs --channel releasetest -t FIREFOX_28_0_RELEASE
python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg --platform macosx64 --output tools/release/updates/mozBeta-firefox-mac64.cfg --release-config-file mozilla/release-firefox-mozilla-release.py -b https://hg.mozilla.org/build/buildbot-configs --channel releasetest -t FIREFOX_28_0_RELEASE
python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg --platform win32 --output tools/release/updates/mozBeta-firefox-win32.cfg --release-config-file mozilla/release-firefox-mozilla-release.py -b https://hg.mozilla.org/build/buildbot-configs --channel releasetest -t FIREFOX_28_0_RELEASE
# hand edit to make the following changes:
#  - remove everything except the last two betas (in this case, that was everything except the top 3 lines)
#  - fix "to" variable to point at valid candidates directory

# update verify, manual launch on appropriate slaves, eg
# linux32 is special: need a slave that last did a 32bit build in the mock env (or set one up)

mock_mozilla -r mozilla-centos6-x86_64 --cwd /builds/slave --unpriv --shell 'PATH="/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin" bash'

# other platforms: no special action
# all:
mkdir bug961831; cd $_
hg clone http://hg.mozilla.org/build/tools
cd tools
hg up -r FIREFOX_28_0_RELEASE_RUNTIME
wget -O- https://bugzilla.mozilla.org/attachment.cgi?id=8389239 | patch -p1 
cd release/updates
# use correct .cfg file for platform
./verify.sh -c mozBeta-firefox-linux64.cfg 2>&1 | tee log

For posterity, both linux update verifies were run on bld-centos6-hp-016 (64-bit outside of mock, 32-bit inside of mock). bld-lion-r5-005 was used for Mac. w64-ix-slave21 was used for Windows.

Update verify results:

• Linux/Linux64 - PASS
  • Only expected diffs found in output:

78813 diff -r source/firefox/defaults/pref/channel-prefs.js target/firefox/defaults/pref/channel-prefs.js
78814 1c1
78815 < //@line 2 "/builds/slave/rel-m-beta-l64_bld-00000000000/build/browser/app/profile/channel-prefs.js"
78816 ---
78817 > //@line 2 "/builds/slave/rel-m-rel-l64_bld-000000000000/build/browser/app/profile/channel-prefs.js"
78818 6c6
78819 < pref("app.update.channel", "beta");
78820 ---
78821 > pref("app.update.channel", "release");
78822 diff -r source/firefox/update-settings.ini target/firefox/update-settings.ini
78823 5c5
78824 < ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-beta,firefox-mozilla-release
78825 ---
78826 > ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-release

• Mac - PASS
  • Same diffs as above
• Win32 - PASS
  • Same diffs as above

Fix up releasetest for beta

ashughes reported failures when testing from 27.0 using releasetest. On aus3, we have a partial.txt from release updates (27.0 -> 28.0), and a complete.txt from beta updates. Ideally we have separate channels for this, but in the meantime

# ffxbld@aus3-staging
for f in /opt/aus2/incoming/3/Firefox/27.0/*/20140127194636/*/releasetest/partial.txt; do 
  mv -v $f{,.for_release};
done

We'll need to undo this for release testing, and fix the complete.txt. Repushing the Firefox-28.0-build1-test snippets would one way to do that.