Releases/Firefox 28.0/BuildNotes

bug 961831 - bhearsum

Build 1

Issues hit:

• 5x win32 update verify failures from network disruption, auto-retried

Updates for beta channel

# using bld-centos6-hp-016 with screen
mkdir ~/bug961831
cd ~/bug961831
hg clone http://hg.mozilla.org/build/tools
# get a signing token for release certs, 6 hour expiry
export PYTHONPATH=tools/lib/python; export PATH=$PATH:`pwd`
wget -O repack.py https://bugzilla.mozilla.org/attachment.cgi?id=8389037

# make sure enough disk space is free
python ~/bug961831/tools/buildfarm/maintenance/purge_builds.py -s 40 -n 'rel-*:45d' -n 'tb-rel-*:45d' /builds/slave

# start mac repack
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P mac 2>&1 | tee mac.log

# in separate screen windows, generate other platforms
cd ~/bug961831
export PYTHONPATH=tools/lib/python; export PATH=$PATH:`pwd`
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-i686 --no-env 2>&1 | tee linux.log
# repeat for these:
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-x86_64 --no-env 2>&1 | tee linux64.log
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P win32 --no-env 2>&1 | tee win32.log
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P mac --no-env 2>&1 | tee mac.log

Unfortunately, not enough disk space was free, and each process failed out because of it. Mac failed at "lij", linux-i686 failed at "tr", linux-x86_64 at "uk", and win32 at "zu". To get things going again, did the following:

• Cleared 14G of disk space by hand
• Modified repack.py to restart where it left off, with this patch:

--- repack.py.orig      2014-03-11 06:24:30.996377161 -0700
+++ repack.py   2014-03-11 06:23:59.384371739 -0700
@@ -329,6 +329,7 @@
 
     # calculate locale overlap
     locales = getLocales(from_build, to_build)
+    firstLocale = {"mac": "lij", "linux-i686": "tr", "linux-x86_64": "uk", "win32": "zu"}
 
     if options.platforms:
         platforms = []
@@ -344,6 +345,11 @@
 
     print "%s locales to do, %s platforms" % (len(locales), len(platforms))
     for p in platforms:
+        newLocales = []
+        for i in sorted(locales):
+            if i >= firstLocale[p]:
+                newLocales.append(i)
+        locales = newLocales
         for l in locales:
             if l == 'ja-JP-mac' and p != 'mac':
                 continue

• Tested the above by commenting out all actual work in the script and adding a print, output looked fine, eg:

[cltbld@bld-centos6-hp-016.build.scl1.mozilla.com bug961831]$  time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-x86_64 --no-env
ERROR: Failed to open cert /etc/ssl/certs/Makefile: (-34).
ERROR: Failed to open cert /etc/ssl/certs/ca-bundle.trust.crt: (-34).
ERROR: Failed to open cert /etc/ssl/certs/make-dummy-cert: (-34).
2014-03-11 06:23:38 URL:https://hg.mozilla.org/releases/mozilla-beta/raw-file/FIREFOX_28_0b9_RELEASE/browser/locales/shipped-locales [365/365] -> "from_shipped-locales" [1]

ERROR: Failed to open cert /etc/ssl/certs/Makefile: (-34).
ERROR: Failed to open cert /etc/ssl/certs/ca-bundle.trust.crt: (-34).
ERROR: Failed to open cert /etc/ssl/certs/make-dummy-cert: (-34).
2014-03-11 06:23:38 URL:https://hg.mozilla.org/releases/mozilla-release/raw-file/FIREFOX_28_0_RELEASE/browser/locales/shipped-locales [359/359] -> "to_shipped-locales" [1]

88 locales to do, 1 platforms
** Working on 28.0-build1 linux-x86_64 uk
------------------------------------------------------------------------------------------------------------------------
** Working on 28.0-build1 linux-x86_64 vi
------------------------------------------------------------------------------------------------------------------------
** Working on 28.0-build1 linux-x86_64 xh
------------------------------------------------------------------------------------------------------------------------
** Working on 28.0-build1 linux-x86_64 zh-CN
------------------------------------------------------------------------------------------------------------------------
** Working on 28.0-build1 linux-x86_64 zh-TW
------------------------------------------------------------------------------------------------------------------------
** Working on 28.0-build1 linux-x86_64 zu
------------------------------------------------------------------------------------------------------------------------

• Removed comment and restart each platforms' MAR generation with these commands (note "tee -a" to append to the original logs):

time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-i686 --no-env 2>&1 | tee -a linux.log
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-x86_64 --no-env 2>&1 | tee -a linux64.log
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P win32 --no-env 2>&1 | tee -a win32.log
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P mac --no-env 2>&1 | tee -a mac.log

Windows completed successfully, but mac and both linux platforms failed to sign most of their MARs due to an expired token. To fix:

• Generated a new signing token & deleted the "nonce" file
• Deleted the files generated for the locales that failed to sign (mai for mac, zh-TW for linux-i686, zu for linux-x86_64)
• Adjusted the firstLocale for mac & linux platforms in the modified repack.py
• Started the mac & linux platforms again (I didn't bother adjusting the locales because only 1 of each platform at most succeeded):

time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-i686 --no-env 2>&1 | tee -a linux.log
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P linux-x86_64 --no-env 2>&1 | tee -a linux64.log
time python -u repack.py -f 28.0b9-build1 -t 28.0-build1 -T 28.0b9-build1 -P mac --no-env 2>&1 | tee -a mac.log

# fix permissions and upload to stage
find repacked/ -type f -exec chmod 644 {} \;
find repacked/ -type d -exec chmod 755 {} \;
# rsync -n is your friend
rsync -av -e 'ssh -i /home/cltbld/.ssh/ffxbld_dsa' \
  repacked/28.0-candidates/build1/ \
  ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/candidates/28.0-candidates/build1/
# reran check_permissions builder to make sure all is correct.

# run av scan on partials, anyone on stage
cd /pub/mozilla.org/firefox/candidates/28.0-candidates/build1/update
nice ionice -c2 -n7 extract_and_run_command.py -j2 clamdscan -m --no-summary -- \
  `find . -name firefox-28.0b9-28.0.partial.mar -o -name firefox-28.0-forbeta.complete.mar` 2>&1 | tee /tmp/av.log

# bump + modify patcher config - I did this on my laptop for 27.0
cd <parent_of_tools_checkout>
export PERL5LIB="tools/lib/perl:$HOME/usr/perl"  # need Config::General v2.44
wget http://hg.mozilla.org/releases/mozilla-release/raw-file/FIREFOX_28_0_RELEASE/browser/locales/shipped-locales
perl tools/release/patcher-config-bump.pl -p firefox -r Firefox -v 28.0 -a 28.0 -o 28.0b9 -b 1 \
  -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg -t stage.mozilla.org -f ftp.mozilla.org \
  -d download.mozilla.org -l shipped-locales --partial-version 28.0b9 --platform linux --platform linux64 \
  --platform macosx64 --platform win32
# then some manual modifications <28.0> block ——> <28.0-build1>, to -> 28.0-build1, add mar-channel-ids, pretty version
# tweak paths for mar filenames + bouncer url. Update verify configs can be generated from this
# upload patch to bug 961831  (update verify harness changes from previous repackings have already landed)

# generate snippets (on bld-centos6-hp-016):
cd /home/cltbld/bug961831/tools
hg pull
hg up -r FIREFOX_28_0_RELEASE_RUNTIME
wget -O- https://bugzilla.mozilla.org/attachment.cgi?id=8389205 | patch -p1 
cd ..
export PYTHONPATH="tools/lib/python:tools/lib/python/vendor"
python tools/scripts/updates/create-snippets.py --config tools/release/patcher-configs/mozBeta-branch-patcher2.cfg \
   --checksums-dir checksums --snippet-dir aus2 --test-snippet-dir aus2.test -v 2>&1 | tee snippets.log

# ditch the betatest snippets, we're going to use releasetest only before pushing to beta
rm -rf aus2.test/Firefox/*/WIN*/*/*/betatest
rm -rf aus2.test/Firefox/*/Lin*/*/*/betatest
rm -rf aus2.test/Firefox/*/Dar*/*/*/betatest

# Sync to aus3-staging 
rsync -av -e 'ssh -i /home/cltbld/.ssh/ffxbld_dsa' aus2.test/ \
  ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-28.0-build1-forbeta-test/
rsync -av -e 'ssh -i /home/cltbld/.ssh/ffxbld_dsa' aus2/ \
  ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-28.0-build1-forbeta/

# push test snippets live, on aus3-staging
~/bin/pushsnip Firefox-28.0-build1-forbeta-test

# Bouncer setup (to match patcher config)
# add Products: Firefox-28.0-build1-complete, Firefox-28.0-build1-Partial-28.0b9
# add Locations: 4 locations for each

# announce to r-d

DONE TO HERE

# update verify, manual launch on appropriate slaves, eg
# linux32 is special: need a slave that last did a 32bit build in the mock env (or set one up)

mock_mozilla -r mozilla-centos6-x86_64 --cwd /builds/slave --unpriv --shell 'PATH="/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin" bash'

# other platforms: no special action
# all:
mkdir bug961831; cd $_
hg clone http://hg.mozilla.org/build/tools
cd tools
hg up -r FIREFOX_28_0_RELEASE_RUNTIME
wget -O- https://bugzilla.mozilla.org/attachment.cgi?id=8389239 | patch -p1 
cd release/updates
# use correct .cfg file for platform
./verify.sh -c mozBeta-firefox-linux64.cfg 2>&1 | tee log