Releases/Firefox 30.0/BuildNotes

From MozillaWiki
Jump to navigation Jump to search

bug 996137 - hwine

Build 1

Issues hit: Just the releasenotes url below

Updates for beta channel (nthomas)

 # using bld-centos6-hp-015 with screen
 mkdir ~/bug978746
 cd ~/bug978746
 hg clone http://hg.mozilla.org/build/tools
 # get a signing token for release certs, 6 hour expiry
 export PYTHONPATH=tools/lib/python; export PATH=$PATH:`pwd`
 # updated since 29.0 to strip out old code, fix defaults
 wget -O repack.py https://bugzilla.mozilla.org/attachment.cgi?id=8433787

 # make sure enough disk space is free
 python ~/bug978746/tools/buildfarm/maintenance/purge_builds.py -s 55 -n 'rel-*:45d' -n 'tb-rel-*:45d' /builds/slave

 # in separate screen windows, generate other platforms
 cd ~/bug978746
 export PYTHONPATH=tools/lib/python; export PATH=$PATH:`pwd`
 time python -u repack.py -f 30.0b9-build1 -t 30.0-build1 -T 30.0b9-build1 -P linux-i686 2>&1 | tee linux.log
 # repeat for these:
 time python -u repack.py -f 30.0b9-build1 -t 30.0-build1 -T 30.0b9-build1 -P linux-x86_64 --no-env 2>&1 | tee linux64.log
 time python -u repack.py -f 30.0b9-build1 -t 30.0-build1 -T 30.0b9-build1 -P mac --no-env 2>&1 | tee mac.log
 time python -u repack.py -f 30.0b9-build1 -t 30.0-build1 -T 30.0b9-build1 -P win32 --no-env 2>&1 | tee win32.log

# fix permissions and upload to stage
find repacked/ -type f -exec chmod 644 {} \;
find repacked/ -type d -exec chmod 755 {} \;
# rsync -n is your friend
rsync -av -e 'ssh -i /home/cltbld/.ssh/ffxbld_dsa' \
  repacked/30.0-candidates/build1/ \
  ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/candidates/30.0-candidates/build1/
# reran check_permissions builder to make sure all is correct.

# run av scan on partials, anyone on stage
cd /pub/mozilla.org/firefox/candidates/30.0-candidates/build1/update
nice ionice -c2 -n7 extract_and_run_command.py -j2 clamdscan -m --no-summary -- \
  `find . -name firefox-30.0b9-30.0.partial.mar` 2>&1 | tee /tmp/av.log

# Generate patcher configs and update verify configs, nthomas's laptop but can be done on a slave too
cd <parent_of_tools_checkout>
export PERL5LIB="tools/lib/perl:$HOME/perl5/lib/perl5"  # need Config::General v2.44
wget http://hg.mozilla.org/releases/mozilla-release/raw-file/FIREFOX_30_0_RELEASE/browser/locales/shipped-locales
perl tools/release/patcher-config-bump.pl -p firefox -r Firefox -v 30.0 -a 30.0 -o 30.0b9 -b 1 \
  -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg -t stage.mozilla.org -f ftp.mozilla.org \
  -d download.mozilla.org -l shipped-locales --partial-version 30.0b9 --platform linux --platform linux64 \
  --platform macosx64 --platform win32
# then some manual modifications: url in complete uses product firefox-30.0-build1-complete, 
# <30.0> block ——> <30.0-build1>, to -> 30.0-build1, add mar-channel-ids, pretty version


# generate update verify configs
python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg --platform linux64 --output tools/release/updates/mozBeta-firefox-linux64.cfg --release-config-file mozilla/release-firefox-mozilla-release.py -b https://hg.mozilla.org/build/buildbot-configs --channel releasetest -t FIREFOX_30_0_RELEASE
python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg --platform linux --output tools/release/updates/mozBeta-firefox-linux.cfg --release-config-file mozilla/release-firefox-mozilla-release.py -b https://hg.mozilla.org/build/buildbot-configs --channel releasetest -t FIREFOX_30_0_RELEASE
python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg --platform macosx64 --output tools/release/updates/mozBeta-firefox-mac64.cfg --release-config-file mozilla/release-firefox-mozilla-release.py -b https://hg.mozilla.org/build/buildbot-configs --channel releasetest -t FIREFOX_30_0_RELEASE
python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg --platform win32 --output tools/release/updates/mozBeta-firefox-win32.cfg --release-config-file mozilla/release-firefox-mozilla-release.py -b https://hg.mozilla.org/build/buildbot-configs --channel releasetest -t FIREFOX_30_0_RELEASE

# fix the to="/path" definitions
for f in tools/scripts/release/updates/mozBeta-firefox-{linux64,linux,mac64,win32}.cfg; do
  sed -i '' -e 's,30.0-build1,30.0,' $f
done
hg -R tools diff > patch
# attach this to bug 996137, attachment 8434029

# generate snippets (on bld-centos6-hp-015):
cd /home/cltbld/bug978746/tools
hg pull
hg up -r FIREFOX_30_0_RELEASE_RUNTIME
wget -O- https://bugzilla.mozilla.org/attachment.cgi?id=8434029 | patch -p1 
cd ..
export PYTHONPATH="tools/lib/python:tools/lib/python/vendor"
python tools/scripts/updates/create-snippets.py --config tools/release/patcher-configs/mozBeta-branch-patcher2.cfg \
   --checksums-dir checksums --snippet-dir aus2 --test-snippet-dir aus2.test -v 2>&1 | tee snippets.log

# ditch the betatest snippets, we're going to use releasetest only before pushing to beta
rm -rf aus2.test/Firefox/*/WIN*/*/*/betatest
rm -rf aus2.test/Firefox/*/Lin*/*/*/betatest
rm -rf aus2.test/Firefox/*/Dar*/*/*/betatest

# Sync to aus3-staging 
rsync -av -e 'ssh -i /home/cltbld/.ssh/ffxbld_dsa' aus2.test/ \
  ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-30.0-build1-forbeta-test/
rsync -av -e 'ssh -i /home/cltbld/.ssh/ffxbld_dsa' aus2/ \
  ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-30.0-build1-forbeta/

# Bouncer setup (to match patcher config)
# add Products: Firefox-30.0-Partial-30.0b9, Firefox-30.0-build1-Complete
# add Locations: 

Product                         OS       Path
Firefox-30.0-Partial-30.0b9	win	/firefox/candidates/30.0-candidates/build1/update/win32/:lang/firefox-30.0b9-30.0.partial.mar
Firefox-30.0-Partial-30.0b9	linux	/firefox/candidates/30.0-candidates/build1/update/linux-i686/:lang/firefox-30.0b9-30.0.partial.mar
Firefox-30.0-Partial-30.0b9	linux64	/firefox/candidates/30.0-candidates/build1/update/linux-x86_64/:lang/firefox-30.0b9-30.0.partial.mar
Firefox-30.0-Partial-30.0b9	osx	/firefox/candidates/30.0-candidates/build1/update/mac/:lang/firefox-30.0b9-30.0.partial.mar 

Firefox-30.0-build1-Complete	win	/firefox/candidates/30.0-candidates/build1/update/win32/:lang/firefox-30.0.complete.mar
Firefox-30.0-build1-Complete	linux	/firefox/candidates/30.0-candidates/build1/update/linux-i686/:lang/firefox-30.0.complete.mar
Firefox-30.0-build1-Complete	linux64	/firefox/candidates/30.0-candidates/build1/update/linux-x86_64/:lang/firefox-30.0.complete.mar
Firefox-30.0-build1-Complete	osx	/firefox/candidates/30.0-candidates/build1/update/mac/:lang/firefox-30.0.complete.mar

# push test snippets live, on aus3-staging
~/bin/pushsnip Firefox-30.0-build1-forbeta-test

# Update verify
# All platforms except linux32:

mkdir bug978746 ; cd $_
hg clone http://hg.mozilla.org/build/tools
cd tools
hg up -r FIREFOX_30_0_RELEASE_RUNTIME
wget -O- https://bugzilla.mozilla.org/attachment.cgi?id=8434029 | patch -p1 
cd release/updates
# use correct .cfg file for platform
./verify.sh -c mozBeta-firefox-linux64.cfg 2>&1 | tee log

For posterity, both linux update verifies were run on bld-centos6-hp-015 (64-bit outside of mock, 32-bit inside of mock), bld-lion-r5-057 was used for Mac, b-2008-ix-0084 was used for Windows. Expected differences look like this (search for calling 'QuitProgressUI'): 

diff -r source/firefox/defaults/pref/channel-prefs.js target/firefox/defaults/pref/channel-prefs.js
1c1
< //@line 2 "/builds/slave/rel-m-beta-l64_bld-00000000000/build/browser/app/profile/channel-prefs.js"
---
> //@line 2 "/builds/slave/rel-m-rel-l64_bld-000000000000/build/browser/app/profile/channel-prefs.js"
6c6
< pref("app.update.channel", "beta");
---
> pref("app.update.channel", "release");
diff -r source/firefox/update-settings.ini target/firefox/update-settings.ini
5c5
< ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-beta,firefox-mozilla-release
---
> ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-release

Fix whatsnew page for release channel (nthomas)

We're pointing at the 29.0 url, eg 

 https://www.mozilla.org/%locale%/firefox/29.0/whatsnew/?oldversion=%OLD_VERSION%

Should be 30.0. bug 1009893 handles this, except for the config being stale already, landed a fix.

Massage the snippets to fix 

 # ffxbld@aus3-staging
 cd /opt/aus2/snippets
 mv pushed/Firefox-30.0-build1-test staging/
 cd staging
 for d in Firefox-30.0-build1-test Firefox-30.0-build1; do
   for f in `find $d -type f`; do
     echo $f
     sed -i'' -e 's,firefox/29.0/whatsnew,firefox/30.0/whatsnew,' $f;
   done
 done
 pushsnip Firefox-30.0-build1-test

Build 2

Issues hit:

Both of those are fallout from ship-it having 30.0build1 in the list of partials (communication issue with RelMan).

Updates for beta channel (nthomas/bhearsum)

Want to do updates from 30.0b9 and 30.0 build1. Ship-it was set to do 30.0build1 partials, which is odd but helpful. Still need the 30.0b9 ones though (2/3 of actively updating users). 

 # using bld-centos6-hp-019 with screen
 mkdir ~/bug978746
 cd ~/bug978746
 hg clone http://hg.mozilla.org/build/tools
 # get a signing token for release certs, 6 hour expiry
 export PYTHONPATH=tools/lib/python; export PATH=$PATH:`pwd`
 # updated since 29.0 to strip out old code, fix defaults
 wget -O repack.py https://bugzilla.mozilla.org/attachment.cgi?id=8435509

 # make sure enough disk space is free, (55G for extra set of extra partial)
 python ~/bug978746/tools/buildfarm/maintenance/purge_builds.py -s 55 -n 'rel-*:45d' -n 'tb-rel-*:45d' /builds/slave

 # in separate screen windows, generate other platforms
 cd ~/bug978746
 export PYTHONPATH=tools/lib/python; export PATH=$PATH:`pwd`
 time python -u repack.py -f 30.0b9-build1 -t 30.0-build2 -T 30.0b9-build1 -P mac 2>&1 | tee mac.log
 # repeat for these:
 time python -u repack.py -f 30.0b9-build1 -t 30.0-build2 -T 30.0b9-build1 -P linux-i686 --no-env 2>&1 | tee linux.log
 time python -u repack.py -f 30.0b9-build1 -t 30.0-build2 -T 30.0b9-build1 -P linux-x86_64 --no-env 2>&1 | tee linux64.log
 time python -u repack.py -f 30.0b9-build1 -t 30.0-build2 -T 30.0b9-build1 -P win32 --no-env 2>&1 | tee win32.log

# fix permissions and upload to stage
find repacked/ -type f -exec chmod 644 {} \;
find repacked/ -type d -exec chmod 755 {} \;
# rsync -n is your friend
rsync -nav -e 'ssh -i /home/cltbld/.ssh/ffxbld_dsa' \
  repacked/30.0-candidates/build2/ \
  ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/candidates/30.0-candidates/build2/
# reran check_permissions builder to make sure all is correct.

# run av scan on partials, anyone on stage
cd /pub/mozilla.org/firefox/candidates/30.0-candidates/build2/update
nice ionice -c2 -n7 extract_and_run_command.py -j2 clamdscan -m --no-summary -- \
  `find . -name firefox-30.0b9-30.0.partial.mar` 2>&1 | tee /tmp/av.log

# Generate patcher configs and update verify configs, nthomas's laptop but can be done on a slave too
cd /path/to/tools/..
export PERL5LIB="tools/lib/perl:$HOME/perl5/lib/perl5"  # need Config::General v2.44
wget http://hg.mozilla.org/releases/mozilla-release/raw-file/FIREFOX_30_0_RELEASE/browser/locales/shipped-locales
perl tools/release/patcher-config-bump.pl -p firefox -r Firefox -v 30.0 -a 30.0 -o 30.0 -b 2 \
  -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg -t stage.mozilla.org -f ftp.mozilla.org \
  -d download.mozilla.org -l shipped-locales --partial-version 30.0b9 --platform linux --platform linux64 \
  --platform macosx64 --platform win32
# then some manual modifications: url in <complete> should use product firefox-30.0-build2-complete, 
# <30.0> block ——> <30.0-build2>, from -> 30.0-build1, add <30.0-build1> to <partials>
# to -> 30.0-build1, add past-update line for 30.0b9
# add mar-channel-ids & pretty version to <30.0-build1> and <30.0-build2>

# generate update verify configs
python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg --platform linux64 --output tools/release/updates/mozBeta-firefox-linux64.cfg --release-config-file mozilla/release-firefox-mozilla-release.py -b https://hg.mozilla.org/build/buildbot-configs --channel releasetest -t FIREFOX_30_0_RELEASE
python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg --platform linux --output tools/release/updates/mozBeta-firefox-linux.cfg --release-config-file mozilla/release-firefox-mozilla-release.py -b https://hg.mozilla.org/build/buildbot-configs --channel releasetest -t FIREFOX_30_0_RELEASE
python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg --platform macosx64 --output tools/release/updates/mozBeta-firefox-mac64.cfg --release-config-file mozilla/release-firefox-mozilla-release.py -b https://hg.mozilla.org/build/buildbot-configs --channel releasetest -t FIREFOX_30_0_RELEASE
python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg --platform win32 --output tools/release/updates/mozBeta-firefox-win32.cfg --release-config-file mozilla/release-firefox-mozilla-release.py -b https://hg.mozilla.org/build/buildbot-configs --channel releasetest -t FIREFOX_30_0_RELEASE

# fix the to="/path" definitions
for f in tools/release/updates/mozBeta-firefox-{linux64,linux,mac64,win32}.cfg; do
  echo $f
  sed -i '' -e 's,30.0-build2,30.0,g' $f
done
# fix the from="/path" and to="/path" definitions
for f in tools/release/updates/mozBeta-firefox-{linux64,linux,mac64,win32}.cfg; do
  echo $f
  sed -i '' -e 's,firefox/releases/30.0-build1,firefox/candidates/30.0-candidates/build1,' -e 's,30.0-build1,30.0,g' $f
done
hg -R tools diff > patch
# attach this to bug 996137, attachment 8435674

# generate snippets (on bld-centos6-hp-019):
cd /home/cltbld/bug978746/tools
hg pull
hg up -r FIREFOX_30_0_RELEASE_RUNTIME
wget -O- https://bugzilla.mozilla.org/attachment.cgi?id=8435674 | patch -p1 
cd ..
export PYTHONPATH="tools/lib/python:tools/lib/python/vendor"
python tools/scripts/updates/create-snippets.py --config tools/release/patcher-configs/mozBeta-branch-patcher2.cfg \
   --checksums-dir checksums --snippet-dir aus2 --test-snippet-dir aus2.test -v 2>&1 | tee snippets.log

# ditch the betatest snippets, we're going to use releasetest only before pushing to beta
rm -rf aus2.test/Firefox/*/WIN*/*/*/betatest
rm -rf aus2.test/Firefox/*/Lin*/*/*/betatest
rm -rf aus2.test/Firefox/*/Dar*/*/*/betatest

# Sync to aus3-staging 
rsync -av -e 'ssh -i /home/cltbld/.ssh/ffxbld_dsa' aus2.test/ \
  ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-30.0-build2-forbeta-test/
rsync -av -e 'ssh -i /home/cltbld/.ssh/ffxbld_dsa' aus2/ \
  ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-30.0-build2-forbeta/

# Bouncer setup (to match patcher config)
Product                         OS       Path
Firefox-30.0-build2-Partial-30.0b9	win	/firefox/candidates/30.0-candidates/build2/update/win32/:lang/firefox-30.0b9-30.0.partial.mar
Firefox-30.0-build2-Partial-30.0b9	linux	/firefox/candidates/30.0-candidates/build2/update/linux-i686/:lang/firefox-30.0b9-30.0.partial.mar
Firefox-30.0-build2-Partial-30.0b9	linux64	/firefox/candidates/30.0-candidates/build2/update/linux-x86_64/:lang/firefox-30.0b9-30.0.partial.mar
Firefox-30.0-build2-Partial-30.0b9	osx	/firefox/candidates/30.0-candidates/build2/update/mac/:lang/firefox-30.0b9-30.0.partial.mar

Firefox-30.0-build2-Partial-30.0-build1	win	/firefox/candidates/30.0-candidates/build2/update/win32/:lang/firefox-30.0-30.0.partial.mar
Firefox-30.0-build2-Partial-30.0-build1	linux	/firefox/candidates/30.0-candidates/build2/update/linux-i686/:lang/firefox-30.0-30.0.partial.mar
Firefox-30.0-build2-Partial-30.0-build1	linux64	/firefox/candidates/30.0-candidates/build2/update/linux-x86_64/:lang/firefox-30.0-30.0.partial.mar
Firefox-30.0-build2-Partial-30.0-build1	osx	/firefox/candidates/30.0-candidates/build2/update/mac/:lang/firefox-30.0-30.0.partial.mar

Firefox-30.0-build2-Complete	win	/firefox/candidates/30.0-candidates/build2/update/win32/:lang/firefox-30.0.complete.mar
Firefox-30.0-build2-Complete	linux	/firefox/candidates/30.0-candidates/build2/update/linux-i686/:lang/firefox-30.0.complete.mar
Firefox-30.0-build2-Complete	linux64	/firefox/candidates/30.0-candidates/build2/update/linux-x86_64/:lang/firefox-30.0.complete.mar
Firefox-30.0-build2-Complete	osx	/firefox/candidates/30.0-candidates/build2/update/mac/:lang/firefox-30.0.complete.mar

# push test snippets live, on aus3-staging
~/bin/pushsnip Firefox-30.0-build2-forbeta-test

# Update verify
# All platforms except linux32:

mkdir bug978746 ; cd $_
hg clone http://hg.mozilla.org/build/tools
cd tools
hg up -r FIREFOX_30_0_RELEASE_RUNTIME
wget -O- https://bugzilla.mozilla.org/attachment.cgi?id=8435674 | patch -p1 
cd release/updates
# use correct .cfg file for platform
./verify.sh -c mozBeta-firefox-linux64.cfg 2>&1 | tee log

Running on bld-centos6-hp-019 (inside and outside mock, in screen), bld-lion-r5-075, w64-ix-slave104.

Update Verify Issue

Getting failures doing 30.0 build1 -> build2 due to the harness never having to cope with that before (urls with same basename overwriting each other, so we try to apply the partial to build2, and it fails bug 1021697).

Workaround:

  • in each stage:/pub/mozilla.org/firefox/candidates/30.0-candidates/build1/{linux-i686,linux-x86_64,mac,win32}/*, create a symlink, eg linux*: firefox-30.0-build1.tar.bz2 -> firefox-30.0.tar.bz2
  • modify the update verify config for this, eg linux64 (see from=)
release="30.0" product="Firefox" platform="Linux_x86_64-gcc3" build_id="20140603140158" locales="ach ..." channel="releasetest" \
patch_types="complete partial" from="/firefox/candidates/30.0-candidates/build1/linux-x86_64/%locale%/firefox-30.0-build1.tar.bz2" \
ftp_server_from="http://stage.mozilla.org/pub/mozilla.org" ftp_server_to="http://stage.mozilla.org/pub/mozilla.org" \
mar_channel_IDs="firefox-mozilla-beta,firefox-mozilla-release" aus_server="https://aus3.mozilla.org" \
ftp_server_to="http://stage.mozilla.org/pub/mozilla.org" to="/firefox/candidates/30.0-candidates/build2/linux-x86_64/%locale%/firefox-30.0.tar.bz2"

To test build1 -> build2 partials, did the following:

  • On FTP, create symlinks for all builds (see above):
cd /pub/mozilla.org/firefox/candidates/30.0-candidates/build1
cd linux-i686
for i in `ls | grep -v xpi | grep -v ach`; do cd $i && ln -s firefox-30.0.tar.bz2 firefox-30.0-build1.tar.bz2 && cd ..; done
cd ../linux-x86_64
for i in `ls | grep -v xpi | grep -v ach`; do cd $i && ln -s firefox-30.0.tar.bz2 firefox-30.0-build1.tar.bz2 && cd ..; done
cd ../mac
for i in `ls | grep -v xpi`; do cd $i && ln -s "Firefox 30.0.dmg" "Firefox 30.0-build1.dmg" && cd ..; done
cd ../win32
for i in `ls | grep -v xpi`; do cd $i && ln -s "Firefox Setup 30.0.exe" "Firefox Setup 30.0-build1.exe" && cd ..; done
  • On each platform, after original update verify run had finished:
    • Modified update verify config:
      • Removed all tests except build1 -> build2
      • Updated build1 installer URLs to point at "-build1" version of files created above.
      • Restarted update verify with new config
      • Configs ended up looking like this:
release="30.0" product="Firefox" platform="Linux_x86_64-gcc3" build_id="20140603140158" locales="ach af an ar as ast be bg bn-BD bn-IN br bs ca cs csb cy da de el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gu-IN he hi-IN hr hu hy-AM id is it ja kk km kn ko ku lij lt lv mai mk ml mr ms nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk vi xh zh-CN zh-TW zu" channel="releasetest" patch_types="complete partial" from="/firefox/candidates/30.0-candidates/build1/linux-x86_64/%locale%/firefox-30.0-build1.tar.bz2" ftp_server_from="http://stage.mozilla.org/pub/mozilla.org" ftp_server_to="http://stage.mozilla.org/pub/mozilla.org" mar_channel_IDs="firefox-mozilla-beta,firefox-mozilla-release" to="/firefox/candidates/30.0-candidates/build2/linux-x86_64/%locale%/firefox-30.0.tar.bz2"

After things finished, verified by looking for "FAIL" in the logs. The original update verify runs each had 88 FAILs (one per locale from the 30.0build1 tests). The second runs had zero fails (redo of the fixed 30.0build1 tests.)

Bouncer uptake

See bug 1023031 - basically we were looking for a non-existent product, then hit a db error.

Fennec Push

See bug 1025501 - fennec push to ftp missed during release.

Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_30.0/BuildNotes&oldid=989430"