Releases/Firefox 11.0/BuildNotes: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(43 intermediate revisions by 2 users not shown)
Line 3: Line 3:


= Bugs hit =
= Bugs hit =
Enter any bugs pre-existing or newly discovered and filed during the release:
* {{Bug|530808}} - support locales in beta (mn, sw were in the changesets on the dashboard)
* {{bug|734644}} - hg.m.o time out during Firefox 11.0 build
* {{bug|734644}} - hg.m.o time out during Firefox 11.0 build
* {{bug|734645}} - l10n verification times out unpacking files
* {{bug|734645}} - l10n verification times out unpacking files
Line 13: Line 13:


=Signed-off Revision(s)=
=Signed-off Revision(s)=
Build 1: [https://hg.mozilla.org/releases/mozilla-release/rev/c081ebf13261 c081ebf13261]
* Build 1: [https://hg.mozilla.org/releases/mozilla-release/rev/c081ebf13261 c081ebf13261]
* Build 2: [http://hg.mozilla.org/releases/mozilla-release/rev/398407cca90c 398407cca90c]


=L10N changesets=
=L10N changesets=
Line 26: Line 27:
| style="background:#efefef" | '''Changeset'''
| style="background:#efefef" | '''Changeset'''
|-  
|-  
| 1
| GECKO110_2012030912_RELBRANCH FIREFOX_11_0_BUILD1 FIREFOX_11_0_RELEASE  
| GECKO110_2012030912_RELBRANCH FIREFOX_11_0_BUILD1 FIREFOX_11_0_RELEASE  
| [http://hg.mozilla.org/releases/mozilla-release/rev/72e3dc061978 72e3dc061978]
| [http://hg.mozilla.org/releases/mozilla-release/rev/72e3dc061978 72e3dc061978]
|-
| 2
| GECKO110_2012031218_RELBRANCH FIREFOX_11_0_BUILD2 FIREFOX_11_0_RELEASE
| [http://hg.mozilla.org/releases/mozilla-release/rev/b967d9c07377 b967d9c07377]
|}
|}


Line 56: Line 62:
| mw32-ix-slave13  
| mw32-ix-slave13  
| 3 hrs, 7 mins, 46 secs
| 3 hrs, 7 mins, 46 secs
|-
| rowspan="4" | 2
| Linux
| rowspan="4" | 20120312181643
| mv-moz2-linux-ix-slave06
| 1 hrs, 16 mins, 53 secs
|-
| Linux64
| linux64-ix-slave05
| 1 hrs, 14 mins, 38 secs
|-
| Mac
| moz2-darwin10-slave50
| 3 hrs, 25 mins, 21 secs
|-
| Windows
| mw32-ix-slave13
| 3 hrs, 25 mins, 26 secs
|}
|}


Line 196: Line 220:


=== Final verification ===
=== Final verification ===
No problems.


=== Push index file to mirrors ===
=== Push index file to mirrors ===
Line 219: Line 244:
for dir in `find . -mindepth 1 -type d `; do cp -pv index.html $dir/; done
for dir in `find . -mindepth 1 -type d `; do cp -pv index.html $dir/; done
</pre>
</pre>
=== Remove from internal mirrors ===
Respin declared for the graphics blocklist crashes not actually being fixed in 11.0b7/11.0 build1 code.
Deleted firefox/releases/11.0 from the mirrors, and filed {{bug|734692}} to speed up clean up of pv-mirror01/02 (the "internal mirrors").
==Build 2==
===Preparing to start Automation===
* Set clobbers for the appropriate masters. Doing this 24-48 hours in advance should speedup the build.
* Reserve slaves
echo "10"> master/reserved_slaves_bm08-build1
* Land automation configs (are you or buildduty going to reconfig? great, merge. otherwise double-land only your configs)
* Tag buildbot-configs, buildbotcustom, & tools with build & release tags (FIREFOX_11_0_RELEASE FIREFOX_11_0_BUILD2)
* update and reconfigure the master
* start automation
# Dry-run
source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u rail \
  -V 11.0 --branch mozilla-release --build-number 2 -c release-firefox-mozilla-release.py \
  --products firefox --l10n-dashboad-version=11 --dryrun localhost:9001
* Failed because beta-only locales was listed on the l10n dashboard:
2012-03-09 11:48:54,898 : ERROR :      locale mn missing in config
2012-03-09 11:48:54,898 : ERROR :      locale sw missing in config
2012-03-09 11:48:54,898 : ERROR : Error verifying l10n dashboard changesets
* reran
# Dry-run
source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u rail \
  -V 11.0 --branch mozilla-release --build-number 2 -c release-firefox-mozilla-release.py \
  --products firefox --l10n-dashboad-version=11 --bypass-l10n-check --dryrun localhost:9001
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u rail \
  -V 11.0 --branch mozilla-release --build-number 2 -c release-firefox-mozilla-release.py \
  --products firefox --l10n-dashboad-version=11 --bypass-l10n-check localhost:9001
=== Edit rsync exclude files ===
Done in build1
=== Tag ===
No problems
=== Bouncer Submitter ===
No problems
=== Source ===
No problems
=== Build ===
No problems
===== L10N repacks =====
No problems
=== Unittests / Talos ===
# Look to see that they ran
# Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
# Make sure there's no red/failures that we need to have a dev look at
=== XULRunner ===
====Source====
No problems
====Build====
No problems
====Signing====
Used [http://hg.mozilla.org/build/braindump/file/cdebd7cd7ba8/releases-related/sign_xulrunner.sh this script]
=== Partner Repack ===
No problems
=== L10N verify ===
No problems
=== Checksums ===
No problems
=== Updates ===
No problems. create_buildN_snippets step went red what is expected.
==== Update verify ====
No problems
=== Reset reserved slaves ===
echo "0" >master/reserved_slaves_bm07-build1
=== Check permissions / AV scan ===
No problems
=== Push to internal mirrors ===
[https://intranet.mozilla.org/ReleaseEngineering/Release/Primer#Mirrors_.28internal_.26_external.29 Mirrors Policy]
Forced
=== Before Release Day ===
Assuming this is not a chemspill.
==== Run backupsnip for release snippets ====
~/bin/backupsnip Firefox-11.0-build2
# 68m3.698s
=== Final verification ===
No problems.
=== Push index file to mirrors ===
N/A


=== Push to external mirrors ===
=== Push to external mirrors ===
Removed previously added exclusion lines.


=== Release Day ===
=== Release Day ===


====Push XULRunner to Mirrors ====
====Push XULRunner to Mirrors ====
<pre>
# ssh -i ~/.ssh/xrbld_dsa xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*-symbols.zip --exclude=*.checksums \
--exclude 'jsshell-*' \
/pub/mozilla.org/xulrunner/nightly/11.0-candidates/build2/ /pub/mozilla.org/xulrunner/releases/11.0/
</pre>


==== Update XULRunner wiki page ====
==== Update XULRunner wiki page ====
Line 230: Line 365:
* https://developer.mozilla.org/en/XULRunner
* https://developer.mozilla.org/en/XULRunner
* https://developer.mozilla.org/en/Gecko_SDK
* https://developer.mozilla.org/en/Gecko_SDK


==== Push to Release Channel ====
==== Push to Release Channel ====
Once there is enough uptake and we get "go" from release driver.
Enabled manual-only-updates throttling by [https://bugzilla.mozilla.org/attachment.cgi?id=603055 this patch]
 
~/bin/pushsnip Firefox-11.0-build2
# 68m46.449s


==== Update symlinks ====
==== Update symlinks ====
Once we're signed off on the release channel.
 
===== Remove index.html files =====
<pre>
If you created them earlier to hide the release.
# ffxbld@stage
cd /pub/mozilla.org/firefox/releases
ln -s 11.0 latest-11.0
rm latest-10.0
rm latest
ln -s latest-11.0 latest
</pre>


==== Update browser choice website ====
==== Update browser choice website ====
Filed {{bug|735348}}
==== Problems with Bouncer/sentry ====
See {{bug|735363}} for the details.
* Forgot to remove "- firefox/11.0/**" line (fallout from build1)
* Forgot to add 11.0 to rsyncd-mozilla-current.exclude
== Detached GPG signing for Mac dmg files ==
* it turned out that we don't upload *dmg.asc files, see {{bug|738349}}
* On keymaster:
mkdir -p ~/signing-work/firefox-11.0/build2
cd ~/signing-work/firefox-11.0
rsync -avP --include='*.dmg' --include='*/' --exclude='*' cltbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/11.0-candidates/build2/mac/ ./build2/mac/
find build2/mac -type f -name '*.dmg' -exec gpg --no-use-agent --batch --passphrase "$(GPGPASS)" -sba '{}' \;
find build2/mac -type f -name '*.dmg.asc' -exec chmod 644 '{}' \;
rsync -e "ssh -i /home/cltsign/.ssh/ffxbld_dsa" -avP --include='*.dmg.asc' --include='*/' --exclude='*' ./build2/mac/ ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/11.0-candidates/build2/mac/
rsync -e "ssh -i /home/cltsign/.ssh/ffxbld_dsa" -avP --include='*.dmg.asc' --include='*/' --exclude='*' ./build2/mac/ ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/releases/11.0/mac/

Latest revision as of 17:39, 30 March 2012

Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

  • bug 530808 - support locales in beta (mn, sw were in the changesets on the dashboard)
  • bug 734644 - hg.m.o time out during Firefox 11.0 build
  • bug 734645 - l10n verification times out unpacking files
  • Slow (4.5h) linux64 build, no ix slaves attached.
  • esr10 was on the same master, aki moved it to bm08

Build Engineers

rail - Tracking bug: bug 718778

Signed-off Revision(s)

L10N changesets

Instructions on how to get them

Tags

Build # Branch, Tags Changeset
1 GECKO110_2012030912_RELBRANCH FIREFOX_11_0_BUILD1 FIREFOX_11_0_RELEASE 72e3dc061978
2 GECKO110_2012031218_RELBRANCH FIREFOX_11_0_BUILD2 FIREFOX_11_0_RELEASE b967d9c07377

Build data

Firefox

Build # Type Build ID Build machine Time to build
1 Linux 20120309131123 mv-moz2-linux-ix-slave09 1 hrs, 4 mins, 46 secs
Linux64 moz2-linux64-slave08 4 hrs, 39 mins, 53 secs
Mac moz2-darwin10-slave45 3 hrs, 13 mins, 56 secs
Windows mw32-ix-slave13 3 hrs, 7 mins, 46 secs
2 Linux 20120312181643 mv-moz2-linux-ix-slave06 1 hrs, 16 mins, 53 secs
Linux64 linux64-ix-slave05 1 hrs, 14 mins, 38 secs
Mac moz2-darwin10-slave50 3 hrs, 25 mins, 21 secs
Windows mw32-ix-slave13 3 hrs, 25 mins, 26 secs

Notes

The release is Firefox only, Fennec is supposed to be released off mozilla-esr10

Build 1

Syncing repos

03-08-2012: DONE mozilla-beta -> mozilla-release & l10n repos sync Note - new locale, lij, didn't have anything in the l10n/mozilla-release/lij repo so I had to push the beta repo to it: 

# in lij.beta
hg push -f -e "ssh -l ffxbld -i ~/.ssh/ffxbld_dsa" ssh://hg.mozilla.org/releases/l10n/mozilla-release/lij

Then continued the script with a subset of locales

  • Sanity check:
diff -Naur --exclude='.hg*' mozilla-beta mozilla-release > beta-release.diff
  • Inspected beta-release.diff. The differences were expected (shipped-locales without mn, sw and android mozconfig changes landed after migration)

Preparing to start Automation

  • Set clobbers for the appropriate masters. Doing this 24-48 hours in advance should speedup the build.
  • For build1 only, create a symlink for your $version (eg 11.0b4, 10.0.2esr, 10.0.2, 3.6.27) (until bug 725839 is resolved)
# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/
mkdir ../candidates/11.0-candidates
ln -s ../candidates/11.0-candidates 11.0-candidates
  • Reserve slaves
echo "10"> master/reserved_slaves_bm08-build1
  • Update l10n changesets for desktop
  • Land automation configs (are you or buildduty going to reconfig? great, merge. otherwise double-land only your configs)
  • Tag buildbot-configs, buildbotcustom, & tools with build & release tags (FIREFOX_11_0_RELEASE FIREFOX_11_0_BUILD1)
  • update and reconfigure the master
  • start automation
# Dry-run
source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u rail \
  -V 11.0 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py \
  --products firefox --l10n-dashboad-version=11 --dryrun localhost:9001
  • Failed because beta-only locales was listed on the l10n dashboard:
2012-03-09 11:48:54,898 : ERROR :       locale mn missing in config
2012-03-09 11:48:54,898 : ERROR :       locale sw missing in config
2012-03-09 11:48:54,898 : ERROR : Error verifying l10n dashboard changesets
  • reran
# Dry-run
source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u rail \
  -V 11.0 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py \
  --products firefox --l10n-dashboad-version=11 --bypass-l10n-check --dryrun localhost:9001

PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u rail \
  -V 11.0 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py \
  --products firefox --l10n-dashboad-version=11 --bypass-l10n-check localhost:9001

E-mail Metrics

Use the address "metrics-alerts < AT > mozilla < PERIOD > org"

And for 11.0 final it's 

Firefox,firefox,firefox,11.0,11.0

E-mail mirrors@mozilla.org

Not necessary for rapid release betas, files only go as far as the 'internal mirrors'.

Coordinate yourself with the person creating the ESR and the latest release which normally go out at the same time.

They want to know approximately what time push to external mirrors will occur. Ideally this is a 24hr notice to them, with the estimated time. 

Subject: Firefox 11.0 [and {VERSION2}] coming to mirrors on {DATE}
Body:
We're planning on pushing our Firefox 11.0 [and {VERSION2}] release(s) to mirrors 24 hours before {DATE}.
# If this is a chemspill, please mention that in the email as it will mean pinging 
# for CDN to be enabled when the push to internal mirrors happens

If you believe these releases or these dates will cause any problems,
please notify release at mozilla.com. If you have any problems with mirror
status/weights/etc, please notify mirror-submissions at mozilla.org.

Edit rsync exclude files

Edit /pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude on stage so that it excludes the current release. This prevents external mirrors from picking up the new release prematurely.

For rapid release betas, this is not necessary as there's already an exclude that matches beta releases.

Tag

  • Failed due to slow hg.m.o. Restarted automation from the beginning.

Bouncer Submitter

No problems

Source

No problems

Build

No problems

L10N Repacks

No problems.

Unittests / Talos

  1. Look to see that they ran
  2. Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
  3. Make sure there's no red/failures that we need to have a dev look at

XULRunner

Source

No problems

Build

No problems

Signing

Used this script

Partner Repack

No problems

L10N verify

win32 timed out. rebuilt

Checksums

No problems

Updates

No problems

Update verify

No problems

Reset reserved slaves

echo "0" >master/reserved_slaves_bm07-build1

Check permissions / AV scan

No problems

Push to internal mirrors

Forced the builder. No problems.

Mirrors Policy

Before Release Day

Run backupsnip for release snippets

Do this the night before release day as it can take up to two hours.

Final verification

No problems.

Push index file to mirrors

Added the following lines to the top of stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude: 

+ firefox
+ firefox/releases
+ firefox/releases/11.0
+ firefox/releases/11.0/*/
+ firefox/releases/11.0/*/*/
+ firefox/releases/11.0/*/*/*/
+ firefox/releases/11.0/**index.html
- firefox/releases/11.0/**

Removed the previously added "- firefox/releases/11.0" line. After that, ran the following to lay out the index files: 

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases/11.0
wget --no-check-certificate -O index.html https://bugzilla.mozilla.org/attachment.cgi?id=540139
sed -i -e 's/5/11/g' index.html 
for dir in `find . -mindepth 1 -type d `; do cp -pv index.html $dir/; done

Remove from internal mirrors

Respin declared for the graphics blocklist crashes not actually being fixed in 11.0b7/11.0 build1 code.

Deleted firefox/releases/11.0 from the mirrors, and filed bug 734692 to speed up clean up of pv-mirror01/02 (the "internal mirrors").

Build 2

Preparing to start Automation

  • Set clobbers for the appropriate masters. Doing this 24-48 hours in advance should speedup the build.
  • Reserve slaves
echo "10"> master/reserved_slaves_bm08-build1
  • Land automation configs (are you or buildduty going to reconfig? great, merge. otherwise double-land only your configs)
  • Tag buildbot-configs, buildbotcustom, & tools with build & release tags (FIREFOX_11_0_RELEASE FIREFOX_11_0_BUILD2)
  • update and reconfigure the master
  • start automation
# Dry-run
source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u rail \
  -V 11.0 --branch mozilla-release --build-number 2 -c release-firefox-mozilla-release.py \
  --products firefox --l10n-dashboad-version=11 --dryrun localhost:9001
  • Failed because beta-only locales was listed on the l10n dashboard:
2012-03-09 11:48:54,898 : ERROR :       locale mn missing in config
2012-03-09 11:48:54,898 : ERROR :       locale sw missing in config
2012-03-09 11:48:54,898 : ERROR : Error verifying l10n dashboard changesets
  • reran
# Dry-run
source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u rail \
  -V 11.0 --branch mozilla-release --build-number 2 -c release-firefox-mozilla-release.py \
  --products firefox --l10n-dashboad-version=11 --bypass-l10n-check --dryrun localhost:9001

PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u rail \
  -V 11.0 --branch mozilla-release --build-number 2 -c release-firefox-mozilla-release.py \
  --products firefox --l10n-dashboad-version=11 --bypass-l10n-check localhost:9001

Edit rsync exclude files

Done in build1

Tag

No problems

Bouncer Submitter

No problems

Source

No problems

Build

No problems

L10N repacks

No problems

Unittests / Talos

  1. Look to see that they ran
  2. Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
  3. Make sure there's no red/failures that we need to have a dev look at

XULRunner

Source

No problems

Build

No problems

Signing

Used this script

Partner Repack

No problems

L10N verify

No problems

Checksums

No problems

Updates

No problems. create_buildN_snippets step went red what is expected.

Update verify

No problems

Reset reserved slaves

echo "0" >master/reserved_slaves_bm07-build1

Check permissions / AV scan

No problems

Push to internal mirrors

Mirrors Policy

Forced

Before Release Day

Assuming this is not a chemspill.

Run backupsnip for release snippets

~/bin/backupsnip Firefox-11.0-build2
# 68m3.698s

Final verification

No problems.

Push index file to mirrors

N/A

Push to external mirrors

Removed previously added exclusion lines.

Release Day

Push XULRunner to Mirrors

# ssh -i ~/.ssh/xrbld_dsa xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*-symbols.zip --exclude=*.checksums \
 --exclude 'jsshell-*' \
 /pub/mozilla.org/xulrunner/nightly/11.0-candidates/build2/ /pub/mozilla.org/xulrunner/releases/11.0/

Update XULRunner wiki page

For major releases or chemspills, update the links on:


Push to Release Channel

Enabled manual-only-updates throttling by this patch 

~/bin/pushsnip Firefox-11.0-build2
# 68m46.449s

Update symlinks

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases
ln -s 11.0 latest-11.0
rm latest-10.0
rm latest
ln -s latest-11.0 latest

Update browser choice website

Filed bug 735348


Problems with Bouncer/sentry

See bug 735363 for the details.

  • Forgot to remove "- firefox/11.0/**" line (fallout from build1)
  • Forgot to add 11.0 to rsyncd-mozilla-current.exclude


Detached GPG signing for Mac dmg files

  • it turned out that we don't upload *dmg.asc files, see bug 738349
  • On keymaster:
mkdir -p ~/signing-work/firefox-11.0/build2
cd ~/signing-work/firefox-11.0
rsync -avP --include='*.dmg' --include='*/' --exclude='*' cltbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/11.0-candidates/build2/mac/ ./build2/mac/
find build2/mac -type f -name '*.dmg' -exec gpg --no-use-agent --batch --passphrase "$(GPGPASS)" -sba '{}' \;
find build2/mac -type f -name '*.dmg.asc' -exec chmod 644 '{}' \;
rsync -e "ssh -i /home/cltsign/.ssh/ffxbld_dsa" -avP --include='*.dmg.asc' --include='*/' --exclude='*' ./build2/mac/ ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/11.0-candidates/build2/mac/
rsync -e "ssh -i /home/cltsign/.ssh/ffxbld_dsa" -avP --include='*.dmg.asc' --include='*/' --exclude='*' ./build2/mac/ ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/releases/11.0/mac/
Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_11.0/BuildNotes&oldid=414387"