MozillaWiki

Releases/Firefox 5.0b5/BuildNotes: Difference between revisions

< Releases
(Created page with "=Notes About Releasing= Please update the [https://intranet.mozilla.org/Build:Release:Primer Release:Primer] for future releases (bug fixes, changes to automation) as needed =Bu...")
 
m (remove incorrect mentions of beta 4)
 
(28 intermediate revisions by 3 users not shown)
Line 1: Line 1:
=Notes About Releasing=
=Notes About Releasing=
Please update the [https://intranet.mozilla.org/Build:Release:Primer Release:Primer] for future releases (bug fixes, changes to automation) as needed
Please update the [https://intranet.mozilla.org/Build:Release:Primer Release:Primer] for future releases (bug fixes, changes to automation) as needed
= Bugs hit =
The following are the list of bugs that we expect to hit or that we did not expect to hit. For the next release some of this will be already fixed.
* We forgot that beta4 was not shipped and update verify failed
These two required the manual work for major updates:
* {{bug|662803}} - Major update automation should support multiple "from" versions
* {{bug|662298}} - major update automation should be able to be run independently, or as part of the "to" release
Also check the "[[Release:Release_Automation_on_Mercurial/Known_Issues|known issues]]" page.
== For postmortem ==
The whole release was rushed up (go to build later on EDT) and expectations of releasing on next afternoon was not explicit. If it was known I could have started things earlier.


=Build Engineers=
=Build Engineers=
Line 6: Line 20:


=Signed-off Revision(s)=
=Signed-off Revision(s)=
Build 1: []
Build 1: [http://hg.mozilla.org/releases/mozilla-beta/rev/3524ac4292a7 3524ac4292a7]


=L10N changesets=
=L10N changesets=
Line 18: Line 32:
|-
|-
| rowspan="2" | 1  
| rowspan="2" | 1  
|
| GECKO50_2011060814_RELBRANCH
|
| [http://hg.mozilla.org/releases/mozilla-beta/rev/cf9c20531b17 cf9c20531b17]
|-  
|-  
|
| FIREFOX_5_0b5_RELEASE FIREFOX_5_0b5_BUILD1
|
| [http://hg.mozilla.org/releases/mozilla-beta/rev/cf9c20531b17 cf9c20531b17]
|}
|}


Line 35: Line 49:
| rowspan="4" | 1  
| rowspan="4" | 1  
| Linux
| Linux
| rowspan="4" |
| rowspan="4" | 20110608151458
|  
| mv-moz2-linux-ix-slave13
|  
| 59 mins, 35 secs
|-
|-
| Linux64
| Linux64
|  
| moz2-linux64-slave02
|  
| 1 hrs, 7 mins, 45 secs
|-
|-
| Mac
| Mac
|  
| moz2-darwin10-slave48
|  
| 4 hrs, 9 mins, 3 secs
|-
|-
| Windows  
| Windows  
|  
| mw32-ix-slave17
|  
| 2 hrs, 42 mins, 39 secs
|}
|}


Line 62: Line 76:
[cltbld@production-master03 builder_master]$ echo 6 > reserved_slaves_pm03  
[cltbld@production-master03 builder_master]$ echo 6 > reserved_slaves_pm03  
</pre>
</pre>
* <i>Hit "ship it" button on "Firefox 5 Beta Build 4"</i> (https://l10n-stage-sj.mozilla.org/shipping/milestones)
* Hit "ship it" button on "Firefox 5 Beta Build 5"</i> (https://l10n-stage-sj.mozilla.org/shipping/milestones)
* Landed configs, merged configs & custom to production.
* Tagged buildbot-configs, buildbotcustom, and tools with FIREFOX_5_0b5_RELEASE and FIREFOX_5_0b5_BUILD1
* Update pm03 checkouts & reconfigured
* Ran release_sanity dry run with:
PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u armenzg -V 5.0b5 --branch mozilla-beta --build-number 1 -c release-firefox-mozilla-beta.py --dryrun localhost:9010
* And then to start the release:
PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u armenzg -V 5.0b5 --branch mozilla-beta --build-number 1 -c release-firefox-mozilla-beta.py localhost:9010


=== Tag ===
=== Tag ===
Worked fine.


=== Start Autosign ===
=== Start Autosign ===
Once mozilla-beta was tagged, on keymaster, started signcodepwd.exe, opened a cygwin terminal and run the following:
<pre>
PRODUCT=firefox
VERSION=5.0b5
BUILD=1
TAG=FIREFOX_5_0b5_RELEASE
REPO='releases/mozilla-beta'
EMAIL=release@mozilla.com
KEYDIR=d:/2010-keys
RELCONFIG=mozilla/release-firefox-mozilla-beta.py
cd ${HOME}/hg-tools/
hg pull && hg up -r ${TAG}
mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG}
</pre>


=== Bouncer Submitter ===
=== Bouncer Submitter ===
Worked fine.


=== Source ===
=== Source ===
Worked fine.


=== Build/Repack ===
=== Build/Repack ===
* Linux: No problems
* Linux64: No problems
* Windows: No problems
* Mac: No problems


=== Reset reserved_slaves ===
=== Reset reserved_slaves ===
[cltbld@production-master03 builder_master]$ pwd
/builds/buildbot/builder_master
[cltbld@production-master03 builder_master]$ echo 0 > reserved_slaves_pm03


=== Signing ===
=== Signing ===
No problems. Closed signcodepwd.exe after signing finished. Got "signature verification succeeded" mail a few hours later.


=== L10n Verify ===
=== L10n Verify ===
A few legitimate looking string changes.


=== Updates (5.0b[1-4] -> 5.0b5) ===
=== Updates (5.0b[1-4] -> 5.0b5) ===
No problems.
==== Update verify ====
Beta4 had not been shipped and therefore update verify failed.
Process stderr:
http://stage-old.mozilla.org/pub/mozilla.org//firefox/releases/5.0b4/linux-i686/af/firefox-5.0b4.tar.bz2:
22:10:02 ERROR 404: Not Found.


==== Update verify ====
* We landed a switch from [https://bugzilla.mozilla.org/attachment.cgi?id=538260 releases/5.0b4 to nightly/5.0b4-candidates/build1].
* Tagged with FIREFOX_5_0b5_{RELEASE,BUILD1}_RUNTIME.
* Re-build each one of the failing jobs
 
It all looks green now.


=== Updates (4.0many -> 5.0b5) ===
=== Updates (4.0many -> 5.0b5) ===
Manually bumped [https://bugzilla.mozilla.org/attachment.cgi?id=538274 patcher config] and [https://bugzilla.mozilla.org/attachment.cgi?id=537843 update verify config].
Bumped FIREFOX_4_0_1_RELEASE tag on buildbot-configs, FIREFOX_4_0_1_{BUILD1,RELEASE}_RUNTIME on tools.
Generation of updates was done on mv-moz2-linux-ix-slave13, using the UPDATE_PACKAGING_R14 tag, and re-using the updates directory from the 5.0b[12] -> 5.0b5 updates run, to save time & energy.
Disabled the slave from slavealloc and gracefully shut it down.
<pre>
cd /builds/slave/rel-beta-updates/build/patcher-configs
cvs up
cd ..
perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/moz20-branch-major-update-patcher2.cfg
cd temp/firefox
ln -s 5.0b5 4.0.1-5.0b5
cd ../..
perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/moz20-branch-major-update-patcher2.cfg
# remove partials for 4.0.1
cd temp/firefox/4.0.1-5.0b5
find aus2{,.test} -name partial.txt -exec rm {} \;
# check releasetest == beta
bash /builds/slave/rel-beta-updates//tools/release/compare-channel-snippets.sh aus2 beta aus2.test releasetest


# push up to aus
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-5.0b5-build1-4.0.1-build1-test
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-5.0b5-build1-4.0.1-build1
# push test snippets live
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/backupsnip Firefox-5.0b5-build1-4.0.1-build1-test'
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/pushsnip Firefox-5.0b5-build1-4.0.1-build1-test'
</pre>
==== Update Verify ====
==== Update Verify ====
To force all of the verify builders, used the following script (warning: not OS X compatible):
<pre>
for p in linux linux64 macosx64 win32; do
  for i in `seq 1 10`; do
    curl "http://production-master1.build.sjc1.mozilla.com:8010/builders/release-mozilla-2.0-${p}_major_update_verify_${i}%2F10/force" >/dev/null
  done
done
</pre>
Differences in all platforms:
<pre>
diff -r source/firefox/defaults/pref/channel-prefs.js target/firefox/defaults/pref/channel-prefs.js
1,2c1,2
< //@line 2 "/builds/slave/rel-2.0-lnx-bld/build/browser/app/profile/channel-prefs.js"
< pref("app.update.channel", "release");
---
> //@line 2 "/builds/slave/rel-beta-lnx-bld/build/browser/app/profile/channel-prefs.js"
> pref("app.update.channel", "beta");
WARN: non-binary files found in diff
</pre>
Reviewing this GREEN logs are so verbose. How could this be improved?


=== Channel switching snippets ===
=== Channel switching snippets ===
We'll use the 4.0.1 snippets for this, which has all of the locales we care about, plus a (harmless) few extra. Like 5.0b2, we put them in a bogus buildID of 30000000000000 to keep the channel switching snippets separate from more specific ones.
NOTE: For this to work you have to trigger the MU first.
<pre>
# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
mkdir -p Firefox-5.0b5-build1-channel-switching/Firefox/
rsync -a Firefox-5.0b5-build1-4.0.1-build1/Firefox/4.0.1 Firefox-5.0b5-build1-channel-switching/Firefox/
cd Firefox-5.0b5-build1-channel-switching/Firefox/
mv 4.0.1 5.0
for d in 5.0/*; do
  mv -v $d/* $d/30000000000000
done
</pre>


=== Permissions check and A/V Scan ===
=== Permissions check and A/V Scan ===
No problems.


=== A/V Notification ===
=== A/V Notification ===
No problems.


=== Push Files ===
=== Push Files ===
As in 5.0b2, we'll be pushing to a few internal mirrors rather than the whole network. To set this up, added the following to stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude:
- firefox/releases/5.0b5
...and waited a few minutes, to ensure that the rsync servers have the file.
Then, forced the "release-mozilla-beta-push_to_mirrors" builder with the following properties:
release_config: mozilla/release-firefox-mozilla-beta.py
script_repo_revision: FIREFOX_5_0b5_RELEASE


=== Final verify ===
=== Final verify ===
Triggered "release-mozilla-beta-$PLATFORM_final_verification" without any problems.
Using internal mirrors was so fast!!


=== Push snippets ===
=== Push snippets ===
"Firefox-5.0b5-build1" and "Firefox-5.0b5-build1-channel-switching" both only have a "5.0" version in them, so we only need one backupsnip between them. "Firefox-5.0b5-build1-4.0.1-build1" has 4.0 versions, and needs its own. As cltbld@aus2-staging, did the following for backups:
<pre>
cd /opt/aus2/snippets/staging
~/bin/backupsnip Firefox-5.0b5-build1
~/bin/backupsnip Firefox-5.0b5-build1-4.0.1-build1
</pre>
And for the pushes:
<pre>
~/bin/pushsnip Firefox-5.0b5-build1
~/bin/pushsnip Firefox-5.0b5-build1-channel-switching
~/bin/pushsnip Firefox-5.0b5-build1-4.0.1-build1
</pre>

Latest revision as of 14:51, 14 June 2011

Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

The following are the list of bugs that we expect to hit or that we did not expect to hit. For the next release some of this will be already fixed.

  • We forgot that beta4 was not shipped and update verify failed

These two required the manual work for major updates:

  • bug 662803 - Major update automation should support multiple "from" versions
  • bug 662298 - major update automation should be able to be run independently, or as part of the "to" release

Also check the "known issues" page.

For postmortem

The whole release was rushed up (go to build later on EDT) and expectations of releasing on next afternoon was not explicit. If it was known I could have started things earlier.

Build Engineers

armenzg - bug 662509

Signed-off Revision(s)

Build 1: 3524ac4292a7

L10N changesets

From Axel's webapp.

Tags

Build # Tag Changeset
1 GECKO50_2011060814_RELBRANCH cf9c20531b17
FIREFOX_5_0b5_RELEASE FIREFOX_5_0b5_BUILD1 cf9c20531b17

Build data

Build # Type Build ID Build machine Time to build
1 Linux 20110608151458 mv-moz2-linux-ix-slave13 59 mins, 35 secs
Linux64 moz2-linux64-slave02 1 hrs, 7 mins, 45 secs
Mac moz2-darwin10-slave48 4 hrs, 9 mins, 3 secs
Windows mw32-ix-slave17 2 hrs, 42 mins, 39 secs

Notes

Build 1

Setup

  • Marked "Any master", "mozilla-beta", "Any builder" for a clobber
  • About an hour ahead of the expected "go", set reserved_slaves to 6:
[cltbld@production-master03 builder_master]$ pwd
/builds/buildbot/builder_master
[cltbld@production-master03 builder_master]$ echo 6 > reserved_slaves_pm03
  • Hit "ship it" button on "Firefox 5 Beta Build 5" (https://l10n-stage-sj.mozilla.org/shipping/milestones)
  • Landed configs, merged configs & custom to production.
  • Tagged buildbot-configs, buildbotcustom, and tools with FIREFOX_5_0b5_RELEASE and FIREFOX_5_0b5_BUILD1
  • Update pm03 checkouts & reconfigured
  • Ran release_sanity dry run with:
PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u armenzg -V 5.0b5 --branch mozilla-beta --build-number 1 -c release-firefox-mozilla-beta.py --dryrun localhost:9010
  • And then to start the release:
PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u armenzg -V 5.0b5 --branch mozilla-beta --build-number 1 -c release-firefox-mozilla-beta.py localhost:9010

Tag

Worked fine.

Start Autosign

Once mozilla-beta was tagged, on keymaster, started signcodepwd.exe, opened a cygwin terminal and run the following: 

PRODUCT=firefox
VERSION=5.0b5
BUILD=1
TAG=FIREFOX_5_0b5_RELEASE
REPO='releases/mozilla-beta'
EMAIL=release@mozilla.com
KEYDIR=d:/2010-keys
RELCONFIG=mozilla/release-firefox-mozilla-beta.py

cd ${HOME}/hg-tools/
hg pull && hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG}

Bouncer Submitter

Worked fine.

Source

Worked fine.

Build/Repack

  • Linux: No problems
  • Linux64: No problems
  • Windows: No problems
  • Mac: No problems

Reset reserved_slaves

[cltbld@production-master03 builder_master]$ pwd
/builds/buildbot/builder_master
[cltbld@production-master03 builder_master]$ echo 0 > reserved_slaves_pm03

Signing

No problems. Closed signcodepwd.exe after signing finished. Got "signature verification succeeded" mail a few hours later.

L10n Verify

A few legitimate looking string changes.

Updates (5.0b[1-4] -> 5.0b5)

No problems.

Update verify

Beta4 had not been shipped and therefore update verify failed. 

Process stderr:
http://stage-old.mozilla.org/pub/mozilla.org//firefox/releases/5.0b4/linux-i686/af/firefox-5.0b4.tar.bz2:
22:10:02 ERROR 404: Not Found.

It all looks green now.

Updates (4.0many -> 5.0b5)

Manually bumped patcher config and update verify config.

Bumped FIREFOX_4_0_1_RELEASE tag on buildbot-configs, FIREFOX_4_0_1_{BUILD1,RELEASE}_RUNTIME on tools.

Generation of updates was done on mv-moz2-linux-ix-slave13, using the UPDATE_PACKAGING_R14 tag, and re-using the updates directory from the 5.0b[12] -> 5.0b5 updates run, to save time & energy.

Disabled the slave from slavealloc and gracefully shut it down. 

cd /builds/slave/rel-beta-updates/build/patcher-configs
cvs up
cd ..
perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/moz20-branch-major-update-patcher2.cfg

cd temp/firefox
ln -s 5.0b5 4.0.1-5.0b5
cd ../..

perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/moz20-branch-major-update-patcher2.cfg

# remove partials for 4.0.1
cd temp/firefox/4.0.1-5.0b5
find aus2{,.test} -name partial.txt -exec rm {} \;

# check releasetest == beta
bash /builds/slave/rel-beta-updates//tools/release/compare-channel-snippets.sh aus2 beta aus2.test releasetest

# push up to aus
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-5.0b5-build1-4.0.1-build1-test
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-5.0b5-build1-4.0.1-build1

# push test snippets live
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/backupsnip Firefox-5.0b5-build1-4.0.1-build1-test'
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/pushsnip Firefox-5.0b5-build1-4.0.1-build1-test'

Update Verify

To force all of the verify builders, used the following script (warning: not OS X compatible): 

for p in linux linux64 macosx64 win32; do
  for i in `seq 1 10`; do
    curl "http://production-master1.build.sjc1.mozilla.com:8010/builders/release-mozilla-2.0-${p}_major_update_verify_${i}%2F10/force" >/dev/null
  done
done

Differences in all platforms: 

 diff -r source/firefox/defaults/pref/channel-prefs.js target/firefox/defaults/pref/channel-prefs.js
1,2c1,2
 < //@line 2 "/builds/slave/rel-2.0-lnx-bld/build/browser/app/profile/channel-prefs.js"
 < pref("app.update.channel", "release");
 ---
 > //@line 2 "/builds/slave/rel-beta-lnx-bld/build/browser/app/profile/channel-prefs.js"
 > pref("app.update.channel", "beta");
 WARN: non-binary files found in diff

Reviewing this GREEN logs are so verbose. How could this be improved?

Channel switching snippets

We'll use the 4.0.1 snippets for this, which has all of the locales we care about, plus a (harmless) few extra. Like 5.0b2, we put them in a bogus buildID of 30000000000000 to keep the channel switching snippets separate from more specific ones.

NOTE: For this to work you have to trigger the MU first. 

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
mkdir -p Firefox-5.0b5-build1-channel-switching/Firefox/
rsync -a Firefox-5.0b5-build1-4.0.1-build1/Firefox/4.0.1 Firefox-5.0b5-build1-channel-switching/Firefox/
cd Firefox-5.0b5-build1-channel-switching/Firefox/
mv 4.0.1 5.0
for d in 5.0/*; do 
  mv -v $d/* $d/30000000000000
done

Permissions check and A/V Scan

No problems.

A/V Notification

No problems.

Push Files

As in 5.0b2, we'll be pushing to a few internal mirrors rather than the whole network. To set this up, added the following to stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude: 

- firefox/releases/5.0b5

...and waited a few minutes, to ensure that the rsync servers have the file.

Then, forced the "release-mozilla-beta-push_to_mirrors" builder with the following properties: 

release_config: mozilla/release-firefox-mozilla-beta.py
script_repo_revision: FIREFOX_5_0b5_RELEASE

Final verify

Triggered "release-mozilla-beta-$PLATFORM_final_verification" without any problems.

Using internal mirrors was so fast!!

Push snippets

"Firefox-5.0b5-build1" and "Firefox-5.0b5-build1-channel-switching" both only have a "5.0" version in them, so we only need one backupsnip between them. "Firefox-5.0b5-build1-4.0.1-build1" has 4.0 versions, and needs its own. As cltbld@aus2-staging, did the following for backups: 

cd /opt/aus2/snippets/staging
~/bin/backupsnip Firefox-5.0b5-build1
~/bin/backupsnip Firefox-5.0b5-build1-4.0.1-build1

And for the pushes: 

~/bin/pushsnip Firefox-5.0b5-build1
~/bin/pushsnip Firefox-5.0b5-build1-channel-switching
~/bin/pushsnip Firefox-5.0b5-build1-4.0.1-build1
Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_5.0b5/BuildNotes&oldid=318722"