Notes About Releasing

Bugs hit

Enter any bugs pre-existing or newly discovered and filed during the release:

no L10n milestone. I took no L10n changes
change l10n-stage-sj.mozilla.org for l10n.mozilla.org in documentation
I had to lock few extra darwin10 slaves since there were 4 attached but only 2 ssh-able to start the mac builds (xulrunner was picked up first)
One of the mac repacks failed around Mon May 28 23:26:41 2012

remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.

One of the mac repacks timed out on signing
- http://buildbot-master13.build.mozilla.org:8001/builders/release-mozilla-beta-macosx64_repack_4%2F6/builds/3/steps/run_script/logs/stdio
- command timed out: 2400 seconds without output, attempting to kill
I started the release before fixing the symlinks
- Once the automation stops we can fix it by doing this:

# do not run until all deliverables and logs are uploaded
cd /pub/mozilla.org/firefox/nightly
rsync -av 13.0b6-candidates/ ../candidates/13.0b6-candidates/
mv 13.0b6-candidates 13.0b6-candidates.old && ln -s ../candidates/13.0b6-candidates 13.0b6-candidates
rm -rf 13.0b6-candidates.old

check permissions failed. I assume I can fix this one I fix the candidates symlinks later on

Error while running: find /pub/mozilla.org/firefox/nightly/13.0b6-candidates/build1/ -mindepth 1 -type d ! -perm 755 ! -path '*/contrib*' ! -path '*/partner-repacks*'
command: START
command: ssh -l ffxbld -i /home/cltbld/.ssh/ffxbld_dsa stage.mozilla.org 'test "0" = "$(find /pub/mozilla.org/firefox/nightly/13.0b6-candidates/build1/ -maxdepth 1 -type d ! -perm 2775 -path \'*/contrib*\' | wc -l)"'
command: cwd: /builds/slave/rel-m-beta-chk-prms
command: output:
Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-chk-prms/scripts/scripts/release/push-to-mirrors.py", line 176, in <module>
    buildNumber=buildNumber)
  File "/builds/slave/rel-m-beta-chk-prms/scripts/scripts/release/push-to-mirrors.py", line 84, in checkStagePermissions
    username=stageUsername, sshKey=stageSshKey)
  File "/builds/slave/rel-m-beta-chk-prms/scripts/scripts/release/../../lib/python/util/commands.py", line 57, in run_remote_cmd
    run_cmd(cmd_prefix + cmd, **kwargs)
  File "/builds/slave/rel-m-beta-chk-prms/scripts/scripts/release/../../lib/python/util/commands.py", line 42, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python-2.5.1/lib/python2.5/subprocess.py", line 461, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ssh', '-l', 'ffxbld', '-i', '/home/cltbld/.ssh/ffxbld_dsa', 'stage.mozilla.org', 'test "0" = "$(find /pub/mozilla.org/firefox/nightly/13.0b6-candidates/build1/ -mindepth 1 -type d ! -perm 755 ! -path \'*/contrib*\' ! -path \'*/partner-repacks*\' | wc -l)"']' returned non-zero exit status 1
command: END (0.18s elapsed)

Build Engineers

armenzg - Tracking bug: bug 744595

Signed-off Revision(s)

Build 1: 046e86aee0c1

L10N changesets

Instructions on how to get them

Firefox: no milestone. I took no L10n changes

Build data

Firefox

Build #	Type	Build ID	Build machine	Time to build
1	Linux		linux-ix-slave28	1 hrs, 6 mins, 14 secs
	Linux64		linux64-ix-slave21	1 hrs, 5 mins, 46 secs
	Mac		moz2-darwin10-slave42	3 hrs, 18 mins, 57 secs
	Windows		w32-ix-slave37	3 hrs, 45 mins, 30 secs

Notes

Build 1

Preparing to start Automation

Set clobbers for the appropriate masters.
- clobberer
- I didn't do it in advance
Reserve slaves (you need ~5 each of linux, linux64, win32 (w32), and macosx64 (moz2-darwin10)). Doing this 2-3 hours in advance should speed things up. e.g.:

[cltbld@buildbot-master13 master]$ echo 8 > /builds/buildbot/build1/master/reserved_slaves

For Firefox build1 only, create a symlink for your $version (eg 11.0b4, 10.0.2esr, 10.0.2, 3.6.27) (until bug 725839 is resolved)

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/
mkdir ../candidates/13.0b6-candidates
ln -s ../candidates/13.0b6-candidates 13.0b6-candidates

Update l10n changesets for desktop
- Instructions on how to get them
- dashboard
Land automation configs
Tag buildbot-configs, buildbotcustom, & tools with build & release tags. (details)
- Sample command for Firefox 10.0 beta 5 build 1:

hg tag -f FIREFOX_13_0b6_RELEASE FIREFOX_13_0b6_BUILD1

update and reconfigure the master

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u armenzg -V 13.0b6 --branch mozilla-beta --build-number 1 \
    --release-config release-firefox-mozilla-beta.py --products firefox  \
    --dryrun localhost:9001

start automation ← monitor progress on buildbot (e.g. beta)
- the same command as above but without the "--dryrun"
- I had to run with -b to skip checking for the L10n dashboard

E-mail Metrics

Use the address "metrics-alerts < AT > mozilla < PERIOD > org". Note for first-time-releasers: your email will get held for moderator approval - that is expected. If it happens more than once, get help on #metrics.

Firefox,firefox,firefox,13.0b6,13.0

Tag

No problems.

Bouncer Submitter

No problems.

Source

No problems.

Build

Firefox

No problems.

Firefox repacks

We had a couple of issues with the mac repacks. I re-triggered them.

Unittests / Talos

details

Look to see that they ran
Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
Make sure there's no red/failures that we need to have a dev look at

XULRunner

No problems.

Source

No problems.

Build

No problems.

Checksums

No problems.

Updates

updates generation worked

Update verify

No problems.

Reset reserved slaves

[cltbld@buildbot-master13 ~]$ echo 0 > /builds/buildbot/build1/master/reserved_slaves

Check permissions / AV scan

It is supposed to be triggered automatically after updates are done.

check permissions failed. I assume I can fix this one I fix the candidates symlinks later on

# do not run until all deliverables and logs are uplodaed
cd /pub/mozilla.org/firefox/nightly
rsync -av 13.0b6-candidates/ ../candidates/13.0b6-candidates/
mv 13.0b6-candidates 13.0b6-candidates.old && ln -s ../candidates/13.0b6-candidates 13.0b6-candidates
rm -rf 13.0b6-candidates.old

After retriggering it failed again

Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-chk-prms/scripts/scripts/release/push-to-mirrors.py", line 176, in <module>
    buildNumber=buildNumber)
  File "/builds/slave/rel-m-beta-chk-prms/scripts/scripts/release/push-to-mirrors.py", line 84, in checkStagePermissions
    username=stageUsername, sshKey=stageSshKey)
  File "/builds/slave/rel-m-beta-chk-prms/scripts/scripts/release/../../lib/python/util/commands.py", line 57, in run_remote_cmd
    run_cmd(cmd_prefix + cmd, **kwargs)
  File "/builds/slave/rel-m-beta-chk-prms/scripts/scripts/release/../../lib/python/util/commands.py", line 42, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python-2.5.1/lib/python2.5/subprocess.py", line 461, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ssh', '-l', 'ffxbld', '-i', '/home/cltbld/.ssh/ffxbld_dsa', 'stage.mozilla.org', 'test "0" = "$(find /pub/mozilla.org/firefox/nightly/13.0b6-candidates/build1/ -mindepth 1 -type d ! -perm 755 ! -path \'*/contrib*\' ! -path \'*/partner-repacks*\' | wc -l)"']' returned non-zero exit status 1

nthomas figured this out:

firefox/nightly is 2775, where 2 is the sticky group bit; firefox/candidates is 755

fix it up

cd /pub/mozilla.org/firefox/candidates/
find 13.0b6-candidates -type d ! -name 'contrib*' -exec chmod g-s {} \;

then the builder passes the tests but is orange due to this expected state

WARN: target directory /pub/mozilla.org/firefox/releases/13.0b6/ exists

redo the rsync to mirrors to fix perms

rsync -av --exclude=*tests* --exclude=*crashreporter* --exclude=*.log --exclude=*.txt --exclude=*unsigned* --exclude=*update-backup* --exclude=*partner-repacks* --exclude=*.checksums --exclude=*.checksums.asc --exclude=logs --exclude=jsshell* --exclude=*.zip --exclude=*.zip.asc /pub/mozilla.org/firefox/nightly/13.0b6-candidates/build1/ /pub/mozilla.org/firefox/releases/13.0b6/

Push to internal mirrors

This is done by automation for rapid release betas. Note: if you have to reconfig the release buildbot master in during the release, the uptake monitoring will fail (bug 629648). That means you'll need to manually send the "ready for releasetest" emails when you believe things are "good enough". Refer to a prior release for email details.

I had to do a reconfig during the day. I sent an email manually
This can be checked by looking the "release-mozilla-beta-ready_for_releasetest_testing" builder

Final Verification

Record receipt of "completed_final_verification" emails (1 per platform):

   Subject: ***SPAM*** [release] Firefox 13.0b6 build1: completed final_verification on macosx64
   Subject: ***SPAM*** [release] Firefox 13.0b6 build1: completed final_verification on linux
   Subject: ***SPAM*** [release] Firefox 13.0b6 build1: completed final_verification on linux64
   Subject: ***SPAM*** [release] Firefox 13.0b6 build1: completed final_verification on win32

Going to Beta

Run backupsnip

NOTE: Remember to do this at least 2 hours ahead of the expected "go to beta" email. details

Done under "screen -R armenzg" on "armenzg" user.

[armenzg@dp-ausstage01 ~]$ sudo su ffxbld
[ffxbld@dp-ausstage01 armenzg]$  cd /opt/aus2/snippets/staging/ 
[ffxbld@dp-ausstage01 staging]$ ls -d Firefox-13.0b6*
Firefox-13.0b6-build1       Firefox-13.0b6-build1-test.time
Firefox-13.0b6-build1-test
[ffxbld@dp-ausstage01 staging]$ ~/bin/backupsnip Firefox-13.0b6-build1

Remember to ssh to aus3-staging using your short LDAP name and then run sudo su - ffxbld once on. If you intend to use screen, initiate your screen session while you are still yourself.

Push to beta

Wait for "go" from release managers
For Firefox:

Publish (and record run time here): details
- missed the time incantation on the pushsnip, but it was only a few minutes
on completion, email r-d that VERSION updates are live on the beta channel

DONE TO HERE

Wait for confirmation from QA that each of the above releases is okay, then close the tracking ticket.

Build #	Branch, Tags	Changeset
1	GECKO130_2012052815_RELBRANCH, FIREFOX_13_0b6_BUILD1 FIREFOX_13_0b6_RELEASE	046e86aee0c1

Releases/Firefox 13.0b6/BuildNotes

Contents

Notes About Releasing

Bugs hit

Build Engineers

Signed-off Revision(s)

L10N changesets

Tags

Build data

Firefox

Notes

Build 1

Preparing to start Automation

E-mail Metrics

Tag

Bouncer Submitter

Source

Build

Firefox

Firefox repacks

Unittests / Talos

XULRunner

Source

Build

Checksums

Updates

Update verify

Reset reserved slaves

Check permissions / AV scan

Push to internal mirrors

Final Verification

Going to Beta

Run backupsnip

Push to beta

Navigation menu

Search