Releases/Firefox 3.5.4/BuildNotes

Build Engineers

coop https://bugzilla.mozilla.org/show_bug.cgi?id=519365

Signed-off Revision(s)

Build1: 09522dd1776e
Build2: e178dbd2a389

Build data

Build #	Type	Build ID	Build machine
1	[Windows installer/zip]	20091007001339	moz2-win32-slave44
	[Mac compressed]	20091006224018	moz2-darwin9-slave13
	[Linux compressed]	20091006225203	moz2-linux-slave38
2	[Windows installer/zip]	20091014223740	moz2-win32-slave47
	[Mac compressed]	20091014210210	moz2-darwin9-slave14
	[Linux compressed]	20091014211016	moz2-linux-slave35
3	[Windows installer/zip]	20091016092926	moz2-win32-slave52
	[Mac compressed]	20091016081620	moz2-darwin9-slave16
	[Linux compressed]	20091016081727	moz2-linux-slave36

Notes

Useful docs and FAQs here.
Done on production-master02

Build 1

Cleaned up previous releases' build dirs
- Went through logs on the master to gather hostnames for linux/win32. Just cleared all possible buildslaves on mac. Used csshX on linux/mac, RDP as Administrator on win32.
  - Linux (moz2-linux-slave[18-23,25])
    - /builds/slave/{tag,source,linux_build,linux_repack,updates,linux_update_verify,final_verification}/*
  - Mac (bm-xserve22, moz2-darwin9-slave[13-18])
    - /builds/slave/{macosx_build,macosx_repack,l10n_verification,macosx_update_verify}/*
  - Win32 (moz2-win32-slave[32-39]
    - /e/builds/moz2_slave/{win32_build,win32_repack,win32_update_verify}/*
l10n-changesets updated in http://hg.mozilla.org/build/buildbot-configs/rev/61377630502c
Land version bump, reconfig production-master02
Kicked-off automation:

buildbot sendchange --username=coop --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5.4 build1" DOOOOEEEET

Manually tagged the latest of buildbot-configs, buildbotcustom, and tools with FIREFOX_3_5_4_RELEASE

Socorro product data updates

Pinged IT to add the following entries to the Socorro db:

insert into productdims (product, version, branch, release) values ('Firefox','3.5.4', '1.9.1', 'major');
insert into productdims (product, version, branch, release) values ('Firefox','3.5.5pre', '1.9.1', 'development');

bug 524558

Tag

(slave: moz2-linux-slave35) No problems

Source

(slave: moz2-linux-slave35) No problems

Build/Repack

Linux: No problems
Mac: No problems
Win32: No problems

Sign

Cleaned up Firefox 3.0.12 signing dirs for a little more space (22GB total)
First attempt failed, exiting with a make error.
- Removed the signed-build1/ dir, and tried again successfully using -j1
instructions

L10nVerify

(slave: moz2-darwin9-slave16) No problems

Generate updates and push betatest snippets

1st attempt: (slave: moz2-linux-slave36) patcher_create_patches failed: failure log

Exception: cmd failed mar -C /builds/slave/updates/build/tmpsVvBD6-fastmode/223 ...

2nd attempt: (slave: moz2-linux-slave30) diff_patcher_config failed (didn't remove bump step) and patcher_create_patches failed: failure log

Exception: cmd failed mar -x /builds/slave/updates/build/temp/firefox/3.5.4/ftp/firefox-3.5.4.sv-SE.win32.complete.mar

3rd attempt: (slave: moz2-linux-slave49) checkout_patcher failed. This slave had never connected to the CVS server before. Updated the linux slave migration doc to include the CVS server.
4th attempt: (slave: moz2-linux-slave47) backupsnip timed out (timeout is 7200s)
at this point, I gave up with automation, took moz2-linux-slave47 out of rotation, checked the updates steps for the 3.5.3 run, and ran the remaining commands (backupsnip/pushsnip/wait/upload_snippets) by hand (updated for 3.5.4, of course)

[cltbld@moz2-linux-slave47 build]$ ssh -l cltbld aus2-staging.mozilla.org ~/bin/backupsnip 20091007-Firefox-3.5.4-test

OK, that failed too. We hit a hard ssh timeout of (2 hours). joduinn filed bug 521139 to track the slowdown on aus2-staging. The next step was running the command directly on aus2-staging in a screen session:

bash-3.2$ time ~/bin/backupsnip 20091007-Firefox3.5.4-test
Running /bin/tar cfvj /opt/aus2/snippets/backup/20091007-4-pre-20091007-Firefox-3.5.4-test.tar.bz2 .
bash-3.2$ time ~/bin/pushsnip 20091007-Firefox-3.5.4-test

The above commands succeeded. backupsnip took 60 minutes, pushsnip took almost 3 minutes.
then re-ran the rsync command (upload_snippets) on the original build slave (moz2-linux-slave47):

[cltbld@moz2-linux-slave47 ~]$ cd /builds/slave/updates/build/temp/firefox/3.5.3-3.5.4
[cltbld@moz2-linux-slave47 3.5.3-3.5.4]$ rsync -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091007-Firefox-3.5.4-beta

Notes: I'm unsure as to why the early mar failures occurred and why the patch generation magically worked on the 4th attempt. I *think* the slaves in question might have been running out of space, but the slaves were being re-used by buildbot so quickly after the failed step that it's hard to tell. The clean_old_builds step passed in all cases, so maybe we need to up the required space?

Update verify

Linux: (slave: moz2-linux-slave46) No problems
Mac: (slave: bm-xserve22) No problems
Win32: (slave: moz2-win32-slave34) No problems

Publish updates to beta channel

# cltbld @ aus2-staging
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20091007-Firefox-3.5.4-beta
Running /bin/tar cfvj /opt/aus2/snippets/backup/20091009-1-pre-20091007-Firefox-3.5.4-beta.tar.bz2 .
real    67m23.979s
user    0m55.489s
sys     2m2.785s
~/bin/pushsnip 20091007-Firefox-3.5.4-beta

XULRunner build

Create the builds by going to the buildbot waterfall for mozilla-1.9.1 and forcing builds for linux (not x64), mac and windows xulrunner. Set the branch to 'releases/mozilla-1.9.1' and revision to 'FIREFOX_3_5_4_RELEASE'.

When they finish, as xrbld@stage move the dir from /pub/mozilla.org/xulrunner/nightly/2009/10/2009-10-xx-xx-mozilla-1.9.1/ to ..../nightly/1.9.1.4-candidates/build1. Remove the broken symlink at xulrunner/nightly/2009-10-xx-xx-mozilla-1.9.1. Contact mfinkle and dtownsend for smoketest QA.

RESPIN

bug 521878

Build 2

Cleaned up build dirs from build 1
- Went through builders on the master to gather hostnames for linux/win32/mac. Used cssh on linux/mac, RDP as Administrator on win32.
  - Linux (moz2-linux-slave[30,35,36,38,40,43-50])
    - /builds/slave/{tag,source,linux_build,linux_repack,updates,linux_update_verify,final_verification}/*
  - Mac (bm-xserve22, moz2-darwin9-slave[13-18])
    - /builds/slave/{macosx_build,macosx_repack,l10n_verification,macosx_update_verify}/*
  - Win32 (moz2-win32-slave[34,44,52-59])
    - /e/builds/moz2_slave/{win32_build,win32_repack,win32_update_verify}/*

Tag

attempt 1 (slave: moz2-linux-slave44): initial reconfig didn't take due to a local typo I introduced while checking that the config was updated. I didn't notice because the reconfig usually times out anyway :/
attempt 2 (slave: moz2-linux-slave44): failed because I hadn't cleaned out the release dirs after the previous attempt.
attempt 3 (slave: moz2-linux-slave33): third time's the charm.

Source

(slave: moz2-linux-slave37) No problems

Build/Repack

Linux: No problems
Mac: No problems
Win32: No problems

Sign

updated ~/signing-work/firefox-3.5.4/signing.py to pick up the fix for bug 521878.
ran with CONCURRENCY=2 for bug 521080
instructions

L10nVerify

(slave: bm-xserve22) No problems

Generate updates and push betatest snippets

first backupsnip completed properly before respin was called, which is better than can be said for build 1.

RESPIN

bug 519589

Build 3

Cleaned up build dirs from build 2
- Went through builders on the master to gather hostnames for linux/win32/mac. Used cssh on linux/mac, RDP as Administrator on win32.
  - Linux (moz2-linux-slave[20,22,33,35,37,44])
    - /builds/slave/{tag,source,linux_build,linux_repack,updates,linux_update_verify,final_verification}/*
  - Mac (bm-xserve[11,19,22], moz2-darwin9-slave14)
    - /builds/slave/{macosx_build,macosx_repack,l10n_verification,macosx_update_verify}/*
  - Win32 (moz2-win32-slave[32,33,39,47])
    - /e/builds/moz2_slave/{win32_build,win32_repack,win32_update_verify}/*

Tag

(slave: moz2-linux-slave39) KaiRo re-tagged mozilla-1.9.1 in the middle of our tagging step, so the tagging step failed on mozilla-1.9.1. All locales were tagged fine though
to recover, I logged into the slave (moz2-linux-slave39), enabled the hg rebase extension in the .hgrc file, and then ran the following commands:

cd /builds/slave/tag/mozilla-1.9.1
hg pull --rebase
hg push -e 'ssh -l ffxbld -i ~cltbld/.ssh/ffxbld_dsa' -f ssh://hg.mozilla.org/releases/mozilla-1.9.1

I then disabled the rebase extension in the .hgrc file again

Source

(slave: moz2-linux-slave37) forced from the builder page, no problems

Build/Repack

Linux: en-US build forced from the builder page
Mac: en-US build forced from the builder page
Win32: en-US build forced from the builder page

to force the l10n repacks, I had to use the same process as bhearsum used for 3.6b1:

# on coop's machine:
hg clone http://hg.mozilla.org/users/bhearsum_mozilla.com/tools tools-bhearsum
cd tools-bhearsum/buildbot-helpers
wget http://people.mozilla.org/~coop/files/3.5.4/shipped-locales-linux
wget http://people.mozilla.org/~coop/files/3.5.4/shipped-locales-macosx
wget http://people.mozilla.org/~coop/files/3.5.4/shipped-locales-win32
# after linux en-US finished:
time python force_release_l10n.py -m http://production-master02.build.mozilla.org:8010 -v -t FIREFOX_3_5_4_RELEASE -s shipped-locales-linux -n coop 2>&1 | tee linux_repack_trigger.log
real	3m11.262s
user	0m0.606s
sys	0m0.274s
# after mac en-US finished
time python force_release_l10n.py -m http://production-master02.build.mozilla.org:8010 -v -t FIREFOX_3_5_4_RELEASE -s shipped-locales-macosx -n coop 2>&1 | tee macosx_repack_trigger.log
real	3m9.023s
user	0m0.609s
sys	0m0.293s
# after win32 en-US finished
time python force_release_l10n.py -m http://production-master02.build.mozilla.org:8010 -v -t FIREFOX_3_5_4_RELEASE -s shipped-locales-win32 -n coop 2>&1 | tee win32_repack_trigger.log
real	3m34.634s
user	0m0.612s
sys	0m0.300s

NOTE: I sent the initial batch of linux l10n re-triggers to the wrong master (production-master), so I had to kill off a bunch of linux_repack builds there.

bug 522989: decided to do a single-locale rebuild on build3 for the Turkish (tr) locale
- re-tagged changeset fdb83f4b08fc as FIREFOX_3_5_4_BUILD3 and FIREFOX_3_5_4_RELEASE
- forced builds from the buildbot waterfall: linux_repack, macosx_repack, win32_repack

param	value
en_REVISION	FIREFOX_3_5_4_RELEASE
l10n_REVISION	FIREFOX_3_5_4_RELEASE
locale	tr

- post_upload.py failed while trying to chmod /home/ftp/pub/firefox/nightly/3.5.4-candidates/build3/contrib/solaris_pkgadd,
  - I modified the post_upload.py command on stage.m.o to ignore the contrib dirs
  - I then had to login to the linux/macosx repack slaves and run the following commands:

cd /builds/slave/{linux|macosx}_repack/build/mozilla-1.9.1/browser/locales
export AB_CD=tr
export MOZ_MAKE_COMPLETE_MAR=1
export MOZ_PKG_PRETTYNAMES=1
export MOZ_PKG_VERSION=3.5.4
export POST_UPLOAD_CMD='post_upload.py -p firefox -v 3.5.4 -n 3 --release-to-candidates-dir'
export UPLOAD_HOST=stage.mozilla.org
export UPLOAD_SSH_KEY=~/.ssh/ffxbld_dsa
export UPLOAD_TO_TEMP=1
export UPLOAD_USER=ffxbld
make l10n-upload-tr

- - I simply re-ran the win32 repack again as that was faster (once the script on staging was fixed) than trying to get all the env settings right to do it by hand

Sign

ran with CONCURRENCY=1 for bug 521080
removed signed-builds[1-2] and unsigned-builds[1-2] for 3.5.4 to free up space (19G)
instructions

signing of build3 single-locale rebuild (tr):
- set variables
- moved old unsigned-build3 dir out of the way
- ran following commands:

cd ~/signing-work/firefox-3.5.4
make download PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
# Removed all non-tr builds/xpis from unsigned-build3
./sign-release.py --first-locale tr --keydir d:/2008-keys -j4 --product firefox -o signed-build3 unsigned-build3 >> win32_signing_build3.log  2>&1
cat ./unsigned-build3/unsigned/win32_info.txt >> ./signed-build3/win32_info.txt
rsync -av ./unsigned-build3/unsigned/win32/xpi ./signed-build3/win32
make checksum-files PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
# Remove existing .asc files for .tr builds from signed-build3, and also for the *SUM files.
make create-sigs PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
# contrib. dirs. already exist, so we just run 'make fix-permissions' rather than 'make stage'
make fix-permissions PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
make verify PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
make fake-upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

L10nVerify

(slave: bm-xserve22) No problems

Generate updates and push betatest snippets

attempt 1: (slave: moz2-linux-slave30) failed while extracting a mar (log link):

Exception: cmd failed mar -x /builds/slave/updates/build/temp/firefox/3.5.4/ftp/firefox-3.5.4.ta.win32.complete.mar

cleaned up slave and forced the step again from the builder page
attempt 2: (slave: moz2-linux-slave32) No problems

need to generate updates from build1 to build3. Followed build notes example from 3.0.11:

# connect as cltbld@moz2-linux-slave32
cd /builds/slave/updates/build/temp/firefox/3.5.3-3.5.4
for dir in aus2.test aus2.beta; do
  # copy complete snippets from 3.5.3 into a new dir for 3.5.4
  mkdir -p b1-b3-${dir}/Firefox
  rsync -a --exclude 'partial.txt' ${dir}/Firefox/3.5.3/ b1-b3-${dir}/Firefox/3.5.4/
  # use the buildid's from build1
  mv b1-b3-${dir}/Firefox/3.5.4/Darwin_Universal-gcc3/20090824085414 b1-b3-${dir}/Firefox/3.5.4/Darwin_Universal-gcc3/20091006224018 
  mv b1-b3-${dir}/Firefox/3.5.4/Linux_x86-gcc3/20090824085743 b1-b3-${dir}/Firefox/3.5.4/Linux_x86-gcc3/20091006225203
  mv b1-b3-${dir}/Firefox/3.5.4/WINNT_x86-msvc/20090824101458 b1-b3-${dir}/Firefox/3.5.4/WINNT_x86-msvc/20091007001339
done
rsync -a -e 'ssh -i ~/.ssh/aus' b1-b3-aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091019-Firefox-3.5.4build1-build3-test
rsync -a -e 'ssh -i ~/.ssh/aus' b1-b3-aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091019-Firefox-3.5.4build1-build3-beta

Then go to aus2-staging and push 20091019-Firefox-3.5.4build1-build3-test live.

Updates for tr re-build required some manual work. Logged on to the slave that generated the updates (moz2-linux-slave32), and

cd /builds/slave/updates/build/patcher-configs.

Narrowed the locale list in moz191-branch-patcher2.cfg to include only 'tr' for the 3.5.3 and 3.5.4 blocks, and then ran the following commands:

cd /builds/slave/updates/build
perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/moz192-branch-patcher2.cfg
perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/moz192-branch-patcher2.cfg
find temp/firefox/3.5.3-3.5.4/ftp/firefox/nightly/3.5.4-candidates/build3 -type f -exec chmod 644 {} \;
find temp/firefox/3.5.3-3.5.4/ftp/firefox/nightly/3.5.4-candidates/build3 -type d -exec chmod 755 {} \;
cd /builds/slave/updates/build/temp/firefox/3.5.3-3.5.4/ftp/firefox/nightly/3.5.4-candidates/build3
rsync -av -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' --exclude=*complete.mar update ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/3.5.4-candidates/build3/
cd /builds/slave/updates/build/temp/firefox/3.5.3-3.5.4
rsync -av aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091019-Firefox-3.5.4-from-3.5.3
rsync -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091019-Firefox-3.5.4-from-3.5.3-test
rsync -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091019-Firefox-3.5.4-from-3.5.3-beta

pushed tr updates live:

# cltbld@aus2-staging
~/bin/pushsnip 20091019-Firefox-3.5.4-from-3.5.3-test

Update verify

Linux: (slave: moz2-linux-slave37) No problems.
Mac: (slave: moz2-darwin9-slave13) No problems.
Win32: (slave: moz2-win32-slave45) No problems.

Publish updates to beta channel

backupsnip was run once see here and encompasses both 3.0.15 and 3.5.4

# cltbld@aus2-staging
# Regular beta snippets
~/bin/pushsnip 20091016-Firefox-3.5.4-beta
# build1->build3 snippets
~/bin/pushsnip 20091019-Firefox-3.5.4build1-build3-beta
# tr build3 rebuild snippets
~/bin/pushsnip 20091019-Firefox-3.5.4-from-3.5.3-beta

Update Bouncer

No problems.

Final checks before push

cd /pub/mozilla.org/firefox/nightly/3.5.4-candidates/build3/
find . ! -user ffxbld
find . ! -group firefox
find . -type f ! -perm 644
find . -type d -mindepth 1 ! -perm 755 ! -name 'contrib*'
find . -type d -maxdepth 1 ! -perm 2775  -name 'contrib*'

Only returned Solaris builds (different user). This was expected.

Push to mirrors

# ffxbld@stage
rsync -av --exclude=*.log --exclude=*.txt --exclude=*unsigned* /pub/mozilla.org/firefox/nightly/3.5.4-candidates/build3/ /pub/mozilla.org/firefox/releases/3.5.4/

# cltbld@stage
vim /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude
# replace 3.5.3 with 3.5.4

Final Verification

Kicked off from pm02 waterfall - no problems with any links.

Push updates to release channel

Backupsnip had been run for 3.0.15.

# cltbld @ aus2-staging
time ~/bin/pushsnip 20091016-Firefox-3.5.4
real    1m54.901s
user    0m0.244s
sys     0m7.906s

~/bin/pushsnip 20091019-Firefox-3.5.4-from-3.5.3
real    0m1.187s
user    0m0.005s
sys     0m0.069s

Release

cd /home/ftp/pub/firefox/releases 
rm latest-3.5 && ln -s 3.5.4 latest-3.5

XULRunner build

Created the builds by going to the buildbot waterfall for mozilla-1.9.1 and forcing builds for linux (not x64), mac and windows xulrunner. Set the branch to 'releases/mozilla-1.9.1' and revision to 'FIREFOX_3_5_4_RELEASE'.

When they finished, as xrbld@stage moved the dir from /pub/mozilla.org/xulrunner/nightly/2009/10/2009-10-28-07-mozilla-1.9.1/ to ..../nightly/1.9.1.4-candidates/build3. Removed the broken symlink at xulrunner/nightly/2009-10-28-07-mozilla-1.9.1. Contacted mfinkle and dtownsend for smoketest QA.

Publishing is done in a similar (but not identical!) way as 3.0.13:

# cltbld@stage
VERSION=1.9.1.4
FX_VERSION=3.5.4
BUILD=1

mkdir -p ~/xulrunner-${VERSION}/batch1/stage-unsigned/{runtimes,source,sdk}
cd ~/xulrunner-${VERSION}/batch1/stage-unsigned
rsync -av --exclude=*sdk* --exclude=*info* \
  /home/ftp/pub/xulrunner/nightly/${VERSION}-candidates/build${BUILD}/ runtimes/
rsync -av --include=*sdk* --exclude=* \
  /home/ftp/pub/xulrunner/nightly/${VERSION}-candidates/build${BUILD}/ sdk/
cp -pv /pub/mozilla.org/firefox/releases/${FX_VERSION}/source/firefox-${FX_VERSION}.source.tar.bz2 \
  source/xulrunner-${VERSION}.source.tar.bz2
cd ~/xulrunner-${VERSION}/batch1
rsync -av stage-unsigned/ stage-signed/

Then create detached signatures per usual process (with PRODUCT=xulrunner, VERSION=1.9.1.4, don't need BUILD or TAG), skip the win32 signing section, do pgp, verify and upload. Back on stage:

cd ~/xulrunner-${VERSION}/batch1/stage-signed
cp /pub/mozilla.org/firefox/releases/${FX_VERSION}/KEY .
~/bin/checksum-files .
chown -R cltbld:xulrunner .
find . -type f -exec chmod -v 644 {} \;
find . -type d -exec chmod -v 755 {} \;
cd ~/xulrunner-${VERSION}/batch1
rsync -nav stage-signed/* /home/ftp/pub/xulrunner/releases/${VERSION}/

If all is fine then

rsync -av stage-signed/* /home/ftp/pub/xulrunner/releases/${VERSION}/

Update the links in these two wiki pages for the version change. NOTE: make sure to update all the links on the page!!!

Build #	Tag	Changeset
1	GECKO1914_20091006_RELBRANCH	09522dd1776e
1	FIREFOX_3_5_4_BUILD1 FIREFOX_3_5_4_RELEASE	f2eb72702e5b
2	GECKO1914_20091006_RELBRANCH	4a35b0f000c7
2	FIREFOX_3_5_4_BUILD2 FIREFOX_3_5_4_RELEASE	e178dbd2a389
3	GECKO1914_20091006_RELBRANCH	49342a1d9d93
3	FIREFOX_3_5_4_BUILD3 FIREFOX_3_5_4_RELEASE	2fe392234f8c

Releases/Firefox 3.5.4/BuildNotes

Contents

Build Engineers

Signed-off Revision(s)

Tags

Build data

Notes

Build 1

Socorro product data updates

Tag

Source

Build/Repack

Sign

L10nVerify

Generate updates and push betatest snippets

Update verify

Publish updates to beta channel

XULRunner build

RESPIN

Build 2

Tag

Source

Build/Repack

Sign

L10nVerify

Generate updates and push betatest snippets

RESPIN

Build 3

Tag

Source

Build/Repack

Sign

L10nVerify

Generate updates and push betatest snippets

Update verify

Publish updates to beta channel

Update Bouncer

Final checks before push

Push to mirrors

Final Verification

Push updates to release channel

Release

XULRunner build

Navigation menu

Search