Build:Release Automation: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(put bootstrap section back together)
No edit summary
 
(48 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{{Obsolete RelEng Page|It was last edited in 2008, and is thus certainly out of date by now.}}
=Intro=
=Intro=
Firefox and Thunderbird releases are currently done using the [http://wiki.mozilla.org/Build:Release_Automation#Bootstrap Bootstrap] automation scripts, which call into [http://mxr.mozilla.org/seamonkey/source/tools/tinderbox/ Tinderbox client] to do the actual build.
Firefox and Thunderbird releases are currently done using the [http://wiki.mozilla.org/Build:Release_Automation#Bootstrap Bootstrap] automation scripts, which call into [http://mxr.mozilla.org/seamonkey/source/tools/tinderbox/ Tinderbox client] to do the actual build.
Line 6: Line 8:
=Bootstrap=
=Bootstrap=


[http://mxr.mozilla.org/seamonkey/source/tools/release/ Bootstrap] is a simple Perl framework intended to take the formerly manual release process and automate it, with as little change to the process as possible.  
[[Build:Release Automation:Bootstrap|Bootstrap]] is a simple Perl framework intended to take the formerly manual release process and automate it, with as little change to the process as possible.
 
Bootstrap is invoked using the "release" command, and supports a set of high-level "steps":
 
<b>Tag</b> - tag, branch, apply version bumps to all relevant files.<br>
<b>TinderConfig</b> - generate tinderbox config files (mozconfig/tinder-config.pl)<br>
<b>Build</b> - invoke Tinderbox client to create and en-US build and publish to FTP<br>
<b>Source</b> - create a source tarball and push it to FTP<br>
<b>Repack</b> - invoke Tinderbox client to create localized versions of en-US build and publish to FTP<br>
<b>PatcherConfig</b> - create a [http://mxr.mozilla.org/seamonkey/source/tools/patcher/ Patcher] config file for generating updates<br>
<b>Updates</b> - invoke Patcher to create partial updates and [http://wiki.mozilla.org/AUS AUS] configuration<br>
<b>Stage</b> - create a staging area and rename files for release<br>
<b>Sign</b> - not implemented<br>
 
==Bootstrap Steps==
 
A Bootstrap "step" must implement 2 required methods:
 
<b>Execute</b> - carry out the actual function of the step, e.g. Build<br>
<b>Verify</b> - run an automated test<br>
 
Additionally, there are 2 optional methods:
 
<b>Push</b> - upload the appropriate changes for testing, e.g. upload build to FTP<br>
<b>Announce</b> - send an email announcing that the step has finished.<br>
 
==Using Bootstrap==
 
If the "release" command is invoked with no parameters, it will attempt to start at the first step and call the methods in this order:
 
# Execute
# Verify
# Push
# Announce
 
As each step completes successfully, the next will be invoked.
 
There are several command-line options, shown by calling "release -h":
 
Usage: release [-l] [-s Step] [-o Step] [-e | -v | -p | -a] [-h]
    -l list all Steps
    -s start at Step
    -o only run one Step
    -e only run Execute
    -v only run Verify
    -p only run Push
    -a only run Announce
    -h this usage message
 
For example, to only run the Push method on the Build step:
 
./release -o Build -p
 
 


=Buildbot=
=Buildbot=
Line 73: Line 22:
=Current status=
=Current status=


For the Firefox 2.0.0.8 release, we are using Buildbot to drive the release. Instead of a human operater logging into [http://wiki.mozilla.org/index.php?title=Build:Release_Automation#Roles_and_resource_requirements each machine] used in the release process, the machines run Buildbot slaves instead. Most of the slaves simply check out and run Bootstrap, at this point.
This system is used for all current Firefox branches. Public documentation for each release is posted under the "Build" links on the [[Releases]] page.


Both the [http://mxr.mozilla.org/mozilla/source/tools/buildbot-configs/automation/staging/ staging] and [http://mxr.mozilla.org/mozilla/source/tools/buildbot-configs/automation/production production] configs are checked into CVS.
All configs are checked into CVS under [http://mxr.mozilla.org/mozilla/source/tools/buildbot-configs/automation/ mozilla/tools/buildbot-configs/automation].
 
There are still several manual processes, which we are working on:
 
* Buildbot config
** if necessary, tag new version of mozilla/tools/release (used RELEASE_AUTOMATION_M5_3)
*** make sure buildbot-configs/automation/production/master.cfg uses this tag
** need to file bug (e.g. {{Bug|393264}}) and attach diff for bootstrap.cfg (e.g. [http://mxr.mozilla.org/mozilla/source/tools/release/configs/fx-moz18-bootstrap.cfg fx-moz18-bootstrap.cfg])
** need to "cvs update" /home/buildmaster/Automation/buildbot-configs/ and /home/buildmaster/Automation/bootstrap-configs/ after checkin
** need to manually insert passwords into master.cfg, as they are intentionally not in the checked-in version.
** make sure Tinderbox is up-to-date on all slaves {{Bug|397554}}
*** used RELEASE_AUTOMATION_M5_3
** ensure that machines have [http://wiki.mozilla.org/Build:Release_Automation#Roles_and_resource_requirements enough resources] {{Bug|393274}}
** kick off buildbot (run as cltbld):
*** buildbot sendchange --master=localhost:9989 -u joduinn -m"Firefox 2.0.0.8 RC1" release
** <strike>it's not possible to do dependent schedulers with a forced tag {{Bug|394963}}</strike>
** <strike>need to disable updateverify depscheduler until auto-config for update verify is done {{Bug|373995}}, and Update step is able to auto-deploy.</strike>
 
* Tag -
** <strike>had to manually tag based on GECKO181_20070712_RELBRANCH {{Bug|396290}}
*** NOTE - RC1/RC2 respin case fixed/tested</strike>; RC1 firedrill should work, not yet tested
 
* Source
** permissions for generated source tarball are incorrect, (0700 should be <strike>0755</strike> 0644). For FF2.0.0.8 release, automation created the source tarball with the correct 0644. Not sure if this was previously fixed, or we're just confused. Leaving here for now to keep an eye on it.
** <strike>must be run on stage, need to rewrite source step {{Bug|394034}}</strike>
** <strike>manually sync build-console and stage {{Bug|396438}}</strike>
 
* Build
** <strike>manually sync build-console and stage {{Bug|396438}}</strike>
 
* Repack
** <strike>had to fall back to cerberus-vm due to EOL problems {{Bug|397842}}</strike>
** <strike>manually sync build-console and stage {{Bug|396438}}</strike>
** <strike>"scp -r" does not work on pacifica-vm, need to upgrade or use something else (e.g. rsync), fixed in tinderbox</strike>
 
* Sign
** still manual
** need to sync signed bits manually back to build-console e.g. as cltbld@build-console:
*** rsync -av stage.mozilla.org:/home/ftp/pub/firefox/nightly/2.0.0.8-candidates/ /home/ftp/pub/firefox/nightly/2.0.0.8-candidates/
 
* Updates
** call push, "./release -o Updates -p", manually
** had to correct permissions for both snippets and MARs
** update verification config is still manual {{Bug|373995}}
** <strike>had to change stagingServer to "stage" and re-run configs {{Bug|396438}}</strike>
** <strike>manually sync build-console and stage {{Bug|396438}}</strike>
 
* Stage
** had to correct permissions
** need to rsync /data/cltbld/firefox-2.0.0.7/ stage:/data/cltbld/firefox-2.0.0.7/ (not covered by {{Bug|396438}}).
** created "latest" and "latest-2.0" symlinks manually after final release
** created bouncer links manually {{Bug|372746}}


=Roles and resource requirements=
=Roles and resource requirements=
Line 161: Line 59:
** e.g. /opt/aus2/incoming/3/Firefox/2.0.0.4/, /opt/aus2/snippets/staging/20070523-Fx-2.0.0.4/, /opt/aus2/snippets/backup/20070611-1-pre-20070611-Fx-2.0.0.4.tar.bz2
** e.g. /opt/aus2/incoming/3/Firefox/2.0.0.4/, /opt/aus2/snippets/staging/20070523-Fx-2.0.0.4/, /opt/aus2/snippets/backup/20070611-1-pre-20070611-Fx-2.0.0.4.tar.bz2


=Notes on staging setup=
=Updating Staging release version=
 
* Bump config versions in mozilla/tools/release/Makefile, mozilla/tools/configs/fx-moz18-staging-bootstrap.cfg, mozilla/tools/buildbot-configs/automation/staging/master.cfg e.g. {{bug|407672}}
Buildbot master basedir is ~buildmaster/TestBot
 
The bootstrap.cfg is pulled from the master dir.
 
Slaves basedirs are in cltbld's home directory on the appropriate machine, e.g. ~cltbld/linux-slave1
 
Changes can be inserted with "buildbot sendchange" on the master e.g.:
buildbot sendchange --master=localhost:9989 -u rhelmer -m"latest bootstrap from CVS" test
 
Bootstrap uses a local CVS mirror, and the "tag", "source", "updates", and "stage" builders are run by a local buildslave.
 
The bootstrap [http://mxr.mozilla.org/mozilla/source/tools/release/Makefile Makefile] has the following targets:
 
* stage/clean_stage
** create/remove basic fileserver/tag/source/updates/stage environment
* cvsmirror/clean_cvsmirror
** create/remove cvsmirror in /builds/cvsmirror
 
These targets are hard-coded to prepare for a 2.0.0.4 release.
 
There must be "cltbld" and "symbols" accounts on the staging FTP server that the build machines' cltbld accounts can connect to via SSH without a password.
 
* must accept staging-build-console's hostkey via this SSH tunnel:
 
* set up staging FTP server
mkdir /home/ftp /builds /data/cltbld
chown cltbld /home/ftp /builds/ /data/cltbld
cvs co /mofo/release/stage/ to /data/cltbld/bin
groupadd firefox
 
* set up staging AUS server
# TODO - auto-update
mkdir -p /opt/aus2/snippets/staging/backup /opt/aus2/incoming /opt/aus2/app
 
# check out aus2
cd /opt/aus2/
cvs -d /builds/cvsmirror/cvsroot/ co -d app/ -r AUS2_PRODUCTION mozilla/webtools/aus/xml
cd app && ln -s ../incoming ./data
# install apache
yum install httpd


==Updating release version (mirror refresh, etc.)==
* As cltbld@staging-build-console, disable cltbld's nightly cronjob to prevent accidently starting mid-way through update.
* Bump config versions in mozilla/tools/release/Makefile, mozilla/tools/configs/fx-moz18-staging-bootstrap.cfg, mozilla/tools/buildbot-configs/automation/staging/master.cfg e.g. {{bug|397425}}


* Disable cltbld's nightly cronjob
* As cltbld@staging-build-console, refresh cvsmirror by doing:
 
* Refresh cvsmirror
 
As cltbld@staging-build-console:
  cd /home/cltbld/mozilla/tools/release
  cd /home/cltbld/mozilla/tools/release
  cvs up
  cvs up
Line 218: Line 70:
  make cvsmirror
  make cvsmirror


* Update bootstrap and buildbot configs. These are symlinked from bootstrap-configs and buildbot-configs checkouts (of mozilla/tools/release/configs/ and mozilla/tools/buildbot-configs/automation/staging/, respectively).
* Update buildbot configs. These are symlinked from the buildbot-configs checkout (of mozilla/tools/buildbot-configs/automation/staging/).


As buildmaster@staging-build-console:
As buildmaster@staging-build-console:
# pull configs and restart buildbot
  cd /home/buildmaster/TestBot
  cd /home/buildmaster/TestBot
  buildbot stop `pwd`
  buildbot stop `pwd`
cd bootstrap-configs && cvs up && cd ../
  cd buildbot-configs && cvs up && cd ../
  cd buildbot-configs && cvs up && cd ../
  buildbot start `pwd`
  buildbot start `pwd`


NOTE - the Talkback symbol server is hardcoded in /builds/cvsmirror.clean/mofo/talkback/fullsoft/Makefile.in, this should be changed like so:
As cltbld@staging-build-console:
 
  # update mofo scripts
FC_TUNNEL      = ssh -$(FC_SSH_VERSION) -f -L 8080:hal:80 $(LSSH_USER)staging-build-console.build.mozilla.org sleep 20
  cd /data/cltbld/bin
SYM_TUNNEL      = ssh -$(SYM_SSH_VERSION) -f -L 2222:localhost:22 $(LSSH_USER)staging-build-console.build.mozilla.org sleep 20
cvs up
 
=Production setup HOWTO for linux/mac/win32=
 
This section describes the changes made to clones of the nightly tinderboxes (which were formerly used exclusively for releases).
 
* build-console setup
** check out /mofo/release/stage to /data/cltbld/bin
*** <strike>''NOTE - this is for the firefox-src-tarball-nobuild script, which checks out a tag from CVS and creates a source archive. This should be reimplemented in the [http://mxr.mozilla.org/mozilla/source/tools/release/Bootstrap/Step/Source.pm bootstrap Source step]''</strike>
 
* (Win32/Mac only) install Config::General
  cd /tools/dist
  wget http://search.cpan.org/CPAN/authors/id/T/TL/TLINDEN/Config-General-2.33.tar.gz
  tar xfvz Config-General-2.33.tar.gz
  cd Config-General-2.33
  perl Makefile.PL
 
its ok to ignore the warning from "perl Makefile.PL":
Warning: the following files are missing in your kit:
t/test.rc.out
 
  sudo make install
 
* (Linux only) prepend custom GCC to the path in ~/.bash_profile
export PATH="/usr/gcc-3.3.2rh/bin:/opt/local/bin:/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:$PYTHONHOME/bin:$PATH"
 
* create logs dir
$ mkdir -p /tools/dist/logs
$ mkdir -p /builds/logs
 
* (Mac only) Install 7z. You can download it. Or you can copy it from bm-xserve01, which is what we did here. By putting the file in /usr/bin, it is automatically on the PATH of cltbld's .profile.
 
  $ cd /usr/bin
  $ sudo rsync -av cltbld@bm-xserve01.build.mozilla.org:/usr/local/bin/7z .
 
* look for Tinderbox directory
#linux: if tinderbox name is not "Fx-Mozilla1.8-Release" exactly, symlink it
ln -s /builds/tinderbox/Fx-Mozilla1.8-release /builds/tinderbox/Fx-Mozilla1.8-Release
 
Check out tinderbox configs:
# win32
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -r MOZILLA_1_8_BRANCH_release -d tinderbox-configs mozilla/tools/tinderbox-configs/firefox/win32
# linux
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -r MOZILLA_1_8_BRANCH_release -d tinderbox-configs mozilla/tools/tinderbox-configs/firefox/linux
# macosx
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -r MOZILLA_1_8_BRANCH_release -d tinderbox-configs mozilla/tools/tinderbox-configs/firefox/macosx
 
 
* set up Tinderbox l10n build directory
# linux
cd /builds/tinderbox/
# win32
cd /cygdrive/c/builds/tinderbox/
 
mkdir Fx-Mozilla-1.8-l10n-Release
cd Fx-Mozilla-1.8-l10n-Release
../mozilla/tools/tinderbox/install-links
rm build-seamonkey.pl
ln -s ../mozilla/tools/tinderbox/build-firefox.pl .
ln -s build-firefox.pl build-seamonkey.pl
rm post-mozilla.pl
ln -s post-mozilla-release.pl post-mozilla.pl
 
Check out tinderbox configs:
# win32
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -r MOZILLA_1_8_BRANCH_l10n_release -d tinderbox-configs mozilla/tools/tinderbox-configs/firefox/win32
# linux
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -r MOZILLA_1_8_BRANCH_l10n_release -d tinderbox-configs mozilla/tools/tinderbox-configs/firefox/linux
# macosx
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -r MOZILLA_1_8_BRANCH_l10n_release -d tinderbox-configs mozilla/tools/tinderbox-configs/firefox/macosx
 
 
ln -s tinderbox-configs/mozconfig .
ln -s tinderbox-configs/tinder-config.pl .
 
* [http://wiki.mozilla.org/ReferencePlatforms/BuildBot/Linux Install buildbot]
* running as "cltbld", install slave
#linux
$ cd ~
$ buildbot create linux-slave1 build-console.build.mozilla.org:9989 linux-slave1 password
  #win32
c:\\buildtools\\python24\\scripts\\buildbot create-slave c:\\win32-slave1 build-console.build.mozilla.org:9989 win32-slave1 password
 
* edit the admin and host pages in ~/linux-slave1/info/
 
* start slave
#linux
buildbot start /home/cltbld/linux-slave1
# win32
c:\\buildtools\\python24\\scripts\\buildbot start c:\\win32-slave1
 
==Just for testing==
 
* build-console
** use "stage" target in [http://mxr.mozilla.org/mozilla/source/tools/release/Makefile#7 bootstrap's Makefile]
 
* Move prod ssh keys out of the way, and copy in "staging" keys:
cd ~
mv ~/.ssh ~/ssh.prod
scp cltbld@staging-prometheus-vm:~/.ssh/id_rsa .ssh/
 
* Move prod tinderbox-configs and put staging-build-console in Root:
# win32
  cd /cygdrive/c/builds/tinderbox/Fx-Mozilla-1.8-Release
# linux
cd /builds/tinderbox/Fx-Mozilla-1.8-Release
 
cp -rp tinderbox-configs tinderbox-configs.prod
# change root to cltbld@staging-build-console.build.mozilla.org:/builds/cvsmirror/cvsroot
vi tinderbox-configs/CVS/Root
 
Same for l10n tinderbox build directories:
 
# win32
cd /cygdrive/c/builds/tinderbox/Fx-Mozilla-1.8-l10n-Release
# linux
cd /builds/tinderbox/Fx-Mozilla-1.8-l10n-Release
 
cp -rp tinderbox-configs tinderbox-configs.prod
# change root to cltbld@staging-build-console.build.mozilla.org:/builds/cvsmirror/cvsroot 
vi tinderbox-configs/CVS/Root
 
* /data/cltbld/bin/firefox-src-tarball-nobuild has a hardcoded CVSROOT; change it to cltbld@staging-build-console.build.mozilla.org:/builds/cvsmirror/cvsroot
 
=Production changes=
 
==Staging/Production Buildbot master differences==


# Signing - prod waits for signed bits, stage fakes w/ symlink ''ok''
* Remove and re-checkout tinderbox-configs directories on build machines. Since the mirror takes checkins, the local checkout thinks that it has a higher revision and won't update to the now-refreshed mirror.
# Bootstrap - prod pulls tag e.g. RELEASE_AUTOMATION_M5, staging pulls tip ''ok''


=Outstanding issues=
# e.g. linux
cd /builds/tinderbox/Fx-Mozilla1.8-release
rm -rf tinderbox-configs
cvs -d :ext:cltbld@staging-build-console.build.mozilla.org:/builds/cvsmirror/cvsroot co -d tinderbox-configs -r MOZILLA_1_8_BRANCH_release mozilla/tools/tinderbox-configs/firefox/linux


# How to handle bootstrap logs.. remove them between runs? Don't want accumulation on slaves ''remove at start''
For l10n use the MOZILLA_1_8_BRANCH_l10n_release branch.
# How to do mock release.. fake version (e.g. 1.2.3.4)? Early 2.0.0.7, that we know we won't release? ''2007 rc1''
# "Source" and "Staging" steps - install a buildslave on stage, or stage everything on build-console? ''use build-console''
# Make sure QA checks e.g. top 5 extensions after Mac Intel switch


==Enhancements==
* re-enable cltbld's nightly cronjob


* ([https://bugzilla.mozilla.org/show_bug.cgi?id=394507  bug 394507]) should set buildbot up to mail based on any failures, currently just depend on bootstrap
=Setup notes (historical)=
* ({{bug|372746}}) Automatically configure bouncer
* [[Build:Release Automation:Notes on Staging Setup|Notes on Staging Setup]]
* ([https://bugzilla.mozilla.org/show_bug.cgi?id=373995 bug 373995]) l10n needs the URL it downloads builds from to be configurable as well
* [[Build:Release Automation:Notes on Production Setup|Notes on Production Setup]]
* ([https://bugzilla.mozilla.org/show_bug.cgi?id=394498 bug 394498]) should report on mirror saturation after release
* [[Build:Release Automation:Trunk|Notes on Trunk Automation Setup]]
* ({{bug|397554}}) Automatically check out, set up, and keep Tinderbox installs up to date
* [http://buildbot.net/trac/ticket/68 buildbot bug#68] buildbot default timeout too short. 5sec isnt always enough, and you can get a "timed out" message in the slave logs, even though slave started "normally".
* [http://buildbot.net/trac/ticket/85 buildbot bug#85] sometimes buildmaster sees buildslave correctly, confirms ping ok, but never assigns pending work to the slave. Doing "buildmaster refresh" is not enough, you need to do "buildmaster stop/start". Restarting the slave does not help.
* [http://buildbot.net/trac/ticket/92 buildbot bug#92] on win32, console output is not logged (goes to the DOS console running buildbot :( )
* [http://buildbot.net/trac/ticket/77 buildbot bug#77] file buildbot bug to handle kill on win32. Add details linking to bsmedberg fix.
* [http://buildbot.net/trac/ticket/67 buildbot bug#67] link to history for old builds at bottom of page (ala tinderbox server).
* [http://buildbot.net/trac/ticket/69 buildbot bug#69] meta-refresh tag for waterfall page
* [http://buildbot.net/trac/ticket/78 buildbot bug#78] buildbot UI to contain way to force build dependent steps instead of just doing current step.
* [http://buildbot.net/trac/ticket/91 buildbot bug#91] When using the CVS Source step on a Mac OSX slave, if a CVS directory is found on the path, buildbot will attempt to use it as if it were a CVS binary.
* [http://buildbot.net/trac/ticket/88 buildbot bug#88] steps which start within a few seconds of each other show as same start time on waterfall page
* (needs bug filed) tinderbox symbol server should be configurable

Latest revision as of 20:39, 17 March 2017

Warning: This RelEng page is obsolete!
It was last edited in 2008, and is thus certainly out of date by now.

Intro

Firefox and Thunderbird releases are currently done using the Bootstrap automation scripts, which call into Tinderbox client to do the actual build.

Buildbot calls bootstrap, parallelizing and serializing where needed.

Bootstrap

Bootstrap is a simple Perl framework intended to take the formerly manual release process and automate it, with as little change to the process as possible.

Buildbot

Buildbot is a continuous integration tool, similar to Tinderbox but written in Python as a client/server Twisted application.

We have a vendor branch in mozilla/tools/buildbot, based on Buildbot's 0.7.5 release.

Mozilla-specific Buildbot install instructions

Buildbot user manual

Current status

This system is used for all current Firefox branches. Public documentation for each release is posted under the "Build" links on the Releases page.

All configs are checked into CVS under mozilla/tools/buildbot-configs/automation.

Roles and resource requirements

  • buildbot master
    • keeps logs, manages overall process
  • ftp/stage.m.o
    • fileserver, both public and private areas
    • FTP candidates - 20GB storage
    • e.g. stage:/home/ftp/pub/firefox/nightly/2.0.0.4-candidates/
    • FTP private staging - 20GB storage
      • e.g. stage:firefox-2.0.0.4/
    • FTP release - 6GB storage
      • e.g. stage:/home/ftp/pub/firefox/releases/2.0.0.4/
  • "tagging" builder
    • checks out source and applies tag
    • 2GB storage
      • e.g. karma:/builds/tags/FIREFOX_2_0_0_4_RELEASE/
  • "source archive" builder
    • builds source archive and pushes for QA
  • "linux/mac/win32 firefox builders"
    • builds firefox and pushes for QA
    • l10n/update verification
    • needs 2GB memory, 10GB storage (each)
      • e.g. prometheus-vm:/builds/tinderbox/Fx-Mozilla1.8-Release/
  • "updates builder"
    • downloads and inventories a set of complete firefox updates, generates partial updates, creates AUS configuration ("snippets")
    • updates - 1GB memory, 5GB storage
    • e.g. prometheus-vm:/builds/updates/firefox-2.0.0.4/
  • "stage builder"
    • creates private staging area on FTP, renames files for release
    • see "fileserver" requirements, above
  • Automatic Update Server (AUS), aus2.m.o
    • 10GB for config files, backups and staging area
    • e.g. /opt/aus2/incoming/3/Firefox/2.0.0.4/, /opt/aus2/snippets/staging/20070523-Fx-2.0.0.4/, /opt/aus2/snippets/backup/20070611-1-pre-20070611-Fx-2.0.0.4.tar.bz2

Updating Staging release version

  • Bump config versions in mozilla/tools/release/Makefile, mozilla/tools/configs/fx-moz18-staging-bootstrap.cfg, mozilla/tools/buildbot-configs/automation/staging/master.cfg e.g. bug 407672
  • As cltbld@staging-build-console, disable cltbld's nightly cronjob to prevent accidently starting mid-way through update.
  • As cltbld@staging-build-console, refresh cvsmirror by doing:
cd /home/cltbld/mozilla/tools/release
cvs up
export CVS_RSH="/home/cltbld/ssh_prod.sh"
make cvsmirror
  • Update buildbot configs. These are symlinked from the buildbot-configs checkout (of mozilla/tools/buildbot-configs/automation/staging/).

As buildmaster@staging-build-console: 

# pull configs and restart buildbot
cd /home/buildmaster/TestBot
buildbot stop `pwd`
cd buildbot-configs && cvs up && cd ../
buildbot start `pwd`

As cltbld@staging-build-console: 

# update mofo scripts
cd /data/cltbld/bin
cvs up
  • Remove and re-checkout tinderbox-configs directories on build machines. Since the mirror takes checkins, the local checkout thinks that it has a higher revision and won't update to the now-refreshed mirror.
# e.g. linux
cd /builds/tinderbox/Fx-Mozilla1.8-release
rm -rf tinderbox-configs
cvs -d :ext:cltbld@staging-build-console.build.mozilla.org:/builds/cvsmirror/cvsroot co -d tinderbox-configs -r MOZILLA_1_8_BRANCH_release mozilla/tools/tinderbox-configs/firefox/linux

For l10n use the MOZILLA_1_8_BRANCH_l10n_release branch.

  • re-enable cltbld's nightly cronjob

Setup notes (historical)

Retrieved from "https://wiki.mozilla.org/index.php?title=Build:Release_Automation&oldid=1166043"