Setup

Prerequisites

python 2.6
virtualenv

NOTE: You will need root access to create a user account for yourself on dev-master01 and a buildbot area under /builds/buildbot/your_name. If you are setting dev-master01 you probably want to use root to set up an account in /builds/buildbot/your_name since you might get this error (I think it has to do with which group your account belongs to):

from twisted.python.zippath import ZipArchive
 ImportError: No module named zippath

General

$ ssh root@dev-master01
# useradd yourname

Instructions specific to dev-master01 (dev-master01.build.mozilla.org)

 # as root
 WHO=coop
 mkdir /builds/buildbot/${WHO} 
 chown ${WHO} /builds/buildbot/${WHO}

Notes:
- BASEDIR refers to where you want the new master to be created
- MASTER_NAME is the name of one of the predefined masters. Search for a name prefixed by "staging".
  - /usr/bin/python2.6 setup-master.py -l -- display a list of enabled masters
  - /usr/bin/python2.6 setup-master.py -l -8 -- display a hardcoded list of 0.8.0 masters
- PYTHON, HG, VIRTUALENV should be adjusted to your local environment

Create a build master

export WHO=coop
export BASEDIR=/builds/buildbot/${WHO}/build-master
export BUILDMASTER=staging-builder_master1
hg clone http://hg.mozilla.org/build/buildbot-configs
cd buildbot-configs
make -f Makefile.setup \
HG=/usr/bin/hg \
VIRTUALENV=/usr/bin/virtualenv-2.6 \
PYTHON=/usr/bin/python2.6 \
MASTER_NAME=${BUILDMASTER} \
BASEDIR=${BASEDIR} \
virtualenv deps install-buildbot master master-makefile

cd ${BASEDIR}/buildbot-configs/mozilla
cp staging_builder_master_sm01_localconfig.py ${WHO}_master_localconfig.py
cp staging_config.py ${WHO}_staging_localconfig.py
cd ../../master
rm localconfig.py && ln -s ${BASEDIR}/buildbot-configs/mozilla/${WHO}_staging_localconfig.py localconfig.py
rm master_localconfig.py && ln -s ${BASEDIR}/buildbot-configs/mozilla/${WHO}_master_localconfig.py master_localconfig.py

build-master path values

Configs: buildbot-configs/mozilla/
master_localconfig: staging_builder_master_sm01_localconfig.py
host: staging-builder_master1

Create a tests master

export WHO=coop
export BASEDIR=/builds/buildbot/${WHO}/tests-master
export TESTSMASTER=staging-tests_master1
export MASTERS_JSON=http://hg.mozilla.org/build/tools/raw-file/tip/buildfarm/maintenance/production-masters.json
hg clone http://hg.mozilla.org/build/buildbot-configs
cd buildbot-configs
make -f Makefile.setup \
HG=/usr/bin/hg \
VIRTUALENV=/usr/bin/virtualenv-2.6 \
PYTHON=/usr/bin/python2.6 \
MASTER_NAME=${TESTSMASTER} \
BASEDIR=${BASEDIR} \
virtualenv deps install-buildbot master master-makefile

cd ${BASEDIR}/buildbot-configs/mozilla-tests
cp staging_tests_master_stm01_localconfig.py ${WHO}_master_localconfig.py
cp staging_config.py ${WHO}_staging_localconfig.py
cd ../../master
rm localconfig.py && ln -s ${BASEDIR}/buildbot-configs/mozilla-tests/${WHO}_staging_localconfig.py localconfig.py
rm master_localconfig.py && ln -s ${BASEDIR}/buildbot-configs/mozilla-tests/${WHO}_master_localconfig.py master_localconfig.py
# Fix in master_localconfig.py WebStatus, QUEUEDIR and comment manhole section

tests-master path values

configs: buildbot-configs/mozilla-test/
master_localconfig: staging_tests_master_stm01_localconfig.py
host: staging-tests_master1

Customize your configs

In master_localconfig.py, adjust ports for slavePortnum, webstatus & manhole
- Check slavealloc to see what ports are in use by others. Search for 'staging-pers' and be sure to Show all Entries.
- Try to use similar ports, i.e. if you choose 9099 for slavePortnum, use 8099 for your webstatus and 1299 for your manhole.
Consider optimizing the number of branches you're running.

Create your own queue dir

To avoid this error:

OSError: [Errno 1] Operation not permitted: '/dev/shm/queue/commands/tmp'

Do this:

 WHO=coop
 mkdir -p /dev/shm/${WHO}/queue
 chown ${WHO} /dev/shm/${WHO}

modify your master/master_localconfig.py so that QUEUEDIR points to your new dir:

QUEUEDIR = "/dev/shm/${WHO}/queue"

Does it work?

If you want production schedulers to run, then symlink /builds/buildbot/${WHO}/{build|tests}-master/master/master.cfg to universal_master_sqlite.cfg. Note: the default scheduler is usually fine.
Run "make checkconfig" to make sure the default configs are in place and good.
Run "make start" to make sure you master starts.
- You should have a Makefile at $BASEDIR/Makefile to start/stop your master
- Visit http://dev-master01.build.mozilla.org:${YOUR_WEBSTATUS_PORT}/waterfall
If it works, congrats! Now run "make stop" to shut it down until you're ready to actually start testing.

Sendchanges to your tests master from your build master

If you're running *both* a build master and a tests master and want to test the sendchanges between them, you should have a similar local change on your build master (replace 9041 with the relevant port on your tests master):

diff --git a/mozilla/staging_config.py b/mozilla/staging_config.py
--- a/mozilla/staging_config.py
+++ b/mozilla/staging_config.py
@@ -49,3 +49,3 @@ GLOBAL_VARS = {
     'talos_masters': [
-        ('dev-master01.build.mozilla.org:9009', True, 1),
+        ('staging-master:9041', True, 1),
     ],
@@ -55,4 +55,3 @@ GLOBAL_VARS = {
     'unittest_masters': [
-        ('staging-master.build.mozilla.org:9009', True, 1),
-        ('talos-staging-master02.build.mozilla.org:9012', True, 1),
+        ('dev-master01:9041', True, 1),
         ],

Adding your master to slavealloc

ssh -l root slavealloc.build.mozilla.org
su slavealloc
cd ~
vim armenzg-master.csv

# create a CSV similar to this one
# make sure you specify 'staging-pers' for staging masters
nickname,fqdn,http_port,pb_port,datacenter,pool
sm-armenzg-build,dev-master01.build.scl1.mozilla.com,8040,9040,sjc1,staging-pers
sm-armenzg-test,dev-master01.build.scl1.mozilla.com,8041,9041,sjc1,staging-pers

# import your master into the DB
# look at history to see what the value of PASSWORD should be
# 10.2.70.105 == tm-b01-master01.mozilla.org
# or ask someone in RelEng for the buildslaves DB password
/tools/slavealloc/bin/slavealloc dbimport \
-D mysql://buildslaves:PASSWORD@10.2.70.105/buildslaves \
--master-data armenzg-master.csv

Modifying your old master in slavealloc

If you have an old master you can modify it:

Visit http://slavealloc.build.mozilla.org/ui/#masters
Modify the FQDN of the master and the port (e.g. dev-master01.build.mozilla.org)

Setting up a local slave

cd /path/to/master
source ./bin/activate
buildslave --help

The buildslave command should be available at this point, if not:

pip install buildbot-slave

Now we create a slave using the directory my-slave in the master directory.

To find the relevant ports, check master/master_localconfig.py
The port to access your master via the browser is under WebStatus.
The port for slave access (used when creating a slave) is under slavePortnum.
Choose a slave name that the master is already looking for. To find one go to 127.0.0.1:{browserport}/builders, select a builder (OS X 10.6.2 mozilla-central nightly for example) and choose one of the names under "Buildslaves"

 
# From the master's directory
buildslave create-slave my-slave 127.0.0.1:<port> <name> pass

If you haven't already, start the master, then start the slave

cd /path/to/master
make start
# Start the slave
cd my-slave
buildslave start

Your slave should be picked up momentarily by the master.

Pointing dev slaves to your master

Using the slaves interface of slavealloc (http://slavealloc.build.mozilla.org/ui/#slaves), search for slaves that are currently in the preprod (or preprod-tests) pool that are not currently locked to someone else's dev master. Edit the slave and lock that slave to your master.

NOTE: the slave will need to be rebooted to pick up this change.

If no slaves are available for your required platform, find out who has them all and ask to borrow one.

Setup a user source repository

Depending on how invasive your changes are going to be, you can choose to setup your own user repo for any of the releng repos or even the Firefox code repos.

Follow the instructions for Mercurial Clones and then replace references to releng repos in the configs with your own repo.

Optimizations

master_localconfig.py

Comment out/change ACTIVE_{BRANCHES,PLATFORMS,PROJECTS} in master_localconfig.py as needed to reduce your workspace.
- e.g. I usually run with only one branch: ACTIVE_BRANCHES = ['mozilla-central']

ReleaseEngineering/How To/Setup Personal Development Master

Contents

Setup

Prerequisites

General

Create a build master

build-master path values

Create a tests master

tests-master path values

Customize your configs

Create your own queue dir

Does it work?

Sendchanges to your tests master from your build master

Adding your master to slavealloc

Modifying your old master in slavealloc

Setting up a local slave

Pointing dev slaves to your master

Setup a user source repository

Optimizations

master_localconfig.py

See Also

Navigation menu

ReleaseEngineering/How To/Setup Personal Development Master

Setup

Prerequisites

General

Create a build master

build-master path values

Create a tests master

tests-master path values

Customize your configs

Create your own queue dir

Does it work?

Sendchanges to your tests master from your build master

Adding your master to slavealloc

Modifying your old master in slavealloc

Setting up a local slave

Pointing dev slaves to your master

Setup a user source repository

Optimizations

master_localconfig.py

See Also

Navigation menu

Search