MozillaWiki

ReleaseEngineering/How To/Setup Personal Development Master: Difference between revisions

Page Discussion

ReleaseEngineering/How To/Setup Personal Development Master (view source)

Revision as of 14:44, 21 November 2017

6,405 bytes added ,  21 November 2017
Updated instructions on how to setup a personal dev master
(Updated instructions on how to setup a personal dev master)
 
(48 intermediate revisions by 14 users not shown)
Line 3: Line 3:
= Setup =
= Setup =
== Prerequisites ==
== Prerequisites ==
* python 2.6
* python 2.7
* virtualenv
* virtualenv
** If you need to create a virtualenv manually, you must call it directly with: <pre>python /tools/misc-python/virtualenv.py </path/to/destination></pre>
** This is not necessary for a buildbot master setup if you use the methods described below


'''NOTE''': You will need root access to create a user account for yourself on dev-master01 and a buildbot area under /builds/buildbot/your_name.
If you are setting dev-master01 you probably want to use root to set up an account in /builds/buildbot/your_name since you might get this error (I think it has to do with which group your account belongs to):
from twisted.python.zippath import ZipArchive
  ImportError: No module named zippath


'''NOTE''': You will need root access to create a user account for yourself on dev-master2 and a buildbot area under /builds/buildbot/your_name - if not using root, you might get this error (I think it has to do with which group your account belongs to):
<pre>
  from twisted.python.zippath import ZipArchive
    ImportError: No module named zippath
</pre>
== General ==
== General ==
* Instructions specific to dev-master01 (dev-master01.build.mozilla.org)
* Instructions specific to dev-master2 (dev-master2.bb.releng.use1.mozilla.com)
<pre>
<pre>
  # as root
  # as root
  WHO=coop
  WHO=$whoami
  mkdir /builds/buildbot/${WHO}  
  mkdir /builds/buildbot/${WHO}  
  chown ${WHO} /builds/buildbot/${WHO}
  chown ${WHO} /builds/buildbot/${WHO}
Line 23: Line 26:
** BASEDIR refers to where you want the new master to be created
** BASEDIR refers to where you want the new master to be created
** MASTER_NAME is the name of one of the predefined masters.  Search for a name prefixed by "staging".
** MASTER_NAME is the name of one of the predefined masters.  Search for a name prefixed by "staging".
*** /usr/bin/python2.6 setup-master.py -l   -- display a list of enabled masters
*** display a list of enabled masters: /usr/bin/python2.6 setup-master.py -l
*** /usr/bin/python2.6 setup-master.py -l -8 -- display a hardcoded list of 0.8.0 masters
*** display a hardcoded list of 0.8.0 masters: /usr/bin/python2.6 setup-master.py -l -8  
** PYTHON, HG, VIRTUALENV should be adjusted to your local environment
** PYTHON, HG, VIRTUALENV should be adjusted to your local environment


== Create a build master ==
== Create a build master ==
 
There are two ways to do it:
Use [https://hg.mozilla.org/users/john.hopkins_mozillamessaging.com/misc/file/default/buildbot-scripts/create-staging-master this automation script] to create a build master, or follow the manual steps below. Add <code>'try'</code> to the list of <code>ACTIVE_BRANCHES</code> in <code>master/master_localconfig.py</code>.
* using an automation [https://hg.mozilla.org/build/braindump/file/tip/buildbot-related/create-staging-master.pl script]:
 
<pre>
<code>create_master.sh</code>
perl create-staging-master.pl -u aselagea --master-kind=build --master-dir=build1 --http-port=8021
</pre>
* following several manual steps:
:* clone the <code>buildbot-configs</code> repository.
<pre>
$ hg clone https://hg.mozilla.org/build/buildbot-configs
</pre>
:* add <code>'try'</code> to the list of <code>ACTIVE_BRANCHES</code> in <code>master/master_localconfig.py</code>.
:* run <code>create_master.sh</code> below
<pre>
<pre>
USERNAME=me
USERNAME=`whoami`
make -f Makefile.setup \
make -f Makefile.setup \
USE_DEV_MASTER=1 \
USE_DEV_MASTER=1 \
MASTER_NAME=$USERNAME \
MASTER_NAME=$USERNAME \
BASEDIR=/builds/buildbot/$USERNAME/build1 \
BASEDIR=/builds/buildbot/$USERNAME/build1 \
PYTHON=python2.6 \
PYTHON=/tools/python27/bin/python \
VIRTUALENV=virtualenv-2.6 \
VIRTUALENV=virtualenv \
BUILDBOTCUSTOM_BRANCH=default \
BUILDBOTCUSTOM_BRANCH=default \
BUILDBOTCONFIGS_BRANCH=default \
BUILDBOTCONFIGS_BRANCH=default \
Line 47: Line 58:
</pre>
</pre>


== Universal Master ==
=== Universal Master ===


To create a universal master:
To create a universal master:
  $ cd master
  $ cd /builds/buildbot/$USERNAME/build1/master
  $ ln -sf ../buildbot-configs/mozilla/universal_master_sqlite.cfg master.cfg
  $ ln -sf ../buildbot-configs/mozilla/universal_master_sqlite.cfg master.cfg


You will probably want to clear the schedulers to avoid a pile of scheduled builds from starting on your personal master.  Add this line to the end of master/master.cfg:
You will probably want to avoid running scheduled builds (nightlies, etc.) from starting on your personal master.  Clear the ACTIVE_* lists in <code>mozilla/build_localconfig.py</code> to achieve this.  You can also set <code>c['schedulers']=[]</code> but take care to do that '''before''' the release schedulers are added to the list or sendchange will fail to kick off a release build.
 
  c['schedulers'] = []


=== build-master path values ===
=== build-master path values ===
* Configs: buildbot-configs/mozilla/
* Configs: buildbot-configs/mozilla/
* master_localconfig: ???
* master_localconfig: build_localconfig.py ?
* host: preprod-builder
* host: preprod-builder


=== Signing server setup ===
=== Signing server setup ===
Ever since we started signing all builds, a proper signing server setup in your dev master is necessary to do proper tests. The signing servers are configured in passwords.py and require real secrets (meaning, do not distribute them publicly). The [https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/passwords.py.template templatized version of passwords.py] will show you the format of the file, but you'll need to get the real values elsewhere. bhearsum's version is generally up-to-date and available at /builds/buildbot/bhearsum/builds/master/passwords.py on dev-master01. Note that in staging you must be careful not to use the Release or Nightly signing server instances, which will sign builds with real certificates (instead of self signed ones) -- so do NOT copy a passwords.py file from a production build master. If you have trouble with this, ask for help.
Ever since we started signing all builds, a proper signing server setup in your dev master is necessary to do proper tests. The signing servers are configured in passwords.py and require real secrets (meaning, do not distribute them publicly). The [https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/passwords.py.template templatized version of passwords.py] will show you the format of the file, but you'll need to get the real values elsewhere. bhearsum's version is generally up-to-date and available at /builds/buildbot/bhearsum/build1/master/passwords.py on dev-master2. Note that in staging you must be careful not to use the Release or Nightly signing server instances, which will sign builds with real certificates (instead of self signed ones) -- so do NOT copy a passwords.py file from a production build master. If you have trouble with this, ask for help.
 
=== Disable periodic builds ===
 
If you are testing changes on your build master, it can be annoying for pgo builds to be invoked and run on your master consuming your build machine resources. To avoid this, comment out lines that define pgo_strategy like this BRANCHES['mozilla-central']['pgo_strategy'] = 'periodic' or BRANCHES['mozilla-esr31']['pgo_strategy'] = 'per-checkin'. Conversely, if you are testing pgo builds, you can change the start_hour and start_minute on a per branch basis.


== Create a tests master ==
== Create a tests master ==


Use [https://hg.mozilla.org/users/john.hopkins_mozillamessaging.com/misc/file/default/buildbot-scripts/create-staging-master this automation script] to create a build master, or use the same approach as for build master.
We can create a test master using the same automation [https://hg.mozilla.org/build/braindump/file/tip/buildbot-related/create-staging-master.pl script] as for a build master:
<pre>
perl create-staging-master.pl -u aselagea --master-kind=test --master-dir=test1 --http-port=8049
</pre>


= Customize your configs =
= Customize your configs =
* In <code>master_localconfig.py</code>, adjust ports for <code>slavePortnum</code>, <code>webstatus</code> & <code>manhole</code>
* In <code>master/master_config.json</code>, adjust ports for <code>pb_port</code>, <code>http_port</code> & <code>ssh_port</code>
** Check [http://slavealloc.build.mozilla.org/ui/#masters slavealloc] to see what ports are in use by others. Search for 'staging-pers' and be sure to Show all Entries.
** Check [https://secure.pub.build.mozilla.org/slavealloc/ui/#masters slavealloc] to see what ports are in use by others. Search for 'staging-pers' and be sure to Show all Entries.
** Try to use similar ports, i.e. if you choose 9099 for slavePortnum, use 8099 for your webstatus and 1299 for your manhole.
** Try to use similar ports, i.e. if you choose 9099 for pb_port, use 8099 for your http_port and 7099 for your ssh_port.
* Consider [[#Optimizations|optimizing the number of branches]] you're running.
* Consider [[#Optimizations|optimizing the number of branches]] you're running.


Line 81: Line 97:
Do this:
Do this:
<pre>
<pre>
  WHO=coop
  WHO=$whoami
  mkdir -p /dev/shm/${WHO}/queue
  mkdir -p /dev/shm/${WHO}/queue
  chown ${WHO} /dev/shm/${WHO}
  chown ${WHO} /dev/shm/${WHO}
</pre>
</pre>
* modify your master/master_localconfig.py so that QUEUEDIR points to your new dir:
* modify your master/master_localconfig.py so that QUEUEDIR points to your new dir:
QUEUEDIR = "/dev/shm/${WHO}/queue"
    QUEUEDIR = "/dev/shm/${WHO}/queue"
* or (newer) <tt>master/master_config.json</tt> to add the line:
    "queuedir": "/dev/shm/${WHO}/queue"
 
= Test your buildbot patches =
Once you have created a build/test master, you can test various buildbot changes quite easily. Basic steps:
:* clone [https://hg.mozilla.org/build/braindump/ braindump] repo
:* cd to master's home folder && activate the virtual environment
:* make sure 'bb-configs' repo is up-to-date
<pre>
hg pull; hg update -C
</pre>
:* compute the initial list of builds/tests
<pre>
python ../braindump/buildbot-related/builder_list.py master/master.cfg > old
</pre>
:* apply your patch
<pre>
patch -p1 < <patch_name>.patch
</pre>
:* compute the list of builds/tests following your changes
<pre>
python ../braindump/buildbot-related/builder_list.py master/master.cfg > new
:* compute the diff between them
<pre>
diff -U 0 -p old new
</pre>
:*make sure your changes are not causing issues on the other masters
<pre>
bash buildbot-configs/test-masters.sh
</pre>


= Does it work? =
= Does it work? =
* If you want production schedulers to run, then "<tt>cd $BASEDIR/master; ln -sf ../buildbot-configs/mozilla/universal_master_sqlite.cfg master.cfg; cd -</tt>" (build master) or "<tt>cd $BASEDIR/master; ln -sf ../buildbot-configs/mozilla-tests/universal_master_sqlite.cfg master.cfg; cd -</tt>" (tests master). <small>Note: the default scheduler <code>builder_master.cfg</code> is usually fine.</small>
If you want production schedulers to run, then run the following:
** Read the above as "you '''always''' want <tt>universal_master_sqlite.cfg</tt> unless you have a good and solid reason for not wanting it <small>(or want to test your antacid options)</small>.
:* build master:
* Run "make checkconfig" to make sure the default configs are in place and good.
<pre>
* Run "make start" to make sure you master starts.
cd $BASEDIR/master; ln -sf ../buildbot-configs/mozilla/universal_master_sqlite.cfg master.cfg; cd -
** You should have a Makefile at $BASEDIR/Makefile to start/stop your master
</pre>
** Visit http://dev-master01.build.mozilla.org:${YOUR_WEBSTATUS_PORT}/waterfall
:* test master:
* If it works, congrats! Now run "make stop" to shut it down until you're ready to actually start testing.
 
== Sendchanges to your tests master from your build master ==
* If you're running *both* a build master and a tests master and want to test the sendchanges between them, you should have a similar local change on your build master (replace 9041 with the relevant port on your tests master):
 
<pre>
<pre>
diff --git a/mozilla/staging_config.py b/mozilla/staging_config.py
cd $BASEDIR/master; ln -sf ../buildbot-configs/mozilla-tests/universal_master_sqlite.cfg master.cfg; cd -
--- a/mozilla/staging_config.py
+++ b/mozilla/staging_config.py
@@ -49,3 +49,3 @@ GLOBAL_VARS = {
    'talos_masters': [
-        ('dev-master01.build.mozilla.org:9009', True, 1),
+        ('staging-master:9041', True, 1),
    ],
@@ -55,4 +55,3 @@ GLOBAL_VARS = {
    'unittest_masters': [
-        ('staging-master.build.mozilla.org:9009', True, 1),
-        ('talos-staging-master02.build.mozilla.org:9012', True, 1),
+        ('dev-master01:9041', True, 1),
        ],
</pre>
</pre>
<small>Note: the default scheduler <code>builder_master.cfg</code> is usually fine.</small>
:* Read the above as "you '''always''' want <tt>universal_master_sqlite.cfg</tt> unless you have a good and solid reason for not wanting it <small>(or want to test your antacid options)</small>.
:* Run "make checkconfig" to make sure the default configs are in place and good.
:* Run "make start" to make sure you master starts.
:** You should have a Makefile at $BASEDIR/Makefile to start/stop your master
:** Visit http://dev-master2.bb.releng.use1.mozilla.com:${YOUR_WEBSTATUS_PORT}/waterfall
:* If it works, congrats! Now run "make stop" to shut it down until you're ready to actually start testing.


= Adding your master to slavealloc =
= Adding your master to slavealloc =
ssh -l root slavealloc.build.mozilla.org
To lock slaves onto your master you need to add it to slavealloc, see [[ReleaseEngineering:Buildduty:Slave_Management#Adding_a_master | adding a master]]
su slavealloc
cd ~
vim armenzg-master.csv
# create a CSV similar to this one
# make sure you specify 'staging-pers' for staging masters
nickname,fqdn,http_port,pb_port,datacenter,pool
sm-armenzg-build,dev-master01.build.scl1.mozilla.com,8040,9040,scl1,staging-pers
sm-armenzg-test,dev-master01.build.scl1.mozilla.com,8041,9041,scl1,staging-pers
# import your master into the DB
# look at history to see what the value of PASSWORD should be
# 10.2.70.105 == tm-b01-master01.mozilla.org
# or ask someone in RelEng for the buildslaves DB password
/tools/slavealloc/bin/slavealloc dbimport \
-D mysql://buildslaves:PASSWORD@10.22.70.59/buildslaves \
--master-data armenzg-master.csv
 
Note: for latest info on csv file format, or to add slaves, see [http://hg.mozilla.org/build/tools/file/tip/lib/python/slavealloc/scripts/dbimport.py dbimport.py]


= Modifying your old master in slavealloc =
= Modifying your old master in slavealloc =
If you have an old master you can modify it:
If you have an old master you can modify it:
* Visit http://slavealloc.build.mozilla.org/ui/#masters
* Visit http://slavealloc.build.mozilla.org/ui/#masters
* Modify the FQDN of the master and the port (e.g. dev-master01.build.mozilla.org)
* Modify the FQDN of the master and the port (e.g. dev-master2.bb.releng.use1.mozilla.com)
 
= Setup/Running local master scheduler on laptop - not dev-master =
<ol><li> Download [http://hg.mozilla.org/build/braindump/raw-file/default/buildbot-related/fresh_bb_setup fresh_bb_setup] to ~/</li>
<li>bash ./fresh_bb_setup Buildbot</li>
<li>cd Buildbot; source bin/activate</li>
<li>cd buildbot-configs; ./test-masters.sh</li>
<ul><li><strong>NOTE:</strong> you can stop here if your only goal is to test config changes. Make them in this directory, then run <code>./test-masters.sh</code>.</li></ul>
<li>cd ~</li>
<li>cd BASEDIR/build-master/</li>
<li>Modify line 2 of Makefile from BUILDBOT=$(PWD)/bin/buildbot to be BUILDBOT=$(PWD)/../bin/buildbot</li>
<li>Modify line 11 of master_config.json so that 'release_branches' contains an empty list. It did contain ["mozilla-1.9.2",  "mozilla-beta"] and they are unnecessary for a local setup.</li>
<li>make checkconfig
* Should say: Config file is good!</li>
<li>make start
<ul><li> It will appear as if start has failed, but fear not! You'll see the message below, but this is simply because the buildbot process is launched as a dameon and buildmaster waits to give the all configuration is complete signal. However, Mozilla's configuration files are way to complicated to be dealt with in under 10 seconds, so it times out and you see the message below:
<pre>(Buildbot)localhost:build-master jzeller$ make start
cd master && /Users/jzeller/Buildbot/build-master/../bin/buildbot  start $PWD
Following twistd.log until startup finished..
2014-01-22 14:47:30-0800 [-] Log opened.
2014-01-22 14:47:30-0800 [-] twistd 12.0.0 (/Users/jzeller/Buildbot/bin/python 2.6.7) starting up.
2014-01-22 14:47:30-0800 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2014-01-22 14:47:30-0800 [-] monkeypatch_twisted_cbLogin applied
2014-01-22 14:47:30-0800 [-] Creating BuildMaster — buildbot.version: 0.8.2
2014-01-22 14:47:30-0800 [-] loading configuration from /Users/jzeller/Buildbot/build-master/master.cfg
2014-01-22 14:47:30-0800 [-] unable to import dnotify, so Maildir will use polling instead
2014-01-22 14:47:30-0800 [-] nextAWSSlave: start
2014-01-22 14:47:30-0800 [-] nextAWSSlave: start
2014-01-22 14:47:37-0800 [-] nextAWSSlave: start
2014-01-22 14:47:37-0800 [-] nextAWSSlave: start
 
The buildmaster took more than 10 seconds to start, so we were unable to
confirm that it started correctly. Please 'tail twistd.log' and look for a
line that says 'configuration update complete' to verify correct startup.
 
make: *** [start] Error 1</pre></li></ul>
<li>To doublecheck that the configuration was successful, just follow the directions and type: less twistd.log | grep "configuration update complete". Make sure that the most recent timestamp matches the time with which you tried a make start, but as long as you see it pop up then you're good!</li>
<li>Go to http://localhost:8501/</li>
<li>If you see a page with "Welcome to the Buildbot for the Firefox project!" then you have been successful!</li>
<li>You can now check out what buildbot has pending by going to http://localhost:8501/waterfall</li></ol>


= Setting up a local slave =
= Setting up a local slave =
Line 182: Line 237:
Using the slaves interface of slavealloc (http://slavealloc.build.mozilla.org/ui/#slaves), search for slaves that are currently in the preprod (or preprod-tests) pool that are not currently locked to someone else's dev master. Edit the slave and lock that slave to your master.  
Using the slaves interface of slavealloc (http://slavealloc.build.mozilla.org/ui/#slaves), search for slaves that are currently in the preprod (or preprod-tests) pool that are not currently locked to someone else's dev master. Edit the slave and lock that slave to your master.  


'''NOTE''': the slave will need to be rebooted to pick up this change.
'''NOTE''': the slave will need to be rebooted to pick up this change. If no slaves are available for your required platform, find out who has them all and ask to borrow one.
 
If no slaves are available for your required platform, find out who has them all and ask to borrow one.


= Setup a user source repository =
= Setup a user source repository =
Depending on how invasive your changes are going to be, you can choose to setup your own user repo for any of the releng repos or even the Firefox code repos.  
Depending on how invasive your changes are going to be, you can choose to setup your own user repo for any of the releng repos or even the Firefox code repos.  


Follow the instructions for [https://developer.mozilla.org/en/Publishing_Mercurial_ClonesPublishing Mercurial Clones] and then replace references to releng repos in the configs with your own repo.
Follow the instructions for [https://developer.mozilla.org/en/Publishing_Mercurial_Clones Publishing Mercurial Clones] and then replace references to releng repos in the configs with your own repo.
 
As of 2012-12-06, you need a mozharness user repo just run b2g compile jobs in staging. It is the only place to change the upload hosts to the staging host.


= Optimizations =
= Optimizations =
Line 196: Line 251:
* Comment out/change ACTIVE_{BRANCHES,PLATFORMS,PROJECTS} in master_localconfig.py as needed to reduce your workspace.
* Comment out/change ACTIVE_{BRANCHES,PLATFORMS,PROJECTS} in master_localconfig.py as needed to reduce your workspace.
** e.g. I usually run with only one branch: ACTIVE_BRANCHES = ['mozilla-central']
** e.g. I usually run with only one branch: ACTIVE_BRANCHES = ['mozilla-central']
==master_config.json==
* Limit the number of platforms your dev-master will run tests on.  For instance, if you only want to run tests for Android devices, add this to master_config.json
<pre>
"limit_fx_platforms": [],
  "limit_mobile_platforms": [
    "android",
    "android-armv6",
    "android-noion"
  ],
  "limit_tb_platforms": [],
  "limit_b2g_platforms": [],
</pre>
==BuildSlaves.py==
* Update with the appropriate staging passwords so your slaves can connect
==staging_config.py==
* Update with the STAGING_SLAVES dictionary with the  appropriate type and range of devices that you want to connect to you connect to your master
== Resetting db state ==
This is generally most useful when you want to get rid of all pending builds. First, add this to your master's Makefile:
<pre>
reset-db:
rm master/state.sqlite
cd master && $(BUILDBOT) upgrade-master $$PWD
</pre>
Then run:
<pre>
  make stop
  make reset-db
  make start
</pre>
== Getting build notifications ==
It might be useful to know when a build finishes or the status of certain builds. This way you don't have to keep an eye on your dev master and move on to other awesome things. There are a few ways to accomplish this:
:* Using Buildbot's IRC bot
:** add the following to your list of status obj in your master.cfg:
<pre>
  from buildbot.status import words
  ...
  irc = words.IRC("irc.mozilla.org", "<your-bot-nick>",
                  channels=["<irc-channel>"],  useSSL=True,
                  port=6697, allowForce=False,
                  notify_events={
                    'exception': 1,
                    'successToFailure': 1,
                    'failureToSuccess': 1,
                  })
  c['status'].append(irc)
</pre>
::* now reconfig and your should see your bot join <irc-channel>. From there you can do a number of things: http://docs.buildbot.net/0.8.0/IRC-Bot.html
::* I like to start a private chat with my bot and request the build status when a job finishes. This gives me actual pings! You can do this like so from irc:
<pre>
  > /query <your-bot-nick>
  > notify on finished
</pre>
:* Using Check4Change Firefox extension:
:** this extension will check if a page changes without having to actually refresh! You can hook this up to your dev master buildbot web UI
:** for more details: http://www.check4change.com/


= See Also =
= See Also =
Line 201: Line 321:
* http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla
* http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla
* http://coop.deadsquid.com/2011/10/thinnerbox-updated/
* http://coop.deadsquid.com/2011/10/thinnerbox-updated/
* [https://hg.mozilla.org/build/braindump/file/d84e1ece72c6/community/create_community_slaves_and_masters.sh Script to setup staging local buildbot masters and slaves]
Aselagea
148

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/484620...1184570"