ReleaseEngineering/How To/Setup Personal Development Master: Difference between revisions

← Older edit

ReleaseEngineering/How To/Setup Personal Development Master (view source)

Revision as of 14:44, 21 November 2017

614 bytes removed , 21 November 2017

Updated instructions on how to setup a personal dev master

Aselagea

148

edits

@@ Line 9: / Line 9: @@
-'''NOTE''': You will need root access to create a user account for yourself on dev-master1 and a buildbot area under /builds/buildbot/your_name.<br />
+'''NOTE''': You will need root access to create a user account for yourself on dev-master2 and a buildbot area under /builds/buildbot/your_name - if not using root, you might get this error (I think it has to do with which group your account belongs to):
-Ask someone with root access to set it up following instructions in [https://intranet.mozilla.org/Build/Mentor_Checklist here].<br />
+<pre>
-If you are setting dev-master1 you probably want to use root to set up an account in /builds/buildbot/your_name since you might get this error (I think it has to do with which group your account belongs to):
+  from twisted.python.zippath import ZipArchive
- from twisted.python.zippath import ZipArchive
+    ImportError: No module named zippath
-  ImportError: No module named zippath
+</pre>
 == General ==
-* Instructions specific to dev-master1 (dev-master1.build.mozilla.org)
+* Instructions specific to dev-master2 (dev-master2.bb.releng.use1.mozilla.com)
 <pre>
   # as root
-  WHO=coop
+  WHO=$whoami
   mkdir /builds/buildbot/${WHO}
   chown ${WHO} /builds/buildbot/${WHO}
@@ Line 27: / Line 26: @@
 ** BASEDIR refers to where you want the new master to be created
 ** MASTER_NAME is the name of one of the predefined masters.  Search for a name prefixed by "staging".
-*** /usr/bin/python2.6 setup-master.py -l    -- display a list of enabled masters
+*** display a list of enabled masters: /usr/bin/python2.6 setup-master.py -l
-*** /usr/bin/python2.6 setup-master.py -l -8 -- display a hardcoded list of 0.8.0 masters
+*** display a hardcoded list of 0.8.0 masters: /usr/bin/python2.6 setup-master.py -l -8
 ** PYTHON, HG, VIRTUALENV should be adjusted to your local environment
 == Create a build master ==
+There are two ways to do it:
-Use [https://hg.mozilla.org/build/braindump/file/tip/buildbot-related/create-staging-master.pl this automation script] to create a build master, or follow the manual steps below.
+* using an automation [https://hg.mozilla.org/build/braindump/file/tip/buildbot-related/create-staging-master.pl script]:
+<pre>
-First, clone the <code>buildbot-configs</code> repository.
+perl create-staging-master.pl -u aselagea --master-kind=build --master-dir=build1 --http-port=8021
+</pre>
+* following several manual steps:
+:* clone the <code>buildbot-configs</code> repository.
 <pre>
 $ hg clone https://hg.mozilla.org/build/buildbot-configs
 </pre>
+:* add <code>'try'</code> to the list of <code>ACTIVE_BRANCHES</code> in <code>master/master_localconfig.py</code>.
-Add <code>'try'</code> to the list of <code>ACTIVE_BRANCHES</code> in <code>master/master_localconfig.py</code>.
+:* run <code>create_master.sh</code> below
-<code>create_master.sh</code>
 <pre>
 USERNAME=`whoami`
@@ Line 69: / Line 68: @@
 === build-master path values ===
 * Configs: buildbot-configs/mozilla/
-* master_localconfig: ???
+* master_localconfig: build_localconfig.py ?
 * host: preprod-builder
 === Signing server setup ===
-Ever since we started signing all builds, a proper signing server setup in your dev master is necessary to do proper tests. The signing servers are configured in passwords.py and require real secrets (meaning, do not distribute them publicly). The [https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/passwords.py.template templatized version of passwords.py] will show you the format of the file, but you'll need to get the real values elsewhere. bhearsum's version is generally up-to-date and available at /builds/buildbot/bhearsum/build1/master/passwords.py on dev-master1. Note that in staging you must be careful not to use the Release or Nightly signing server instances, which will sign builds with real certificates (instead of self signed ones) -- so do NOT copy a passwords.py file from a production build master. If you have trouble with this, ask for help.
+Ever since we started signing all builds, a proper signing server setup in your dev master is necessary to do proper tests. The signing servers are configured in passwords.py and require real secrets (meaning, do not distribute them publicly). The [https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/passwords.py.template templatized version of passwords.py] will show you the format of the file, but you'll need to get the real values elsewhere. bhearsum's version is generally up-to-date and available at /builds/buildbot/bhearsum/build1/master/passwords.py on dev-master2. Note that in staging you must be careful not to use the Release or Nightly signing server instances, which will sign builds with real certificates (instead of self signed ones) -- so do NOT copy a passwords.py file from a production build master. If you have trouble with this, ask for help.
 === Disable periodic builds ===
@@ Line 81: / Line 80: @@
 == Create a tests master ==
-Use [https://hg.mozilla.org/build/braindump/file/tip/buildbot-related/create-staging-master.pl this automation script] (recommended) to create a build master, or use the same approach as for build master.
+We can create a test master using the same automation [https://hg.mozilla.org/build/braindump/file/tip/buildbot-related/create-staging-master.pl script] as for a build master:
+<pre>
- perl create-staging-master.pl -u coop --master-kind=test --master-dir=tests-master --http-port=8045
+perl create-staging-master.pl -u aselagea --master-kind=test --master-dir=test1 --http-port=8049
+</pre>
 = Customize your configs =
@@ Line 97: / Line 97: @@
 Do this:
 <pre>
-  WHO=coop
+  WHO=$whoami
   mkdir -p /dev/shm/${WHO}/queue
   chown ${WHO} /dev/shm/${WHO}
 </pre>
 * modify your master/master_localconfig.py so that QUEUEDIR points to your new dir:
-       QUEUEDIR = "/dev/shm/${WHO}/queue"
+    QUEUEDIR = "/dev/shm/${WHO}/queue"
 * or (newer) <tt>master/master_config.json</tt> to add the line:
-      "queuedir": "/dev/shm/${WHO}/queue"
+    "queuedir": "/dev/shm/${WHO}/queue"
 = Test your buildbot patches =
 Once you have created a build/test master, you can test various buildbot changes quite easily. Basic steps:
-* clone 'braindump' repo: https://hg.mozilla.org/build/braindump/
+:* clone [https://hg.mozilla.org/build/braindump/ braindump] repo
-* cd to master's home folder && activate virtual env
+:* cd to master's home folder && activate the virtual environment
-* make sure 'bb-configs' repo is up-to-date
+:* make sure 'bb-configs' repo is up-to-date
-** simply run 'hg pull; hg update -C'
+<pre>
-* compute the initial list of builds/tests
+hg pull; hg update -C
-** e.g. "../braindump/buildbot-related/builder_list.py master/master.cfg > old"
+</pre>
-* apply your patch
+:* compute the initial list of builds/tests
-* compute the list of builds/tests following your changes
+<pre>
-** e.g. "../braindump/buildbot-related/builder_list.py master/master.cfg > new"
+python ../braindump/buildbot-related/builder_list.py master/master.cfg > old
-* compute the diff between them
+</pre>
+:* apply your patch
+<pre>
+patch -p1 < <patch_name>.patch
+</pre>
+:* compute the list of builds/tests following your changes
+<pre>
+python ../braindump/buildbot-related/builder_list.py master/master.cfg > new
+:* compute the diff between them
+<pre>
+diff -U 0 -p old new
+</pre>
+:*make sure your changes are not causing issues on the other masters
+<pre>
+bash buildbot-configs/test-masters.sh
+</pre>
 = Does it work? =
-* If you want production schedulers to run, then "<tt>cd $BASEDIR/master; ln -sf ../buildbot-configs/mozilla/universal_master_sqlite.cfg master.cfg; cd -</tt>" (build master) or "<tt>cd $BASEDIR/master; ln -sf ../buildbot-configs/mozilla-tests/universal_master_sqlite.cfg master.cfg; cd -</tt>" (tests master). <small>Note: the default scheduler <code>builder_master.cfg</code> is usually fine.</small>
+If you want production schedulers to run, then run the following:
-** Read the above as "you '''always''' want <tt>universal_master_sqlite.cfg</tt> unless you have a good and solid reason for not wanting it <small>(or want to test your antacid options)</small>.
+:* build master:
-* Run "make checkconfig" to make sure the default configs are in place and good.
+<pre>
-* Run "make start" to make sure you master starts.
+cd $BASEDIR/master; ln -sf ../buildbot-configs/mozilla/universal_master_sqlite.cfg master.cfg; cd -
-** You should have a Makefile at $BASEDIR/Makefile to start/stop your master
+</pre>
-** Visit http://dev-master1.build.mozilla.org:${YOUR_WEBSTATUS_PORT}/waterfall
+:* test master:
-* If it works, congrats! Now run "make stop" to shut it down until you're ready to actually start testing.
-== Sendchanges to your tests master from your build master ==
-* If you're running *both* a build master and a tests master and want to test the sendchanges between them, you should have a similar local change on your build master (replace 9041 with the relevant port on your tests master):
 <pre>
-diff --git a/mozilla/staging_config.py b/mozilla/staging_config.py
+cd $BASEDIR/master; ln -sf ../buildbot-configs/mozilla-tests/universal_master_sqlite.cfg master.cfg; cd -
---- a/mozilla/staging_config.py
-+++ b/mozilla/staging_config.py
-@@ -49,3 +49,3 @@ GLOBAL_VARS = {
-     'talos_masters': [
--        ('dev-master1.build.mozilla.org:9009', True, 1),
-+        ('staging-master:9041', True, 1),
-     ],
-@@ -55,4 +55,3 @@ GLOBAL_VARS = {
-     'unittest_masters': [
--        ('staging-master.build.mozilla.org:9009', True, 1),
--        ('talos-staging-master02.build.mozilla.org:9012', True, 1),
-+        ('dev-master1:9041', True, 1),
-         ],
 </pre>
+<small>Note: the default scheduler <code>builder_master.cfg</code> is usually fine.</small>
+:* Read the above as "you '''always''' want <tt>universal_master_sqlite.cfg</tt> unless you have a good and solid reason for not wanting it <small>(or want to test your antacid options)</small>.
+:* Run "make checkconfig" to make sure the default configs are in place and good.
+:* Run "make start" to make sure you master starts.
+:** You should have a Makefile at $BASEDIR/Makefile to start/stop your master
+:** Visit http://dev-master2.bb.releng.use1.mozilla.com:${YOUR_WEBSTATUS_PORT}/waterfall
+:* If it works, congrats! Now run "make stop" to shut it down until you're ready to actually start testing.
 = Adding your master to slavealloc =
@@ Line 154: / Line 158: @@
 If you have an old master you can modify it:
 * Visit http://slavealloc.build.mozilla.org/ui/#masters
-* Modify the FQDN of the master and the port (e.g. dev-master1.build.mozilla.org)
+* Modify the FQDN of the master and the port (e.g. dev-master2.bb.releng.use1.mozilla.com)
 = Setup/Running local master scheduler on laptop - not dev-master =
@@ Line 233: / Line 237: @@
 Using the slaves interface of slavealloc (http://slavealloc.build.mozilla.org/ui/#slaves), search for slaves that are currently in the preprod (or preprod-tests) pool that are not currently locked to someone else's dev master. Edit the slave and lock that slave to your master.
-'''NOTE''': the slave will need to be rebooted to pick up this change.
+'''NOTE''': the slave will need to be rebooted to pick up this change. If no slaves are available for your required platform, find out who has them all and ask to borrow one.
-If no slaves are available for your required platform, find out who has them all and ask to borrow one.
 = Setup a user source repository =
@@ Line 275: / Line 277: @@
 == Resetting db state ==
 This is generally most useful when you want to get rid of all pending builds. First, add this to your master's Makefile:
- reset-db:
+<pre>
+reset-db:
   	rm master/state.sqlite
   	cd master && $(BUILDBOT) upgrade-master $$PWD
+</pre>
 Then run:
- make stop
+<pre>
- make reset-db
+  make stop
- make start
+  make reset-db
+  make start
+</pre>
 == Getting build notifications ==
 It might be useful to know when a build finishes or the status of certain builds. This way you don't have to keep an eye on your dev master and move on to other awesome things. There are a few ways to accomplish this:
-* Using Buildbot's IRC bot
+:* Using Buildbot's IRC bot
-**first add the following to your list of status obj in your master.cfg:
+:** add the following to your list of status obj in your master.cfg:
+<pre>
    from buildbot.status import words
    ...
@@ Line 300: / Line 305: @@
                    })
    c['status'].append(irc)
+</pre>
-now reconfig and your should see your bot join <irc-channel>. From there you can do a number of things: http://docs.buildbot.net/0.8.0/IRC-Bot.html
+::* now reconfig and your should see your bot join <irc-channel>. From there you can do a number of things: http://docs.buildbot.net/0.8.0/IRC-Bot.html
+::* I like to start a private chat with my bot and request the build status when a job finishes. This gives me actual pings! You can do this like so from irc:
-I like to start a private chat with my bot and request the build status when a job finishes. This gives me actual pings!
+<pre>
-you can do this like so from irc:
    > /query <your-bot-nick>
    > notify on finished
+</pre>
-* Using Check4Change Firefox extension:
+:* Using Check4Change Firefox extension:
-** this extension will check if a page changes without having to actually refresh! You can hook this up to your dev master buildbot web UI
+:** this extension will check if a page changes without having to actually refresh! You can hook this up to your dev master buildbot web UI
-** for more details: http://www.check4change.com/
+:** for more details: http://www.check4change.com/
 = See Also =