List builders added/removed
You can read the blog post in here.
Download your patch and run it against list_builder_differences.sh. It will create a clean buildbot environment under $HOME/.mozilla/releng and determine what your patch adds/removes.
NOTE: This only works with buildbot-configs patches.
hg clone http://hg.mozilla.org/build/braindump braindump/community/list_builder_differences.sh -j path_to_patch.diff
Did you know vim has a diff mode? Try running vimdiff file1 file2
Most of the suggestions below combine nicely with vimdiff, or some other diff utility. Get a dump of the output prior to your changes, and then a dump of the output after your changes and run them through vimdiff or your favourite diff utility. Since the diff lines can be lengthy, "wdiff -3" can be helpful to hilight not just what lines changed, but exactly what changed on each line.
buildbot-configs includes a script called test-masters.sh. It will iterate through the list of buildbot master configurations and run buildbot checkconfig on each.
NOTE: You want your PYTHONPATH to point to the path that contains your buildbotcustom checkout and lib/python inside of the tools repo. For example:
where ~/repos is where buildbotcustom and tools are checked out.
Run the tests like this:
If you don't have buildbot on your path, then you should do
./test-masters.sh -b /path/to/buildbot
eg .../bin/buildbot in a master you already have set up (see below).
Please run this before asking for review, or landing changes!
config.py is executable!
You can run config.py in the mozilla/ or mozilla-tests/ directories and have it spit out the complete configuration for all branches. e.g.
ln -sf production_config localconfig.py python config.py > orig.txt hg qpush # Apply my awesome changes python config.py > new.txt vimdiff orig.txt new.txt
setup one master and output the steps for it
cd ~/repos/buildbot-confgs # run ./setup-master.py --list to see the list of masters rm -rf master_dir; mkdir master_dir; python setup-master.py master_dir pm01-builder cd master_dir python ~/repos/releng/braindump/buildbot-related/dump_master.py master.cfg > old # apply your changes python ~/repos/releng/braindump/buildbot-related/dump_master.py master.cfg > new # compare old and new
and here is the changes I do to brainddump:
diff --git a/buildbot-related/dump_master.py b/buildbot-related/dump_master.py --- a/buildbot-related/dump_master.py +++ b/buildbot-related/dump_master.py @@ -31,4 +31,5 @@ print "Builders:" for b in g['c']['builders']: print b['name'], b['factory'].__class__.__name__ +''' for s in b['factory'].steps: step_class, args = s @@ -65,2 +66,3 @@ for s in g['c']['status']: d[a] = getattr(s, a) print format_objs(s, d) +'''
builder_list.py / dump_master.py
Our braindump repo has a few useful scripts to help with testing: builder_list.py and dump_master.py.
builder_list.py will take a buildbot master.cfg file as input and output the list of all builders with steps to stdout.
export PYTHONPATH=:/builds/buildbot/kmoir/test2:/builds/buildbot/kmoir/tools/lib/python:/builds/buildbot/kmoir/test2:/builds/buildbot/kmoir/tools/lib/python source ./bin/activate /builds/buildbot/kmoir/test2/braindump/buildbot-related/builder_list.py master/master.cfg > new hg up -C /builds/buildbot/kmoir/test2/braindump/buildbot-related/builder_list.py master/master.cfg > old diff -pU 8 old new > differences
dump_master.py will take a buildbot master.cfg file as input and output the list of all builders with steps, all change sources, all schedulers, and all status plugins to stdout.
To be really thorough, you can run dump_masters.sh, which is like test-masters.sh, except instead of doing just a buildbot checkconfig, it runs dump_master.py on each master configuration. This can take a while to run (over a minute), but is a great way to make sure your changes don't have unintended side-effects.
This is an example on how to use it:
cd ~/repos/buildbot-configs ~/repos/releng/braindump/buildbot-related/dump_masters.sh > new hg up -C ~/repos/releng/braindump/buildbot-related/dump_masters.sh > old diff -pU 8 old new > differences # or vimdiff old new
Be not afraid!
... of hacking things to make testing easier. Here's an example of a simple script to output the command for certain status objects. I used this to make sure that my patch was only changing the command for a subset of the status objects.
import re # Load the buildbot master config execfile("master.cfg") for s in c['status']: # We only care about status plugins with a 'command' attribute here if hasattr(s, 'command'): # Strip out 'instance at 0x12345678', it changes between runs s = "%s %s %s" % (s, s.command, s.builders) s = re.sub("instance at 0x[0-9a-f]+", "", s) print s
How to test a talos change and use compare-talos
- update your mozilla-central checkout & take note of the revision (oldRevision)
- push a build through the try server (this gives you a unique &rev= on Try - newRevision) indicating the platform you want (see TryChooser builder site) and *no* talos results (we want to force the talos testing on our preprod slave)
hg ci -m "try: -b o -p win32 -u none -t none" hg push -f try # Verify
- make your personal master to point to the Try tree and to the production graphs DB
- TODO document this step
- do a talos sendchange to your personal master pointing to the try build once completed
- TODO document this step
- once your talos jobs are completed use compare-talos to compare the old and new reviions
- load http://perf.snarkfest.net/compare-talos/
- add the oldRevision and newRevision and submit
- you now have the talos variances