ReleaseEngineering:TestingTechniques

vimdiff

Did you know vim has a diff mode? Try running vimdiff file1 file2

Most of the suggestions below combine nicely with vimdiff, or some other diff utility. Get a dump of the output prior to your changes, and then a dump of the output after your changes and run them through vimdiff or your favourite diff utility.

test-masters.sh

buildbot-configs includes a script called test-masters.sh. It will iterate through the list of buildbot master configurations and run buildbot checkconfig on each.

Please run this before asking for review, or landing changes!

config.py is executable!

You can run config.py in the mozilla/ or mozilla-tests/ directories and have it spit out the complete configuration for all branches. e.g.

ln -sf production_config localconfig.py
python config.py > orig.txt
hg qpush # Apply my awesome changes
python config.py > new.txt
vimdiff orig.txt new.txt

builder_list.py / dump_master.py

Our braindump repo has a few useful scripts to help with testing: builder_list.py and dump_master.py.

builder_list.py will take a buildbot master.cfg file as input and output the list of all builders with steps to stdout.

dump_master.py will take a buildbot master.cfg file as input and output the list of all builders with steps, all change sources, all schedulers, and all status plugins to stdout.

To be really thorough, you can run dump_masters.sh, which is like test-masters.sh, except instead of doing just a buildbot checkconfig, it runs dump_master.py on each master configuration. This can take a while to run (over a minute), but is a great way to make sure your changes don't have unintended side-effects.

Be not afraid!

... of hacking things to make testing easier. Here's an example of a simple script to output the command for certain status objects. I used this to make sure that my patch was only changing the command for a subset of the status objects.

import re

# Load the buildbot master config
execfile("master.cfg")

for s in c['status']:
    # We only care about status plugins with a 'command' attribute here
    if hasattr(s, 'command'):
        # Strip out 'instance at 0x12345678', it changes between runs
        s = "%s %s %s" % (s, s.command, s.builders)
        s = re.sub("instance at 0x[0-9a-f]+", "", s)
        print s