ReleaseEngineering/Applications/BuildbotBridge

Buildbot Bridge (BBB)

The Buildbot Bridge is a set of services that allow us to schedule jobs in Taskcluster but run them in Buildbot. The Taskcluster Listener listens for pending Taskcluster Tasks and creates BuildRequests for them. The Buildbot Listener listens for Buildbot events for those jobs, and updates the Taskcluster Tasks accordingly. The Reflector reclaims Taskcluster Tasks and polls for changes that can't be detected by listening for Taskcluster on Buildbot events.

Interactions with other systems

The Bridge interacts with and has credentials for many different systems:

• Taskcluster - To claim and resolve Tasks
• Buildbot Scheduler DB - To create BuildRequests
• Buildbot Bridge Database - To track ongoing jobs
• Pulse - To subscribe to Taskcluster and Buildbot exchanges
• Self Serve - To cancel Builds and BuildRequests

Development

To do development of the Buildbot Bridge is it recommended that you run your own versions of each of its components. You will need a few other things before this will work:

• Taskcluster credentials
• ReleaseEngineering/How_To/Setup_Personal_Development_Master A Build and/or Test Buildbot master
  • You must be running command_runner.py and pulse_publisher.py as well.
  • If you need both, they must be pointed at the same database
• [A Pulse account]

Once you have those, adjust the config to use them as well as your own provisioner id, worker group, and worker id. For example, here is the config that bhearsum uses:

{
    "taskcluster_queue_config": {
        "credentials": {
            "clientId": "<redacted>",
            "accessToken": "<redacted>"
        }
    },
    "buildbot_scheduler_db": "sqlite:////builds/buildbot/bhearsum/build1/master/state.sqlite",
    "bbb_db": "sqlite:///bbb.db",

    "pulse_user": "bhearsum-publisher",
    "pulse_password": "<redacted>",
    "pulse_queue_basename": "queue/bhearsum-publisher",

    "allowed_builders": [
        ".*"
    ],

    "tclistener": {
        "pulse_exchange_basename": "exchange/taskcluster-queue/v1",
        "worker_type": "buildbot-bridge-bhearsum",
        "provisioner_id": "buildbot-bridge-bhearsum",
        "selfserve_url": "http://foo.com/junk",
        "logfile": "tclistener.log"
    },
    "bblistener": {
        "pulse_exchange": "exchange/bhearsum-publisher/buildbot",
        "tc_worker_group": "buildbot-bridge-bhearsum",
        "tc_worker_id": "buildbot-bridge-bhearsum",
        "logfile": "bblistener.log"
    },
    "reflector": {
        "interval": 60,
        "logfile": "reflector.log"
    }
}

Note that the selfserve_url is junk. This is because there is no staging vesrion of BuildAPI (like most of the rest of our CI infrastructure), so you'll be unable to test changes that depend on it in dev unless you set-up your own instance.

Deployment

The Buildbot Bridge services run on multiple machines for redundancy and increased throughput. The installations are fully deployed and managed by Puppet. The running services are managed by supervisord. You can find them in "/builds/bbb" on the following Buildbot masters:

• buildbot-master70.bb.releng.use1.mozilla.com
• buildbot-master72.bb.releng.usw2.mozilla.com
• buildbot-master82.bb.releng.scl3.mozilla.com

How to update

To deploy new Buildbot Bridge code you must generate a new Python package and have Puppet deploy it. Once your code has been reviewed and landed, do the following to deploy it:

1. Bump the version in setup.py
2. Run "python setup.py sdist" to generate a new tarball.
3. Copy the tarball to the puppet server
4. Update the package list in the Buildbot Bridge puppet module
5. Wait for Puppet to update the installations and restart the instances

How to restart the services

If you need to restart the Buildbot Bridge for any reason, you can do so by running the following as root (don't forget it runs on three different machines):

for i in bblistener tclistener reflector; do supervisorctl restart buildbot_bridge_$i; done

What to expect in /builds/bbb

root@buildbot-master70.bb.releng.use1.mozilla.com bbb]# ls -tlr
total 2756
drwxr-xr-x 3 cltbld cltbld    4096 May  5 08:18 lib
lrwxrwxrwx 1 cltbld cltbld      15 May  5 08:18 lib64 -> /builds/bbb/lib
drwxr-xr-x 2 cltbld cltbld    4096 May  5 08:18 include
-rw------- 1 cltbld cltbld    1241 May 21 05:41 config.json
drwxr-xr-x 2 cltbld cltbld    4096 May 25 07:07 bin
-rw-r--r-- 1 cltbld cltbld 2372446 May 25 10:55 reflector.log
-rw-r--r-- 1 cltbld cltbld  359635 May 25 11:48 bblistener.log
-rw-r--r-- 1 cltbld cltbld   56047 May 26 07:12 tclistener.log

As you can see, each service has its own log file. The supervisord logs sometimes have additional information in error cases, and can be found in /var/log/supervisord.