User:Catlee/BuildScripts

This page is an exploration of some ideas of how to clean up our build infrastructure, in particular the set of commands that get run to produce a build, or run a set of tests.

Motivation

• module and class dependencies in buildbotcustom has become very complicated
• testing changes to classes in buildbotcustom isn't feasible outside of a buildbot environment
• buildbotcustom is opaque to outside contributors
• Making minor changes that affect only one platform or branch is very complicated in the current setup, requiring a lot of testing across all potentially affected platforms and branches.

Idea 1

Have a new repository, let's call it build/build-scripts. This repository is laid out similarly to how our mozconfigs are organized:

build-scripts/mozilla-central/linux/debug
build-scripts/mozilla-central/linux/nightly

Each directory would contain a config file which would specify the steps to run, as well as things like the mozconfig and helper scripts.

To run a build, the slave would checkout a harness script, along with the build scripts. The harness is then pointed at the config file to execute the required steps.

The "build configuration", which contains things like which revision to build, gets written somewhere (like /tmp?). An environment variable is set to indicate this location ($BUILD_CONFIG). It could be nice to have this configuration written out in multiple formats, e.g. config.sh suitable for source'ing in a bash script, config.py suitable for importing in a python script, and config.json suitable for other purposes.

The harness will also create a temporary working directory where scripts are free to save data for use later in the build, e.g. recording changesets, hashes, or package urls.

Example

mozilla-central/linux/depend/config.ini:

[general]
steps = cleanup, checkout, mozconfig, build, package, upload

[env]
MOZ_OBJDIR = obj-firefox
PATH = /usr/local/bin:/usr/bin:/bin

[cleanup]
timeout = 1200

[checkout]
timeout = 1200

[mozconfig]
command = cp %(CONFIG_DIR)s/mozconfig mozilla-central/.mozconfig

[build]
command = make -C mozilla-central -f client.mk

[package]
command = make -C mozilla-central/obj-firefox package

[package:env]
UPLOAD_HOST = staging-stage.build.mozilla.org

mozilla-central/linux/depend/checkout:

#!/bin/sh
set -e
test -f ${BUILD_CONFIG}.sh && . ${BUILD_CONFIG}.sh

if [ -n $REV ]; then
    REV=default
fi

if [ -d mozilla-central ]; then
    cd mozilla-central
    hg pull $HGURL
    hg update -C -r $REV
else
    hg clone $HGURL mozilla-central
fi

steps with no 'command' set expect to find a script with the same name of the step in the configuration directory.

To run this build, you would run:

harness.py -f mozilla-central/linux/depend/config.ini -C build/mozilla-central/depend HGURL=http://hg.mozilla.org/mozilla-central REV=123456

Open Questions

• We'd lose per-step logging and status as far as buildbot is concerned. Does this matter?
• How to share common scripts and configs?