MozillaWiki

User:Bhearsum:Build/Buildserver ref image: Difference between revisions

User page Discussion

User:Bhearsum:Build/Buildserver ref image (view source)

Revision as of 21:43, 13 July 2007

8,653 bytes added ,  13 July 2007
m
removed wip
(wip)
m (removed wip)
 
(6 intermediate revisions by the same user not shown)
Line 1: Line 1:
<span style="font-size: 150%">WIP</span>
= Buildserver Ref Image =
= Buildserver Ref Image =
This document describes how to create a "Build server" ref image. More info is in [https://bugzilla.mozilla.org/show_bug.cgi?id=385911 bug#385911].
This document describes how to create a "Build server" ref image. More info is in [https://bugzilla.mozilla.org/show_bug.cgi?id=385911 bug#385911].
Line 21: Line 19:
# Choose location, Finish
# Choose location, Finish


Boot up the VM and 'mkdir /cvs'.
Create a single partition on the new hard drive (/dev/sdb) and run the following commands:
Create a single partition on the new hard drive (/dev/sdb).
mkdir /cvs
mkfs -t ext3 -j /dev/sdb1
mkfs -t ext3 -j /dev/sdb1
echo "/dev/sdb1 /cvs ext3 defaults 0 0" >> /etc/fstab
echo "/dev/sdb1 /cvs ext3 defaults 0 0" >> /etc/fstab
mount -a
mount -a


== Install and setup CVSD ==
== Install and setup CVSD ==
Install CVSD:
CVSD is a wrapper for pserver that allows it to run in a rootjail.
Here are the installation instructions:
  wget http://ch.tudelft.nl/~arthur/cvsd/cvsd-1.0.13.tar.gz
  wget http://ch.tudelft.nl/~arthur/cvsd/cvsd-1.0.13.tar.gz
  tar -zvxf cvsd-1.0.13.tar.gz
  tar -zvxf cvsd-1.0.13.tar.gz
Line 40: Line 39:
  cp /usr/local/etc/init.d/cvsd /etc/init.d
  cp /usr/local/etc/init.d/cvsd /etc/init.d
  chkconfig --add cvsd
  chkconfig --add cvsd
Edit /etc/init.d/cvsd and make sure 'CVSD_CFG' is set to '/etc/cvsd/cvsd.conf'.
Edit /etc/cvsd/cvsd.conf and change the following parameters:
Edit /etc/cvsd/cvsd.conf and change the following parameters:
  RootJail /cvs
  RootJail /cvs
Line 47: Line 47:
Run these commands:
Run these commands:
  cvsd-buildroot /cvs
  cvsd-buildroot /cvs
mkdir -p /cvs/var/lock
  adduser cvsd
  adduser cvsd


Line 57: Line 56:
  cd cvs-1.11.22/src
  cd cvs-1.11.22/src
  patch -p0 < ../../cvs.noreadlocks.patch
  patch -p0 < ../../cvs.noreadlocks.patch
cd ../
  ./configure --prefix=/cvs
  ./configure --prefix=/cvs
  make
  make
Line 80: Line 80:


== Create a buildbot master ==
== Create a buildbot master ==
  mkdir -p /buildbot/master
  mkdir -p /buildbot/default
  chown -R buildbot /buildbot
  chown -R buildbot /buildbot
  su - buildbot
  su - buildbot
  cd /buildbot/master
  cd /buildbot/default
  buildbot create-master .
  buildbot create-master .
  mv Makefile.sample Makefile
  mv Makefile.sample Makefile
rm master.cfg.sample
master.cfg:
master.cfg:
<pre>
<pre>
Line 107: Line 108:
####### PROJECT IDENTITY
####### PROJECT IDENTITY
c['projectName'] = "VMware Team Buildbot"
c['projectName'] = "VMware Team Buildbot"
c['buildbotURL'] = "[IP ADDRESS HERE]:8810"
c['slavePortnum'] = 9990
c['slavePortnum'] = 9990


Line 140: Line 140:
c['builders'] = []
c['builders'] = []


import mozbuild
reload(mozbuild)
from mozbuild import *
from mozbuild import *
from buildbot.process import factory
from buildbot.process import factory
Line 195: Line 197:


## END OF DEFAULT MASTER.CFG
## END OF DEFAULT MASTER.CFG
</pre>
mozbuild.py:
<pre>
# -*- Python -*-
from buildbot.process import step
from buildbot.process.step import ShellCommand
MozillaEnvironments = { }
# standard vc8 express build env; vc8 normal will be very similar, just different
# platform SDK location.  we can build both from one generic template.
MozillaEnvironments['vc8_express'] = {
    "MOZ_TOOLS": '/cygdrive/d/moztools',
    "VSINSTALLDIR": 'C:\\Program Files\\Microsoft Visual Studio 8',
    "VS80COMMTOOLS": 'C:\\Program Files\\Microsoft Visual Studio 8\\Common7\\Tools\\',
    "VCINSTALLDIR": 'C:\\Program Files\\Microsoft Visual Studio 8\\VC',
    "FrameworkDir": 'C:\\WINDOWS\\Microsoft.NET\\Framework',
    "FrameworkVersion": 'v2.0.50727',
    "FrameworkSDKDir": 'C:\\Program Files\\Microsoft Visual Studio 8\\SDK\\v2.0',
    "DevEnvDir": "C:\\Program Files\\Microsoft Visual Studio 8\\VC\\Common7\\IDE",
    "MSVCDir": 'C:\\Program Files\\Microsoft Visual Studio 8\\VC',
    "PATH": 'C:\\Program Files\\Microsoft Visual Studio 8\\Common7\\IDE;' + \
            'C:\\Program Files\\Microsoft Visual Studio 8\\VC\\bin;' + \
            'C:\\Program Files\\Microsoft Visual Studio 8\\VC\\PlatformSDK\\bin;' + \
            'C:\\Program Files\\Microsoft Visual Studio 8\\VC;' + \
            'C:\\Program Files\\Microsoft Visual Studio 8\\Common7\\Tools;' + \
            'C:\\Program Files\\Microsoft Visual Studio 8\\Common7\\Tools\\bin;' + \
            'd:\\moztools\\bin;' + \
            'd:\\cygwin\\bin;' + \
            'd:\\buildtools\\NSIS;' + \
            'd:\\buildtools\\7-zip;' + \
            'd:\\buildtools\\upx;' + \
    'C:\\WINDOWS\system32;',
    "INCLUDE": 'C:\\Program Files\\Microsoft Visual Studio 8\\VC\\ATLMFC\\INCLUDE;' + \
              'C:\\Program Files\\Microsoft Visual Studio 8\\VC\\INCLUDE;' + \
              'C:\\Program Files\\Microsoft Visual Studio 8\\VC\\PlatformSDK\\include',
    "LIB": 'C:\\Program Files\\Microsoft Visual Studio 8\\VC\\ATLMFC\\LIB;' + \
          'C:\\Program Files\\Microsoft Visual Studio 8\\VC\\LIB;' + \
          'C:\\Program Files\\Microsoft Visual Studio 8\\VC\\PlatformSDK\\lib'
}
class MozillaCheckoutClientMk(ShellCommand):
    haltOnFailure = True
    cvsroot = ":pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot"
   
    def __init__(self, **kwargs):
        if 'cvsroot' in kwargs:
            self.cvsroot = kwargs['cvsroot']
        if not 'command' in kwargs:
            kwargs['command'] = ["cvs", "-d", self.cvsroot, "co", "mozilla/client.mk"]
        ShellCommand.__init__(self, **kwargs)
    def describe(self, done=False):
        return ["client.mk update"]
class MozillaClientMkPull(ShellCommand):
    haltOnFailure = True
    def __init__(self, **kwargs):
        if not 'project' in kwargs or kwargs['project'] is None:
            self.project = "browser"
        else:
            self.project = kwargs['project']
            del kwargs['project']
        if not 'workdir' in kwargs:
            kwargs['workdir'] = "mozilla"
        if not 'command' in kwargs:
            kwargs['command'] = ["make", "-f", "client.mk", "pull_all"]
        env = {}
        if 'env' in kwargs:
            env = kwargs['env'].copy()
        env['MOZ_CO_PROJECT'] = self.project
        kwargs['env'] = env
        ShellCommand.__init__(self, **kwargs)
    def describe(self, done=False):
        if not done:
            return ["pulling (" + self.project + ")"]
        return ["pull (" + self.project + ")"]
class MozillaPackage(ShellCommand):
    name = "package"
    warnOnFailure = True
    description = ["packaging"]
    descriptionDone = ["package"]
    command = ["make"]
</pre>
== Set-up Buildbot to start with the system ==
Create the file '/etc/default/buildbot' with the following contents:
<pre>
# buildbots to manage
# add a new set of variables for each buildbot to start
# BB_NUMBER    -> index for the buildbot
# BB_NAME      -> short name that is printed when starting/stopping
# BB_USER      -> user to run the buildbot as
# BB_BASEDIR  -> the absolute path to the buildbot master or slave
# BB_OPTIONS  -> extra options to pass to buildbot
# BB_PREFIXCMD -> prefix command, ie. nice, linux32, etc.
#
# Each of the preceeding are arrays. Each Buildbot you wish to run should
# increase the index of each. For example, the first Buildbot should use
# [0] on each array. The next one uses [1], etc.
BB_NUMBER[0]=0
BB_NAME[0]="Default Buildbot"
BB_USER[0]="buildbot"
BB_BASEDIR[0]="/buildbot/default/"
BB_OPTIONS[0]=""
BB_PREFIXCMD[0]=""
</pre>
Create the file '/etc/init.d/buildbot' with the following contents:
<pre>
#! /bin/bash
# initscript for buildbot
### BEGIN INIT INFO
# Provides:          cvsd
# Required-Start:    $local_fs $network
# Required-Stop:    $local_fs
# Should-Start:      $remote_fs
# Should-Stop:      $remote_fs
# Default-Start:    2 3 4 5
# Default-Stop:      S 0 1 6
# Short-Description: Buildbot
# Description:      Buildbot
### END INIT INFO
PATH=/sbin:/bin:/usr/sbin:/usr/bin
DESC="BuildBot"
USER=buildbot
DAEMON=/tools/buildbot/bin/buildbot
DNAME=buildbot
PROCNAME=$NAME
[ -r /etc/default/buildbot ] && . /etc/default/buildbot
test -x ${DAEMON} || exit 0
. /lib/lsb/init-functions
check_config()
{
    errors=0
    for i in ${BB_NUMBER[@]}; do
[ $i -ge 0 ] || continue
if [ -z "${BB_NAME[$i]}" ]; then
    echo >&2 "buildbot $i: no name"
    errors=$(($errors+1))
fi
if [ -z "${BB_USER[$i]}" ]; then
    echo >&2 "buildbot $i: no user"
    errors=$(($errors+1))
elif ! getent passwd ${BB_USER[$i]} >/dev/null; then
    echo >&2 "buildbot $i: unknown user ${BB_USER[$i]}"
    errors=$(($errors+1))
fi
if [ ! -d "${BB_BASEDIR[$i]}" ]; then
    echo >&2 "buildbot $i: no base directory ${BB_BASEDIR[$i]}"
    errors=$(($errors+1))
fi
    done
    [ $errors -eq 0 ] || exit 1
}
check_config
start_buildbot() {
    NAME="$1"
    USER="$2"
    BASEDIR="$3"
    PREFIXCMD="$4"
    OPTIONS="$5"
    #START="--start --quiet --exec ${DAEMON} --name ${NAME} --pidfile ${BASEDIR}/twistd.pid"
    #[ -n "${USER}" ] && START="${START} --chuid ${USER}"
    #START="${START} -- start ${BASEDIR} ${OPTIONS}"
    #${PREFIXCMD} start-stop-daemon ${START} >/dev/null 2>&1
    ${PREFIXCMD} su -s /bin/bash -c "${DAEMON} start ${BASEDIR} ${OPTIONS}" - ${USER}
    return $?
}
stop_buildbot() {
    NAME="$1"
    USER="$2"
    BASEDIR="$3"
    PREFIXCMD="$4"
    ${PREFIXCMD} su -s /bin/bash -c "${DAEMON} stop ${BASEDIR}" - ${USER}
    return $?
}
reload_buildbot() {
    NAME="$1"
    USER="$2"
    BASEDIR="$3"
    PREFIXCMD="$4"
    ${PREFIXCMD} su -s /bin/bash -c "${DAEMON} sighup ${BASEDIR}" - ${USER}
    return $?
}
do_start () {
    errors=0
    for i in ${BB_NUMBER[@]}; do
[ $i -ge 0 ] || continue
echo "Starting buildbot ${BB_NAME[$i]}"
if start_buildbot "${BB_NAME[$i]}" "${BB_USER[$i]}" "${BB_BASEDIR[$i]}" \
    "${BB_PREFIXCMD[$i]}" "${BB_OPTIONS[$i]}"
then
    echo "started"
else
    echo "not started"
    errors=$(($errors+1))
fi
    done
    return $errors
}
do_stop () {
    errors=0
    for i in ${BB_NUMBER[@]}; do
[ $i -ge 0 ] || continue
echo "Stopping buildbot ${BB_NAME[$i]}"
if stop_buildbot "${BB_NAME[$i]}" "${BB_USER[$i]}" "${BB_BASEDIR[$i]}" \
    "${BB_PREFIXCMD[$i]}"
then
    echo "stopped"
else
    echo "not stopped"
    errors=$(($errors+1))
fi
    done
    return $errors
}
do_reload () {
    errors=0
    for i in ${BB_NUMBER[@]}; do
[ $i -ge 0 ] || continue
echo "Reload buildbot ${BB_NAME[$i]}"
if reload_buildbot "${BB_NAME[$i]}" "${BB_USER[$i]}" "${BB_BASEDIR[$i]}" \
    "${BB_PREFIXCMD[$i]}"
then
    echo "reloaded"
else
    echo "not reloaded"
    errors=$(($errors+1))
fi
    done
    return $errors
}
do_restart () {
    errors=0
    for i in ${BB_NUMBER[@]}; do
[ $i -ge 0 ] || continue
echo "Restarting buildbot ${BB_NAME[$i]}"
stop_buildbot "${BB_NAME[$i]}" "${BB_USER[$i]}" "${BB_BASEDIR[$i]}" \
    "${BB_PREFIXCMD[$i]}" || true
if start_buildbot "${BB_NAME[$i]}" "${BB_USER[$i]}" "${BB_BASEDIR[$i]}" \
    "${BB_PREFIXCMD[$i]}" "${BB_OPTIONS[$i]}"
then
    echo "restarted"
else
    echo "not restarted"
    errors=$(($errors+1))
fi
    done
    return $errors
}
case "$1" in
  start)
  do_start
  exit $?
;;
  stop)
  do_stop
  exit $?
;;
  reload)
  do_reload
  exit $?
;;
  restart|force-reload)
  do_restart
  exit $?
;;
  *)
log_warning_msg "Usage: $0 {start|stop|restart|reload|force-reload}"
exit 1
;;
esac
exit 0
</pre>
</pre>


TODO: Have Buildbot start with the machine.
Run the following command to add buildbot to the init scripts:
chkconfig --add buildbot
Bhearsum
canmove, Confirmed users
6,441

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/60840...62127"