Buildbot/IT Mozilla2 Support Document

From MozillaWiki
Jump to: navigation, search

Machines

Mozilla2 Waterfall
Mozilla2 Buildslave list

All of following machines live in the build network. This makes their FQDN $name.build.mozilla.org.

Name Platform Tree Support Tier*
production-master (master) linux N/A X
moz2-linux-slave1
moz2-linux-slave02
moz2-linux-slave03
linux Mozilla2 Actionmonkey X
moz2-win32-slave1
moz2-win32-slave02
moz2-win32-slave03
win32 Mozilla2 Actionmonkey X
bm-xserve16
bm-xserve17
bm-xserve18
Mac OS X 10.5 Mozilla2 Actionmonkey X

- * - tiers explained

Fixing failing builds

Clobber the failing slave and force a new build. (Note: forcing a new build will not necessarily cause a build on the failing slave. This is OK.)

  1. Find the hostname in the Tinderbox log (look for 'Building on: XXX' where 'XXX' is a hostname)
  2. Login to the slave using provided credentials (SSH for Linux, RDP for Windows, VNC for Mac).
    1. On Windows, kill any ssh/sh/make/other unix-type processes in the task manager.
    2. Go to the builder directory (e:\builds\moz2_slave\$builder_name on win32, /builds/moz2_slave/$builder_name on linux/mac [where $builder_name matches the name of the failing build on Tinderbox])
    3. Delete the subdirectory named 'build'.
    4. Force a new build
      1. Click on the appropriate column title on the Waterfall (eg. 'mozilla-central-win32')
      2. Fill out the Force Build form. 'Branch to build' MUST be specified (branch name is part of the builders name. eg, mozilla-central or actionmonkey).
      3. Click 'Force Build'.
    5. If the next build doesn't go green, contact RelEng.

Builds not happening at all

Make sure all slaves are connected (Note: do NOT use the 'Ping Builder' button on the Waterfall, it will break future builds):

  1. Go to the Buildslave List page.
  2. For any slaves that are listed as 'NOT connected' we need to connect the slave:
    1. Login to the slave using provided credentials (SSH for Linux, RDP for Windows, VNC for Mac).
    2. For Windows:
      1. Kill any ssh/sh/make/other unix-type processes in the task manager.
      2. If there is not an existing MSYS window start one with the desktop icon.
      3. Start Buildbot with '/d/mozilla-build/python25/scripts/buildbot start /e/builds/moz2_slave'.
      4. Minimize the MSYS window
    3. For Mac:
      1. Start Terminal if not already running.
      2. Start Buildbot with 'buildbot start /builds/moz2_slave'
      3. Minimize the Terminal window (do NOT close it).
    4. For Linux:
      1. Start Buildbot with 'buildbot start /builds/moz2_slave'.
    5. Wait 30 seconds and check the Buildslave List page again. If slaves still aren't connected contact RelEng.


It is almost always a bad idea to reboot the buildmaster VM. It runs multiple Buildbot masters. Unless it is completely unreachable please do not restart this machine without talking to someone from RelEng first.