Notes and Caveats

You must connect to the Windows boxes with a RDP client set to 24-bit color mode otherwise the reftests in /mozilla/modules/libpr0n/test/reftest/ will start failing, until the display is reset back to 24-bit mode (eg, by disconnecting and/or rebooting).

Steps to Try

Please try these steps in order - if the third step (manual clobber) doesn't work contact bhearsum, robcee, or coop.

1. Check Slave is Connected/Force a Build

• check waterfall at: http://qm-rhel02.mozilla.org:2005/ (mpt-vpn)
• see if slave is connected.
• if so, click the machine name link, try a "Force Build"
  • fill out the name and reason fields, click the button

2. Restarting the Slave

• login to machine using provided credentials using Windows Remote Desktop
• on windows, ctrl-C in the command window, answer Yes to terminate buildbot process
• check the Task Manager to see if there are any stuck sh.exe and make.exe processes
  • if so, reboot the machine using whatever means necessary (the stuck sh.exe and make.exe processes can make shutting down tricky. Kill them if you can)
  • when machine is rebooted, log back in and open a command prompt
  • restart buildbot:
    • cd c:\
    • buildbot start slave (command does not return)
    • minimize the cmd.exe window
• check the waterfall at: http://qm-rhel02.mozilla.org:2005/
• the slave sometimes takes a couple of minutes to reconnect
• if it does, and is necessary, click the machine name link and force a build as above (fill name and reason fields, click the button)

3. Clobbering Manually

• Sometimes a machine will need to be "clobbered" (have its build directory removed inside the slave dir)
• login to the machine.
• stop the slave (as above, ctrl-C in the command window)
• check the task manager to make sure there are no errant sh.exe or make.exe files.
  • If there are, kill them (End process on sh.exe) or reboot the machine
• from the Command Line:
  • cd C:\slave
  • rmdir /s /q trunk_2k3 for qm-win2k3-01
• restart the slave
  • cd \
  • buildbot start slave (command does not return)
  • minimize cmd.exe window

Contact Robcee or Bhearsum if you need to go to Step 4

4. Restarting the Farm

• In the worst case, the entire buildbot farm needs to be restarted
• shutdown each slave as per the instructions: Unittest:Linux:ITSupport Unittest:Mac:ITSupport
• shutdown master on qm-rhel02
  • cd /build
  • buildbot stop master
• reboot qm-rhel02 and slave machines if necessary (stuck processes, strange behavior)
• restart master on qm-rhel02
  • cd /build
  • buildbot start master
• restart slaves as above
  • qm-centos5-01, qm-centos5-02, qm-centos5-04, qm-xserve01, qm-xserve06, qm-win2k3-01, qm-win2k3-02, qm-win2k3-pgo01
• verify waterfall at http://qm-rhel02.mozilla.org:2005/ is visible and slaves are connected