Buildbot/IT Talos Support Document - Revision history

Anodelman: Replacing page with ''''This document has been phased out -- please refer to ReleaseEngineering:ITSupport for your IT Support needs''''

2008-09-05T21:42:50Z

Replacing page with ''''This document has been phased out -- please refer to ReleaseEngineering:ITSupport for your IT Support needs''''

Show changes

Anodelman: /* Mac (Tiger/Leopard) */

2008-09-05T01:06:32Z

Mac (Tiger/Leopard)

@@ Line 119: / Line 119: @@
 ==== Mac (Tiger/Leopard) ====
-. Stop the buildbot slave
+reboot the machine
 ==== WinXP  ====

Lukasblakk at 19:26, 5 June 2008

2008-06-05T19:26:09Z

← Older revision		Revision as of 19:26, 5 June 2008
Line 1:		Line 1:
			'''This document is being phased out -- please refer to [[ReleaseEngineering:ITSupport]] for your IT Support needs'''

	== Machines ==		== Machines ==

Reed: /* Machines */

2008-04-27T05:20:20Z

Machines

← Older revision		Revision as of 05:20, 27 April 2008
Line 5:		Line 5:

	slaves:		slaves:
	* qm-~~pxp01-05~~		* qm-p*
	* qm-mini*		* qm-mini*

Anodelman: /* Steps to Try */

2008-02-12T23:06:52Z

Steps to Try

← Older revision		Revision as of 23:06, 12 February 2008
Line 10:		Line 10:
	see also: [[Buildbot/Talos/Machines]]		see also: [[Buildbot/Talos/Machines]]

	== ~~Steps to Try~~ ==		== What's the Problem? ==

	=== The Talos Machines Are Green, But I'm Not Seeing Any Results ===		=== The Talos Machines Are Green, But I'm Not Seeing Any Results ===

Anodelman: /* list of steps to try */

2008-02-12T23:04:33Z

list of steps to try

← Older revision		Revision as of 23:04, 12 February 2008
Line 10:		Line 10:
	see also: [[Buildbot/Talos/Machines]]		see also: [[Buildbot/Talos/Machines]]

	== ~~list of steps~~ to ~~try~~ ==		== Steps to Try ==

	=== ~~For blocked talos~~ ===		=== The Talos Machines Are Green, But I'm Not Seeing Any Results ===

	* <b>Symptoms</b>		* <b>Symptoms</b>
	** A given ~~talos~~ slave hasn't reported any numbers in a long time (upwards of 6 - 8 hours)		** Talos boxes are reporting green, but there are no graph links. It appears that no results from the tests were collected.
	** A given ~~talos~~ slave from a set has been consuming a lot of builds rapidly and failing out on browser download/installation
			1. Turn on '--debug' for talos machines
			* ssh to qm-rhel02
			* in /build/perfmaster edit master.cfg
			* edit command=['python', 'run_tests.py', '--noisy'] to command=['python', 'run_tests.py', '--noisy', '--debug']
			* restart the buildmaster
			** in /build
			** buildbot reconfig perfmaster
			2. Wait for machines to cycle

			3. Read the logs, there should be graph server errors that will indicate why sending data is failing

			=== Talos Seems to be Blocked on Something ===

			* <b>Symptoms</b>
			** A given Talos slave hasn't reported any numbers in a long time (upwards of 6 - 8 hours)
			** A given Talos slave from a set has been consuming a lot of builds rapidly and failing out on browser download/installation

	1. Check waterfall at: http://qm-rhel02.mozilla.org:2006/ (mpt-vpn)		1. Check waterfall at: http://qm-rhel02.mozilla.org:2006/ (mpt-vpn)
Line 38:		Line 54:
	'''note''' builds are triggered by finished builds on the Tinderbox (Firefox for trunk, Mozilla1.8 for branch). Then, depending on when the master was started, may take up to 10 minutes to recognize a change. If the master is restarted, first completed tinderbox builds are often missed so sometimes it can take upwards of 30-40 minutes to verify that systems are working as expected.		'''note''' builds are triggered by finished builds on the Tinderbox (Firefox for trunk, Mozilla1.8 for branch). Then, depending on when the master was started, may take up to 10 minutes to recognize a change. If the master is restarted, first completed tinderbox builds are often missed so sometimes it can take upwards of 30-40 minutes to verify that systems are working as expected.

	=== ~~For a talos machine reporting strange numbers~~ ===		=== A Talos Machine is Reporting Strange Numbers ===

	* <b>Symptoms</b>		* <b>Symptoms</b>
	** A given ~~talos~~ machine is reporting significantly higher/lower numbers than matching machines.		** A given Talos machine is reporting significantly higher/lower numbers than matching machines.
	*** Talos machines reporting to trunk come in sets of three (qm-mini-ubuntu01/02/03, qm-mini-vista01/02/03, etc) so that outlier results can be spotted. If we see an outlier we try and fix the configuration on that given machine to have it match it's equals.		*** Talos machines reporting to trunk come in sets of three (qm-mini-ubuntu01/02/03, qm-mini-vista01/02/03, etc) so that outlier results can be spotted. If we see an outlier we try and fix the configuration on that given machine to have it match it's equals.

	==== Linux ~~talos machines~~ ====		==== Linux ====
	1. ~~stop~~ the build slave		1. Stop the build slave
	~$ buildbot stop talos-slave		~$ buildbot stop talos-slave
	2. Is throttling on/correct?		2. Is throttling on/correct?
Line 100:		Line 116:
	~$ buildbot start talos-slave		~$ buildbot start talos-slave

	==== Mac ~~talos machines~~ (Tiger/Leopard) ====		==== Mac (Tiger/Leopard) ====
	1. Stop the buildbot slave		1. Stop the buildbot slave
	~$ buildbot stop talos-slave		~$ buildbot stop talos-slave
Line 120:		Line 136:
	~$ buildbot start talos-slave		~$ buildbot start talos-slave

	==== WinXP ~~talos machines~~ ====		==== WinXP ====
	1. Stop the buildbot slave		1. Stop the buildbot slave
	* In the open cmd Ctrl-C, 'y' to stop slave		* In the open cmd Ctrl-C, 'y' to stop slave
Line 147:		Line 163:
	* buildbot start talos-slave		* buildbot start talos-slave

	==== Vista ~~talos machines~~ ====		==== Vista ====
	1. Stop the buildbot slave		1. Stop the buildbot slave
	* in the open cmd Ctrl-c, 'y' to stop slave		* in the open cmd Ctrl-c, 'y' to stop slave

Anodelman: /* list of steps to try */

2008-02-12T19:59:46Z

list of steps to try

Show changes

Bhearsum: /* Machines */

2007-10-26T20:42:18Z

Machines

← Older revision		Revision as of 20:42, 26 October 2007
Line 2:		Line 2:

	master:		master:
	* qm-~~buildbot01~~		* qm-rhel02

	slaves:		slaves:

Bhearsum: /* list of steps to try */

2007-10-26T19:53:54Z

list of steps to try

← Older revision		Revision as of 19:53, 26 October 2007
Line 13:		Line 13:

	1. first check		1. first check
	* check waterfall at: http://qm-~~buildbot01~~.mozilla.org:~~2004~~/ (mpt-vpn)		* check waterfall at: http://qm-rhel02.mozilla.org:2006/ (mpt-vpn)
	* see if slave is connected.		* see if slave is connected.

Line 38:		Line 38:
	** Ctrl-C in Command window on Windows		** Ctrl-C in Command window on Windows
	** buildbot stop slave in Terminal on Mac and Linux		** buildbot stop slave in Terminal on Mac and Linux
	* shutdown master on qm-~~buildbot01~~		* shutdown master on qm-rhel02
	** cd /build		** cd /build
	** buildbot stop perfmaster		** buildbot stop perfmaster
	* reboot qm-~~buildbot01~~ and slave machines if necessary (stuck processes, strange behavior)		* reboot qm-rhel02 and slave machines if necessary (stuck processes, strange behavior)
	* restart master on qm-~~buildbot01~~		* restart master on qm-rhel02
	** cd /build		** cd /build
	** buildbot start perfmaster		** buildbot start perfmaster
	* restart slaves as above		* restart slaves as above
	* verify waterfall at http://qm-~~buildbot01~~.mozilla.org:~~2004~~/ is visible and slaves are connected		* verify waterfall at http://qm-rhel02.mozilla.org:2006/ is visible and slaves are connected

Bhearsum: /* list of steps to try */

2007-09-14T16:44:33Z

list of steps to try

← Older revision		Revision as of 16:44, 14 September 2007
Line 19:		Line 19:
	* login to machine using provided credentials		* login to machine using provided credentials
	** VNC (qm-pxp01-05, qm-mini*):		** VNC (qm-pxp01-05, qm-mini*):
	* close running instances of firefox or dialog windows		* close running instances of firefox or dialog windows (make sure to check the taskbar)
	* on windows, ctrl-C in the command window, answer Yes to terminate buildbot process		* on windows, ctrl-C in the command window, answer Yes to terminate buildbot process
	* ~~restart~~ buildbot:		** ctrl-c in the command window, answer yes to terminate buildbot process
	** cd c:\		** cd c:\
	** buildbot start slave (command does not return)		** on qm-pxp*, 'buildbot start slave' (command does not return)
			** on qm-mini*, 'buildbot start talos-slave' (command does not return)
			* on linux/mac
			** login via ssh
			** 'buildbot stop talos-slave' (ignore 'never saw slave...' message on mac)
			** 'buildbot start talos-slave' (ignore 'never saw slave...' message on mac)
	* verify slave reappears on buildbot waterfall page		* verify slave reappears on buildbot waterfall page