ReleaseEngineering/How To/Unstick a Stuck Slave From A Master - Revision history

Jhopkins: /* The Hard Way */

2012-10-31T15:07:37Z

The Hard Way

← Older revision		Revision as of 15:07, 31 October 2012
Line 9:		Line 9:
	http://buildbot-master11.build.scl1.mozilla.com:8201/buildslaves?no_builders=1		http://buildbot-master11.build.scl1.mozilla.com:8201/buildslaves?no_builders=1

	If any slaves have a "Last heard from" time of more than a few hours, those are likely stuck.		If any slaves have a "Last heard from" time of more than a few hours (or > 10 connections in the past hour), those are likely stuck.

	Then, on the master, use lsof to figure out what file descriptor the socket it on for that slave:		Then, on the master, use lsof to figure out what file descriptor the socket it on for that slave:

ChrisCooper: /* The Hard Way */

2011-10-28T19:41:36Z

The Hard Way

← Older revision		Revision as of 19:41, 28 October 2011
Line 5:		Line 5:

	= The Hard Way =		= The Hard Way =
	First, use lsof to figure out what file descriptor the socket it on:		First you'll need to find slaves that may be stuck. Check the list of slaves on the master you're trying to reconfig using the no_builders=1 pragma, e.g.:

			http://buildbot-master11.build.scl1.mozilla.com:8201/buildslaves?no_builders=1

			If any slaves have a "Last heard from" time of more than a few hours, those are likely stuck.

			Then, on the master, use lsof to figure out what file descriptor the socket it on for that slave:
	$ /usr/sbin/lsof -p $master_pid \| grep linux-ix-slave05		$ /usr/sbin/lsof -p $master_pid \| grep linux-ix-slave05
	buildbot 2788 cltbld 16u IPv4 471638980 TCP staging-master.build.mozilla.org:9012->linux-ix-slave05.build.mozilla.org:54714 (ESTABLISHED)		buildbot 2788 cltbld 16u IPv4 471638980 TCP staging-master.build.mozilla.org:9012->linux-ix-slave05.build.mozilla.org:54714 (ESTABLISHED)

Djmitche at 15:41, 13 June 2011

2011-06-13T15:41:29Z

← Older revision		Revision as of 15:41, 13 June 2011
Line 14:		Line 14:
	>>> os.close(16)		>>> os.close(16)

	This will cause some weird tracebacks in the master log, but ~~should come out fine~~ in the ~~end~~.		This will cause some weird tracebacks in the master log, but will let the reconfig finish. It's black magic, and leaves the master in a potentially-very-corrupted state (eventually Twisted will try to close fd 16, which will close something completely unrelated), so '''disable the master in slavealloc and do a clean shutdown of this master once the reconfig is back on track'''.

Catlee: /* The Hard Way */

2011-04-21T21:39:48Z

The Hard Way

← Older revision		Revision as of 21:39, 21 April 2011
Line 5:		Line 5:

	= The Hard Way =		= The Hard Way =
	First, use ~~netstat~~ to figure out what ~~inode~~ the socket it on:		First, use lsof to figure out what file descriptor the socket it on:
	$ /usr/sbin/lsof -p $master_pid \| grep linux-ix-slave05		$ /usr/sbin/lsof -p $master_pid \| grep linux-ix-slave05
	buildbot 2788 cltbld 16u IPv4 471638980 TCP staging-master.build.mozilla.org:9012->linux-ix-slave05.build.mozilla.org:54714 (ESTABLISHED)		buildbot 2788 cltbld 16u IPv4 471638980 TCP staging-master.build.mozilla.org:9012->linux-ix-slave05.build.mozilla.org:54714 (ESTABLISHED)

Djmitche: Created page with "{{Release Engineering How To|Unstick a Stuck Slave From A Master}} Sometimes slaves can be in various wedged states, which prevents a master reconfig. If this is the case, then ..."

2011-03-24T15:27:03Z

Created page with "{{Release Engineering How To|Unstick a Stuck Slave From A Master}} Sometimes slaves can be in various wedged states, which prevents a master reconfig. If this is the case, then ..."

New page

{{Release Engineering How To|Unstick a Stuck Slave From A Master}}
Sometimes slaves can be in various wedged states, which prevents a master reconfig.

If this is the case, then you need to convince the master to drop the connection to that slave, with prejudice.

= The Hard Way =
First, use netstat to figure out what inode the socket it on:
$ /usr/sbin/lsof -p $master_pid | grep linux-ix-slave05
buildbot 2788 cltbld 16u IPv4 471638980 TCP staging-master.build.mozilla.org:9012->linux-ix-slave05.build.mozilla.org:54714 (ESTABLISHED)
The '16u' here gives the file descriptor within the master process (without the u)

Then, open up the manhole and:
>>> import os
>>> os.close(16)

This will cause some weird tracebacks in the master log, but should come out fine in the end.