Changes

I attacked the pool of broken/dead Tegras this week, and with a lot ofguidance from Callek, managed to get a bunch of them back online. TheTegra wiki page(https://wiki.mozilla.org/ReleaseEngineering:How_To:Android_Tegras)isn't very helpful in terms of mapping "state it's in" -> "action i needto take", so I wanted to let you all know the simple steps I took:

1) # Look for Tegras with alerts onhttp://admin1.infra.scl1.mozilla.com/nagios/cgi-bin/status.cgi?host=all&servicestatustypes=28&hoststatustypes=15&serviceprops=2703462) # Find the Tegra on http://mobile-dashboard1.build.mtv1.mozilla.com/tegras/3) # Click the "tegra-xxx" link to find the Tegra's Buildbot page4) # If the Tegra is offline, try to PDU reboot it(https://wiki.mozilla.org/ReleaseEngineering:How_To:Android_Tegras#Reboot_a_Tegra_through_the_PDU)5) # If the Tegra isn't online 30min later, open its bug and block itagainst the "tegra-recovery" bug. You may need to file one or both of these.6) # If that doesn't work, talk to Callek ;).

And that's it! More than half the Tegras I touched came back onlineeither through the PDU reboot or IT recovery. I implore you all to keepon top of Tegras when you're on buildduty - it's not as hard as it seems.