View source for CIDuty/How To/Deprecated / Archived/Slave Management

In general, slave management involves:
* keeping as many slaves up as possible, including
** proactively checking for hung/broken slaves - see [https://secure.pub.build.mozilla.org/builddata/reports/slave_health/index.html slave health dashboard].
** returning re-imaged slaves to production
* handling [[ReleaseEngineering:Buildduty:Nagios|nagios alerts]] for slaves
* interacting with IT regarding slave maintenance

= Known failure modes =
* talos-r4-snow, talos-mtnlion-r5
** all r4 and r5 slaves are connected to [[#PDU|PDUs for power-cycling]]
* AWS slaves
** a common failure is running out of disk space.  They have default disk allocations of 150GB versus our which have 250GB.  Catlee is working on changing that. 
*** To clean them, you can run mock_mozilla -v -r mozilla-centos6-i386 --scrub=all See {{bug|829186}} for an example.  
** Rail wrote a tool [[ReleaseEngineering/How_To/Manage_AWS_slaves | to manage aws slaves]] - enable or disable automatic reboot and automatic shutdown.

= Automated =
There are currently no automated mechanisms for recovering individual slaves.

AWS instances will automatically terminate when idle.

= Manual =
== Rebooting slaves ==
Find the slave page on slave health. There's a button to reboot the machine.

== Filing bugs for IT ==
* File a bug using the link in the slave health page for the slave - it will "do the right thing" to set up a new bug if needed.
* File a [https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org&component=Server%20Operations%3A%20DCOps&short_desc=HOST%20is%20unreachable "slave is unreachable bug"] for IT.
* Create dependent bugs for any IT actions. (As of 2014, we should file per-slave bugs for reboots instead of grouping together machines in the same DC into one bug.)
** should block both the datacenter bug & the per host bug (for record keeping)
** consider whether the slave should be disabled in slavealloc, and note that in bug (no slave without a detailed bug should be disabled)
** dcops assumes if there is no separate bug, they only need to reboot and see the machine come online.
** Examples: https://bugzilla.mozilla.org/show_bug.cgi?id=966954, https://bugzilla.mozilla.org/show_bug.cgi?id=828602

== Slave Tracking ==
Slave tracking is done via the [http://slavealloc.build.mozilla.org/ui/#slaves Slave Allocator]. Please disable/enable slaves in slavealloc. You can also disable them directly from the slave page in slave health, e.g. [https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=t-w732-ix-001 t-w732-ix-001].

'''NOTE:''' you no longer need to add the slave-specific bug number to the Notes field. Clicking on the https://secure.pub.build.mozilla.org/slavealloc/ui/icons/help.png icon in slavealloc will look up the bug number and status for you, or create a template you can use to file a new bug. If there is another bug, e.g. for IT re-imaging, please add that extra bug number to the Notes field instead using the format: 'bug #######.'

=== Slavealloc ===
==== Connecting ====
Slaves are added to slavealloc via the 'dbimport' subcommand of the 'slavealloc' command.

You will need to ssh as your own user onto the server which hosts slavealloc:

<pre>
ssh <your user>@relengwebadm.private.scl3.mozilla.com 
</pre>
==== Staging vs production ====
The DB urls for staging and production are shared in a PGP encrypted file used by the Release Engineering team. Ask someone else in the team if you do not have this file.

==== Adding a slave ====
Once you connect to relengwebadm (see above), to see the help for the slavealloc dbimport command, run:
<pre>
/data/releng/www/slavealloc/slavealloc dbimport -h
</pre>

To import data, first you need to create a CSV file, like this one:

<pre>
name,distro,bitlength,speed,datacenter,trustlevel,environment,purpose,pool,basedir
t-w864-ix-236,win8,64,ix,scl3,try,prod,tests,tests-inhouse-windows,C:\slave
t-w864-ix-237,win8,64,ix,scl3,try,prod,tests,tests-inhouse-windows,C:\slave
t-w864-ix-238,win8,64,ix,scl3,try,prod,tests,tests-inhouse-windows,C:\slave
</pre>

You'll want a command line something like:
<pre>
/data/releng/www/slavealloc/slavealloc dbimport -D mysql://user:password@host/DB_name --slave-data <the csv file you just created containing slaves>
</pre>

==== Adding a master ====
Adding masters is similar to adding a slave:
<pre>
/data/releng/www/slavealloc/slavealloc dbimport -D mysql://user:password@host/DB_name  --master-data <csv file containing masters>
</pre>

The following example shows the required fields, and example values:
<pre>
nickname,fqdn,http_port,pb_port,datacenter,pool
bm89-tests1-panda,buildbot-master89.srv.releng.scl3.mozilla.com,8201,9201,scl3,tests-panda
</pre>

To get a full list of allowed values for the various normalized fields to use in both import files, you can connect to the mysql database and query the tables directly:

<pre>
SELECT name FROM bitlengths;
SELECT name FROM datacenters;
SELECT name FROM distros;
SELECT name FROM environments;
SELECT name FROM pools;
SELECT name FROM purposes;
SELECT name FROM speeds;
SELECT name FROM trustlevels;
</pre>

Please note you'll need to set values in your CSV file that correspond to these allowed values.

The slavealloc dbimport mechanism will convert lines of the CSV file into INSERT sql statements. Non specified fields will essentially be set to NULL. To see how the fields are mapped and normalized, see: https://hg.mozilla.org/build/tools/file/5439f10a7127/lib/python/slavealloc/scripts/dbimport.py#l111 (lines 111-137).

==== Moving slaves ====
Connect to relengwebadmn and then connect to the mysql DB.

You have to determine the correct poolid and trustid values.
 UPDATE slaves SET poolid=43, trustid=4 WHERE notes LIKE 'bug 917923 - to be converted into try hosts';

==== Removing slaves ====
Connect to relengwebadmn and then connect to the mysql DB.
<pre>
 SELECT name FROM slaves WHERE notes LIKE '%bumblebumble%';
 DELETE name FROM slaves WHERE notes LIKE '%bumblebumble%';
</pre>

== Returning a re-imaged slave to production ==

(aka. post-imaging)

See [[ReleaseEngineering/How_To/Set_Up_a_Freshly_Imaged_Slave]]

== How to decommission a slave ==
* disable the slave in slavealloc, also setting its environment to "decomm"
* if the hardware has failed:
** file a bug against Server Ops:Releng to decomm the slave. They should (at the very least) make sure the nagios alerts are updated, DNS updated, and the hardware recovered from the dc.
* if the hardware is still viable and can be used by another pool (e.g. r3 mini)
** file a bug against Server Ops:Releng to have the slave re-imaged to another OS with bad wait times (usually Windows)
** add the new slave to the buildbot configs, and make sure nagios monitoring is setup for the new slave (may require a new bug against relops)
* Open a bug for the release engineering changes to decommission a machine which include
** remove the slave from the buildbot configs example {{bug|798460}}
** remove the slave from puppet or opsi configs, if it exists in one (This doesn't apply to puppet if you unless you are removing an entire pool of machines, not single one)
** remove the slave from slavealloc see https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Slave_Management#Removing_slaves 
** In the bug you can attach patches for the buildbot configs and mysql changes for slavealloc for review by another relenger

= Windows =
I'm hoping to add enough info to demystify Windows and allow anyone to debug a Windows machine.

== Start up flow ==
This is how buildbot starts:
 scheduled task (after login) -> start talos bat -> C:\slave\runslave.py

We started logging the start up of runslave.py under C:\slave\runslave.log
We do some clean up steps inside of the .bat file.

TODO: correct file names and paths
=== Trigger buildbot the natural way ===
You're logged in and you want to trigger buildbot the same way as if the machine had come back from a reboot.

Go to the Task Library, change the property of the scheduled task to allow running manually and hit "run" on the task (more or less).

== Infra setup ==
The Windows machines are managed via GPO.

The Windows test machines have the on-board graphics card and a third party graphic card.
The screenshot below shows two devices listed:
[[File:Xp_-_two_graphic_cards.png]]

=== root vs .\root ===
You want to use .\root to use the local admin user rather than the remote one.

=== Fix 2nd monitor ===
(From Q)
On all of the machines there is a script c:\monitor_config\fakemon.vbs that
will detect if the second screen is missing. Add it if necessary then adjust
the resolution.

== Windows basics ==
=== Command Prompt ===
Aka cmd.exe, you can start it by clicking on the "start" button and then click on "Run..."

=== Quick edit mode ===
You can change the properties of a Command Prompt to allow you to do these neat things:
* right-click to paste
* select with mouse and press enter to copy from selected text

You can do so by doing a right click on the Command Prompt window and changing the properties.
You can also change the defaults settings for Command Prompts being generated in the future.

If I recall correctly, this feature was requested for RelOps to deploy to all of our Windows machines.

=== runas ===
In many places you can right click and run a process as root.
However, sometimes you would want to do that from the command prompt.

 runas /user:root command_that_you_want

=== Screen resolution ===
Manually: You can do a right click on the desktop and click on "Properties".
You can then click on the "Settings" tab.

A while ago I wrote a script that adjusts the screen resolution on Win7 machines:
http://hg.mozilla.org/build/tools/file/default/scripts/support/mouse_and_screen_resolution.py

There is code to query screen resolutions.

We should find a way to prevent starting machines up with not big enough screen resolutions. We could use runslave.py or start-buildbot.bat to prevent that (since we don't have pre-flight tasks yet).

=== Registry ===
You can start the registry editor by running "regedit".

=== Rebooting ===
You can run this command (Start->Run...):
 shutdown -f -r -t 0

=== Computer Management ===
Do a "right click" on "My Computer" and click on "Manage"<br />
[[File:Xp_-_computer_management.png]]

==== Check logs ====
You can review the logs of the Windows machine to debug issues.
You should things like reboot times and others.
[[File:Xp_-_looking_at_logs.png]]

== Task Library - talosslave task ==
[[Image:W7_-_task_library.png|thumbnail]]
[[Image:W8 - task library.png|thumbnail]]

On Win7 & Win8 you can right click on the "Computer" icon and click on "Manage".
For Win8, you will need to enter the admin credentials.

This will take you to the "Computer Management" window. Click on the following to reach to task library:
* System Tools
* Task Scheduler
* Task Scheduler Library

You should "talosslave" listed there which takes care of staring buildbot/runslave.py.<br />
NOTE: To run manually the talosslave task you might need to change the property of the task.<br />
NOTE2: I have not figured out WinXp.<br />

== Working graphical setup ==
=== Xp ===
NOTE: You see two monitors unlike win7.<br />
[[Image:Xp - working screen setup.png]]
=== Win7 ===
NOTE: Only one monitor is showing up. <br />
[[Image:W7 - screen resolution.png]]
=== Win8 ===
NOTE: You see two monitors in here; unlike win7.<br />
[[Image:W8 - screen resolution.png]]