541
edits
Buildbot/IT Talos Support Document: Difference between revisions
Jump to navigation
Jump to search
Buildbot/IT Talos Support Document (view source)
Revision as of 23:04, 12 February 2008
, 12 February 2008→list of steps to try
| Line 10: | Line 10: | ||
see also: [[Buildbot/Talos/Machines]] | see also: [[Buildbot/Talos/Machines]] | ||
== | == Steps to Try == | ||
=== | === The Talos Machines Are Green, But I'm Not Seeing Any Results === | ||
* <b>Symptoms</b> | * <b>Symptoms</b> | ||
** A given | ** Talos boxes are reporting green, but there are no graph links. It appears that no results from the tests were collected. | ||
** A given | |||
1. Turn on '--debug' for talos machines | |||
* ssh to qm-rhel02 | |||
* in /build/perfmaster edit master.cfg | |||
* edit command=['python', 'run_tests.py', '--noisy'] to command=['python', 'run_tests.py', '--noisy', '--debug'] | |||
* restart the buildmaster | |||
** in /build | |||
** buildbot reconfig perfmaster | |||
2. Wait for machines to cycle | |||
3. Read the logs, there should be graph server errors that will indicate why sending data is failing | |||
=== Talos Seems to be Blocked on Something === | |||
* <b>Symptoms</b> | |||
** A given Talos slave hasn't reported any numbers in a long time (upwards of 6 - 8 hours) | |||
** A given Talos slave from a set has been consuming a lot of builds rapidly and failing out on browser download/installation | |||
1. Check waterfall at: http://qm-rhel02.mozilla.org:2006/ (mpt-vpn) | 1. Check waterfall at: http://qm-rhel02.mozilla.org:2006/ (mpt-vpn) | ||
| Line 38: | Line 54: | ||
'''note''' builds are triggered by finished builds on the Tinderbox (Firefox for trunk, Mozilla1.8 for branch). Then, depending on when the master was started, may take up to 10 minutes to recognize a change. If the master is restarted, first completed tinderbox builds are often missed so sometimes it can take upwards of 30-40 minutes to verify that systems are working as expected. | '''note''' builds are triggered by finished builds on the Tinderbox (Firefox for trunk, Mozilla1.8 for branch). Then, depending on when the master was started, may take up to 10 minutes to recognize a change. If the master is restarted, first completed tinderbox builds are often missed so sometimes it can take upwards of 30-40 minutes to verify that systems are working as expected. | ||
=== | === A Talos Machine is Reporting Strange Numbers === | ||
* <b>Symptoms</b> | * <b>Symptoms</b> | ||
** A given | ** A given Talos machine is reporting significantly higher/lower numbers than matching machines. | ||
*** Talos machines reporting to trunk come in sets of three (qm-mini-ubuntu01/02/03, qm-mini-vista01/02/03, etc) so that outlier results can be spotted. If we see an outlier we try and fix the configuration on that given machine to have it match it's equals. | *** Talos machines reporting to trunk come in sets of three (qm-mini-ubuntu01/02/03, qm-mini-vista01/02/03, etc) so that outlier results can be spotted. If we see an outlier we try and fix the configuration on that given machine to have it match it's equals. | ||
==== Linux | ==== Linux ==== | ||
1. | 1. Stop the build slave | ||
~$ buildbot stop talos-slave | ~$ buildbot stop talos-slave | ||
2. Is throttling on/correct? | 2. Is throttling on/correct? | ||
| Line 100: | Line 116: | ||
~$ buildbot start talos-slave | ~$ buildbot start talos-slave | ||
==== Mac | ==== Mac (Tiger/Leopard) ==== | ||
1. Stop the buildbot slave | 1. Stop the buildbot slave | ||
~$ buildbot stop talos-slave | ~$ buildbot stop talos-slave | ||
| Line 120: | Line 136: | ||
~$ buildbot start talos-slave | ~$ buildbot start talos-slave | ||
==== WinXP | ==== WinXP ==== | ||
1. Stop the buildbot slave | 1. Stop the buildbot slave | ||
* In the open cmd Ctrl-C, 'y' to stop slave | * In the open cmd Ctrl-C, 'y' to stop slave | ||
| Line 147: | Line 163: | ||
* buildbot start talos-slave | * buildbot start talos-slave | ||
==== Vista | ==== Vista ==== | ||
1. Stop the buildbot slave | 1. Stop the buildbot slave | ||
* in the open cmd Ctrl-c, 'y' to stop slave | * in the open cmd Ctrl-c, 'y' to stop slave | ||