ReleaseEngineering/NoReboots

From MozillaWiki
Jump to: navigation, search

Why no reboots:

Turning off reboots saves machine time in several ways: 70-120 seconds of reboot time, greater potential for FS cache utilization, and opportunities for sharing work (like clobbering) across jobs more effectively. As such, if successful, we should expect to see less instances launched per day -- as the throughput of the infrastructure rises.

Results:

A preliminary survey of the time saved by non-rebooting spot instances suggests that -- if jobs are not failing more often as a result of the changes -- a great deal of wasted time is being recovered:

   minimum time saved (seconds) = (<iterations_seen> - <halts_seen>) * <reboot_sec_avg>
One hour sample from 01-13-2015 01 CST Raw Data

The test/try machines spend more time rebooting (79s) and less time doing pre-flight tasks (~45s) on average while builders have an opposite skew (82s pre-flight tasks and 67s reboot time). We seem to have saved around 60 hours of machine time during this interval.

start_to_bb_sec_avg reboot_sec_avg start_to_bb_sec_total halts_seen reboot_percentage iterations_seen type reboot_sec_total
47 79 132104 178 6 2804 all spot instances 14070
45 79 121606 167 6 2676 test and try 13329
82 67 10498 11 8 128 builders 741
Three hour sample from 01-13-2015 02 CST Raw Data

During the longer samples builder reboot times increased dramatically, however, this is likely the result of builder issues which were occuring around the time the data was taken; test/try results remained stable. We seem to have saved around 204 hours of machine time during this interval.

start_to_bb_sec_avg reboot_sec_avg start_to_bb_sec_total halts_seen reboot_percentage iterations_seen type reboot_sec_total
47 92 410104 605 7 8547 all spot instances 55769
45 80 362177 553 7 7884 test and try 44648
72 213 47927 52 7 663 builders 11121

NOTE: The data above was gathered via runners influxdb logging and verified by spot checking runner logs (/var/log/runner.log) The reports themselves were generated by this script: http://pastebin.mozilla.org/8191890

No reboots timeline (puppet):

 date:        Fri Jan 16 17:47:33 2015 +0000
 summary:     Bug 1122601 - Coerce runner to reboot after particular job types; r=rail
 date:        Tue Jan 13 21:07:53 2015 +0000
 summary:     Bug 1109932 - Enable reboots for all try, talos, and test slaves; r=Callek
 date:        Tue Jan 06 14:26:55 2015 -0600
 summary:     Bug 1118125 - Turn off osx reboots; r=Callek
 date:        Tue Dec 30 19:40:23 2014 +0000
 summary:     Bug 1103123 - Turn off rebooting of all linux slaves; r=callek
 date:        Thu Dec 18 18:45:10 2014 +0000
 summary:     Bug 1113245 - Remove cleanslate process list on Linux and Mac machines during reboots with halt.py; r=rail
 date:        Fri Dec 12 21:33:04 2014 +0000
 summary:     Bug 1103123 - Turn off rebooting of talos machines; r=catlee

How no-reboot mode is enabled (idllizer, post_flight)

Buildbot is now started/managed by runner, which runs tasks in an infinite loop according to some specified order [each task is blocking]. As such, buildbot initiates a graceful shutdown immediately after accepting any job so that the runner tasks may loop around again after it’s finished. A single runner loop looks like this:

   <tasks before buildbot> -> buildbot.py [graceful shutdown] -> <tasks after buildbot> -> post_flight.py

The graceful shutdown is initiated by idelizer.py, then, post_flight.py decides whether or not to shut down the machine or go forward with another loop.

post_flight checks:

hostname blacklist

Any machine with a hostname that matches some regular expression found in this list will be rebooted by post_flight. For example: [“^tst-“, “^t-"] would reboot all test machines after any job.

build api

BuildAPI is used to fetch data about the most recent job, if the job fails the slave is rebooted. This feature may need to be disabled, since it could mask failures. The thinking, on turning it on, was that we could track problems via logging and avoid tree closures in the case of problems. This is likely too optimistic. Failing hard may be better in the end.

jobname blacklist=====

Works like the hostname blacklist, except, acting on the name of the most recently run job (which is known about by BuildAPI).

What issues have been noted since no-reboot work started?

These bugs have been noted, since December '14, as having possible connection to Runner/NoReboots:

bug 1114541 bug 989048 bug 1109932 bug 1114688 bug 1111137

How are we tracking the status of machines, and measuring effectiveness?

Runner constantly uploads task stats to influxdb, for dashboards see: https://stats.taskcluster.net/grafana/#/dashboard/db/runner