Confirmed users
3,104
edits
| Line 3: | Line 3: | ||
=== How to investigate release runner failures === | === How to investigate release runner failures === | ||
==== "[release-runner] failed" ==== | ==== "[release-runner] failed" ==== | ||
Release runner can fail to start a release for many reasons (eg, release sanity failures, network issues). Unless something very unusual happens, you will receive an e-mail with the subject line "[release-runner] failed" when it encounters an issue. The e-mail should have brief details on the failure - for example, it may contain an excerpt from release sanity. If this doesn't give you enough information to debug the problem, you can get more detailed information by logging onto buildbot- | Release runner can fail to start a release for many reasons (eg, release sanity failures, network issues). Unless something very unusual happens, you will receive an e-mail with the subject line "[release-runner] failed" when it encounters an issue. The e-mail should have brief details on the failure - for example, it may contain an excerpt from release sanity. If this doesn't give you enough information to debug the problem, you can get more detailed information by logging onto buildbot-master81 and inspecting /builds/releaserunner/release-runner.log. | ||
If you're unable to resolve the issue on your own ask someone for help. Once you believe the issue has been resolved you need to mark the release as "ready" again on Ship It and restart the release runner process. This can be done with the following command on buildbot- | If you're unable to resolve the issue on your own ask someone for help. Once you believe the issue has been resolved you need to mark the release as "ready" again on Ship It and restart the release runner process. This can be done with the following command on buildbot-master81 (as root): | ||
supervisorctl restart releaserunner | supervisorctl restart releaserunner | ||
==== "[release-runner] WARNING: Reconfig exceeded (time)" ==== | ==== "[release-runner] WARNING: Reconfig exceeded (time)" ==== | ||
If release runner is unable to reconfig the required masters after 15min you'll receive a mail like this. This initial mail is just a heads up that something may need some intervention. If after 30min the reconfig still isn't complete, you should have a look at buildbot- | If release runner is unable to reconfig the required masters after 15min you'll receive a mail like this. This initial mail is just a heads up that something may need some intervention. If after 30min the reconfig still isn't complete, you should have a look at buildbot-master81:/builds/releaserunner/release-runner.log and see which master(s) to see what's stuck, and go deal with it as you would if you were doing a reconfig by hand. | ||
=== Restarting the automation from a specific point === | === Restarting the automation from a specific point === | ||