Release:Release Automation on Mercurial:Troubleshooting: Difference between revisions

Jump to navigation Jump to search
Line 3: Line 3:
=== How to investigate release runner failures ===
=== How to investigate release runner failures ===
==== "[release-runner] failed" ====
==== "[release-runner] failed" ====
Release runner can fail to start a release for many reasons (eg, release sanity failures, network issues). Unless something very unusual happens, you will receive an e-mail with the subject line "[release-runner] failed" when it encounters an issue. The e-mail should have brief details on the failure - for example, it may contain an excerpt from release sanity. If this doesn't give you enough information to debug the problem, you can get more detailed information by logging onto buildbot-master36 and inspecting /var/log/supervisor/release-runner.log.
Release runner can fail to start a release for many reasons (eg, release sanity failures, network issues). Unless something very unusual happens, you will receive an e-mail with the subject line "[release-runner] failed" when it encounters an issue. The e-mail should have brief details on the failure - for example, it may contain an excerpt from release sanity. If this doesn't give you enough information to debug the problem, you can get more detailed information by logging onto buildbot-master81 and inspecting /builds/releaserunner/release-runner.log.


If you're unable to resolve the issue on your own ask someone for help. Once you believe the issue has been resolved you need to mark the release as "ready" again on Ship It and restart the release runner process. This can be done with the following command on buildbot-master36 (as root):
If you're unable to resolve the issue on your own ask someone for help. Once you believe the issue has been resolved you need to mark the release as "ready" again on Ship It and restart the release runner process. This can be done with the following command on buildbot-master81 (as root):
  supervisorctl restart releaserunner
  supervisorctl restart releaserunner


==== "[release-runner] WARNING: Reconfig exceeded (time)" ====
==== "[release-runner] WARNING: Reconfig exceeded (time)" ====
If release runner is unable to reconfig the required masters after 15min you'll receive a mail like this. This initial mail is just a heads up that something may need some intervention. If after 30min the reconfig still isn't complete, you should have a look at buildbot-master36:/var/log/supervisor/release-runner.log and see which master(s) to see what's stuck, and go deal with it as you would if you were doing a reconfig by hand.
If release runner is unable to reconfig the required masters after 15min you'll receive a mail like this. This initial mail is just a heads up that something may need some intervention. If after 30min the reconfig still isn't complete, you should have a look at buildbot-master81:/builds/releaserunner/release-runner.log and see which master(s) to see what's stuck, and go deal with it as you would if you were doing a reconfig by hand.


=== Restarting the automation from a specific point ===
=== Restarting the automation from a specific point ===
Confirmed users
3,104

edits

Navigation menu