Release:Release Automation on Mercurial:Troubleshooting: Difference between revisions

Jump to navigation Jump to search

Release:Release Automation on Mercurial:Troubleshooting (view source)

Revision as of 14:09, 28 April 2020

1,956 bytes removed ,  28 April 2020
→‎How to investigate release runner failures
No edit summary
Line 2: Line 2:
=== Paperwork ===
=== Paperwork ===
Some releases require creation of build notes when failures occur, others should already have a page created at release start - [[Release:Release_Automation_on_Mercurial:Starting_a_Release#Paperwork | requirements]]. Either way, document all problems and their solutions.
Some releases require creation of build notes when failures occur, others should already have a page created at release start - [[Release:Release_Automation_on_Mercurial:Starting_a_Release#Paperwork | requirements]]. Either way, document all problems and their solutions.
=== How to investigate release runner failures ===
==== "[release-runner] failed" ====
Release runner can fail to start a release for many reasons (eg, release sanity failures, network issues). Unless something very unusual happens, you will receive an e-mail with the subject line "[release-runner] failed" when it encounters an issue. The e-mail should have brief details on the failure - for example, it may contain an excerpt from release sanity. If this doesn't give you enough information to debug the problem, you can get more detailed information by logging onto buildbot-master81 and inspecting /builds/releaserunner/release-runner.log.
If you're unable to resolve the issue on your own ask someone for help. Once you believe the issue has been resolved you need to mark the release as "ready" again on Ship It and restart the release runner process. This can be done with the following command on buildbot-master81 (as root):
supervisorctl restart releaserunner
The release might fail again, with an error similar to:
subprocess.CalledProcessError: Command '['hg', 'commit', '-m', 'Update release config for Fennec-27.0b9-build1', '-u', 'ffxbld']' returned non-zero exit status 1
This is because the prior failure happened after the release configs were already updated. The solution is to:
# revert the changes on both default & production branches of "<tt>buildbot-configs</tt>"
# re-mark the releases as "ready" again, and hit "do eeet"
==== "[release-runner] WARNING: Reconfig exceeded (time)" ====
If release runner is unable to reconfig the required masters after 15min you'll receive a mail like this. This initial mail is just a heads up that something may need some intervention. If after 30min the reconfig still isn't complete, you should have a look at buildbot-master81:/builds/releaserunner/release-runner.log and see which master(s) to see what's stuck, and go deal with it as you would if you were doing a reconfig by hand.


=== Addressing Disk Space Issues ===
=== Addressing Disk Space Issues ===
Rail
Confirmed users
3,104

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1226575"

Navigation menu