CIDuty/Cronmail: Difference between revisions

Jump to navigation Jump to search

CIDuty/Cronmail (view source)

Revision as of 16:29, 9 October 2013

3,031 bytes removed ,  9 October 2013
Blanked the page
No edit summary
(Blanked the page)
Line 1: Line 1:
These are cron jobs that release engineers get to receive.
It informs of which actions to take (if any).


= Cron <bhearsum@cruncher> /bin/bash /home/bhearsum/nonstarted-buildrequests.sh =
We noticed an issue this morning where some builds were scheduled but never actually started. We're still not sure what's causing it - it could be a freak event, could be related to https://bugzilla.mozilla.org/show_bug.cgi?id=893859 (though the evidence doesn't support that). It's not bad enough that trees have been closed, but it is something we need to keep an eye on. I've written a quick cronjob that will check for builds such as this every 15min. It's being sent to release@mozilla.com with "Cron <bhearsum@cruncher> /bin/bash /home/bhearsum/nonstarted-buildrequests.sh" as the subject.
If you see this e-mail list any builds in it I'd ask that you take a few minutes to do the following to unstick the build(s), if you're comfortable doing so:
1) Take note of the buildrequests.id(s) listed in the mail
2) Open a r/w mysql session to the buildbot_schedulers db. You can find those credentials on any master in master/passwords.py
3) Run this query to double check that the build requests are stuck:
select buildrequests.id, from_unixtime(claimed_at), from_unixtime(submitted_at), buildername,
claimed_by_name from buildrequests LEFT OUTER JOIN builds on builds.brid =
buildrequests.id WHERE builds.brid IS NULL and buildrequests.complete = 0 and
buildrequests.claimed_at != 0 and (buildrequests.claimed_at < (unix_timestamp() - 90)) order by submitted_at desc;
4) If they are, run the following to unstick them.
start transaction;
select buildrequests.id, from_unixtime(claimed_at), from_unixtime(submitted_at), buildername,
claimed_by_name from buildrequests LEFT OUTER JOIN builds on builds.brid =
buildrequests.id WHERE builds.brid IS NULL and buildrequests.complete = 0 and
buildrequests.claimed_at != 0 and (buildrequests.claimed_at < (unix_timestamp() - 90)) order by submitted_at desc;
#update buildrequests set claimed_at=0, claimed_by_name=NULL, claimed_by_incarnation=NULL where id in (<insert list of ids from the report>);
5) Once you see that the update command looks pretty remove the "#" from the beginning and run it again.  ¡¡¡¡¡¡¡¡WARNING!!!!!! Do NOT run this without a WHERE clause listing specific buildrequest ids. You will break all of the things.
update buildrequests set claimed_at=0, claimed_by_name=NULL, claimed_by_incarnation=NULL where id in (<insert list of ids from the report>);
6) Verify that they got removed:
select buildrequests.id, from_unixtime(claimed_at), from_unixtime(submitted_at), buildername,
claimed_by_name from buildrequests LEFT OUTER JOIN builds on builds.brid =
buildrequests.id WHERE builds.brid IS NULL and buildrequests.complete = 0 and
buildrequests.claimed_at != 0 and (buildrequests.claimed_at < (unix_timestamp() - 90)) order by submitted_at desc;
7) If so run "commit" or "rollback" if not;
commit;
= TBD 1 =
= TBD 2 =
= TBD 3 =
= TBD 4 =
Bhearsum
canmove, Confirmed users
6,441

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/725568"

Navigation menu