Intermittent Failures Robot

Comments and other changes to intermittent-failure bugs by the "Intermittent Failures Robot" are made by the code at

https://github.com/mozilla/treeherder/blob/master/treeherder/intermittents_commenter/commenter.py

The commenter runs in two modes, "daily" and "weekly". The commenter runs in weekly mode on Sunday and in daily mode every other day of the week. The commenter may add a comment to a bug and/or update its whiteboard and/or priority.

Bugs should be filed in Tree Management : Intermittent Failures View.

Comments

Weekly

• A comment is posted to every bug with 1 or more failures in the last week.

• If there were 150 failures in the last 21 days, and the whiteboard does not contain [stockwell disabled], [stockwell disable-recommended], or [stockwell infra] an extra comment is added:

 ** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

• Otherwise, if there were 75 or more failures in the last week, an extra comment is added:

 ** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **
 ** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

• Otherwise, if there were 30 or more failures in the last week, an extra comment is added:

 ** This failure happened more than 30 times this week! Resolving this bug is a high priority. **
 ** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Daily

• A comment is posted to every bug with 15 or more failures in the last day.

Whiteboard

• If the "150 failures in the last 21 days" comment was made, the whiteboard is updated to [stockwell disable-recommended].
• If the bug is in a bugzilla component that is participating in the stockwell owner-triage trial, and the whiteboard does not contain [stockwell disabled], [stockwell disable-recommended], [stockwell fixed], or [stockwell infra], and a frequent-failure comment has been made, the whiteboard is updated to [stockwell needswork:owner].
• It appears that [stockwell needswork] is no longer set by the robot.

Weekly

• If a bug already has [stockwell needswork] or [stockwell needswork:owner] and the weekly failure count is less than 20, the whiteboard is updated to [stockwell unknown].

Priority

• If the whiteboard was updated to [stockwell needswork:owner] and the priority is not one of ("--", "P1", "P2", "P3"), the priority is set to "--" to indicate this bug needs triage.