canmove, Confirmed users
1,126
edits
Sheriffing/Test Disabling Policy: Difference between revisions
Jump to navigation
Jump to search
→Identifying problematic tests: Making the real-world application of this policy clearer
(→Identifying problematic tests: Making the real-world application of this policy clearer) |
|||
| Line 3: | Line 3: | ||
== Identifying problematic tests == | == Identifying problematic tests == | ||
This policy will define an escalation path for when a single test case is identified to be leaking or failing and is causing enough disruption on the trees. Disruption is defined as: | This policy will define an escalation path for when a single test case is identified to be leaking or failing and is causing enough disruption on the trees. Disruption is defined as any of: | ||
* Test case is on the list of top 20 intermittent failures on [[http://brasstacks.mozilla.com/orangefactor/index.html Orange Factor]] | * Test case is on the list of top 20 intermittent failures on [[http://brasstacks.mozilla.com/orangefactor/index.html Orange Factor]] | ||
* It is causing oranges >=8% of the time | * It is causing oranges >=8% of the time | ||
* We have >100 instances of this failure in the bug in the last 30 days | * We have >100 instances of this failure in the bug in the last 30 days | ||
Note: Whilst priority is tests which meet several of the conditions, meeting one is still sufficient to be be considered worthy of escalation. | |||
== Escalation == | == Escalation == | ||