canmove, Confirmed users
1,126
edits
Sheriffing/Test Disabling Policy: Difference between revisions
m
Correct OrangeFactor link
(→Identifying problematic tests: Making the real-world application of this policy clearer) |
m (Correct OrangeFactor link) |
||
| (One intermediate revision by the same user not shown) | |||
| Line 4: | Line 4: | ||
This policy will define an escalation path for when a single test case is identified to be leaking or failing and is causing enough disruption on the trees. Disruption is defined as any of: | This policy will define an escalation path for when a single test case is identified to be leaking or failing and is causing enough disruption on the trees. Disruption is defined as any of: | ||
* Test case is on the list of top 20 intermittent failures on [[http://brasstacks.mozilla.com/orangefactor/ | * Test case is on the list of top 20 intermittent failures on [[http://brasstacks.mozilla.com/orangefactor/ Orange Factor]] | ||
* It is causing oranges >=8% of the time | * It is causing oranges >=8% of the time | ||
* We have >100 instances of this failure in the bug in the last 30 days | * We have >100 instances of this failure in the bug in the last 30 days | ||
| Line 29: | Line 29: | ||
This is not intended to be perfect, but here are some common exceptions we should keep in mind: | This is not intended to be perfect, but here are some common exceptions we should keep in mind: | ||
* If this test has landed (or been modified) in the last | * If this test has landed (or been modified) in the last 7 days, we will most likely back out the patch with the test. | ||
* If we can identify a non test related change to the product by failure patterns and retriggers, we will push to patch the change or back it out | * If we can identify a non test related change to the product by failure patterns and retriggers, we will push to patch the change or back it out. | ||
* If a test is failing at least 30% of the time, we will file a bug and disable the test first | * If a test is failing at least 30% of the time, we will file a bug and disable the test first. | ||
* When we are bringing a new platform online (Android 2.3, b2g, etc.) many tests will need to be disabled prior to getting the tests on | * When we are bringing a new platform online (Android 2.3, b2g, etc.) many tests will need to be disabled prior to getting the tests on TBPL. | ||
* In the rare case we are disabling the majority of the tests (either at once or slowly over time) for a given feature, we need to get the module owner to sign off on the current state of the tests. | * In the rare case we are disabling the majority of the tests (either at once or slowly over time) for a given feature, we need to get the module owner to sign off on the current state of the tests. | ||