canmove, Confirmed users
1,126
edits
Sheriffing/Test Disabling Policy: Difference between revisions
→Exceptions: 48 hours is too short to notice, file, notice frequent & recently landed; changing to 7 days
(→Identifying problematic tests: Making the real-world application of this policy clearer) |
(→Exceptions: 48 hours is too short to notice, file, notice frequent & recently landed; changing to 7 days) |
||
| Line 29: | Line 29: | ||
This is not intended to be perfect, but here are some common exceptions we should keep in mind: | This is not intended to be perfect, but here are some common exceptions we should keep in mind: | ||
* If this test has landed (or been modified) in the last | * If this test has landed (or been modified) in the last 7 days, we will most likely back out the patch with the test. | ||
* If we can identify a non test related change to the product by failure patterns and retriggers, we will push to patch the change or back it out | * If we can identify a non test related change to the product by failure patterns and retriggers, we will push to patch the change or back it out. | ||
* If a test is failing at least 30% of the time, we will file a bug and disable the test first | * If a test is failing at least 30% of the time, we will file a bug and disable the test first. | ||
* When we are bringing a new platform online (Android 2.3, b2g, etc.) many tests will need to be disabled prior to getting the tests on | * When we are bringing a new platform online (Android 2.3, b2g, etc.) many tests will need to be disabled prior to getting the tests on TBPL. | ||
* In the rare case we are disabling the majority of the tests (either at once or slowly over time) for a given feature, we need to get the module owner to sign off on the current state of the tests. | * In the rare case we are disabling the majority of the tests (either at once or slowly over time) for a given feature, we need to get the module owner to sign off on the current state of the tests. | ||