75
edits
New test environments: Difference between revisions
Jump to navigation
Jump to search
→Green up tests: expand the test disable section
(→Green up tests: expand the test disable section) |
|||
| Line 297: | Line 297: | ||
* meta bugs created in Bugzilla? | * meta bugs created in Bugzilla? | ||
* responsibilities for suites/subsuites divided amongst engineers (if multiple engineers assigned)? | * responsibilities for suites/subsuites/chunks divided amongst engineers (if multiple engineers assigned)? | ||
* for each | * for each failure in a chunk, associate an existing bug or create a new bug. | ||
* for each bug created, has triage owner been notified with ''needinfo'' describing the nature of the problem? | * for each bug created, has triage owner been notified with ''needinfo'' describing the nature of the problem? | ||
== Submit patches == | |||
It is not always possible for developers to address an issue, especially if nature of the failure is deemed to be lower in priority. | |||
Given that the new test environment setup should be time-bounded, this means that often it becomes necessary to disable the particular test and/or manifest files. | |||
=== Example === | |||
[https://phabricator.services.mozilla.com/D36985 macosx1014 - D36985] | |||
In the example above, it became necessary to disable certain tests that continued to fail on macosx1014 despite the test/feature owner being notified of the failure for several weeks. | |||
Generally speaking, the following steps should be taken prior to test being disabled: | |||
* if a bug has not been filed, please do so. | |||
* ''needinfo'' the triage/test/module owner. | |||
* comment on the bug to maintain activity every 2-3 weeks, preferably with status update and newer logs. | |||
If no movement occurs: | |||
* annotate the test case with narrowest possible criteria. | |||
* include comment that specifies platform shorthand and bug number. | |||
* submit to phabricator. | |||
* submit a Try push with the proposed patch applied to verify the work. | |||
Be sure to include in the review the test/module/triage owner and ''intermittent-reviewers'' group. | |||
== Checklist == | |||
* has triage/test/module owner been ''needinfo'''d? | |||
* has consistent activity been reported on the bug for a few weeks? | |||
* is the annotation using narrowest possible criteria? | |||
* has a bug number been added to the comment for the annotation? | |||
* has the owner been notified in the phabricator patch? | |||