New test environments: Difference between revisions

← Older edit

New test environments (view source)

Revision as of 18:58, 14 January 2020

397 bytes added , 14 January 2020

→‎Green up tests: added needinfo section, WIP migrate over a test suite section

Egao

75

edits

@@ Line 285: / Line 285: @@
 ** compiled tests
 ** reftest/crashtest
-** web-platform-tests
+** web-platform-tests (including web-platform variants of reftests/crashtests)
+** miscellaneous tests
-and so on.
+Mochitest is a particularly large suite and has many subsuites, so in practice a separate meta bug for each mochitest subsuite could be useful.
-Mochitest is a particularly large suite and has many subsuites, so it may be good practice to create a meta bug for each mochitest subsuite.
+Reference: [https://bugzilla.mozilla.org/showdependencytree.cgi?id=1572242&hide_resolved=1 Ubuntu1804] meta bug.
-A useful reference is the [https://bugzilla.mozilla.org/showdependencytree.cgi?id=1572242&hide_resolved=1 Debian 10 meta] bug.
 == Reporting failures ==
@@ Line 297: / Line 296: @@
 Use Treeherder where possible when logging failures - this allows all sorts of backend connections to be made, and also nicely includes the debug logs and such if available.
-There are some nuances to consider when logging failures; notably, <b>''has this bug been reported before?''</b> This is critical as everyone should strive to reduce the number of duplicate bugs in the system as much as possible.
+There are some nuances to consider when logging failures; notably, <b>''has this bug been reported before?''</b> Where possible, reduce duplicate bugs in the system as much as possible.
 Example workflows in different situations will be given below.
@@ Line 320: / Line 319: @@
 * in the body, specify the platform, subsuite and paste the relevant portion of the log.
 * fill in the <code>blocks</code>, <code>depends on</code> and <code>see also</code> bug as appropriate.
-* when bug is successfully created, ''needinfo'' request with a brief description of the reason for the request and provide additional context such as failure rates on new platform.
+=== Needinfo test owners ===
+This is the most important part of the process.
+Get the bug in front of the triage owner, feature owner or the test developer and provide as much information as possible relating to nature of the bug, reproduction steps, any extra commands or steps necessary, etc.
+Set a ''needinfo'' request and ask them to chime in.
+If no response is received after some time, usually a week, mark some activity to 'bump' the bug.
+Depending on the migration timeframe, this may happen anywhere between 2-4 times.
 === Checklist ===
@@ Line 363: / Line 372: @@
 * has a bug number been added to the comment for the annotation?
 * has the owner been notified in the phabricator patch?
+== Migrate over a test suite ==