75
edits
New test environments: Difference between revisions
Jump to navigation
Jump to search
Added greening section
(→Test configuration: collapse by default) |
(Added greening section) |
||
| Line 105: | Line 105: | ||
=== Checklist === | === Checklist === | ||
* has meta bug been created in Bugzilla in Firefox Build System::Task Configuration component? | |||
* does <code>./mach taskgraph full</code> succeed? | * does <code>./mach taskgraph full</code> succeed? | ||
* does build successfully complete on Tryserver? | * does build successfully complete on Tryserver? | ||
| Line 215: | Line 216: | ||
Once build and tests are enabled on Tryserver, it is time to run a baseline push to take inventory of suites that pass and fail. | Once build and tests are enabled on Tryserver, it is time to run a baseline push to take inventory of suites that pass and fail. | ||
Using <code>./mach try fuzzy --no-artifact</code>, push tests belonging to the new platform to the Tryserver. Do not use artifact builds for the baseline push as some test results (outside of the compiled tests such as <code>cppunit</code>) are affected by the artifact build. | Using <code>./mach try fuzzy --no-artifact --rebuild 10</code>, push tests belonging to the new platform to the Tryserver. Do not use artifact builds for the baseline push as some test results (outside of the compiled tests such as <code>cppunit</code>) are affected by the artifact build. | ||
Once the checklist is complete, create a diff on Phabricator and have it reviewed by | Once the checklist is complete, create a diff on Phabricator and have it reviewed by | ||
| Line 228: | Line 229: | ||
* does the build succeed? | * does the build succeed? | ||
* are | * are all tests scheduled? | ||
* | * how many tests abort/retry? | ||
* | * how many tests pass? | ||
* | * how many tests fail? | ||
* | * what test chunks appear intermittent? | ||
* for failed tests, is it due to platform configuration issues or legitimate failures? | |||
= Green up tests = | = Green up tests = | ||
Once the build and tests are running on Tryserver, it is time to begin greening the tests. | Once the build and tests are running on Tryserver, it is time to begin greening the tests. | ||
This task is typically the one that consumes the most time as well as having subtle nuances in how the task should be handled. | |||
For this step, it is possible to have several engineers each tackle a selected suite/subsuite: | |||
* engineer A | |||
** mochitest-browser-chrome | |||
** gtest | |||
* engineer B | |||
** web-platform-tests | |||
** reftests | |||
Make sure to create meta bugs for each broad category in Bugzilla. A typical breakdown: | |||
* platform meta bug | |||
** xpcshell | |||
** mochitest | |||
** compiled test | |||
** reftest/crashtest | |||
** raptor | |||
and so on. | |||
An example reference is [https://bugzilla.mozilla.org/showdependencytree.cgi?id=1572242&hide_resolved=1 Debian 10 meta]. | |||
== Checklist == | |||
* meta bugs created in Bugzilla? | |||