Auto-tools/Projects/TestHarnesses: Difference between revisions
(created initial wiki for harnesses) |
(→Milestones and Dates: - fixed bug) |
||
| (One intermediate revision by the same user not shown) | |||
| Line 28: | Line 28: | ||
As of March 2015, our current priorities are: | As of March 2015, our current priorities are: | ||
* [https://bugzilla.mozilla.org/show_bug.cgi?id= | * [https://bugzilla.mozilla.org/show_bug.cgi?id=1124182 fine grain test selection] | ||
* json manifests [https://bugzilla.mozilla.org/show_bug.cgi?id=1083347 android], webgl, imptests | * json manifests [https://bugzilla.mozilla.org/show_bug.cgi?id=1083347 android], webgl, imptests | ||
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1116187 mochitest chrome on b2g] | * [https://bugzilla.mozilla.org/show_bug.cgi?id=1116187 mochitest chrome on b2g] | ||
| Line 35: | Line 35: | ||
* [https://bugzilla.mozilla.org/show_bug.cgi?id=928983 windows process ctrl+c] | * [https://bugzilla.mozilla.org/show_bug.cgi?id=928983 windows process ctrl+c] | ||
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1058923 run from tests.zip] | * [https://bugzilla.mozilla.org/show_bug.cgi?id=1058923 run from tests.zip] | ||
= Implementation = | = Implementation = | ||
| Line 61: | Line 60: | ||
"quicksearch": "status:new,assigned,reopened,unconfirmed", | "quicksearch": "status:new,assigned,reopened,unconfirmed", | ||
"product": "Testing", | "product": "Testing", | ||
"whiteboard": "ateam_harness_work" | "whiteboard": "ateam_harness_work" | ||
} | } | ||
</bugzilla> | </bugzilla> | ||
Latest revision as of 13:38, 11 March 2015
Team
Problem
Mozilla has thousands of tests which run 10+ harnesses. As Firefox evolves in features and supported platforms, the test harnesses need to accommodate the needs and challenges faced to keep our new and existing tests running reliably.
Goals & Considerations
Our primary goal is to ensure our harnesses share as much code as possible and remove old code which depended on older features (of python, firefox, etc.) or was not written in a scalable way. Other goals are to simplify our manifests used while allowing the manifest solution to have more flexibility in scheduling tests.
Non-Goals
We are not interested in rewriting our harnesses from scratch or making the perfect harness. We are not looking to switch from Python and Javascript.
Dependencies / Who will use this
Harnesses are primarily used by automation and developers. We need to support harnesses from up to 9 revisions ago (almost a year) when we look at ESR and older B2G branches.
Design and Approach
Overview of test jobs and their harnesses]
Milestones and Dates
As of March 2015, our current priorities are:
- fine grain test selection
- json manifests android, webgl, imptests
- mochitest chrome on b2g
- ssltunnel to android
- error reporting for asan xpcshell
- windows process ctrl+c
- run from tests.zip
Implementation
Getting Involved
Getting Started
The best way is to find a bug and ask the mentor. In general, it is best to have a local build of Firefox
Expectations
For the majority of our work, it lives in mozilla-central (hg, not git) and we follow these guidelines.
Bugs
| ID | Summary | Priority | Status |
|---|---|---|---|
| 1112322 | Improve error reporting on xpcshell ASAN failures | -- | NEW |
| 1124182 | [Tracking] Fine grained test selection | -- | NEW |
2 Total; 2 Open (100%); 0 Resolved (0%); 0 Verified (0%);