Electrolysis/Test Coverage: Difference between revisions
Jump to navigation
Jump to search
(mochitest-push merged back into mochitest-plain (instead of a standalone job)) |
(add more Ash coverage info) |
||
| Line 53: | Line 53: | ||
|- | |- | ||
!style='text-align:left;'|R(C) crash tests | !style='text-align:left;'|R(C) crash tests | ||
|style='text-align:center;'|Ash: | |style='text-align:center;'|Ash: 0 | ||
|style='text-align:center;'|Ash: 100 | |style='text-align:center;'|Ash: 100 | ||
|style='text-align:center;'|100 | |style='text-align:center;'|100 | ||
| Line 61: | Line 61: | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
|style='text-align:center;'|Ash: | |style='text-align:center;'|Ash: 0 | ||
|style='text-align:center;'|99 | |style='text-align:center;'|99 | ||
|style='text-align:center;'|100 | |style='text-align:center;'|100 | ||
| Line 171: | Line 171: | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
|- | |- | ||
!style='text-align:left;'| | !style='text-align:left;'|M(1-5) | ||
|style='text-align:center;'|Ash: | |style='text-align:center;'|Ash: Y | ||
|style='text-align:center;'|Ash: | |style='text-align:center;'|Ash: Y | ||
|style='text-align:center;'| | |style='text-align:center;'|Ash: Y | ||
|style='text-align:center;'| | |style='text-align:center;'|trunk: 82<br/>aurora: 0<br/>beta: 0 | ||
|style='text-align:center;'|Ash: | |style='text-align:center;'|Ash: Y | ||
|style='text-align:center;'|Ash: | |style='text-align:center;'|Ash: Y | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
|style='text-align:center;'|Ash: Y | |||
|style='text-align:center;'|trunk: 82<br/>aurora: 0<br/>beta: 0 | |||
|style='text-align:center;'|trunk: 81<br/>aurora: 81<br/>beta: 79 | |||
|style='text-align:center;'|trunk: 82<br/>aurora: 82<br/>beta: 80 | |||
|style='text-align:center;'|trunk: 81<br/>aurora: 81<br/>beta: 79 | |||
|style='text-align:center;'|trunk: 82<br/>aurora: 82<br/>beta: 80 | |||
|style='text-align:center;'|Ash: | |style='text-align:center;'|trunk: 82<br/>aurora: 82<br/>beta: 80 | ||
|style='text-align:center;'| | |||
|style='text-align:center;'| | |||
|style='text-align:center;'| | |||
|style='text-align:center;'| | |||
|style='text-align:center;'| | |||
|style='text-align:center;'| | |||
|- | |- | ||
!style='text-align:left;'|M(bc) | !style='text-align:left;'|M(bc) | ||
|style='text-align:center;'| | |style='text-align:center;'|Ash: 84 | ||
|style='text-align:center;'| | |style='text-align:center;'|Ash: 84 | ||
|style='text-align:center;'| | |style='text-align:center;'|Ash: 84 | ||
|style='text-align:center;'|trunk: 84<br/>aurora: 82<br/>beta: 75 | |style='text-align:center;'|trunk: 84<br/>aurora: 82<br/>beta: 75 | ||
|style='text-align:center;'| | |style='text-align:center;'|Ash: 84 | ||
|style='text-align:center;'|trunk: 84<br/>aurora: 82<br/>beta: 75 | |style='text-align:center;'|trunk: 84<br/>aurora: 82<br/>beta: 75 | ||
|style='text-align:center;'| | |style='text-align:center;'|n/a | ||
|style='text-align:center;'|trunk: 84<br/>aurora: 82<br/>beta: 75 | |style='text-align:center;'|trunk: 84<br/>aurora: 82<br/>beta: 75 | ||
|style='text-align:center;'|trunk: 84<br/>aurora: 0<br/>beta: 0 | |style='text-align:center;'|trunk: 84<br/>aurora: 0<br/>beta: 0 | ||
| Line 240: | Line 206: | ||
|- | |- | ||
!style='text-align:left;'|M(dt) | !style='text-align:left;'|M(dt) | ||
|style='text-align:center;'| | |style='text-align:center;'|Ash: 75 | ||
|style='text-align:center;'| | |style='text-align:center;'|Ash: 89 | ||
|style='text-align:center;'|trunk: 75<br/>aurora: 0<br/>beta: 0 | |style='text-align:center;'|trunk: 75<br/>aurora: 0<br/>beta: 0 | ||
|style='text-align:center;'|trunk: 89<br/>aurora: 0<br/>beta: 0 | |style='text-align:center;'|trunk: 89<br/>aurora: 0<br/>beta: 0 | ||
|style='text-align:center;'|Ash: 75 | |||
|style='text-align:center;'|Ash: 89 | |||
|style='text-align:center;'|n/a | |||
|style='text-align:center;'|n/a | |||
|style='text-align:center;'|Ash: 75 | |||
|style='text-align:center;'|trunk: 89<br/>aurora: 0<br/>beta: 0 | |||
|style='text-align:center;'|Ash: 75 | |||
|style='text-align:center;'|trunk: 90<br/>aurora: 88<br/>beta: 87 | |||
|style='text-align:center;'|Ash: 75 | |||
|style='text-align:center;'|trunk: 90<br/>aurora: 88<br/>beta: 87 | |||
|style='text-align:center;'|Ash: 90 | |||
|- | |||
!style='text-align:left;'|M(gl) | |||
|style='text-align:center;'|Ash: 100 | |||
|style='text-align:center;'|Ash: 100 | |||
|style='text-align:center;'|100 | |||
|style='text-align:center;'|100 | |||
|style='text-align:center;'|Ash: 100 | |||
|style='text-align:center;'|Ash: 100 | |||
|style='text-align:center;'|n/a | |||
|style='text-align:center;'|n/a | |||
|style='text-align:center;'|Ash: 0 | |||
|style='text-align:center;'|100 | |||
|style='text-align:center;'|100 | |||
|style='text-align:center;'|100 | |||
|style='text-align:center;'|100 | |||
|style='text-align:center;'|100 | |||
|style='text-align:center;'|100 | |||
|- | |||
!style='text-align:left;'|M(JP) jetpack [{{bug|1246453}}] | |||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
| Line 249: | Line 245: | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
| Line 266: | Line 255: | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|- | |- | ||
!style='text-align:left;'|M( | !style='text-align:left;'|M(oth-a11y) [[#Mochitest_a11y|[2]]] | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
| Line 290: | Line 273: | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|- | |- | ||
!style='text-align:left;'|M( | !style='text-align:left;'|M(oth-chrome) [[#Mochitest_chrome|[3]]] | ||
|style='text-align:center;'| | |||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
| Line 300: | Line 283: | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|style='text-align:center;'| | |style='text-align:center;'| | ||
|- | |||
!style='text-align:left;'|W(1-8) web platform tests | |||
|style='text-align:center;'|Ash: 99 | |||
|style='text-align:center;'|Ash: 99 | |||
|style='text-align:center;'|99 | |||
|style='text-align:center;'|99 | |||
|style='text-align:center;'|Ash: 99 | |||
|style='text-align:center;'|Ash: 99 | |||
|style='text-align:center;'|n/a | |||
|style='text-align:center;'|n/a | |||
|style='text-align:center;'|99(ab) | |style='text-align:center;'|99(ab) | ||
|style='text-align:center;'|99(ab) | |style='text-align:center;'|99(ab) | ||
| Line 322: | Line 305: | ||
|style='text-align:center;'|99(ab) | |style='text-align:center;'|99(ab) | ||
|style='text-align:center;'|99(ab) | |style='text-align:center;'|99(ab) | ||
|style='text-align:center;'| | |style='text-align:center;'|Ash: 99 | ||
|- | |||
!style='text-align:left;'|Wr web platform reftest | |||
|style='text-align:center;'|Ash: 100 | |||
|style='text-align:center;'|Ash: 100 | |||
|style='text-align:center;'|100 | |||
|style='text-align:center;'|100 | |||
|style='text-align:center;'|Ash: 100 | |||
|style='text-align:center;'|Ash: 100 | |||
|style='text-align:center;'|n/a | |||
|style='text-align:center;'|n/a | |||
|style='text-align:center;'|100(ab) | |||
|style='text-align:center;'|100(ab) | |||
|style='text-align:center;'|100 | |||
|style='text-align:center;'|100(ab) | |||
|style='text-align:center;'|100 | |||
|style='text-align:center;'|100(ab) | |||
|style='text-align:center;'|Ash: 100 | |||
|} | |} | ||
Base values indicate coverage on mozilla-central. annotations (a, b, r) indicate support for other releases. Current coverage here reflects 47 on mc, 46 on ma. Tests suite coverage is rolling out with the trains. | Base values indicate coverage on mozilla-central. annotations (a, b, r) indicate support for other releases. Current coverage here reflects 47 on mc, 46 on ma. Tests suite coverage is rolling out with the trains.<br> | ||
Entries with "Ash" next to them are for suites that are running on the Ash project branch, which is | Entries with "Ash" next to them are for suites that are running on the Ash project branch, which is staying synced with mozilla-central. | ||
== Related Bugs == | == Related Bugs == | ||
Revision as of 22:48, 18 March 2016
Enabled Test Suites By Platform
| Test Suite | Windows | OSX | Linux | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| XP [1] | 7 | 8 [1] | 10.6 | 10.10 | 32-bit | 64-bit | asan | |||||||||
| Dbg | Opt | Dbg | Opt | Dbg | Opt | Dbg | Opt | Dbg | Opt | Dbg | Opt | Dbg | Opt | Opt | ||
| R(J) js-ref tests | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 | Ash: 100 | n/a | n/a | Ash: 100 | 100 | 100 | 100 | 100 | 100 | 100 | |
| R(C) crash tests | Ash: 0 | Ash: 100 | 100 | 100 | Ash: 100 | Ash: 100 | n/a | n/a | Ash: 0 | 99 | 100 | 100 | 100 | 100 | 100 | |
| R(R) ref tests | Ash: 99 | Ash: 99 | 99 | 99 | Ash: 99 | Ash: 99 | n/a | n/a | Ash: 99 | 99 | 99 | 99 | 99 | 99 | [0] Ash: 99 | |
| R(Ru) ref tests | n/a | n/a | 0 | Ash: 99 | 0 | Ash: 99 | n/a | n/a | n/a | n/a | 0 | Ash: 99 | 0 | Ash: 99 | Ash: 99 | |
| Cpp | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | |
| X xpc shell tests | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | |
| Mn marionette | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 | Ash: 100 | n/a | n/a | 100 | 100 | 100 | 100 | 100 | 100 | 0 | |
| JIT jit tests | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | |
| M(1-5) | Ash: Y | Ash: Y | Ash: Y | trunk: 82 aurora: 0 beta: 0 |
Ash: Y | Ash: Y | n/a | n/a | Ash: Y | trunk: 82 aurora: 0 beta: 0 |
trunk: 81 aurora: 81 beta: 79 |
trunk: 82 aurora: 82 beta: 80 |
trunk: 81 aurora: 81 beta: 79 |
trunk: 82 aurora: 82 beta: 80 |
trunk: 82 aurora: 82 beta: 80 | |
| M(bc) | Ash: 84 | Ash: 84 | Ash: 84 | trunk: 84 aurora: 82 beta: 75 |
Ash: 84 | trunk: 84 aurora: 82 beta: 75 |
n/a | trunk: 84 aurora: 82 beta: 75 |
trunk: 84 aurora: 0 beta: 0 |
trunk: 84 aurora: 82 beta: 75 |
trunk: 83 aurora: 81 beta: 77 |
trunk: 84 aurora: 82 beta: 76 |
trunk: 83 aurora: 81 beta: 77 |
trunk: 84 aurora: 82 beta: 76 |
trunk: 84 aurora: 82 beta: 76 | |
| M(dt) | Ash: 75 | Ash: 89 | trunk: 75 aurora: 0 beta: 0 |
trunk: 89 aurora: 0 beta: 0 |
Ash: 75 | Ash: 89 | n/a | n/a | Ash: 75 | trunk: 89 aurora: 0 beta: 0 |
Ash: 75 | trunk: 90 aurora: 88 beta: 87 |
Ash: 75 | trunk: 90 aurora: 88 beta: 87 |
Ash: 90 | |
| M(gl) | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 | Ash: 100 | n/a | n/a | Ash: 0 | 100 | 100 | 100 | 100 | 100 | 100 | |
| M(JP) jetpack [bug 1246453] | ||||||||||||||||
| M(oth-a11y) [2] | ||||||||||||||||
| M(oth-chrome) [3] | ||||||||||||||||
| W(1-8) web platform tests | Ash: 99 | Ash: 99 | 99 | 99 | Ash: 99 | Ash: 99 | n/a | n/a | 99(ab) | 99(ab) | 99(ab) | 99(ab) | 99(ab) | 99(ab) | Ash: 99 | |
| Wr web platform reftest | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 | Ash: 100 | n/a | n/a | 100(ab) | 100(ab) | 100 | 100(ab) | 100 | 100(ab) | Ash: 100 | |
Base values indicate coverage on mozilla-central. annotations (a, b, r) indicate support for other releases. Current coverage here reflects 47 on mc, 46 on ma. Tests suite coverage is rolling out with the trains.
Entries with "Ash" next to them are for suites that are running on the Ash project branch, which is staying synced with mozilla-central.
Related Bugs
- bug 1194533 - Schedule e10s tests on OSX 10.10
- bug 1215233 - Schedule e10s tests on Windows 7
- bug 1253849 - Get e10s tests passing on all Windows versions in CI
- Due to capacity limitations, non-Win7 versions will run on a reduced interval
Individual Test Issues
Mochitest a11y
a11y is known to not work under e10s, and the tests in this suite largely fail, so there are no short-term plans to enable running this suite in continuous integration.
Mochitest chrome
These tests are thought to be largely non-e10s specific, so it doesn't make sense to run them in the content process in e10s mode. There is work underway to identify tests which are relevant to e10s to and to figure out what to do with them. There are no short-term plans to enable this suite in continuous integration.
Windows Coverage
Because of constraints on our Windows hardware test pools, we're not planning to enable any e10s suites on Windows XP or Windows 8. Instead, Windows 7 is the focus of all our e10s test coverage.