Electrolysis/Test Coverage: Difference between revisions
Jump to navigation
Jump to search
(all osx 10.10 debug tests are now running on 48+) |
(OSX 10.6 support has been dropped, remove references to it) |
||
| Line 5: | Line 5: | ||
!colspan="1"|Test Suite | !colspan="1"|Test Suite | ||
!colspan="6"|Windows | !colspan="6"|Windows | ||
!colspan=" | !colspan="2"|OSX | ||
!colspan=" | !colspan="5"|Linux | ||
|- | |- | ||
!colspan="1"| | !colspan="1"| | ||
| Line 12: | Line 12: | ||
!colspan="2"|7 (32-bit) | !colspan="2"|7 (32-bit) | ||
!colspan="2"|8 (64-bit) | !colspan="2"|8 (64-bit) | ||
!colspan="2"|10.10 (64-bit) | !colspan="2"|10.10 (64-bit) | ||
!colspan="2"|32-bit | !colspan="2"|32-bit | ||
| Line 18: | Line 17: | ||
|- | |- | ||
!colspan="1"| | !colspan="1"| | ||
!colspan="1"|Debug | !colspan="1"|Debug | ||
!colspan="1"|Opt | !colspan="1"|Opt | ||
| Line 35: | Line 32: | ||
|- | |- | ||
!style='text-align:left;'|Cpp | !style='text-align:left;'|Cpp | ||
|style='text-align:center;'| n/a | |style='text-align:center;'| n/a | ||
|style='text-align:center;'| n/a | |style='text-align:center;'| n/a | ||
| Line 52: | Line 47: | ||
|- | |- | ||
!style='text-align:left;'|JIT jit tests | !style='text-align:left;'|JIT jit tests | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
| Line 69: | Line 62: | ||
|- | |- | ||
!style='text-align:left;'|X xpcshell tests | !style='text-align:left;'|X xpcshell tests | ||
|style='text-align:center;'| n/a | |style='text-align:center;'| n/a | ||
|style='text-align:center;'| n/a | |style='text-align:center;'| n/a | ||
| Line 92: | Line 83: | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
| Line 109: | Line 98: | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
| Line 126: | Line 113: | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
| Line 143: | Line 128: | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
| Line 160: | Line 143: | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
| Line 177: | Line 158: | ||
|style='text-align:center;background:lightgreen;'|Ash: 99 | |style='text-align:center;background:lightgreen;'|Ash: 99 | ||
|style='text-align:center;background:lightgreen;'|Ash: 99 | |style='text-align:center;background:lightgreen;'|Ash: 99 | ||
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 0 | |style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 0 | ||
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 81 | |style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 81 | ||
| Line 194: | Line 173: | ||
|style='text-align:center;background:yellow;'|Ash: [https://bugzilla.mozilla.org/show_bug.cgi?id=1253956 80] | |style='text-align:center;background:yellow;'|Ash: [https://bugzilla.mozilla.org/show_bug.cgi?id=1253956 80] | ||
|style='text-align:center;background:lightgreen;'|trunk: 98<br/>aurora: 98<br/>beta: 82 | |style='text-align:center;background:lightgreen;'|trunk: 98<br/>aurora: 98<br/>beta: 82 | ||
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 82 | |style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 82 | ||
|style='text-align:center;background:lightgreen;'|trunk: 98<br/>aurora: 98<br/>beta: 82 | |style='text-align:center;background:lightgreen;'|trunk: 98<br/>aurora: 98<br/>beta: 82 | ||
| Line 211: | Line 188: | ||
|style='text-align:center;background:yellow;'|Ash: 72 | |style='text-align:center;background:yellow;'|Ash: 72 | ||
|style='text-align:center;background:yellow;'|Ash: 84 | |style='text-align:center;background:yellow;'|Ash: 84 | ||
|style='text-align:center;background:yellow;'|trunk: 72<br/>aurora: 72<br/>beta: 0 | |style='text-align:center;background:yellow;'|trunk: 72<br/>aurora: 72<br/>beta: 0 | ||
|style='text-align:center;background:yellow;'|trunk: 84<br/>aurora: 84<br/>beta: 80 | |style='text-align:center;background:yellow;'|trunk: 84<br/>aurora: 84<br/>beta: 80 | ||
| Line 228: | Line 203: | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
| Line 245: | Line 218: | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
| Line 262: | Line 233: | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
| Line 273: | Line 242: | ||
|- | |- | ||
!style='text-align:left;'|M(oth/chrome) [[#Mochitest Chrome|[2]]] | !style='text-align:left;'|M(oth/chrome) [[#Mochitest Chrome|[2]]] | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
| Line 296: | Line 263: | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
| Line 313: | Line 278: | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
| Line 334: | Line 297: | ||
=== Other Notes === | === Other Notes === | ||
* Yellow cells have incomplete test coverage (<85%) or serious issues that prevent the suite from running reliably (asserts, leaks, crashes, frequent intermittent failures). | * Yellow cells have incomplete test coverage (<85%) or serious issues that prevent the suite from running reliably (asserts, leaks, crashes, frequent intermittent failures). | ||
== Individual Test Issues == | == Individual Test Issues == | ||
Revision as of 19:24, 17 May 2016
Enabled Test Suites By Platform
| Test Suite | Windows | OSX | Linux | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| XP (32-bit) | 7 (32-bit) | 8 (64-bit) | 10.10 (64-bit) | 32-bit | 64-bit | ||||||||
| Debug | Opt | Debug | Opt | Debug | Opt | Debug | Opt | Debug | Opt | Debug | Opt | ASAN | |
| Cpp | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
| JIT jit tests | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
| X xpcshell tests | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
| Mn marionette | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 | Ash: 100 | 100 | 100 | 100 | 100 | 100 | 100 | Ash: 100 |
| R(C) crashtests | Ash: [0] | Ash: 100 | 100 | 100 | Ash: 100 | Ash: 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 |
| R(J) jsreftests | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 | Ash: 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 |
| R(R) reftests | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 | Ash: 100 | 100 | 100 | 100 | 100 | 100 | 100 | [0] Ash: 100 |
| R(Ru) reftests (no accel) | n/a | n/a | Ash: 100 | Ash: 100 | Ash: 100 | Ash: 100 | n/a | n/a | Ash: 100 | Ash: 100 | Ash: 100 | Ash: 100 | Ash: 100 |
| M(1-5) | Ash: 99 | Ash: 99 | Ash: 99 | trunk: 99 aurora: 99 beta: 0 |
Ash: 99 | Ash: 99 | trunk: 99 aurora: 99 beta: 0 |
trunk: 99 aurora: 99 beta: 81 |
trunk: 99 aurora: 99 beta: 81 |
trunk: 99 aurora: 99 beta: 82 |
trunk: 99 aurora: 99 beta: 81 |
trunk: 99 aurora: 99 beta: 82 |
trunk: 99 aurora: 99 beta: 82 |
| M(bc) | Ash: 80 | Ash: 98 | Ash: 80 | trunk: 98 aurora: 98 beta: 82 |
Ash: 80 | trunk: 98 aurora: 98 beta: 82 |
trunk: 97 aurora: 97 beta: 82 |
trunk: 98 aurora: 98 beta: 82 |
trunk: 97 aurora: 97 beta: 81 |
trunk: 97 aurora: 97 beta: 82 |
trunk: 97 aurora: 97 beta: 81 |
trunk: 97 aurora: 97 beta: 82 |
trunk: 97 aurora: 97 beta: 82 |
| M(dt) | Ash: 72 | Ash: 84 | trunk: 72 aurora: 72 beta: 70 |
trunk: 84 aurora: 84 beta: 80 |
Ash: 72 | Ash: 84 | trunk: 72 aurora: 72 beta: 0 |
trunk: 84 aurora: 84 beta: 80 |
Ash: 72 | trunk: 90 aurora: 90 beta: 85 |
Ash: 72 | trunk: 90 aurora: 90 beta: 85 |
Ash: 90 |
| M(gl) | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 | Ash: 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 |
| M(JP) jetpack [1] | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| M(oth/a11y) [1] | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| M(oth/chrome) [2] | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
| W(1-8) web platform tests | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 | Ash: 100 | 100 | 100 | 100 | 100 | 100 | 100 | Ash: 100 |
| Wr web platform reftest | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 | Ash: 100 | 100 | 100 | 100 | 100 | 100 | 100 | Ash: 100 |
Base values indicate coverage on mozilla-central. Current coverage here reflects 49 on central, 48 on aurora, and 47 on beta. Most production test suite coverage is riding the trains with a specific Gecko version.
Entries with "Ash" next to them are for suites that are running on the Ash project branch, which is being kept in sync with mozilla-central.
Mochitest a11y/Jetpack
a11y is known to not work under e10s and the tests in this suite largely fail. Jetpack tests are also known to be broken at this point in time (bug 1246453). However, initial rollout plans for e10s don't include users with a11y enabled or with addons installed, so there are no short-term plans to enable these suites in continuous integration.
Mochitest Chrome
These tests are thought to be largely non-e10s specific, so it doesn't make sense to run them in the content process in e10s mode. There is work underway to identify tests which are relevant to e10s to and to figure out what to do with them. There are no short-term plans to enable this suite in continuous integration.
Other Notes
- Yellow cells have incomplete test coverage (<85%) or serious issues that prevent the suite from running reliably (asserts, leaks, crashes, frequent intermittent failures).
Individual Test Issues
- mochitest coverage spreadsheet
- partially disabled mochitest spreadsheet
- other test suite coverage spreadsheet
Related Bugs
- bug 1194533 - Schedule e10s tests on OSX 10.10
- bug 1215233 - Schedule e10s tests on Windows 7
- bug 1253849 - Get e10s tests passing on all Windows versions in CI