Electrolysis/Test Coverage: Difference between revisions
Jump to navigation
Jump to search
(Improve table readability) |
(update windows debug mochitest-bc numbers) |
||
| (24 intermediate revisions by the same user not shown) | |||
| Line 5: | Line 5: | ||
!colspan="1"|Test Suite | !colspan="1"|Test Suite | ||
!colspan="6"|Windows | !colspan="6"|Windows | ||
!colspan=" | !colspan="2"|OSX | ||
!colspan=" | !colspan="5"|Linux | ||
|- | |- | ||
!colspan="1"| | !colspan="1"| | ||
!colspan="2"|XP | !colspan="2"|XP (32-bit) | ||
!colspan="2"|7 | !colspan="2"|7 (32-bit) | ||
!colspan="2"|8 | !colspan="2"|8 (64-bit) | ||
!colspan="2"|10.10 (64-bit) | |||
!colspan="2"|10.10 | |||
!colspan="2"|32-bit | !colspan="2"|32-bit | ||
!colspan="3"|64-bit | !colspan="3"|64-bit | ||
|- | |- | ||
!colspan="1"| | !colspan="1"| | ||
!colspan="1"|Debug | !colspan="1"|Debug | ||
!colspan="1"|Opt | !colspan="1"|Opt | ||
| Line 35: | Line 32: | ||
|- | |- | ||
!style='text-align:left;'|Cpp | !style='text-align:left;'|Cpp | ||
|style='text-align:center;'| n/a | |style='text-align:center;'| n/a | ||
|style='text-align:center;'| n/a | |style='text-align:center;'| n/a | ||
| Line 52: | Line 47: | ||
|- | |- | ||
!style='text-align:left;'|JIT jit tests | !style='text-align:left;'|JIT jit tests | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
| Line 68: | Line 61: | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
|- | |- | ||
!style='text-align:left;'|X | !style='text-align:left;'|X xpcshell tests | ||
|style='text-align:center;'| n/a | |style='text-align:center;'| n/a | ||
|style='text-align:center;'| n/a | |style='text-align:center;'| n/a | ||
| Line 90: | Line 81: | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | ||
|style='text-align:center; | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
| Line 100: | Line 90: | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|- | |- | ||
!style='text-align:left;'|R(C) crashtests | !style='text-align:left;'|R(C) crashtests | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | |||
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | |||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
| Line 124: | Line 111: | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | |||
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|- | |||
!style='text-align:left;'|R(R) reftests | |||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | |||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
| Line 135: | Line 134: | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|[https://bugzilla.mozilla.org/show_bug.cgi?id=1223198 [0]]<br>Ash: 100 | |||
|style='text-align:center;background:lightgreen;'|[https://bugzilla.mozilla.org/show_bug.cgi?id=1223198 [0]]<br>Ash: | |||
|- | |- | ||
!style='text-align:left;'|R(Ru) reftests (no accel) | !style='text-align:left;'|R(Ru) reftests (no accel) | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
|style='text-align:center;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;'| | |style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: | |style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
|style='text-align:center;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|Ash: | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align: | |style='text-align:center;background:lightgreen;'|[https://bugzilla.mozilla.org/show_bug.cgi?id=1223198 [0]]<br>Ash: 100 | ||
|- | |||
!style='text-align:left;'|M(1-5) | |||
|style='text-align:center;background:lightgreen;'|Ash: 99 | |style='text-align:center;background:lightgreen;'|Ash: 99 | ||
|style='text-align:center;background:lightgreen;'|Ash: 99 | |style='text-align:center;background:lightgreen;'|Ash: 99 | ||
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99 | |||
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99 | |||
|style='text-align:center;background:lightgreen;'| | |style='text-align:center;background:lightgreen;'|Ash: 99<br/>aurora: 99<br/>beta: 99 | ||
|style='text-align:center;background:lightgreen;'|Ash: 99<br/>aurora: 99<br/>beta: 99 | |||
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99 | |||
|style='text-align:center;background:lightgreen;'|trunk: | |style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99 | ||
|style='text-align:center;background:lightgreen;'|Ash: | |style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99 | ||
|style='text-align:center;background:lightgreen;'|Ash: | |style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99 | ||
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99 | |||
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99 | |||
|style='text-align:center;background:lightgreen;'| | |style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99 | ||
|style='text-align:center;background:lightgreen;'|trunk: | |||
|style='text-align:center;background:lightgreen;'|trunk: | |||
|style='text-align:center;background:lightgreen;'|trunk: | |||
|style='text-align:center;background:lightgreen;'|trunk: | |||
|style='text-align:center;background:lightgreen;'|trunk: | |||
|style='text-align:center;background:lightgreen;'|trunk: | |||
|- | |- | ||
!style='text-align:left;'|M(bc) | !style='text-align:left;'|M(bc) | ||
|style='text-align:center;background: | |style='text-align:center;background:lightgreen;'|Ash: 97 | ||
|style='text-align:center;background:lightgreen;'|Ash: | |style='text-align:center;background:lightgreen;'|Ash: 98 | ||
|style='text-align:center;background: | |style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97 | ||
|style='text-align:center;background:lightgreen;'|trunk: | |style='text-align:center;background:lightgreen;'|trunk: 98<br/>aurora: 98<br/>beta: 98 | ||
|style='text-align:center;background: | |style='text-align:center;background:lightgreen;'|Ash: 97<br/>aurora: 97<br/>beta: 97 | ||
|style='text-align:center;background:lightgreen;'|trunk: 98<br/>aurora: 98<br/>beta: 98 | |||
|style='text-align:center; | |style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97 | ||
|style='text-align:center;background:lightgreen;'|trunk: 98<br/>aurora: 98<br/>beta: 98 | |||
|style='text-align:center;background:lightgreen;'|trunk: | |style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97 | ||
|style='text-align:center;background:lightgreen;'|trunk: | |style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97 | ||
|style='text-align:center;background:lightgreen;'|trunk: | |style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97 | ||
|style='text-align:center;background:lightgreen;'|trunk: | |style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97 | ||
|style='text-align:center;background:lightgreen;'|trunk: | |style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97 | ||
|style='text-align:center;background:lightgreen;'|trunk: | |||
|style='text-align:center;background:lightgreen;'|trunk: | |||
|- | |- | ||
!style='text-align:left;'|M(dt) | !style='text-align:left;'|M(dt) | ||
|style='text-align:center;background:yellow;'|Ash: | |style='text-align:center;background:yellow;'|Ash: 72 | ||
|style='text-align:center;background: | |style='text-align:center;background:yellow;'|Ash: 84 | ||
|style='text-align:center;background:yellow;'|trunk: | |style='text-align:center;background:yellow;'|trunk: 72<br/>aurora: 72<br/>beta: 72 | ||
|style='text-align:center;background: | |style='text-align:center;background:yellow;'|trunk: 84<br/>aurora: 84<br/>beta: 84 | ||
|style='text-align:center;background:yellow;'|Ash: | |style='text-align:center;background:yellow;'|Ash: 72<br/>aurora: 72<br/>beta: 72 | ||
|style='text-align:center;background: | |style='text-align:center;background:yellow;'|Ash: 84<br/>aurora: 84<br/>beta: 84 | ||
|style='text-align:center;background:yellow;'|trunk: 72<br/>aurora: 72<br/>beta: 72 | |||
|style='text-align:center;background:yellow;'|trunk: 84<br/>aurora: 84<br/>beta: 84 | |||
|style='text-align:center;background:yellow;'| | |style='text-align:center;background:yellow;'|Ash: 72 | ||
|style='text-align:center;background: | |style='text-align:center;background:lightgreen;'|trunk: 90<br/>aurora: 90<br/>beta: 90 | ||
|style='text-align:center;background:yellow;'|Ash: | |style='text-align:center;background:yellow;'|Ash: 72 | ||
|style='text-align:center;background:lightgreen;'|trunk: 90<br/>aurora: | |style='text-align:center;background:lightgreen;'|trunk: 90<br/>aurora: 90<br/>beta: 90 | ||
|style='text-align:center;background:yellow;'|Ash: | |style='text-align:center;background:lightgreen;'|trunk: 90<br/>aurora: 90<br/>beta: 90 | ||
|style='text-align:center;background:lightgreen;'|trunk: 90<br/>aurora: | |||
|style='text-align:center;background: | |||
|- | |- | ||
!style='text-align:left;'|M(gl) | !style='text-align:left;'|M(gl) | ||
| Line 226: | Line 201: | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | |||
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|- | |||
!style='text-align:left;'|M(mda) | |||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background: | |style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
| Line 238: | Line 226: | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|- | |- | ||
!style='text-align:left;'|M(JP) jetpack | !style='text-align:left;'|M(JP) jetpack [[#Mochitest a11y/jetpack|[1]]] | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
| Line 245: | Line 233: | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
| Line 255: | Line 241: | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
|- | |- | ||
!style='text-align:left;'|M(oth | !style='text-align:left;'|M(oth/a11y) [[#Mochitest a11y/Jetpack|[1]]] | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
| Line 262: | Line 248: | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
| Line 272: | Line 256: | ||
|style='text-align:center;background:lightcyan;'|0 | |style='text-align:center;background:lightcyan;'|0 | ||
|- | |- | ||
!style='text-align:left;'|M(oth | !style='text-align:left;'|M(oth/chrome) [[#Mochitest Chrome|[2]]] | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
|style='text-align:center;'|n/a | |style='text-align:center;'|n/a | ||
| Line 290: | Line 272: | ||
|- | |- | ||
!style='text-align:left;'|W(1-8) web platform tests | !style='text-align:left;'|W(1-8) web platform tests | ||
|style='text-align:center;background:lightgreen;'|Ash: | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: | |style='text-align:center;background:lightgreen;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|Ash: | |style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: | |style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | ||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'|100 | |||
|style='text-align:center;background:lightgreen;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'| | |style='text-align:center;background:yellow;'|Ash: 100 | ||
|style='text-align:center;background:lightgreen;'| | |||
|style='text-align:center;background:yellow;'|Ash: | |||
|- | |- | ||
!style='text-align:left;'|Wr web platform reftest | !style='text-align:left;'|Wr web platform reftest | ||
| Line 311: | Line 291: | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | ||
|style='text-align:center;background:lightgreen;'|Ash: 100 | |style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100 | ||
|style='text-align:center;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;'| | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:lightgreen;'|100 | |style='text-align:center;background:lightgreen;'|100 | ||
|style='text-align:center;background:yellow;'|Ash: 100 | |style='text-align:center;background:yellow;'|Ash: 100 | ||
|} | |} | ||
Base values indicate coverage on mozilla-central. Current coverage here reflects 50 on central, 49 on aurora, and 48 on beta. Most production test suite coverage is riding the trains with a specific Gecko version.<br> | |||
Entries with "Ash" next to them are for suites that are running on the [https://treeherder.mozilla.org/#/jobs?repo=ash Ash project branch], which is being kept in sync with mozilla-central, in lieu of running on trunk.<br> | |||
=== Mochitest a11y/Jetpack === | |||
a11y is known to not work under e10s and the tests in this suite largely fail. Jetpack tests are also known to be broken at this point in time ({{bug|1246453}}). However, initial rollout plans for e10s don't include users with a11y enabled or with addons installed, so there are no short-term plans to enable these suites in continuous integration. | |||
=== Mochitest Chrome === | |||
These tests are thought to be largely non-e10s specific, so it doesn't make sense to run them in the content process in e10s mode. There is work underway to identify tests which are relevant to e10s to and to figure out what to do with them. There are no short-term plans to enable this suite in continuous integration. | |||
=== Other Notes === | |||
* Yellow cells have incomplete test coverage (<85%) or serious issues that prevent the suite from running reliably (asserts, leaks, crashes, frequent intermittent failures). | |||
* | == Individual Test Issues == | ||
* | * [https://docs.google.com/spreadsheets/d/10UeyRoiWV2HjkWwAU51HXyXAV7YLi4BjDm55mr5Xv6c/edit?usp=sharing mochitest coverage spreadsheet] | ||
* | * [https://docs.google.com/spreadsheets/d/1oBnRct8FuTnAoVfbE08m10UHi1_5kOFaOPglJ6ittWc/edit?usp=sharing partially disabled mochitest spreadsheet] | ||
* [https://docs.google.com/spreadsheets/d/1YQ6HHcg14iPtA5hvNklnCrinfPmoxXyC_iqAczX9U2Y/edit?usp=sharing other test suite coverage spreadsheet] | |||
== Related Bugs == | == Related Bugs == | ||
| Line 333: | Line 322: | ||
* {{bug|1215233}} - Schedule e10s tests on Windows 7 | * {{bug|1215233}} - Schedule e10s tests on Windows 7 | ||
* {{bug|1253849}} - Get e10s tests passing on all Windows versions in CI | * {{bug|1253849}} - Get e10s tests passing on all Windows versions in CI | ||
Latest revision as of 16:25, 23 June 2016
Enabled Test Suites By Platform
| Test Suite | Windows | OSX | Linux | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| XP (32-bit) | 7 (32-bit) | 8 (64-bit) | 10.10 (64-bit) | 32-bit | 64-bit | ||||||||
| Debug | Opt | Debug | Opt | Debug | Opt | Debug | Opt | Debug | Opt | Debug | Opt | ASAN | |
| Cpp | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
| JIT jit tests | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
| X xpcshell tests | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
| Mn marionette | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 aurora: 100 beta: 100 |
Ash: 100 aurora: 100 beta: 100 |
100 | 100 | 100 | 100 | 100 | 100 | 100 |
| R(C) crashtests | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 aurora: 100 beta: 100 |
Ash: 100 aurora: 100 beta: 100 |
100 | 100 | 100 | 100 | 100 | 100 | 100 |
| R(J) jsreftests | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 aurora: 100 beta: 100 |
Ash: 100 aurora: 100 beta: 100 |
100 | 100 | 100 | 100 | 100 | 100 | 100 |
| R(R) reftests | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 aurora: 100 beta: 100 |
Ash: 100 aurora: 100 beta: 100 |
100 | 100 | 100 | 100 | 100 | 100 | [0] Ash: 100 |
| R(Ru) reftests (no accel) | n/a | n/a | 100 | 100 | Ash: 100 aurora: 100 beta: 100 |
Ash: 100 aurora: 100 beta: 100 |
n/a | n/a | 100 | 100 | 100 | 100 | [0] Ash: 100 |
| M(1-5) | Ash: 99 | Ash: 99 | trunk: 99 aurora: 99 beta: 99 |
trunk: 99 aurora: 99 beta: 99 |
Ash: 99 aurora: 99 beta: 99 |
Ash: 99 aurora: 99 beta: 99 |
trunk: 99 aurora: 99 beta: 99 |
trunk: 99 aurora: 99 beta: 99 |
trunk: 99 aurora: 99 beta: 99 |
trunk: 99 aurora: 99 beta: 99 |
trunk: 99 aurora: 99 beta: 99 |
trunk: 99 aurora: 99 beta: 99 |
trunk: 99 aurora: 99 beta: 99 |
| M(bc) | Ash: 97 | Ash: 98 | trunk: 97 aurora: 97 beta: 97 |
trunk: 98 aurora: 98 beta: 98 |
Ash: 97 aurora: 97 beta: 97 |
trunk: 98 aurora: 98 beta: 98 |
trunk: 97 aurora: 97 beta: 97 |
trunk: 98 aurora: 98 beta: 98 |
trunk: 97 aurora: 97 beta: 97 |
trunk: 97 aurora: 97 beta: 97 |
trunk: 97 aurora: 97 beta: 97 |
trunk: 97 aurora: 97 beta: 97 |
trunk: 97 aurora: 97 beta: 97 |
| M(dt) | Ash: 72 | Ash: 84 | trunk: 72 aurora: 72 beta: 72 |
trunk: 84 aurora: 84 beta: 84 |
Ash: 72 aurora: 72 beta: 72 |
Ash: 84 aurora: 84 beta: 84 |
trunk: 72 aurora: 72 beta: 72 |
trunk: 84 aurora: 84 beta: 84 |
Ash: 72 | trunk: 90 aurora: 90 beta: 90 |
Ash: 72 | trunk: 90 aurora: 90 beta: 90 |
trunk: 90 aurora: 90 beta: 90 |
| M(gl) | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 aurora: 100 beta: 100 |
Ash: 100 aurora: 100 beta: 100 |
100 | 100 | 100 | 100 | 100 | 100 | 100 |
| M(mda) | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 aurora: 100 beta: 100 |
Ash: 100 aurora: 100 beta: 100 |
100 | 100 | 100 | 100 | 100 | 100 | 100 |
| M(JP) jetpack [1] | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| M(oth/a11y) [1] | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| M(oth/chrome) [2] | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
| W(1-8) web platform tests | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 aurora: 100 beta: 100 |
Ash: 100 aurora: 100 beta: 100 |
100 | 100 | 100 | 100 | 100 | 100 | Ash: 100 |
| Wr web platform reftest | Ash: 100 | Ash: 100 | 100 | 100 | Ash: 100 aurora: 100 beta: 100 |
Ash: 100 aurora: 100 beta: 100 |
100 | 100 | 100 | 100 | 100 | 100 | Ash: 100 |
Base values indicate coverage on mozilla-central. Current coverage here reflects 50 on central, 49 on aurora, and 48 on beta. Most production test suite coverage is riding the trains with a specific Gecko version.
Entries with "Ash" next to them are for suites that are running on the Ash project branch, which is being kept in sync with mozilla-central, in lieu of running on trunk.
Mochitest a11y/Jetpack
a11y is known to not work under e10s and the tests in this suite largely fail. Jetpack tests are also known to be broken at this point in time (bug 1246453). However, initial rollout plans for e10s don't include users with a11y enabled or with addons installed, so there are no short-term plans to enable these suites in continuous integration.
Mochitest Chrome
These tests are thought to be largely non-e10s specific, so it doesn't make sense to run them in the content process in e10s mode. There is work underway to identify tests which are relevant to e10s to and to figure out what to do with them. There are no short-term plans to enable this suite in continuous integration.
Other Notes
- Yellow cells have incomplete test coverage (<85%) or serious issues that prevent the suite from running reliably (asserts, leaks, crashes, frequent intermittent failures).
Individual Test Issues
- mochitest coverage spreadsheet
- partially disabled mochitest spreadsheet
- other test suite coverage spreadsheet
Related Bugs
- bug 1194533 - Schedule e10s tests on OSX 10.10
- bug 1215233 - Schedule e10s tests on Windows 7
- bug 1253849 - Get e10s tests passing on all Windows versions in CI