Electrolysis/Test Coverage: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(→‎11/03/2015: update tests running on linux)
(update windows debug mochitest-bc numbers)
 
(68 intermediate revisions by 3 users not shown)
Line 1: Line 1:


== 10/19/2015 ==
== Enabled Test Suites By Platform ==
{| class="wikitable"
{| class="wikitable"
|-
|-
!colspan="1"|Test Suite
!colspan="1"|Test Suite
!colspan="6"|Windows
!colspan="6"|Windows
!colspan="6"|OSX
!colspan="2"|OSX
!colspan="6"|Linux
!colspan="5"|Linux
|-
|-
!colspan="1"|
!colspan="1"|
!colspan="2"|XP
!colspan="2"|XP (32-bit)
!colspan="2"|7
!colspan="2"|7 (32-bit)
!colspan="2"|8
!colspan="2"|8 (64-bit)
!colspan="2"|10.6
!colspan="2"|10.10 (64-bit)
!colspan="2"|10.7
!colspan="2"|10.10
!colspan="2"|32-bit
!colspan="2"|32-bit
!colspan="2"|64-bit
!colspan="3"|64-bit
!colspan="1"|asan
|-
|-
!colspan="1"|
!colspan="1"|
!colspan="1"|Dbg
!colspan="1"|Debug
!colspan="1"|Opt
!colspan="1"|Opt
!colspan="1"|Dbg
!colspan="1"|Debug
!colspan="1"|Opt
!colspan="1"|Opt
!colspan="1"|Dbg
!colspan="1"|Debug
!colspan="1"|Opt
!colspan="1"|Opt
!colspan="1"|Dbg
!colspan="1"|Debug
!colspan="1"|Opt
!colspan="1"|Opt
!colspan="1"|Dbg
!colspan="1"|Debug
!colspan="1"|Opt
!colspan="1"|Opt
!colspan="1"|Dbg
!colspan="1"|Debug
!colspan="1"|Opt
!colspan="1"|Opt
!colspan="1"|Dbg
!colspan="1"|ASAN
!colspan="1"|Opt
!colspan="1"|Dbg
!colspan="1"|Opt
!colspan="1"|
|-
|-
!style='text-align:left;'|R(J) js-ref tests
!style='text-align:left;'|Cpp
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|-
|-
!style='text-align:left;'|R(C) crash tests
!style='text-align:left;'|JIT jit tests
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|99
|style='text-align:center;'|n/a
|style='text-align:center;'|99
|style='text-align:center;'|99
|style='text-align:center;'|99
|style='text-align:center;'|
|-
|-
!style='text-align:left;'|R(R) ref tests
!style='text-align:left;'|X xpcshell tests
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|
|style='text-align:center;'| n/a
|style='text-align:center;'|99
|style='text-align:center;'| n/a
|style='text-align:center;'|99
|style='text-align:center;'|99
|style='text-align:center;'|99
|style='text-align:center;'|
|-
|-
!style='text-align:left;'|Cpp
!style='text-align:left;'|Mn marionette
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|-
|-
!style='text-align:left;'|X xpc shell tests
!style='text-align:left;'|R(C) crashtests
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|-
|-
!style='text-align:left;'|Mn marionette
!style='text-align:left;'|R(J) jsreftests
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|99
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|-
!style='text-align:left;'|JIT jit tests
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|-
|-
!style='text-align:left;'|Wr web platform reftest
!style='text-align:left;'|R(R) reftests
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|99
|style='text-align:center;background:lightgreen;'|[https://bugzilla.mozilla.org/show_bug.cgi?id=1223198 [0]]<br>Ash: 100
|style='text-align:center;'|99
|style='text-align:center;'|99
|style='text-align:center;'|99
|style='text-align:center;'|
|-
|-
!style='text-align:left;'|M(oth-a11y)
!style='text-align:left;'|R(Ru) reftests (no accel)
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|[https://bugzilla.mozilla.org/show_bug.cgi?id=1223198 [0]]<br>Ash: 100
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|-
|-
!style='text-align:left;'|M(oth-chrome)
!style='text-align:left;'|M(1-5)
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 99
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 99
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|-
|-
!style='text-align:left;'|M(bc)
!style='text-align:left;'|M(bc)
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 97
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 98
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97
|style='text-align:center;'|49
|style='text-align:center;background:lightgreen;'|trunk: 98<br/>aurora: 98<br/>beta: 98
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 97<br/>aurora: 97<br/>beta: 97
|style='text-align:center;'|49
|style='text-align:center;background:lightgreen;'|trunk: 98<br/>aurora: 98<br/>beta: 98
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97
|style='text-align:center;'|48
|style='text-align:center;background:lightgreen;'|trunk: 98<br/>aurora: 98<br/>beta: 98
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97
|style='text-align:center;'|48
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97
|style='text-align:center;'|47
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97
|style='text-align:center;'|48
|style='text-align:center;'|47
|style='text-align:center;'|48
|style='text-align:center;'|48
|-
|-
!style='text-align:left;'|M(dt)
!style='text-align:left;'|M(dt)
|style='text-align:center;'|
|style='text-align:center;background:yellow;'|Ash: 72
|style='text-align:center;'|
|style='text-align:center;background:yellow;'|Ash: 84
|style='text-align:center;'|
|style='text-align:center;background:yellow;'|trunk: 72<br/>aurora: 72<br/>beta: 72
|style='text-align:center;'|
|style='text-align:center;background:yellow;'|trunk: 84<br/>aurora: 84<br/>beta: 84
|style='text-align:center;'|
|style='text-align:center;background:yellow;'|Ash: 72<br/>aurora: 72<br/>beta: 72
|style='text-align:center;'|
|style='text-align:center;background:yellow;'|Ash: 84<br/>aurora: 84<br/>beta: 84
|style='text-align:center;'|
|style='text-align:center;background:yellow;'|trunk: 72<br/>aurora: 72<br/>beta: 72
|style='text-align:center;'|
|style='text-align:center;background:yellow;'|trunk: 84<br/>aurora: 84<br/>beta: 84
|style='text-align:center;'|
|style='text-align:center;background:yellow;'|Ash: 72
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 90<br/>aurora: 90<br/>beta: 90
|style='text-align:center;'|
|style='text-align:center;background:yellow;'|Ash: 72
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 90<br/>aurora: 90<br/>beta: 90
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|trunk: 90<br/>aurora: 90<br/>beta: 90
|style='text-align:center;'|49
|style='text-align:center;'|
|style='text-align:center;'|49
|style='text-align:center;'|
|-
|-
!style='text-align:left;'|M(gl)
!style='text-align:left;'|M(gl)
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|-
|-
!style='text-align:left;'|M(JP) jetpack
!style='text-align:left;'|M(mda)
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|-
|-
!style='text-align:left;'|M(1-5)
!style='text-align:left;'|M(JP) jetpack [[#Mochitest a11y/jetpack|[1]]]
|style='text-align:center;'|
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;'|
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;'|
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;'|
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;'|
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;'|
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;'|
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;'|
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;'|
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;'|
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;'|
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;'|
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;'|85
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;'|87
|-
|style='text-align:center;'|85
!style='text-align:left;'|M(oth/a11y) [[#Mochitest a11y/Jetpack|[1]]]
|style='text-align:center;'|87
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;'|87
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;background:lightcyan;'|0
|style='text-align:center;background:lightcyan;'|0
|-
|-
!style='text-align:left;'|M(p) push
!style='text-align:left;'|M(oth/chrome) [[#Mochitest Chrome|[2]]]
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|n/a
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|style='text-align:center;'|
|-
|-
!style='text-align:left;'|W(1-8) web platform tests
!style='text-align:left;'|W(1-8) web platform tests
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;'|99
|style='text-align:center;background:yellow;'|Ash: 100
|style='text-align:center;'|99
|-
|style='text-align:center;'|99
!style='text-align:left;'|Wr web platform reftest
|style='text-align:center;'|99
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;'|
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:yellow;'|Ash: 100
|}
|}
Base values indicate coverage on mozilla-central. Current coverage here reflects 50 on central, 49 on aurora, and 48 on beta. Most production test suite coverage is riding the trains with a specific Gecko version.<br>
Entries with "Ash" next to them are for suites that are running on the [https://treeherder.mozilla.org/#/jobs?repo=ash Ash project branch], which is being kept in sync with mozilla-central, in lieu of running on trunk.<br>


== Related Bugs ==
=== Mochitest a11y/Jetpack ===
* {{bug|1194533}} - Run e10s mochitest on OSX
a11y is known to not work under e10s and the tests in this suite largely fail. Jetpack tests are also known to be broken at this point in time ({{bug|1246453}}). However, initial rollout plans for e10s don't include users with a11y enabled or with addons installed, so there are no short-term plans to enable these suites in continuous integration.
* {{bug|1215233}} - Schedule e10s tests on Windows
 
== Needed Suites ==


What additional suites need to be run under e10s and why.
=== Mochitest Chrome ===
These tests are thought to be largely non-e10s specific, so it doesn't make sense to run them in the content process in e10s mode. There is work underway to identify tests which are relevant to e10s to and to figure out what to do with them. There are no short-term plans to enable this suite in continuous integration.


* The e10s team is planning to hold a "future of" mochitest-chrome discussion at Mozlando.
=== Other Notes ===
* Yellow cells have incomplete test coverage (<85%) or serious issues that prevent the suite from running reliably (asserts, leaks, crashes, frequent intermittent failures).


== Individual Test Issues ==
== Individual Test Issues ==
* [https://docs.google.com/spreadsheets/d/10UeyRoiWV2HjkWwAU51HXyXAV7YLi4BjDm55mr5Xv6c/edit?usp=sharing mochitest coverage spreadsheet]
* [https://docs.google.com/spreadsheets/d/1oBnRct8FuTnAoVfbE08m10UHi1_5kOFaOPglJ6ittWc/edit?usp=sharing partially disabled mochitest spreadsheet]
* [https://docs.google.com/spreadsheets/d/1YQ6HHcg14iPtA5hvNklnCrinfPmoxXyC_iqAczX9U2Y/edit?usp=sharing other test suite coverage spreadsheet]


Individual tests under currently running suites that need to be investigated.
== Related Bugs ==
 
* {{bug|1194533}} - Schedule e10s tests on OSX 10.10
43 disabled testcases in browser/base/content/test/
* {{bug|1215233}} - Schedule e10s tests on Windows 7
 
* {{bug|1253849}} - Get e10s tests passing on all Windows versions in CI
===.../alerts/ (2) ===
* [browser_notification_remove_permission.js] - no bug listed
* [browser_notification_tab_switching.js] Bug 1100662 - content access causing uncaught exception - Error: cannot ipc non-cpow object at chrome://mochitests/content/browser/browser/base/content/test/general/browser_notification_tab_switching.js:32 (or in RemoteAddonsChild.jsm)
 
=== …/general/ (32) ===
* [browser_aboutHome.js] - Bug 1093153 - no about:home support yet (difficult)
* [browser_search_favicon.js] -  Bug 1212647 (not ready to be turned on)
* [browser_autocomplete_a11y_label.js] Bug 1101993 - times out for unknown reasons when run in the dir (works on its own)
* [browser_beforeunload_duplicate_dialogs.js] - bug 967873 means permitUnload doesn't work in e10s mode (depends on bug 1191143)
* [browser_bug435325.js] - Bug 1099156 - test directly manipulates content
* [browser_bug462289.js] - Bug 1102017 - middle-button mousedown on selected tab2 does not activate tab - Didn't expect [object XULElement], but got it (same 3 tests fail in non-e10s as well)
* [browser_bug517902.js] - bug 1161699 (linux only)
* [browser_bug537013.js] - Bug 1134458 - Find bar doesn't work correctly in a detached tab (5 test failures)
* [browser_bug906190.js] - Bug 1093642 - test manipulates content and relies on content focus, Bug 1212520 - Re-enable on Linux (unable to run scripts internally?)
* [browser_bug1015721.js] -  Bug 1159268 - Need a content-process safe version of synthesizeWheel (is this code or test work?)
* [browser_canonizeURL.js] - Bug 1094510 - test hits the network in e10s mode only (WFM on mac)
* [browser_contextSearchTabPosition.js] -bug 1094761 - test hits the network in e10s, causing next test to crash  (WFM on Mac)
* [browser_datachoices_notification.js] - bug 1113930 (WFM on Mac)
* [browser_devices_get_user_media_about_urls.js] -  Bug 1071623 (one of five tests timed out)
* [browser_devices_get_user_media_in_frame.js] -  Bug 1071623 (one of two tests timed out)
* [browser_double_close_tab.js] - no bug listed (test timed out)
* [browser_fullscreen-window-open.js] - Bug 933103 - mochitest's (gTestBrowser.contentWindow.location is undefined) EventUtils.synthesizeMouse functions not e10s friendly. Linux: Intermittent failures - bug 941575.
* [browser_gestureSupport.js] - Bug 863514 - no gesture support.
* [browser_keywordBookmarklets.js] - Bug 1102025 - different principals for the bookmarklet only in e10s mode (unclear if test or 'real' issue)
* [browser_offlineQuotaNotification.js] - Bug 1093603 - test breaks with PopupNotifications.panel.firstElementChild is null (times out)
* [browser_popup_blocker.js] - Frequent bug 1081925 and bug 1125520 failures (WFM on Mac)
* [browser_printpreview.js] - Bug 1101973 - breaks the next test in e10s, and may be responsible for later timeout after logging "Error: Channel closing: too late to send/recv, messages will be lost”
* [browser_save_link-perwindowpb.js] - Bug 933103 - mochitest's (times out) EventUtils.synthesizeMouse functions not e10s friendly
* [browser_save_private_link_perwindowpb.js] - Bug 933103 - mochitest's EventUtils.synthesizeMouse functions not e10s friendly
* [browser_save_link_when_window_navigates.js] - Bug 933103 - mochitest's EventUtils.synthesizeMouse functions not e10s friendly
* [browser_tabMatchesInAwesomebar_perwindowpb.js] - Bug 1093373, bug 1104755 (empty string passed)
* [browser_tabkeynavigation.js] - no bug listed (can’t pass CPOW to native)
* [browser_unloaddialogs.js] - Bug 1100700 - test relies on unload event firing on closed tabs, which it doesn’t (before unload work?)
* [browser_urlbarEnterAfterMouseOver.js] - Bug 1073339 - Investigate autocomplete test unreliability on Linux/e10s (broken on Mac too)
* [browser_zbug569342.js] - Bug 1094240 - has findbar-related failures (code disabled this?)
* [browser_addCertException.js] - Bug 1100687 - test directly manipulates content (content.document.getElementById) (timed out)
* [browser_blockHPKP.js] - bug 1100687 - test directly manipulates content (content.document.getElementById) ("textElement is null”)
 
=== …/popupNotifications/ (4) ===
 
4 of the 5 test cases in this folder are skipped for e10s
 
=== .../social/ (5) ===
* [browser_addons.js] - Leaking docshells (bug 1150147)
* [browser_blocklist.js] - Leaking docshells (bug 1150147)
* [browser_social_activation.js] - Leaking docshells (bug 1150147)
* [browser_social_chatwindowfocus.js] - tab crash on data url used in this test
* [browser_social_contextmenu.js] - Bug 1072669 context menu relies on target element

Latest revision as of 16:25, 23 June 2016

Enabled Test Suites By Platform

Test Suite Windows OSX Linux
XP (32-bit) 7 (32-bit) 8 (64-bit) 10.10 (64-bit) 32-bit 64-bit
Debug Opt Debug Opt Debug Opt Debug Opt Debug Opt Debug Opt ASAN
Cpp n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a
JIT jit tests n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a
X xpcshell tests n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a
Mn marionette Ash: 100 Ash: 100 100 100 Ash: 100
aurora: 100
beta: 100 		Ash: 100
aurora: 100
beta: 100 		100 100 100 100 100 100 100
R(C) crashtests Ash: 100 Ash: 100 100 100 Ash: 100
aurora: 100
beta: 100 		Ash: 100
aurora: 100
beta: 100 		100 100 100 100 100 100 100
R(J) jsreftests Ash: 100 Ash: 100 100 100 Ash: 100
aurora: 100
beta: 100 		Ash: 100
aurora: 100
beta: 100 		100 100 100 100 100 100 100
R(R) reftests Ash: 100 Ash: 100 100 100 Ash: 100
aurora: 100
beta: 100 		Ash: 100
aurora: 100
beta: 100 		100 100 100 100 100 100 [0]
Ash: 100
R(Ru) reftests (no accel) n/a n/a 100 100 Ash: 100
aurora: 100
beta: 100 		Ash: 100
aurora: 100
beta: 100 		n/a n/a 100 100 100 100 [0]
Ash: 100
M(1-5) Ash: 99 Ash: 99 trunk: 99
aurora: 99
beta: 99 		trunk: 99
aurora: 99
beta: 99 		Ash: 99
aurora: 99
beta: 99 		Ash: 99
aurora: 99
beta: 99 		trunk: 99
aurora: 99
beta: 99 		trunk: 99
aurora: 99
beta: 99 		trunk: 99
aurora: 99
beta: 99 		trunk: 99
aurora: 99
beta: 99 		trunk: 99
aurora: 99
beta: 99 		trunk: 99
aurora: 99
beta: 99 		trunk: 99
aurora: 99
beta: 99
M(bc) Ash: 97 Ash: 98 trunk: 97
aurora: 97
beta: 97 		trunk: 98
aurora: 98
beta: 98 		Ash: 97
aurora: 97
beta: 97 		trunk: 98
aurora: 98
beta: 98 		trunk: 97
aurora: 97
beta: 97 		trunk: 98
aurora: 98
beta: 98 		trunk: 97
aurora: 97
beta: 97 		trunk: 97
aurora: 97
beta: 97 		trunk: 97
aurora: 97
beta: 97 		trunk: 97
aurora: 97
beta: 97 		trunk: 97
aurora: 97
beta: 97
M(dt) Ash: 72 Ash: 84 trunk: 72
aurora: 72
beta: 72 		trunk: 84
aurora: 84
beta: 84 		Ash: 72
aurora: 72
beta: 72 		Ash: 84
aurora: 84
beta: 84 		trunk: 72
aurora: 72
beta: 72 		trunk: 84
aurora: 84
beta: 84 		Ash: 72 trunk: 90
aurora: 90
beta: 90 		Ash: 72 trunk: 90
aurora: 90
beta: 90 		trunk: 90
aurora: 90
beta: 90
M(gl) Ash: 100 Ash: 100 100 100 Ash: 100
aurora: 100
beta: 100 		Ash: 100
aurora: 100
beta: 100 		100 100 100 100 100 100 100
M(mda) Ash: 100 Ash: 100 100 100 Ash: 100
aurora: 100
beta: 100 		Ash: 100
aurora: 100
beta: 100 		100 100 100 100 100 100 100
M(JP) jetpack [1] 0 0 0 0 0 0 0 0 0 0 0 0 0
M(oth/a11y) [1] 0 0 0 0 0 0 0 0 0 0 0 0 0
M(oth/chrome) [2] n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a
W(1-8) web platform tests Ash: 100 Ash: 100 100 100 Ash: 100
aurora: 100
beta: 100 		Ash: 100
aurora: 100
beta: 100 		100 100 100 100 100 100 Ash: 100
Wr web platform reftest Ash: 100 Ash: 100 100 100 Ash: 100
aurora: 100
beta: 100 		Ash: 100
aurora: 100
beta: 100 		100 100 100 100 100 100 Ash: 100

Base values indicate coverage on mozilla-central. Current coverage here reflects 50 on central, 49 on aurora, and 48 on beta. Most production test suite coverage is riding the trains with a specific Gecko version.
Entries with "Ash" next to them are for suites that are running on the Ash project branch, which is being kept in sync with mozilla-central, in lieu of running on trunk.

Mochitest a11y/Jetpack

a11y is known to not work under e10s and the tests in this suite largely fail. Jetpack tests are also known to be broken at this point in time (bug 1246453). However, initial rollout plans for e10s don't include users with a11y enabled or with addons installed, so there are no short-term plans to enable these suites in continuous integration.

Mochitest Chrome

These tests are thought to be largely non-e10s specific, so it doesn't make sense to run them in the content process in e10s mode. There is work underway to identify tests which are relevant to e10s to and to figure out what to do with them. There are no short-term plans to enable this suite in continuous integration.

Other Notes

  • Yellow cells have incomplete test coverage (<85%) or serious issues that prevent the suite from running reliably (asserts, leaks, crashes, frequent intermittent failures).

Individual Test Issues

Related Bugs

Retrieved from "https://wiki.mozilla.org/index.php?title=Electrolysis/Test_Coverage&oldid=1137464"