Electrolysis/Test Coverage: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(more tests enabled on Aurora48)
(update for merge day version number changes and release branches running more test suites)
Line 81: Line 81:
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
Line 89: Line 90:
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|trunk: 100<br/>aurora: 100<br/>beta: 0
|-
|-
!style='text-align:left;'|R(C) crashtests
!style='text-align:left;'|R(C) crashtests
Line 96: Line 96:
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
Line 111: Line 111:
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
Line 126: Line 126:
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
Line 139: Line 139:
|style='text-align:center;'|n/a
|style='text-align:center;'|n/a
|style='text-align:center;'|n/a
|style='text-align:center;'|n/a
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;'|n/a
|style='text-align:center;'|n/a
|style='text-align:center;'|n/a
|style='text-align:center;'|n/a
|style='text-align:center;background:lightgreen;'|trunk: 100<br/>aurora: 100<br/>beta: 0
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|trunk: 100<br/>aurora: 100<br/>beta: 0
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|trunk: 100<br/>aurora: 100<br/>beta: 0
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|trunk: 100<br/>aurora: 100<br/>beta: 0
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|[https://bugzilla.mozilla.org/show_bug.cgi?id=1223198 [0]]<br>Ash: 100
|style='text-align:center;background:lightgreen;'|[https://bugzilla.mozilla.org/show_bug.cgi?id=1223198 [0]]<br>Ash: 100
|-
|-
Line 154: Line 154:
|style='text-align:center;background:lightgreen;'|Ash: 99
|style='text-align:center;background:lightgreen;'|Ash: 99
|style='text-align:center;background:lightgreen;'|Ash: 99
|style='text-align:center;background:lightgreen;'|Ash: 99
|style='text-align:center;background:lightgreen;'|Ash: 99
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 0
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;background:lightgreen;'|Ash: 99
|style='text-align:center;background:lightgreen;'|Ash: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;background:lightgreen;'|Ash: 99
|style='text-align:center;background:lightgreen;'|Ash: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 0
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 81
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 81
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 82
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 81
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 82
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 82
|style='text-align:center;background:lightgreen;'|trunk: 99<br/>aurora: 99<br/>beta: 99
|-
|-
!style='text-align:left;'|M(bc)
!style='text-align:left;'|M(bc)
|style='text-align:center;background:yellow;'|Ash: [https://bugzilla.mozilla.org/show_bug.cgi?id=1253956 80]
|style='text-align:center;background:yellow;'|Ash: [https://bugzilla.mozilla.org/show_bug.cgi?id=1253956 80]
|style='text-align:center;background:lightgreen;'|Ash: 98
|style='text-align:center;background:lightgreen;'|Ash: 98
|style='text-align:center;background:yellow;'|Ash: [https://bugzilla.mozilla.org/show_bug.cgi?id=1253956 80]
|style='text-align:center;background:yellow;'|trunk: [https://bugzilla.mozilla.org/show_bug.cgi?id=1253956 80]<br/>aurora: 80<br/>beta: 80
|style='text-align:center;background:lightgreen;'|trunk: 98<br/>aurora: 98<br/>beta: 82
|style='text-align:center;background:lightgreen;'|trunk: 98<br/>aurora: 98<br/>beta: 98
|style='text-align:center;background:yellow;'|Ash: [https://bugzilla.mozilla.org/show_bug.cgi?id=1253956 80]
|style='text-align:center;background:yellow;'|Ash: [https://bugzilla.mozilla.org/show_bug.cgi?id=1253956 80]<br/>aurora: 80<br/>beta: 80
|style='text-align:center;background:lightgreen;'|trunk: 98<br/>aurora: 98<br/>beta: 82
|style='text-align:center;background:lightgreen;'|trunk: 98<br/>aurora: 98<br/>beta: 98
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 82
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97
|style='text-align:center;background:lightgreen;'|trunk: 98<br/>aurora: 98<br/>beta: 82
|style='text-align:center;background:lightgreen;'|trunk: 98<br/>aurora: 98<br/>beta: 98
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 81
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 82
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 81
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 82
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 82
|style='text-align:center;background:lightgreen;'|trunk: 97<br/>aurora: 97<br/>beta: 97
|-
|-
!style='text-align:left;'|M(dt)
!style='text-align:left;'|M(dt)
|style='text-align:center;background:yellow;'|Ash: 72
|style='text-align:center;background:yellow;'|Ash: 72
|style='text-align:center;background:yellow;'|Ash: 84
|style='text-align:center;background:yellow;'|Ash: 84
|style='text-align:center;background:yellow;'|trunk: 72<br/>aurora: 72<br/>beta: 70
|style='text-align:center;background:yellow;'|trunk: 72<br/>aurora: 72<br/>beta: 72
|style='text-align:center;background:yellow;'|trunk: 84<br/>aurora: 84<br/>beta: 80
|style='text-align:center;background:yellow;'|trunk: 84<br/>aurora: 84<br/>beta: 84
|style='text-align:center;background:yellow;'|Ash: 72
|style='text-align:center;background:yellow;'|Ash: 72<br/>aurora: 72<br/>beta: 72
|style='text-align:center;background:yellow;'|Ash: 84
|style='text-align:center;background:yellow;'|Ash: 84<br/>aurora: 84<br/>beta: 84
|style='text-align:center;background:yellow;'|trunk: 72<br/>aurora: 72<br/>beta: 0
|style='text-align:center;background:yellow;'|trunk: 72<br/>aurora: 72<br/>beta: 72
|style='text-align:center;background:yellow;'|trunk: 84<br/>aurora: 84<br/>beta: 80
|style='text-align:center;background:yellow;'|trunk: 84<br/>aurora: 84<br/>beta: 84
|style='text-align:center;background:yellow;'|Ash: 72
|style='text-align:center;background:yellow;'|Ash: 72
|style='text-align:center;background:lightgreen;'|trunk: 90<br/>aurora: 90<br/>beta: 85
|style='text-align:center;background:lightgreen;'|trunk: 90<br/>aurora: 90<br/>beta: 90
|style='text-align:center;background:yellow;'|Ash: 72
|style='text-align:center;background:yellow;'|Ash: 72
|style='text-align:center;background:lightgreen;'|trunk: 90<br/>aurora: 90<br/>beta: 85
|style='text-align:center;background:lightgreen;'|trunk: 90<br/>aurora: 90<br/>beta: 90
|style='text-align:center;background:lightgreen;'|trunk: 90<br/>aurora: 90<br/>beta: 0
|style='text-align:center;background:lightgreen;'|trunk: 90<br/>aurora: 90<br/>beta: 90
|-
|-
!style='text-align:left;'|M(gl)
!style='text-align:left;'|M(gl)
Line 201: Line 201:
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
Line 214: Line 214:
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
Line 276: Line 276:
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
Line 291: Line 291:
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|Ash: 100
|style='text-align:center;background:lightgreen;'|Ash: 100<br/>aurora: 100<br/>beta: 100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
|style='text-align:center;background:lightgreen;'|100
Line 301: Line 301:
|style='text-align:center;background:yellow;'|Ash: 100
|style='text-align:center;background:yellow;'|Ash: 100
|}
|}
Base values indicate coverage on mozilla-central. Current coverage here reflects 49 on central, 48 on aurora, and 47 on beta. Most production test suite coverage is riding the trains with a specific Gecko version.<br>
Base values indicate coverage on mozilla-central. Current coverage here reflects 50 on central, 49 on aurora, and 48 on beta. Most production test suite coverage is riding the trains with a specific Gecko version.<br>
Entries with "Ash" next to them are for suites that are running on the [https://treeherder.mozilla.org/#/jobs?repo=ash Ash project branch], which is being kept in sync with mozilla-central.<br>
Entries with "Ash" next to them are for suites that are running on the [https://treeherder.mozilla.org/#/jobs?repo=ash Ash project branch], which is being kept in sync with mozilla-central, in lieu of running on trunk.<br>


=== Mochitest a11y/Jetpack ===
=== Mochitest a11y/Jetpack ===

Revision as of 12:04, 14 June 2016

Enabled Test Suites By Platform

Test Suite Windows OSX Linux
XP (32-bit) 7 (32-bit) 8 (64-bit) 10.10 (64-bit) 32-bit 64-bit
Debug Opt Debug Opt Debug Opt Debug Opt Debug Opt Debug Opt ASAN
Cpp n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a
JIT jit tests n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a
X xpcshell tests n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a
Mn marionette Ash: 100 Ash: 100 100 100 Ash: 100
aurora: 100
beta: 100
Ash: 100
aurora: 100
beta: 100
100 100 100 100 100 100 100
R(C) crashtests Ash: 100 Ash: 100 100 100 Ash: 100
aurora: 100
beta: 100
Ash: 100
aurora: 100
beta: 100
100 100 100 100 100 100 100
R(J) jsreftests Ash: 100 Ash: 100 100 100 Ash: 100
aurora: 100
beta: 100
Ash: 100
aurora: 100
beta: 100
100 100 100 100 100 100 100
R(R) reftests Ash: 100 Ash: 100 100 100 Ash: 100
aurora: 100
beta: 100
Ash: 100
aurora: 100
beta: 100
100 100 100 100 100 100 [0]
Ash: 100
R(Ru) reftests (no accel) n/a n/a 100 100 Ash: 100
aurora: 100
beta: 100
Ash: 100
aurora: 100
beta: 100
n/a n/a 100 100 100 100 [0]
Ash: 100
M(1-5) Ash: 99 Ash: 99 trunk: 99
aurora: 99
beta: 99
trunk: 99
aurora: 99
beta: 99
Ash: 99
aurora: 99
beta: 99
Ash: 99
aurora: 99
beta: 99
trunk: 99
aurora: 99
beta: 99
trunk: 99
aurora: 99
beta: 99
trunk: 99
aurora: 99
beta: 99
trunk: 99
aurora: 99
beta: 99
trunk: 99
aurora: 99
beta: 99
trunk: 99
aurora: 99
beta: 99
trunk: 99
aurora: 99
beta: 99
M(bc) Ash: 80 Ash: 98 trunk: 80
aurora: 80
beta: 80
trunk: 98
aurora: 98
beta: 98
Ash: 80
aurora: 80
beta: 80
trunk: 98
aurora: 98
beta: 98
trunk: 97
aurora: 97
beta: 97
trunk: 98
aurora: 98
beta: 98
trunk: 97
aurora: 97
beta: 97
trunk: 97
aurora: 97
beta: 97
trunk: 97
aurora: 97
beta: 97
trunk: 97
aurora: 97
beta: 97
trunk: 97
aurora: 97
beta: 97
M(dt) Ash: 72 Ash: 84 trunk: 72
aurora: 72
beta: 72
trunk: 84
aurora: 84
beta: 84
Ash: 72
aurora: 72
beta: 72
Ash: 84
aurora: 84
beta: 84
trunk: 72
aurora: 72
beta: 72
trunk: 84
aurora: 84
beta: 84
Ash: 72 trunk: 90
aurora: 90
beta: 90
Ash: 72 trunk: 90
aurora: 90
beta: 90
trunk: 90
aurora: 90
beta: 90
M(gl) Ash: 100 Ash: 100 100 100 Ash: 100
aurora: 100
beta: 100
Ash: 100
aurora: 100
beta: 100
100 100 100 100 100 100 100
M(mda) Ash: 100 Ash: 100 100 100 Ash: 100
aurora: 100
beta: 100
Ash: 100
aurora: 100
beta: 100
100 100 100 100 100 100 100
M(JP) jetpack [1] 0 0 0 0 0 0 0 0 0 0 0 0 0
M(oth/a11y) [1] 0 0 0 0 0 0 0 0 0 0 0 0 0
M(oth/chrome) [2] n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a
W(1-8) web platform tests Ash: 100 Ash: 100 100 100 Ash: 100
aurora: 100
beta: 100
Ash: 100
aurora: 100
beta: 100
100 100 100 100 100 100 Ash: 100
Wr web platform reftest Ash: 100 Ash: 100 100 100 Ash: 100
aurora: 100
beta: 100
Ash: 100
aurora: 100
beta: 100
100 100 100 100 100 100 Ash: 100

Base values indicate coverage on mozilla-central. Current coverage here reflects 50 on central, 49 on aurora, and 48 on beta. Most production test suite coverage is riding the trains with a specific Gecko version.
Entries with "Ash" next to them are for suites that are running on the Ash project branch, which is being kept in sync with mozilla-central, in lieu of running on trunk.

Mochitest a11y/Jetpack

a11y is known to not work under e10s and the tests in this suite largely fail. Jetpack tests are also known to be broken at this point in time (bug 1246453). However, initial rollout plans for e10s don't include users with a11y enabled or with addons installed, so there are no short-term plans to enable these suites in continuous integration.

Mochitest Chrome

These tests are thought to be largely non-e10s specific, so it doesn't make sense to run them in the content process in e10s mode. There is work underway to identify tests which are relevant to e10s to and to figure out what to do with them. There are no short-term plans to enable this suite in continuous integration.

Other Notes

  • Yellow cells have incomplete test coverage (<85%) or serious issues that prevent the suite from running reliably (asserts, leaks, crashes, frequent intermittent failures).

Individual Test Issues

Related Bugs