Electrolysis/Test Coverage: Difference between revisions

Revision as of 03:15, 19 March 2016

Enabled Test Suites By Platform

Test Suite	Windows						OSX				Linux
	XP		7		8		10.6		10.10		32-bit		64-bit
	Debug	Opt	Debug	Opt	Debug	Opt	Debug	Opt	Debug	Opt	Debug	Opt	Debug	Opt	ASAN
Cpp	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a
JIT jit tests	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a
X xpcshell tests	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a
Mn marionette	Ash: 100	Ash: 100	100	100	Ash: 100	Ash: 100	n/a	n/a	100	100	100	100	100	100	[0]
R(C) crashtests	Ash: [0]	Ash: 100	100	100	Ash: 100	Ash: 100	n/a	n/a	Ash: [0]	99	100	100	100	100	100
R(J) jsreftests	Ash: 100	Ash: 100	100	100	Ash: 100	Ash: 100	n/a	n/a	Ash: 100	100	100	100	100	100	100
R(R) reftests	Ash: 99	Ash: [0]	99	99	Ash: 99	Ash: 99	n/a	n/a	Ash: 99	99	99	99	99	99	[0] Ash: 99
R(Ru) reftests (no accel)	n/a	n/a	Ash: 99	Ash: 99	Ash: 99	Ash: 99	n/a	n/a	n/a	n/a	Ash: 99	Ash: 99	Ash: 99	Ash: 99	Ash: 99
M(1-5)	Ash: Y	Ash: Y	Ash: Y	trunk: 82 aurora: 0 beta: 0	Ash: Y	Ash: Y	n/a	n/a	Ash: Y	trunk: 82 aurora: 0 beta: 0	trunk: 81 aurora: 81 beta: 79	trunk: 82 aurora: 82 beta: 80	trunk: 81 aurora: 81 beta: 79	trunk: 82 aurora: 82 beta: 80	trunk: 82 aurora: 82 beta: 80
M(bc)	Ash: 84	Ash: 84	Ash: 84	trunk: 84 aurora: 82 beta: 75	Ash: 84	trunk: 84 aurora: 82 beta: 75	n/a	trunk: 84 aurora: 82 beta: 75	trunk: 84 aurora: 0 beta: 0	trunk: 84 aurora: 82 beta: 75	trunk: 83 aurora: 81 beta: 77	trunk: 84 aurora: 82 beta: 76	trunk: 83 aurora: 81 beta: 77	trunk: 84 aurora: 82 beta: 76	trunk: 84 aurora: 82 beta: 76
M(dt)	Ash: 75	Ash: 89	trunk: 75 aurora: 0 beta: 0	trunk: 89 aurora: 0 beta: 0	Ash: 75	Ash: 89	n/a	n/a	Ash: 75	trunk: 89 aurora: 0 beta: 0	Ash: 75	trunk: 90 aurora: 88 beta: 87	Ash: 75	trunk: 90 aurora: 88 beta: 87	Ash: 90
M(gl)	Ash: 100	Ash: 100	100	100	Ash: 100	Ash: 100	n/a	n/a	Ash: [0]	100	100	100	100	100	100
M(JP) jetpack [1]	0	0	0	0	0	0	n/a	n/a	0	0	0	0	0	0	0
M(oth/a11y) [1]	0	0	0	0	0	0	n/a	n/a	0	0	0	0	0	0	0
M(oth/chrome) [2]	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a
W(1-8) web platform tests	Ash: 99	Ash: 99	99	99	Ash: 99	Ash: 99	n/a	n/a	99	99	99	99	99	99	Ash: 99
Wr web platform reftest	Ash: 100	Ash: 100	100	100	Ash: 100	Ash: 100	n/a	n/a	100	100	100	100	100	100	Ash: 100

Base values indicate coverage on mozilla-central. Current coverage here reflects 48 on central, 47 on aurora, and 46 on beta. Most production test suite coverage is riding the trains with a specific Gecko version.
Entries with "Ash" next to them are for suites that are running on the Ash project branch, which is being kept in sync with mozilla-central.

Mochitest a11y/Jetpack

a11y is known to not work under e10s and the tests in this suite largely fail. Jetpack tests are also known to be broken at this point in time (bug 1246453). However, initial rollout plans for e10s don't include users with a11y enabled or with addons installed, so there are no short-term plans to enable these suites in continuous integration.

Mochitest Chrome

These tests are thought to be largely non-e10s specific, so it doesn't make sense to run them in the content process in e10s mode. There is work underway to identify tests which are relevant to e10s to and to figure out what to do with them. There are no short-term plans to enable this suite in continuous integration.

Other Notes

Yellow cells have incomplete test coverage (<80%) or serious issues that prevent the suite from running reliably (asserts, leaks, crashes, frequent intermittent failures).
Support for OSX 10.6 is likely to be dropped soon, so test coverage is not being tracked nor actively enabled.

Individual Test Issues

Related Bugs

bug 1194533 - Schedule e10s tests on OSX 10.10
bug 1215233 - Schedule e10s tests on Windows 7
bug 1253849 - Get e10s tests passing on all Windows versions in CI

@@ Line 68: / Line 68: @@
 |style='text-align:center;'|n/a
 |-
-!style='text-align:left;'|X xpc shell tests
+!style='text-align:left;'|X xpcshell tests
 |style='text-align:center;'| n/a
 |style='text-align:center;'| n/a
@@ Line 100: / Line 100: @@
 |style='text-align:center;background:lightgreen;'|100
 |style='text-align:center;background:lightgreen;'|100
-|style='text-align:center;background:red;'|0
+|style='text-align:center;background:red;'|[https://bugzilla.mozilla.org/show_bug.cgi?id=1257922 [0]]
 |-
 !style='text-align:left;'|R(C) crashtests
-|style='text-align:center;background:red;'|Ash: 0
+|style='text-align:center;background:red;'|Ash: [https://bugzilla.mozilla.org/show_bug.cgi?id=1253838 [0]]
 |style='text-align:center;background:lightgreen;'|Ash: 100
 |style='text-align:center;background:lightgreen;'|100
@@ Line 111: / Line 111: @@
 |style='text-align:center;'|n/a
 |style='text-align:center;'|n/a
-|style='text-align:center;background:red;'|Ash: 0
+|style='text-align:center;background:red;'|Ash: [https://bugzilla.mozilla.org/show_bug.cgi?id=1252237 [0]]
 |style='text-align:center;background:lightgreen;'|99
 |style='text-align:center;background:lightgreen;'|100
@@ Line 138: / Line 138: @@
 !style='text-align:left;'|R(R) reftests
 |style='text-align:center;background:lightgreen;'|Ash: 99
-|style='text-align:center;background:red;'|Ash: 0
+|style='text-align:center;background:red;'|Ash: [https://bugzilla.mozilla.org/show_bug.cgi?id=1254829 [0]]
 |style='text-align:center;background:lightgreen;'|99
 |style='text-align:center;background:lightgreen;'|99
@@ Line 156: / Line 156: @@
 |style='text-align:center;'|n/a
 |style='text-align:center;'|n/a
-|style='text-align:center;'|0
 |style='text-align:center;background:lightgreen;'|Ash: 99
-|style='text-align:center;'|0
+|style='text-align:center;background:lightgreen;'|Ash: 99
+|style='text-align:center;background:lightgreen;'|Ash: 99
 |style='text-align:center;background:lightgreen;'|Ash: 99
 |style='text-align:center;'|n/a
@@ Line 164: / Line 164: @@
 |style='text-align:center;'|n/a
 |style='text-align:center;'|n/a
-|style='text-align:center;'|0
 |style='text-align:center;background:lightgreen;'|Ash: 99
-|style='text-align:center;'|0
+|style='text-align:center;background:lightgreen;'|Ash: 99
+|style='text-align:center;background:lightgreen;'|Ash: 99
 |style='text-align:center;background:lightgreen;'|Ash: 99
 |style='text-align:center;background:lightgreen;'|Ash: 99
@@ Line 230: / Line 230: @@
 |style='text-align:center;'|n/a
 |style='text-align:center;'|n/a
-|style='text-align:center;background:red;'|Ash: 0
+|style='text-align:center;background:red;'|Ash: [https://bugzilla.mozilla.org/show_bug.cgi?id=1252263 [0]]
 |style='text-align:center;background:lightgreen;'|100
 |style='text-align:center;background:lightgreen;'|100
@@ Line 238: / Line 238: @@
 |style='text-align:center;background:lightgreen;'|100
 |-
-!style='text-align:left;'|M(JP) jetpack [{{bug|1246453}}][[#Mochitest_a11y|[2]]]
+!style='text-align:left;'|M(JP) jetpack [[#Mochitest a11y/jetpack|[1]]]
 |style='text-align:center;background:lightcyan;'|0
 |style='text-align:center;background:lightcyan;'|0
@@ Line 255: / Line 255: @@
 |style='text-align:center;background:lightcyan;'|0
 |-
-!style='text-align:left;'|M(oth-a11y) [[#Mochitest_a11y|[2]]]
+!style='text-align:left;'|M(oth/a11y) [[#Mochitest a11y/Jetpack|[1]]]
 |style='text-align:center;background:lightcyan;'|0
 |style='text-align:center;background:lightcyan;'|0
@@ Line 272: / Line 272: @@
 |style='text-align:center;background:lightcyan;'|0
 |-
-!style='text-align:left;'|M(oth-chrome) [[#Mochitest_chrome|[3]]]
+!style='text-align:left;'|M(oth/chrome) [[#Mochitest Chrome|[2]]]
 |style='text-align:center;'|n/a
 |style='text-align:center;'|n/a
@@ Line 298: / Line 298: @@
 |style='text-align:center;'|n/a
 |style='text-align:center;'|n/a
-|style='text-align:center;background:lightgreen;'|99(ab)
+|style='text-align:center;background:lightgreen;'|99
-|style='text-align:center;background:lightgreen;'|99(ab)
+|style='text-align:center;background:lightgreen;'|99
-|style='text-align:center;background:lightgreen;'|99(ab)
+|style='text-align:center;background:lightgreen;'|99
-|style='text-align:center;background:lightgreen;'|99(ab)
+|style='text-align:center;background:lightgreen;'|99
-|style='text-align:center;background:lightgreen;'|99(ab)
+|style='text-align:center;background:lightgreen;'|99
-|style='text-align:center;background:lightgreen;'|99(ab)
+|style='text-align:center;background:lightgreen;'|99
 |style='text-align:center;background:yellow;'|Ash: 99
 |-
@@ Line 315: / Line 315: @@
 |style='text-align:center;'|n/a
 |style='text-align:center;'|n/a
-|style='text-align:center;background:lightgreen;'|100(ab)
-|style='text-align:center;background:lightgreen;'|100(ab)
 |style='text-align:center;background:lightgreen;'|100
-|style='text-align:center;background:lightgreen;'|100(ab)
 |style='text-align:center;background:lightgreen;'|100
-|style='text-align:center;background:lightgreen;'|100(ab)
+|style='text-align:center;background:lightgreen;'|100
+|style='text-align:center;background:lightgreen;'|100
+|style='text-align:center;background:lightgreen;'|100
+|style='text-align:center;background:lightgreen;'|100
 |style='text-align:center;background:yellow;'|Ash: 100
 |}
+Base values indicate coverage on mozilla-central. Current coverage here reflects 48 on central, 47 on aurora, and 46 on beta. Most production test suite coverage is riding the trains with a specific Gecko version.<br>
+Entries with "Ash" next to them are for suites that are running on the Ash project branch, which is being kept in sync with mozilla-central.<br>
-* Base values indicate coverage on mozilla-central. annotations (a, b, r) indicate support for other releases. Current coverage here reflects 47 on mc, 46 on ma. Tests suite coverage is rolling out with the trains.<br>
+=== Mochitest a11y/Jetpack ===
-* Entries with "Ash" next to them are for suites that are running on the Ash project branch, which is staying synced with mozilla-central.<br>
+a11y is known to not work under e10s and the tests in this suite largely fail. Jetpack tests are also known to be broken at this point in time ({{bug|1246453}}). However, initial rollout plans for e10s don't include users with a11y enabled or with addons installed, so there are no short-term plans to enable these suites in continuous integration.
-* Yellow cells are for suites that have incomplete test coverage or serious issues that prevent them from running reliably.
-* OSX 10.6 support is likely to be dropped soon, so test coverage is not being tracked.
+=== Mochitest Chrome ===
+These tests are thought to be largely non-e10s specific, so it doesn't make sense to run them in the content process in e10s mode. There is work underway to identify tests which are relevant to e10s to and to figure out what to do with them. There are no short-term plans to enable this suite in continuous integration.
+=== Other Notes ===
+* Yellow cells have incomplete test coverage (<80%) or serious issues that prevent the suite from running reliably (asserts, leaks, crashes, frequent intermittent failures).
+* Support for OSX 10.6 is likely to be dropped soon, so test coverage is not being tracked nor actively enabled.
+== Individual Test Issues ==
+* [https://docs.google.com/spreadsheets/d/10UeyRoiWV2HjkWwAU51HXyXAV7YLi4BjDm55mr5Xv6c/edit?pref=2&pli=1#gid=0 mochitest coverage spreadsheet]
+* [https://docs.google.com/spreadsheets/d/1YQ6HHcg14iPtA5hvNklnCrinfPmoxXyC_iqAczX9U2Y/edit?usp=sharing other test suite coverage spreadsheet]
 == Related Bugs ==
@@ Line 333: / Line 344: @@
 * {{bug|1215233}} - Schedule e10s tests on Windows 7
 * {{bug|1253849}} - Get e10s tests passing on all Windows versions in CI
-** Due to capacity limitations, non-Win7 versions will run on a reduced interval
-== Individual Test Issues ==
-* [https://docs.google.com/spreadsheets/d/10UeyRoiWV2HjkWwAU51HXyXAV7YLi4BjDm55mr5Xv6c/edit?pref=2&pli=1#gid=0 browser and mochitest plain spreadsheet]
-=== Mochitest a11y===
-a11y is known to not work under e10s, and the tests in this suite largely fail, so there are no short-term plans to enable running this suite in continuous integration.
-=== Mochitest chrome===
-These tests are thought to be largely non-e10s specific, so it doesn't make sense to run them in the content process in e10s mode. There is work underway to identify tests which are relevant to e10s to and to figure out what to do with them. There are no short-term plans to enable this suite in continuous integration.
-== Windows Coverage ==
-Because of constraints on our Windows hardware test pools, we're not planning to enable any e10s suites on Windows XP or Windows 8. Instead, Windows 7 is the focus of all our e10s test coverage.

Electrolysis/Test Coverage: Difference between revisions

Revision as of 03:15, 19 March 2016

Contents

Enabled Test Suites By Platform

Mochitest a11y/Jetpack

Mochitest Chrome

Other Notes

Individual Test Issues

Related Bugs

Navigation menu

Electrolysis/Test Coverage: Difference between revisions

Revision as of 03:15, 19 March 2016

Enabled Test Suites By Platform

Mochitest a11y/Jetpack

Mochitest Chrome

Other Notes

Individual Test Issues

Related Bugs

Navigation menu

Search