B2G/QA/2.2/2015-03-16Status - Revision history

Echang: /* Regression Testing */

2015-03-16T14:12:40Z

Regression Testing

← Older revision		Revision as of 14:12, 16 March 2015
Line 48:		Line 48:
	* Partial Regression/V2.2 Nexus-5-L		* Partial Regression/V2.2 Nexus-5-L
	** [http://mzl.la/1EXfMQQ Moztrap result] - [https://moztrap.mozilla.org/results/cases/?&pagenumber=1&pagesize=100&sortfield=created_on&sortdirection=desc&filter-resultstatus=failed&filter-run=6552 17 fail], 1103 pass, 82% complete		** [http://mzl.la/1EXfMQQ Moztrap result] - [https://moztrap.mozilla.org/results/cases/?&pagenumber=1&pagesize=100&sortfield=created_on&sortdirection=desc&filter-resultstatus=failed&filter-run=6552 17 fail], 1103 pass, 82% complete

			* Bluetooth PTS/V2.2 Flame-KK
			** [https://bugzilla.mozilla.org/showdependencytree.cgi?id=1139349&hide_resolved=1 Meta dependency tree] - 3 open bugs

	==== Functional Testing ====		==== Functional Testing ====

Echang: /* V2.2 Highlight & Risk */

2015-03-16T13:57:29Z

V2.2 Highlight & Risk

← Older revision		Revision as of 13:57, 16 March 2015
Line 2:		Line 2:
	* Highlight		* Highlight
	** Availability of Flame-L is unknown.		** Availability of Flame-L is unknown.
			** Currently we are running [[B2G/QA/2.2/2015-03-16Status#Regression_Testing\|regression test]] for FLAME-KK, NEXUS-5-L.

	== [[Release_Management/FirefoxOS/2_2_Schedule\|V2.2 Milestone]] ==		== [[Release_Management/FirefoxOS/2_2_Schedule\|V2.2 Milestone]] ==

Echang: /* Regression Testing */

2015-03-16T13:56:29Z

Regression Testing

← Older revision		Revision as of 13:56, 16 March 2015
Line 43:		Line 43:

	* Full Regression/V2.2 Flame-KK		* Full Regression/V2.2 Flame-KK
	** [http://mzl.la/1wtR60Y Moztrap result] - [https://moztrap.mozilla.org/results/cases/?&pagenumber=1&pagesize=100&sortfield=created_on&sortdirection=desc&filter-resultstatus=failed&filter-run=6551 25 fail], 2011 pass, 52% complete]		** [http://mzl.la/1wtR60Y Moztrap result] - [https://moztrap.mozilla.org/results/cases/?&pagenumber=1&pagesize=100&sortfield=created_on&sortdirection=desc&filter-resultstatus=failed&filter-run=6551 25 fail], 2011 pass, 52% complete

	* Partial Regression/V2.2 Nexus-5-L		* Partial Regression/V2.2 Nexus-5-L
	** [http://mzl.la/1EXfMQQ Moztrap result] - [https://moztrap.mozilla.org/results/cases/?&pagenumber=1&pagesize=100&sortfield=created_on&sortdirection=desc&filter-resultstatus=failed&filter-run=6552 17 fail], 1103 pass, 82% complete]		** [http://mzl.la/1EXfMQQ Moztrap result] - [https://moztrap.mozilla.org/results/cases/?&pagenumber=1&pagesize=100&sortfield=created_on&sortdirection=desc&filter-resultstatus=failed&filter-run=6552 17 fail], 1103 pass, 82% complete

	==== Functional Testing ====		==== Functional Testing ====

Echang: /* Regression Testing */

2015-03-16T13:53:49Z

Regression Testing

← Older revision		Revision as of 13:53, 16 March 2015
Line 38:		Line 38:
	** Automation - are available [http://mozilla-twqa.github.io/Gaiatest-Reports/2015/index.html on Github.IO] & [https://mail.mozilla.org/listinfo/fxosqa-report-watchlist fxosqa-report-watchlist].		** Automation - are available [http://mozilla-twqa.github.io/Gaiatest-Reports/2015/index.html on Github.IO] & [https://mail.mozilla.org/listinfo/fxosqa-report-watchlist fxosqa-report-watchlist].

	==== Regression Testing ====		==== Regression Testing ====
			* RTL/V2.2 Flame-KK
			** [https://moztrap.mozilla.org/results/runs/?&pagenumber=1&pagesize=20&sortfield=start&sortdirection=desc&filter-status=active&filter-name=rtl+full+test Moztrap result] - [https://moztrap.mozilla.org/results/cases/?&pagenumber=1&pagesize=100&sortfield=created_on&sortdirection=desc&filter-resultstatus=failed&filter-run=6626 76 fail], 326 pass, check with Josh for detail.

			* Full Regression/V2.2 Flame-KK
			** [http://mzl.la/1wtR60Y Moztrap result] - [https://moztrap.mozilla.org/results/cases/?&pagenumber=1&pagesize=100&sortfield=created_on&sortdirection=desc&filter-resultstatus=failed&filter-run=6551 25 fail], 2011 pass, 52% complete]

			* Partial Regression/V2.2 Nexus-5-L
			** [http://mzl.la/1EXfMQQ Moztrap result] - [https://moztrap.mozilla.org/results/cases/?&pagenumber=1&pagesize=100&sortfield=created_on&sortdirection=desc&filter-resultstatus=failed&filter-run=6552 17 fail], 1103 pass, 82% complete]

	==== Functional Testing ====		==== Functional Testing ====

Echang: /* V2.2 Testing by Type */

2015-03-16T13:40:55Z

V2.2 Testing by Type

← Older revision		Revision as of 13:40, 16 March 2015
Line 37:		Line 37:
	** Manual - are available [http://bit.ly/15rM9JC on Google Drive] & [https://mail.mozilla.org/listinfo/fxosqa-report-watchlist fxosqa-report-watchlist].		** Manual - are available [http://bit.ly/15rM9JC on Google Drive] & [https://mail.mozilla.org/listinfo/fxosqa-report-watchlist fxosqa-report-watchlist].
	** Automation - are available [http://mozilla-twqa.github.io/Gaiatest-Reports/2015/index.html on Github.IO] & [https://mail.mozilla.org/listinfo/fxosqa-report-watchlist fxosqa-report-watchlist].		** Automation - are available [http://mozilla-twqa.github.io/Gaiatest-Reports/2015/index.html on Github.IO] & [https://mail.mozilla.org/listinfo/fxosqa-report-watchlist fxosqa-report-watchlist].

			==== Regression Testing ====

	==== Functional Testing ====		==== Functional Testing ====

Echang: /* Exploratory Testing */

2015-03-16T13:31:20Z

Exploratory Testing

← Older revision		Revision as of 13:31, 16 March 2015
Line 138:		Line 138:
	==== Exploratory Testing ====		==== Exploratory Testing ====
	'''Flame-KK''' <span style="color:#FFFFFF; background:#32CD32"> On Track </span>		'''Flame-KK''' <span style="color:#FFFFFF; background:#32CD32"> On Track </span>
	* [http://mzl.la/1Cat1tY ~~223~~ bugs found] - includes every bug in bugzilla affecting v2.2, not limited to the bugs after branch day.		* [http://mzl.la/1Cat1tY 227 bugs found] - includes every bug in bugzilla affecting v2.2, not limited to the bugs after branch day.
	** UNCONFIRMED, NEW, ASSIGNED, REOPENED - [http://mzl.la/1IB4Ith ~~107~~ bugs found].		** UNCONFIRMED, NEW, ASSIGNED, REOPENED - [http://mzl.la/1IB4Ith 106 bugs found].
	** UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - [http://mzl.la/1ATnw3C 14 bugs found].		** UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - [http://mzl.la/1ATnw3C 14 bugs found].
	** RESOLVED, VERIFIED, CLOSED - [http://mzl.la/17KvaU8 ~~116~~ bugs found].		** RESOLVED, VERIFIED, CLOSED - [http://mzl.la/17KvaU8 121 bugs found].
	** RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - [http://mzl.la/14Dhet5 8 bugs found].		** RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - [http://mzl.la/14Dhet5 9 bugs found].

	* Regression		* Regression bugs
	** UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - [http://mzl.la/1Afjcx9 31 bugs found].		** UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - [http://mzl.la/1Afjcx9 24 bugs found].
	** RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - [http://mzl.la/1M98z3H '''~~207~~''' bugs found].		** RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - [http://mzl.la/1M98z3H '''220''' bugs found].

	==== Performance Testing ====		==== Performance Testing ====

Echang: /* Smoke Testing */

2015-03-16T13:28:33Z

Smoke Testing

← Older revision		Revision as of 13:28, 16 March 2015
Line 22:		Line 22:
	'''Flame-KK''' <span style="color:#FFFFFF; background:#32CD32"> On Track </span>		'''Flame-KK''' <span style="color:#FFFFFF; background:#32CD32"> On Track </span>

	* OTA issue - [https://bugzilla.mozilla.org/show_bug.cgi?id=1142778 Bug 1142778 - OTA for Flame Kitkat 2.2, 2.1, and 2.0 builds does not download the latest Nightly builds.], 2.2 OTA is okay on 3/16~~, bug not updated~~.		* OTA issue - [https://bugzilla.mozilla.org/show_bug.cgi?id=1142778 Bug 1142778 - OTA for Flame Kitkat 2.2, 2.1, and 2.0 builds does not download the latest Nightly builds.], 2.2 OTA is okay on 3/16.

	* Existing Bugs Breaking the Automation smoketests: Flame-KK		* Existing Bugs Breaking the Automation smoketests: Flame-KK

Echang: /* Smoke Testing */

2015-03-16T13:26:33Z

Smoke Testing

← Older revision		Revision as of 13:26, 16 March 2015
Line 22:		Line 22:
	'''Flame-KK''' <span style="color:#FFFFFF; background:#32CD32"> On Track </span>		'''Flame-KK''' <span style="color:#FFFFFF; background:#32CD32"> On Track </span>

	* OTA issue - [https://bugzilla.mozilla.org/show_bug.cgi?id=1142778 Bug 1142778 - OTA for Flame Kitkat 2.2, 2.1, and 2.0 builds does not download the latest Nightly builds.]		* OTA issue - [https://bugzilla.mozilla.org/show_bug.cgi?id=1142778 Bug 1142778 - OTA for Flame Kitkat 2.2, 2.1, and 2.0 builds does not download the latest Nightly builds.], 2.2 OTA is okay on 3/16, bug not updated.

	* Existing Bugs Breaking the Automation smoketests: Flame-KK		* Existing Bugs Breaking the Automation smoketests: Flame-KK
Line 29:		Line 29:
	** [Disabled] test_cost_control_data_alert_mobile.py - Bug 1061698 - Need ability to work in the shadow root/shadow DOM		** [Disabled] test_cost_control_data_alert_mobile.py - Bug 1061698 - Need ability to work in the shadow root/shadow DOM
	** [Disabled] test_call_log_all_calls.py - Bug 1096385 - Investigate failure in test_call_log_all_calls.py, call log is displayed erroneously		** [Disabled] test_call_log_all_calls.py - Bug 1096385 - Investigate failure in test_call_log_all_calls.py, call log is displayed erroneously
			** [Reproduced manually] test_ftu_skip_tour.py - Bug 1134445 - [FTU] About Firefox OS page has "Yes! Send data." checked primarily.
			** [Reproduced manually] test_IMAP_email_notification.py - Bug 1133446 - [Email] No new email notification - NEED Patch uplift - tree cluster error, will check

	'''Nexus-5-L''' <span style="color:#FFFFFF; background:#32CD32"> On Track </span>		'''Nexus-5-L''' <span style="color:#FFFFFF; background:#32CD32"> On Track </span>
	* 1 blocker.
	** [https://bugzilla.mozilla.org/show_bug.cgi?id=1118711 Bug 1118711 - Email app fails to fully launch due to worker JS runtime problem (reported as "InternalError: too much recursion") breaking startup process]

	* Daily Reports can be found here.		* Daily Reports can be found here.

Echang: /* V2.2 Testing by Type */

2015-03-16T10:06:56Z

V2.2 Testing by Type

← Older revision		Revision as of 10:06, 16 March 2015
Line 21:		Line 21:
	==== Smoke Testing ====		==== Smoke Testing ====
	'''Flame-KK''' <span style="color:#FFFFFF; background:#32CD32"> On Track </span>		'''Flame-KK''' <span style="color:#FFFFFF; background:#32CD32"> On Track </span>

			* OTA issue - [https://bugzilla.mozilla.org/show_bug.cgi?id=1142778 Bug 1142778 - OTA for Flame Kitkat 2.2, 2.1, and 2.0 builds does not download the latest Nightly builds.]

	* Existing Bugs Breaking the Automation smoketests: Flame-KK		* Existing Bugs Breaking the Automation smoketests: Flame-KK

Echang at 09:51, 16 March 2015

2015-03-16T09:51:04Z

← Older revision		Revision as of 09:51, 16 March 2015
Line 15:		Line 15:
	* Focus on bugs in [https://docs.google.com/a/mozilla.com/spreadsheets/d/1l_HJcd3xcWgvM-VsDOzPUQPR3kPP36gs84lGXxC-TIY/edit?usp=sharing agreed list].		* Focus on bugs in [https://docs.google.com/a/mozilla.com/spreadsheets/d/1l_HJcd3xcWgvM-VsDOzPUQPR3kPP36gs84lGXxC-TIY/edit?usp=sharing agreed list].
	* [[B2G/QA/2.2#V2.2_QA_Owners\|B2G V2.2 Overview Page]] for more info.		* [[B2G/QA/2.2#V2.2_QA_Owners\|B2G V2.2 Overview Page]] for more info.
	* Current [https://bugzilla.mozilla.org/buglist.cgi?f1=cf_feature_b2g&list_id=11963255&o1=substring&resolution=---&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&v1=2.2%2B feature-b2g=2.2+] 10 bugs		* Current [https://bugzilla.mozilla.org/buglist.cgi?f1=cf_feature_b2g&list_id=11963255&o1=substring&resolution=---&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&v1=2.2%2B feature-b2g=2.2+] 8 bugs
	* Current [https://bugzilla.mozilla.org/buglist.cgi?f1=cf_blocking_b2g&list_id=11963250&o1=substring&resolution=---&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&v1=2.2%2B blocking-b2g=2.2+] 95 bugs		* Current [https://bugzilla.mozilla.org/buglist.cgi?f1=cf_blocking_b2g&list_id=11963250&o1=substring&resolution=---&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&v1=2.2%2B blocking-b2g=2.2+] 87 bugs

	== V2.2 Testing by Type ==		== V2.2 Testing by Type ==
Line 40:		Line 40:
	* QA focus - [http://mzl.la/150Afqw 42 bugs found].		* QA focus - [http://mzl.la/150Afqw 42 bugs found].

	* ~~RTL~~ (~~Marigold~~)		* General (eric)
	** ~~Health~~: ~~Yellow~~		** Flame-L might be late for FC, roughly 6 weeks or more needed after SOW (SOW, legal review, porting, build time), PM will have an internal discussion to decide what to do.
	** ~~Test case coverage~~: ~~https~~://~~wiki~~.~~mozilla.org~~/RTL		** feature-b2g\|blocking-b2g == 2.2?\|+ [not resolved] http://mzl.la/1C4X1KT Check if code uplifted
	~~5 P2 still ongoing, ETA 3/6~~		** feature-b2g\|blocking-b2g == 2.2?\|+ [resolved] http://mzl.la/1E5mQqM Need verification
	** ~~Details~~:		** status-b2g-v2.2 affected not blocking any bugs http://mzl.la/1E5ArOQ
	TC: ~~Keyboard, Setting, and Search are ongoing~~		* RTL (Marigold)
	~~Bug: 14 bugs still unfix:~~ http://mzl.la/~~1DTR6HJ~~		** Health: Red
			** Test case coverage: 100%
			** Details: 15 blockers
			*** http://mzl.la/1b2ddSG

	* Privacy (Gerry)		* Privacy (Gerry)
	** Health: Yellow		** Health: Yellow
	** Test case coverage:		** Test case coverage:
	** Details: 4 blockers		** Details: 5 blockers
	*** https://bugzilla.mozilla.org/show_bug.cgi?id=1111992		*** https://bugzilla.mozilla.org/show_bug.cgi?id=1111992
	*** https://bugzilla.mozilla.org/show_bug.cgi?id=1136595
	*** https://bugzilla.mozilla.org/show_bug.cgi?id=1093688		*** https://bugzilla.mozilla.org/show_bug.cgi?id=1093688
			*** https://bugzilla.mozilla.org/show_bug.cgi?id=1134711
			*** https://bugzilla.mozilla.org/show_bug.cgi?id=1136286
	*** https://bugzilla.mozilla.org/show_bug.cgi?id=1137083		*** https://bugzilla.mozilla.org/show_bug.cgi?id=1137083
			*** http://mzl.la/1vvsAfY
			* Josh to help talk with dave/eitan on the backout for the compile time config
	* Cut, Copy, and Paste (Gerry, Edward, Teri, Eric)		* Cut, Copy, and Paste (Gerry, Edward, Teri, Eric)
	** Health: Yellow		** Health: Yellow
	** Test case coverage:		** Test case coverage:
	** Details: 2 blockers		** Details: 2 blockers
			*** https://bugzilla.mozilla.org/show_bug.cgi?id=1021499
	*** https://bugzilla.mozilla.org/show_bug.cgi?id=1130982		*** https://bugzilla.mozilla.org/show_bug.cgi?id=1130982
	*** https://bugzilla.mozilla.org/show_bug.cgi?id=1136041
	* NFC (Alison)		* NFC (Alison)
	** Health: Green		** Health: Green
	** Test case coverage:100%		** Test case coverage:100%
	** Details:		** Details:
	*** ~~Full run testing now~~		*** No blockers
	*** One regression blocker: https://bugzilla.mozilla.org/show_bug.cgi?id=1120849 (patch wait for checkin)
	* short_name support in manifest.webapp (Hermes)		* short_name support in manifest.webapp (Hermes)
	** Health: Yellow		** Health: Yellow
	** Test case coverage: 90%		** Test case coverage: 100%
	** Details: Not fully landed. 3 WIP blockers		** Details: Not fully landed. 1 WIP blockers
	*** https://bugzilla.mozilla.org/show_bug.cgi?id=1137498		*** https://bugzilla.mozilla.org/show_bug.cgi?id=1137498
	*** https://bugzilla.mozilla.org/show_bug.cgi?id=1137501		*** https://bugzilla.mozilla.org/show_bug.cgi?id=1137501 (Resolved, but not uplift to 2.2)
	*** https://bugzilla.mozilla.org/show_bug.cgi?id=1131338 (Resolved, but not uplift to 2.2)
	*** Test cases in Moztrap will be done this week. (test data in qa-testcase-data repository)
	* Search App (Hermes)		* Search App (Hermes)
	** Health: ~~Green~~		** Health: Yellow
	** Test case coverage: 100%		** Test case coverage: 100%
	** Details: 3 WIP blockers		** Details: 5 WIP blockers
	*** https://bugzilla.mozilla.org/show_bug.cgi?id=~~1138631~~		*** https://bugzilla.mozilla.org/show_bug.cgi?id=1140325
	*** https://bugzilla.mozilla.org/show_bug.cgi?id=~~1134350~~		*** https://bugzilla.mozilla.org/show_bug.cgi?id=1141315
	*** https://bugzilla.mozilla.org/show_bug.cgi?id=1132262		*** https://bugzilla.mozilla.org/show_bug.cgi?id=1140525 (RTL)
			*** https://bugzilla.mozilla.org/show_bug.cgi?id=1139807 (RTL)
			*** https://bugzilla.mozilla.org/show_bug.cgi?id=1132262 (patch will be landed this week)
	* App-grouping (Hermes)		* App-grouping (Hermes)
	** Health: Green		** Health: Green
Line 90:		Line 95:
	** Health: Green		** Health: Green
	** Test case coverage: 100%		** Test case coverage: 100%
	** Details: 2 Blockers		** Details: 1 Blockers
	*** https://bugzilla.mozilla.org/show_bug.cgi?id=1100610
	*** https://bugzilla.mozilla.org/show_bug.cgi?id=1107139		*** https://bugzilla.mozilla.org/show_bug.cgi?id=1107139
	*** https://bugzilla.mozilla.org/show_bug.cgi?id=1120974 (resolved not to 2.2)
	* L-porting (Hermes)		* L-porting (Hermes)
	** Health: Green		** Health: Green
	** Details:		** Details:
	*** ~~Marigold has started to perform manual smoke test & regression test.~~		*** Patch of camera recording is waiting for review. https://bugzilla.mozilla.org/show_bug.cgi?id=1120780
	*** Camera recording ~~is still not ready, and its patch~~ is waiting for review. https://bugzilla.mozilla.org/show_bug.cgi?id=1120780
	* CAF bugs (Eric)		* CAF bugs (Eric)
	** Health: Green		** Health: Green
Line 110:		Line 112:
	** Health: Yellow		** Health: Yellow
	** Test case coverage: 100%		** Test case coverage: 100%
	** Details: ~~Will start~~ PTS certification ~~test~~ on flame-kk this week. Need to manually turn on bluetooth daemon~~(adb push an updated prop)~~. Currently the new code with daemon is off by default due to bluetooth function break after FOTA, http://bugzil.la/1134821 a patch is under review.		** Details:
			** PTS certification tested on flame-kk this week. 5/6 profile test completed. Couple cases failed but we need to know they are due to steps error or they are actually bugs http://bit.ly/1FLiZl8 https://bugzil.la/1139349
			** Need to manually turn on bluetooth daemon. Currently the new code with daemon is off by default due to bluetooth function break after FOTA, http://bugzil.la/1134821 a patch is under review.
			** What is Bluetooth PTS? https://wiki.mozilla.org/B2G/QA/Bluetooth_PTS
	* Metrics (Shing)		* Metrics (Shing)
	** Health: Yellow		** Health: Yellow
	** Test case coverage: 80%		** Test case coverage: 95%
	** Details:		** Details:
	*** 1 Blocker ~~http://bugzil~~.~~la/1137063~~		*** 3 Blocker (2.2?) http://mzl.la/1EbkJ7r
	~~***~~ 2 ~~WIP http://bugzil.la/1130013,~~ http://~~bugzil~~.la/~~1126524~~
	* 256M RAM memory optimization for WFVGA, 5MP. (Teri)		* 256M RAM memory optimization for WFVGA, 5MP. (Teri)
	** Health: Green		** Health: Green
	** Test case coverage: 100%		** Test case coverage: 100%
	** Details: ~~Test~~ in progress ~~50%~~, no blocker.		** Details: Testing in progress, no blocker.
	* Multicore performance (Edward)		* Multicore performance (Edward)
	** Health:		** Health: