B2G/QA/2.2/2015-03-16Status: Difference between revisions

Revision as of 09:51, 16 March 2015

V2.2 Highlight & Risk

Highlight
- Availability of Flame-L is unknown.

V2.2 Milestone

2015/02/23 V2.2 FL
2015/04/06 V2.2 FC

QA Shared calendar
- You can add it to your Google Calendar view by clicking on this link and then clicking on the small blue "+ Google Calendar" button at the bottom, right hand of the screen.
- If you are signed in with your @mozilla.com account, you can also access it this way:
  - In Google Calendar, you should go to Other calendars > Browse Interesting Calendars > More > Resources for mozilla.com > Public (9) > Public - FxOS QA Schedule > Subscribe

V2.2 Testing Scope

Focus on bugs in agreed list.
B2G V2.2 Overview Page for more info.
Current feature-b2g=2.2+ 8 bugs
Current blocking-b2g=2.2+ 87 bugs

V2.2 Testing by Type

Smoke Testing

Flame-KK On Track

Existing Bugs Breaking the Automation smoketests: Flame-KK
- [Disabled] test_setup_and_send_active_sync_email.py - Bug 1028192 - Problems with ActiveSync accounts
- [Disabled] test_receive_active_sync_email.py - Bug 1028192 - Problems with ActiveSync accounts
- [Disabled] test_cost_control_data_alert_mobile.py - Bug 1061698 - Need ability to work in the shadow root/shadow DOM
- [Disabled] test_call_log_all_calls.py - Bug 1096385 - Investigate failure in test_call_log_all_calls.py, call log is displayed erroneously

Nexus-5-L On Track

1 blocker.
- Bug 1118711 - Email app fails to fully launch due to worker JS runtime problem (reported as "InternalError: too much recursion") breaking startup process

Daily Reports can be found here.
- Manual - are available on Google Drive & fxosqa-report-watchlist.
- Automation - are available on Github.IO & fxosqa-report-watchlist.

Functional Testing

Yellow

QA focus - 42 bugs found.

General (eric)
- Flame-L might be late for FC, roughly 6 weeks or more needed after SOW (SOW, legal review, porting, build time), PM will have an internal discussion to decide what to do.
- feature-b2g|blocking-b2g == 2.2?|+ [not resolved] http://mzl.la/1C4X1KT Check if code uplifted
- feature-b2g|blocking-b2g == 2.2?|+ [resolved] http://mzl.la/1E5mQqM Need verification
- status-b2g-v2.2 affected not blocking any bugs http://mzl.la/1E5ArOQ
RTL (Marigold)
- Health: Red
- Test case coverage: 100%
- Details: 15 blockers
  - http://mzl.la/1b2ddSG

Privacy (Gerry)
- Health: Yellow
- Test case coverage:
- Details: 5 blockers
Josh to help talk with dave/eitan on the backout for the compile time config
Cut, Copy, and Paste (Gerry, Edward, Teri, Eric)
- Health: Yellow
- Test case coverage:
- Details: 2 blockers
  - https://bugzilla.mozilla.org/show_bug.cgi?id=1021499
  - https://bugzilla.mozilla.org/show_bug.cgi?id=1130982
NFC (Alison)
- Health: Green
- Test case coverage:100%
- Details:
  - No blockers
short_name support in manifest.webapp (Hermes)
- Health: Yellow
- Test case coverage: 100%
- Details: Not fully landed. 1 WIP blockers
  - https://bugzilla.mozilla.org/show_bug.cgi?id=1137498
  - https://bugzilla.mozilla.org/show_bug.cgi?id=1137501 (Resolved, but not uplift to 2.2)
Search App (Hermes)
- Health: Yellow
- Test case coverage: 100%
- Details: 5 WIP blockers
  - https://bugzilla.mozilla.org/show_bug.cgi?id=1140325
  - https://bugzilla.mozilla.org/show_bug.cgi?id=1141315
  - https://bugzilla.mozilla.org/show_bug.cgi?id=1140525 (RTL)
  - https://bugzilla.mozilla.org/show_bug.cgi?id=1139807 (RTL)
  - https://bugzilla.mozilla.org/show_bug.cgi?id=1132262 (patch will be landed this week)
App-grouping (Hermes)
- Health: Green
- Test case coverage: 100%
- Details: No 2.2+ blockers for now
Task Manager (Hermes)
- Health: Green
- Test case coverage: 100%
- Details: 1 Blockers
  - https://bugzilla.mozilla.org/show_bug.cgi?id=1107139
L-porting (Hermes)
- Health: Green
- Details:
  - Patch of camera recording is waiting for review. https://bugzilla.mozilla.org/show_bug.cgi?id=1120780
CAF bugs (Eric)
- Health: Green
- Test case coverage: 100%
- Details: Running regression test now.
TCL bugs (Alison, Eric)
- Health: Green
- Test case coverage: 100%
- Details: Running regression test now.
Bluetooth (Eric, Teri)
- Health: Yellow
- Test case coverage: 100%
- Details:
- PTS certification tested on flame-kk this week. 5/6 profile test completed. Couple cases failed but we need to know they are due to steps error or they are actually bugs http://bit.ly/1FLiZl8 https://bugzil.la/1139349
- Need to manually turn on bluetooth daemon. Currently the new code with daemon is off by default due to bluetooth function break after FOTA, http://bugzil.la/1134821 a patch is under review.
- What is Bluetooth PTS? https://wiki.mozilla.org/B2G/QA/Bluetooth_PTS
Metrics (Shing)
- Health: Yellow
- Test case coverage: 95%
- Details:
  - 3 Blocker (2.2?) http://mzl.la/1EbkJ7r
256M RAM memory optimization for WFVGA, 5MP. (Teri)
- Health: Green
- Test case coverage: 100%
- Details: Testing in progress, no blocker.
Multicore performance (Edward)
- Health:
- Test case coverage:
- Better performance on Multicore (App startup time for core apps is comparable to Android
- Set up raptor for v2.2
- https://bugzilla.mozilla.org/show_bug.cgi?id=1132515
  - analyze and narrow down result from QA jenkins, http://bit.ly/1Glz30h. The problem should be between 4044 and 4045.

Exploratory Testing

Flame-KK On Track

223 bugs found - includes every bug in bugzilla affecting v2.2, not limited to the bugs after branch day.
- UNCONFIRMED, NEW, ASSIGNED, REOPENED - 107 bugs found.
- UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - 14 bugs found.
- RESOLVED, VERIFIED, CLOSED - 116 bugs found.
- RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - 8 bugs found.

Regression
- UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - 31 bugs found.
- RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - 207 bugs found.

Performance Testing

Flame-KK Yellow

will be on soon, link (VPN required)
- Target platforms - flame-KK, flame-L

V2.2 Test Case Creation

On Track

Moztrap link of V2.2 New Features suite.

Test Case Creation WIP

[2.2-feature-qa+] & in-moztrap? (0 bugs) --> WORK DONE
feature-b2g=2.2+ & in-moztrap? (0 bugs) --> WORK DONE

Test Case Creation DONE

[2.2-feature-qa+] & in-moztrap+ (19 bugs) --> WORK DONE
feature-b2g=2.2+ & in-moztrap+ (78 bugs) --> WORK DONE

V2.2 Testing Verification Status

Overall Verification Summary

Need Verification

Full Query

ID	Qa contact	Status	Resolution	Component	Assigned to	Summary
987532	No-Jun Park [:njpark]	RESOLVED	WORKSFORME	Performance	Jerry Shih[:jerry] (UTC+8) (inactive)	[meta] Project Silk on B2G.
1018320		RESOLVED	FIXED	DOM: Core & HTML	Andrea Marchesini [:baku]	Implement RequestSync API for FirefoxOS
1020757		RESOLVED	FIXED	Gaia::Settings	EJ Chen [:eragonj][:小龍哥][Contributor mode on](inactive)	[FDN] Enable/Disable FDN asks to enter PUK if the enable/disable FDN failed
1043558		RESOLVED	FIXED	Graphics	Sotaro Ikeda [:sotaro]	Use gralloc buffers for WebRTC local camera preview
1061130		RESOLVED	FIXED	Gaia::Dialer	Gabriele Svelto [:gsvelto]	[STK] 3GPP TS 22.030 6.6.4 An abbreviated dialling code shall be able to be read using the following procedure: N(N)(N)#
1068963		RESOLVED	INCOMPLETE	Audio/Video	Blake Wu [:bwu][:blakewu]	Frame drops during end-to-end playback
1071484		RESOLVED	FIXED	Audio/Video	Bobby Chien	[Meta][User story] Support *.mkv container format
1073419		RESOLVED	WONTFIX	General		[ALA] All gecko code needed to support adjustable location accuracy
1082268	Edward Chen[:edchen]	RESOLVED	WORKSFORME	Gaia::Music	Bobby Chien	[meta] Music app launch latency is worse compared to Android on quad core 1 GB device
1082290		RESOLVED	FIXED	Performance	Dave Hylands [:dhylands]	Implement cgroup swappiness feature for low memory target (256 MB)
1083953		RESOLVED	FIXED	Gaia	marta	[META] Privacy Panel move from dev_apps to apps
1086963	Edward Chen[:edchen]	RESOLVED	WORKSFORME	Gaia::Gallery	Bobby Chien	[meta] Gallery app launch latency is worse than Android on quad core device
1092427	Eric Chang [:ericcc] [:echang]	RESOLVED	FIXED	Gaia::Dialer	Tamara Hills [:thills]	Dialer app should follow text selection pattern.
1092437	Eric Chang [:ericcc] [:echang]	RESOLVED	FIXED	Gaia::SMS		Messages app should follow text selection pattern
1092960		RESOLVED	FIXED	Gaia::Camera	Wilson Page [:wilsonpage]	Camera app to follow text selection pattern
1092966		RESOLVED	FIXED	Gaia::FMRadio	Justin D'Arcangelo [:justindarc]	FM Radio app to follow text selection pattern
1092967		RESOLVED	FIXED	Gaia::Gallery	Punam Dahiya [:pdahiya]	Gallery app to follow text selection pattern
1092969	Eric Chang [:ericcc] [:echang]	RESOLVED	FIXED	Gaia::Music	Dominic Kuo [:dkuo]	Music app to follow text selection pattern
1092972		RESOLVED	FIXED	Gaia::Video	Russ Nicoletti [:russn]	Video app to follow text selection pattern
1094121		RESOLVED	FIXED	General	Steven Yang [:styang]	[meta] Android L 5.0 Porting for B2G
1097865	Shing Lyu [:shinglyu]	RESOLVED	WONTFIX	Gaia::System	Marshall Culpepper [:marshall_law]	[Telemetry] Collect metrics for voice call info
1098970		RESOLVED	FIXED	GonkIntegration	Boris Chiou [:boris]	[lollipop] Porting NativeWindow from LL to gonk
1099076	Shing Lyu [:shinglyu]	RESOLVED	WONTFIX	WebRTC		Screen sharing support on B2G
1099079		RESOLVED	WONTFIX	Gaia		Implement permission dialog for screen sharing
1107291		RESOLVED	FIXED	MTP/UMS	viral [:viralwang]	SD/Storage (fuse) for L on Nexus 5
1107296		RESOLVED	WORKSFORME	General	Jamin Liu [:jaliu][:Jamin]	[meta] GPS Android L Porting
1107298		RESOLVED	FIXED	General	viral [:viralwang]	[meta] Sensors Android L Porting
1107305		RESOLVED	FIXED	General	Kai-Zhen Li [:seinlin][:kli]	[gonk-l] the error message: could not load library "libsigchain.so" from LD_PRELOAD
1109422	Shing Lyu [:shinglyu]	RESOLVED	FIXED	General	Marshall Culpepper [:marshall_law]	AppUsage Metric collection enhancements for MonthlyActiveUsers computation
1115356		RESOLVED	INVALID	Gaia::Settings	Josh Cheng [:josh]	[meta]Privacy Bugs Meta

30 Total; 0 Open (0%); 30 Resolved (100%); 0 Verified (0%);

Verification Done