B2G/QA/2.2/2015-04-27Status

V2.2 Highlight & Risk

• Highlight
  • V2.2 FC moved to 2015/04/29.
  • NEXUS-5-L 5.1 PVT build MTBF 38.5 hrs & counting.
  • FLAME-KK 4.4 PVT build MTBF 34.42 hrs.
  • NEXUS-5-L 5.1 Regression test done.
  • 48 bugs with blocking=2.2+ http://mzl.la/1CDEaqi
  • 12 bugs with blocking=2.2? http://mzl.la/1CDEKV4

V2.2 Milestone

• 2015/06/08 V2.2 CC
• 2015/04/29 V2.2 Updated FC
• 2015/04/06 V2.2 FC
• 2015/02/23 V2.2 FL

V2.2 Testing Scope

• Focus on bugs in agreed list.
• B2G V2.2 Overview Page for more info.

• Status NEW: blocking-b2g=2.2+ 49 bugs

Bugzilla query error

Array ( [type] => error [message] => http-timed-out [params] => Array ( [0] => https://bugzilla.mozilla.org/bzapi/bug?cf_blocking_b2g=2.2%2B&cf_blocking_b2g_type=contains&include_fields=id&include_fields=assigned_to&include_fields=summary&include_fields=status&include_fields=target_milestone ) ) 1

V2.2 Testing by Type

Smoke Testing

Flame-KK Green

• No bug found blocking manual smoketests but we have couple automation bugs need to investigate.
• New Bugs Breaking the Automation smoketests
  • [Investigating] test_clock_create_new_alarm.py - Bug 1138278- [Clock] After running test_clock_create_new_alarm.py, it should set the alarm time to 1 min more than the current time
  • [Investigating] [Not reproduced manually] test_ftu_skip_tour.py - Bug 1138278 - test_ftu_skip_tour should reflect new Send Data initial state and separate SIM- and network-available conditions
  • [Investigating] test_marketplace_packaged_app.py - Bug 1151172 - [marketplace-tests-gaia] "test_marketplace_search_and_install_app" is failing on b2g v2.2
  • [Not reproduced] test_sms_with_attachments.py - Bug 1151696 - [v2.2] Investigate test_sms_with_attachments.py failure . (Cannot reproduce locally)
  • [Not reproduced manually] test_music_artist_mp3.py - Bug 1149886 - using to replace at Music app results in unable to locate <span.list-song-index> error

• Automation reports - are available on Github.IO & fxosqa-report-watchlist.

Nexus-5-L 5.1 Red

• Testing on PVT build from 2015/04/14
• Cases blocking manual test
  • OTA not working
• Manual - are available on Google Drive
• Known issues - no FM radio support, no DSDS support and no UMS support.

2nd Run (2015/04/14)

• NEXUS-5-L 2nd Run (focus on essential functionality) Done
  • Test cases http://mzl.la/1D76gXG

• Result
  • 1119(total), 1090(passed), 8(failed), 21(invalid)
  • 0.007% fail rate

• bugs
  • Total 5
  • http://mzl.la/1cYZZau (Bugs related to software home button, privacy or ui polish will be skipped for now)

• FLAME-KK 2nd Run (focus on dsds and gaia apps) Done
  • ​Test cases http://mzl.la/1aKYIC8

• Result
  • ​342(total), 334(passed), 5(failed), 3(invalid)
  • ​0.01% fail rate

• bugs
  • ​Total 5
  • ​http://mzl.la/1GifEOz

1st Run (2015/03/03)

• RTL/V2.2 Flame-KK Done
  • Moztrap result - 76 fail, 326 pass, check with Josh for detail.

• Full Regression/V2.2 Flame-KK Done
  • Moztrap result - 55 fail, 3426 pass, 100% complete

• Partial Regression/V2.2 Nexus-5-L Done
  • Moztrap result - 30 fail, 1216 pass, 100% complete
  • Bug list

• Bluetooth PTS/V2.2 Flame-KK Done
  • Bug Meta dependency tree - 0 open bugs

Functional Testing

Red

• General
  • Details:
  • 2nd regression, NEXUS-5-L 1119(total), 8(failed) bugs http://mzl.la/1cYZZau
  • FLAME-KK 342(total), 5(failed) bugs http://mzl.la/1GifEOz
  • NEXUS-5-L MTBF Average: ​29.18 hrs
  • Flame-L, RD has just received complete code, but building fail.

• RTL (Marigold)
  • Health Status: Green
  • Details: 2 blockers http://mzl.la/1Juz6oH

• Cut, Copy, and Paste (Gerry, Edward, Teri)
  • Health Status:
  • Details: 1 blocker
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1021499

• System Platform (Gerry)
  • Health Status:
  • Details: 7 blockers

• OTA/FOTA (Gerry)
  • Health Status:
  • Details:
    • 2.0/2.1 can’t upgrade to 2.2 via FOTA - Bug 1155035
    • 2.0 can upgrade to 2.2 via OTA with some issues
      • Bluetooth issue - Bug 1132388 & Bug 1134821
      • Bug 1153787 & Bug 1153778
    • 2.1 can’t upgrade to 2.2 via OTA - Bug 1151006

• NFC (Alison)
  • Health Status: Yellow
  • Details: 2 blockers
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1155873
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1156605 (r+, wait to land code)

• Ringtones (Alison)
  • Health Status: Green
  • Details: No blockers

• MTP/UMS (Alison)
  • Health Status: Green
  • Details: No blockers

• System Frontend bugs (Hermes)
  • Health Status: RED
  • Details: 7 blockers, 2 nom

• L-porting V5.1 (Hermes)
  • Health Status: Green
  • Details: no blockers

• CAF bugs (Eric)
  • Health Status: Red
  • Details: 3 new bluetooth bugs opened this week, RDs are working on them, http://mzl.la/1HWrTQa 28 in total

• COMMS, RIL bugs (Eric)
  • Health Status: Yellow
  • Details: 2WIP http://mzl.la/1HWsscJ

• MUSIC (Eric)
  • Health Status: Yellow
  • Details: lots of regression recently http://mzl.la/1HWsSzS
    • dominic kuo/jim porter (cc Hema)

• Bluetooth (Eric, Teri)
  • Health Status: Yellow
  • Details: https://bugzil.la/1143925 Patch in review

• Metrics (Shing)
  • Health Status: Yellow
  • Details: WIP https://bugzil.la/1134998

• 256M RAM memory optimization for WFVGA, 5MP. (Teri)
  • Health Status: Green
  • Details:

Exploratory Test

Flame-KK & NEXUS-5-L

• 268 bugs found
  • UNCONFIRMED, NEW, ASSIGNED, REOPENED - 253 bugs found.
  • UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - 12 bugs found.
  • RESOLVED, VERIFIED, CLOSED - 191 bugs found.
  • RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - 19 bugs found.

• Regression bugs
  • UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - 17 bugs found.
  • RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - 256 bugs found.

Stability

NEXUS-5-L 5.1

• Environment
  • Display DIM
  • Rest time with dummy case (30 sec)
  • Running tests ** 14 case http://bit.ly/1aWkJyY
  • NEXUS *4, ZTE *2

- Result

• NEXUS-5-L 5.1
  • Current run - NEXUS-5-L 5.1 *4: 38.5 hrs and counting = [55+93(running)+93(running) +67(running)] / (4 devices + 4 crashes) = 38.5
  • Current run - ZTE-L 5.1 * 2: Just started
  • Previous run - NEXUS-5-L 5.1 *4: MTBF 34.42 hrs = (41.3 + 53.2 + 43.8 + 6.1)/4
  • NEXUS-5-L 5.1 Rest time: 10%
  • [script issue] OSError: [Errno 24] Too many open files.
  • [adb issue] https://bugzil.la/1154629

FLAME-KK 4.4

• Environment
  • Display DIM
  • Rest time with dummy case (30 sec)
  • FLAME *9

- Result

• FLAME-KK 4.4
  • LAME-KK 4.4 * 4: MTBF 25.81 hrs = 50.77/4.96/25.81 - max/min/avg

V2.2 Test Case Creation

On Track

• Moztrap link of V2.2 New Features suite.

Test Case Creation WIP

• [2.2-feature-qa+] & in-moztrap? (0 bugs) --> WORK DONE
• feature-b2g=2.2+ & in-moztrap? (0 bugs) --> WORK DONE

Test Case Creation DONE

• [2.2-feature-qa+] & in-moztrap+ (19 bugs) --> WORK DONE
• feature-b2g=2.2+ & in-moztrap+ (78 bugs) --> WORK DONE

V2.2 Testing Verification Status

Overall Verification Summary

Need Verification

Full Query

ID	Qa contact	Status	Resolution	Component	Assigned to	Summary
987532	No-Jun Park [:njpark]	RESOLVED	WORKSFORME	Performance	Jerry Shih[:jerry] (UTC+8) (inactive)	[meta] Project Silk on B2G.
1018320		RESOLVED	FIXED	DOM: Core & HTML	Andrea Marchesini [:baku]	Implement RequestSync API for FirefoxOS
1020757		RESOLVED	FIXED	Gaia::Settings	EJ Chen [:eragonj][:小龍哥][Contributor mode on](inactive)	[FDN] Enable/Disable FDN asks to enter PUK if the enable/disable FDN failed
1043558		RESOLVED	FIXED	Graphics	Sotaro Ikeda [:sotaro]	Use gralloc buffers for WebRTC local camera preview
1061130		RESOLVED	FIXED	Gaia::Dialer	Gabriele Svelto [:gsvelto]	[STK] 3GPP TS 22.030 6.6.4 An abbreviated dialling code shall be able to be read using the following procedure: N(N)(N)#
1068963		RESOLVED	INCOMPLETE	Audio/Video	Blake Wu [:bwu][:blakewu]	Frame drops during end-to-end playback
1071484		RESOLVED	FIXED	Audio/Video	Bobby Chien	[Meta][User story] Support *.mkv container format
1073419		RESOLVED	WONTFIX	General		[ALA] All gecko code needed to support adjustable location accuracy
1082268	Edward Chen[:edchen]	RESOLVED	WORKSFORME	Gaia::Music	Bobby Chien	[meta] Music app launch latency is worse compared to Android on quad core 1 GB device
1082290		RESOLVED	FIXED	Performance	Dave Hylands [:dhylands]	Implement cgroup swappiness feature for low memory target (256 MB)
1083953		RESOLVED	FIXED	Gaia	marta	[META] Privacy Panel move from dev_apps to apps
1086963	Edward Chen[:edchen]	RESOLVED	WORKSFORME	Gaia::Gallery	Bobby Chien	[meta] Gallery app launch latency is worse than Android on quad core device
1092427	Eric Chang [:ericcc] [:echang]	RESOLVED	FIXED	Gaia::Dialer	Tamara Hills [:thills]	Dialer app should follow text selection pattern.
1092437	Eric Chang [:ericcc] [:echang]	RESOLVED	FIXED	Gaia::SMS		Messages app should follow text selection pattern
1092960		RESOLVED	FIXED	Gaia::Camera	Wilson Page [:wilsonpage]	Camera app to follow text selection pattern
1092966		RESOLVED	FIXED	Gaia::FMRadio	Justin D'Arcangelo [:justindarc]	FM Radio app to follow text selection pattern
1092967		RESOLVED	FIXED	Gaia::Gallery	Punam Dahiya [:pdahiya]	Gallery app to follow text selection pattern
1092969	Eric Chang [:ericcc] [:echang]	RESOLVED	FIXED	Gaia::Music	Dominic Kuo [:dkuo]	Music app to follow text selection pattern
1092972		RESOLVED	FIXED	Gaia::Video	Russ Nicoletti [:russn]	Video app to follow text selection pattern
1094121		RESOLVED	FIXED	General	Steven Yang [:styang]	[meta] Android L 5.0 Porting for B2G
1097865	Shing Lyu [:shinglyu]	RESOLVED	WONTFIX	Gaia::System	Marshall Culpepper [:marshall_law]	[Telemetry] Collect metrics for voice call info
1098970		RESOLVED	FIXED	GonkIntegration	Boris Chiou [:boris]	[lollipop] Porting NativeWindow from LL to gonk
1099076	Shing Lyu [:shinglyu]	RESOLVED	WONTFIX	WebRTC		Screen sharing support on B2G
1099079		RESOLVED	WONTFIX	Gaia		Implement permission dialog for screen sharing
1107291		RESOLVED	FIXED	MTP/UMS	viral [:viralwang]	SD/Storage (fuse) for L on Nexus 5
1107296		RESOLVED	WORKSFORME	General	Jamin Liu [:jaliu][:Jamin]	[meta] GPS Android L Porting
1107298		RESOLVED	FIXED	General	viral [:viralwang]	[meta] Sensors Android L Porting
1107305		RESOLVED	FIXED	General	Kai-Zhen Li [:seinlin][:kli]	[gonk-l] the error message: could not load library "libsigchain.so" from LD_PRELOAD
1109422	Shing Lyu [:shinglyu]	RESOLVED	FIXED	General	Marshall Culpepper [:marshall_law]	AppUsage Metric collection enhancements for MonthlyActiveUsers computation
1115356		RESOLVED	INVALID	Gaia::Settings	Josh Cheng [:josh]	[meta]Privacy Bugs Meta

30 Total; 0 Open (0%); 30 Resolved (100%); 0 Verified (0%);

Verification Done

Full Query

ID	Qa contact	Status	Resolution	Component	Assigned to	Summary
1107304		VERIFIED	DUPLICATE	General		[meta] BT Android L Porting

1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);