B2G/QA/2.2/2015-04-27Status: Difference between revisions

Revision as of 09:12, 27 April 2015

V2.2 Highlight & Risk

Highlight
- V2.2 FC moved to 2015/04/29
- NEXUS-5-L 5.1 PVT build MTBF 34.42 hrs
- NEXUS-5-L 5.1 Regression test started from 2015/04/14 for 2 wks
- 48 bugs with blocking=2.2+ http://mzl.la/1CDEaqi
- 22 bugs with blocking=2.2? http://mzl.la/1CDEKV4

V2.2 Milestone

2015/06/08 V2.2 CC
2015/04/29 V2.2 Updated FC
~~2015/04/06 V2.2 FC~~
2015/02/23 V2.2 FL

QA Shared calendar
- You can add it to your Google Calendar view by clicking on this link and then clicking on the small blue "+ Google Calendar" button at the bottom, right hand of the screen.
- If you are signed in with your @mozilla.com account, you can also access it this way:
  - In Google Calendar, you should go to Other calendars > Browse Interesting Calendars > More > Resources for mozilla.com > Public (9) > Public - FxOS QA Schedule > Subscribe

V2.2 Testing Scope

Focus on bugs in agreed list.
B2G V2.2 Overview Page for more info.

NEW blocking-b2g=2.2+ 49 bugs

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1

V2.2 Testing by Type

Smoke Testing

Flame-KK Green

No bug found blocking manual smoketests but we have couple automation bugs need to investigate.
New Bugs Breaking the Automation smoketests
- [Investigating] test_clock_create_new_alarm.py - Bug 1138278- [Clock] After running test_clock_create_new_alarm.py, it should set the alarm time to 1 min more than the current time
- [Investigating] [Not reproduced manually] test_ftu_skip_tour.py - Bug 1138278 - test_ftu_skip_tour should reflect new Send Data initial state and separate SIM- and network-available conditions
- [Investigating] test_marketplace_packaged_app.py - Bug 1151172 - [marketplace-tests-gaia] "test_marketplace_search_and_install_app" is failing on b2g v2.2
- [Not reproduced] test_sms_with_attachments.py - Bug 1151696 - [v2.2] Investigate test_sms_with_attachments.py failure . (Cannot reproduce locally)
- [Not reproduced manually] test_music_artist_mp3.py - Bug 1149886 - using to replace at Music app results in unable to locate <span.list-song-index> error

Automation reports - are available on Github.IO & fxosqa-report-watchlist.

Nexus-5-L 5.1 Red

Testing on PVT build from 2015/04/14
Cases blocking manual test
- OTA not working
Manual - are available on Google Drive
Known issues - no FM radio support, no DSDS support and no UMS support.

Regression Testing

2nd Run (2015/04/14)

NEXUS-5-L (80% done)
- http://mzl.la/1D76gXG
Result
1076(total), 1049(passed), 6(failed), 21(invalid)
0.005% fail rate

FLAME-KK (done)
- http://mzl.la/1aKYIC8
Result
342(total), 334(passed), 5(failed), 3(invalid)
0.01% fail rate

1st Run (2015/03/03)

RTL/V2.2 Flame-KK Done
- Moztrap result - 76 fail, 326 pass, check with Josh for detail.

Full Regression/V2.2 Flame-KK Done
- Moztrap result - 55 fail, 3426 pass, 100% complete

Partial Regression/V2.2 Nexus-5-L Done
- Moztrap result - 30 fail, 1216 pass, 100% complete
- Bug list

Bluetooth PTS/V2.2 Flame-KK Done
- Bug Meta dependency tree - 0 open bugs

Functional Testing

Red

General
- Details:
- 2nd regression, NEXUS-5-L 1119(total), 8(failed) bugs http://mzl.la/1cYZZau
- FLAME-KK 342(total), 5(failed) bugs http://mzl.la/1GifEOz
- NEXUS-5-L MTBF Average: 29.18 hrs
- Flame-L, RD has just received complete code, but building fail.

RTL (Marigold)
- Health Status: Green
- Details: 3 blockers http://mzl.la/1Juz6oH

Cut, Copy, and Paste (Gerry, Edward, Teri)
- Health Status:
- Details: 1 blocker
  - https://bugzilla.mozilla.org/show_bug.cgi?id=1021499

System Platform (Gerry)
- Health Status:
- Details: 7 blockers

OTA/FOTA (Gerry)
- Health Status:
- Details:
  - 2.0/2.1 can’t upgrade to 2.2 via FOTA - Bug 1155035
  - 2.0 can upgrade to 2.2 via OTA with some issues
    - Bluetooth issue - Bug 1132388 & Bug 1134821
    - Bug 1153787 & Bug 1153778
  - 2.1 can’t upgrade to 2.2 via OTA - Bug 1151006

NFC (Alison)
- Health Status: Yellow
- Details: 2 blockers
  - https://bugzilla.mozilla.org/show_bug.cgi?id=1155873
  - https://bugzilla.mozilla.org/show_bug.cgi?id=1156605 (r+, wait to land code)

Ringtones (Alison)
- Health Status: Green
- Details: No blockers

MTP/UMS (Alison)
- Health Status: Green
- Details: No blockers

System Frontend bugs (Hermes)
- Health Status: RED
- Details: 7 blockers, 2 nom

L-porting V5.1 (Hermes)
- Health Status: Green
- Details: no blockers

CAF bugs (Eric)
- Health Status: Red
- Details: 3 new bluetooth bugs opened this week, RDs are working on them, http://mzl.la/1HWrTQa 28 in total

COMMS, RIL bugs (Eric)
- Health Status: Yellow
- Details: 2WIP http://mzl.la/1HWsscJ

MUSIC (Eric)
- Health Status: Yellow
- Details: lots of regression recently http://mzl.la/1HWsSzS
  - dominic kuo/jim porter (cc Hema)

Bluetooth (Eric, Teri)
- Health Status: Yellow
- Details: https://bugzil.la/1143925 Patch in review

Metrics (Shing)
- Health Status: Yellow
- Details: WIP https://bugzil.la/1134998

256M RAM memory optimization for WFVGA, 5MP. (Teri)
- Health Status: Green
- Details:

Exploratory Test

Flame-KK & NEXUS-5-L

240 bugs found
- UNCONFIRMED, NEW, ASSIGNED, REOPENED - 224 bugs found.
- UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - 14 bugs found.
- RESOLVED, VERIFIED, CLOSED - 184 bugs found.
- RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - 16 bugs found.

Regression bugs
- UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - 15 bugs found.
- RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - 249 bugs found.

Stability

NEXUS-5-L 5.1

Environment
- Display DIM
- Rest time with dummy case (30 sec)
- Running tests
- 14 case http://bit.ly/1aWkJyY
- NEXUS *3

- Result

NEXUS-5-L 5.1
- NEXUS-5-L 5.1 * 4: MTBF 34.42 hrs = (41.3 + 53.2 + 43.8 + 6.1)/4
- NEXUS-5-L 5.1 Rest time: 10%
- [script issue] OSError: [Errno 24] Too many open files.
- [adb issue] https://bugzil.la/1154629

V2.2 Test Case Creation

On Track

Moztrap link of V2.2 New Features suite.

Test Case Creation WIP

[2.2-feature-qa+] & in-moztrap? (0 bugs) --> WORK DONE
feature-b2g=2.2+ & in-moztrap? (0 bugs) --> WORK DONE

Test Case Creation DONE

[2.2-feature-qa+] & in-moztrap+ (19 bugs) --> WORK DONE
feature-b2g=2.2+ & in-moztrap+ (78 bugs) --> WORK DONE

V2.2 Testing Verification Status

Overall Verification Summary

Need Verification

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1

Verification Done

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1

@@ Line 86: / Line 86: @@
 * General
-** Ongoing regression test on NEXUS-5-L,  FLAME-KK, results expected by 24th.
+** Details:
-** 5.1 has landed, please flash LMY47D then v2.2 on NEXUS-5-L.
+** 2nd regression, NEXUS-5-L 1119(total), 8(failed) bugs http://mzl.la/1cYZZau
-*** Do we have pvt nightlies available
+** FLAME-KK 342(total), 5(failed) bugs http://mzl.la/1GifEOz
-** https://wiki.mozilla.org/B2G/QA/2.2
+** NEXUS-5-L MTBF Average: 29.18 hrs
-** Flame-L, RD has not received complete code yet, porting estimation: 2wks, new branch published on PVT: another 2+ wks.
+**Flame-L, RD has just received complete code, but building fail.
 * RTL (Marigold)
-** Health Status: Yellow
+** Health Status: Green
-** Details: 5 blockers
+** Details: 3 blockers http://mzl.la/1Juz6oH
-*** https://wiki.mozilla.org/RTL#RTL_blocking-b2g:2.2.2B
 * Cut, Copy, and Paste  (Gerry, Edward, Teri)
 ** Health Status:
 ** Details: 1 blocker
 *** https://bugzilla.mozilla.org/show_bug.cgi?id=1021499
 * System Platform (Gerry)
 ** Health Status:
-** Details: 4 blockers
+** Details: 7 blockers
+* OTA/FOTA (Gerry)
+** Health Status:
+** Details:
+*** 2.0/2.1 can’t upgrade to 2.2 via FOTA - Bug 1155035
+*** 2.0 can upgrade to 2.2 via OTA with some issues
+**** Bluetooth issue - Bug 1132388 & Bug 1134821
+**** Bug 1153787 & Bug 1153778
+*** 2.1 can’t upgrade to 2.2 via OTA - Bug 1151006
 * NFC  (Alison)
-** Health Status: Green
+** Health Status: Yellow
-** Details: No blockers
+** Details: 2 blockers
+*** https://bugzilla.mozilla.org/show_bug.cgi?id=1155873
+*** https://bugzilla.mozilla.org/show_bug.cgi?id=1156605 (r+, wait to land code)
 * Ringtones  (Alison)
@@ Line 119: / Line 129: @@
 * System Frontend bugs (Hermes)
-** Health Status: Yellow
+** Health Status: RED
-** Details: 4 blockers, 2 nom
+** Details: 7 blockers, 2 nom
 * L-porting V5.1 (Hermes)
 ** Health Status: Green
-** Details:
+** Details: no blockers
-*** Update to 5.1 mr1: done on pvt. We have started to perform test runs on Nexus-5 L 5.1
 * CAF bugs (Eric)
 ** Health Status: Red
-** Details: 11 opened since Monday, no owner yet. other bugs 39 in total http://mzl.la/1zgJ8EG Check with kkuo for these new bugs
+** Details: 3 new bluetooth bugs opened this week, RDs are working on them, http://mzl.la/1HWrTQa 28 in total
 * COMMS, RIL  bugs (Eric)
 ** Health Status: Yellow
-** Details: 2 WIP, 1 Investigation http://mzl.la/1zgFMl1
+** Details: 2WIP http://mzl.la/1HWsscJ
+* MUSIC (Eric)
+** Health Status: Yellow
+** Details: lots of regression recently  http://mzl.la/1HWsSzS
+*** dominic kuo/jim porter (cc Hema)
 * Bluetooth (Eric, Teri)
 ** Health Status: Yellow
-** Details:
+** Details: https://bugzil.la/1143925 Patch in review
-*** Waiting for code to land https://bugzilla.mozilla.org/show_bug.cgi?id=1152098
-*** ETA 4/20 https://bugzilla.mozilla.org/show_bug.cgi?id=1143925
 * Metrics (Shing)
 ** Health Status: Yellow
-** Details: 1 WIP https://bugzil.la/1134998
+** Details: WIP https://bugzil.la/1134998
 * 256M RAM memory optimization for WFVGA, 5MP.  (Teri)
 ** Health Status: Green
-** Details: No blocker
+** Details:
 ==== Exploratory Test ====

B2G/QA/2.2/2015-04-27Status: Difference between revisions

Revision as of 09:12, 27 April 2015

Contents

V2.2 Highlight & Risk

V2.2 Milestone

V2.2 Testing Scope

Bugzilla query error

V2.2 Testing by Type

Smoke Testing

Regression Testing

2nd Run (2015/04/14)

1st Run (2015/03/03)

Functional Testing

Exploratory Test

Stability

V2.2 Test Case Creation

Test Case Creation WIP

Test Case Creation DONE

V2.2 Testing Verification Status

Overall Verification Summary

Bugzilla query error

Bugzilla query error

Navigation menu

B2G/QA/2.2/2015-04-27Status: Difference between revisions

Revision as of 09:12, 27 April 2015

V2.2 Highlight & Risk

V2.2 Milestone

V2.2 Testing Scope

Bugzilla query error

V2.2 Testing by Type

Smoke Testing

Regression Testing

2nd Run (2015/04/14)

1st Run (2015/03/03)

Functional Testing

Exploratory Test

Stability

V2.2 Test Case Creation

Test Case Creation WIP

Test Case Creation DONE

V2.2 Testing Verification Status

Overall Verification Summary

Bugzilla query error

Bugzilla query error

Navigation menu

Search