B2G/QA/2.2/2015-04-27Status: Difference between revisions

Latest revision as of 14:56, 27 April 2015

V2.2 Highlight & Risk

Highlight
- V2.2 FC moved to 2015/04/29.
- NEXUS-5-L 5.1 PVT build MTBF 38.5 hrs & counting.
- FLAME-KK 4.4 PVT build MTBF 34.42 hrs.
- NEXUS-5-L 5.1 Regression test done.
- 48 bugs with blocking=2.2+ http://mzl.la/1CDEaqi
- 12 bugs with blocking=2.2? http://mzl.la/1CDEKV4

V2.2 Milestone

2015/06/08 V2.2 CC
2015/04/29 V2.2 Updated FC
~~2015/04/06 V2.2 FC~~
2015/02/23 V2.2 FL

V2.2 Testing Scope

Focus on bugs in agreed list.
B2G V2.2 Overview Page for more info.

Status NEW: blocking-b2g=2.2+ 49 bugs

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1

V2.2 Testing by Type

Smoke Testing

Flame-KK Green

No bug found blocking manual smoketests but we have couple automation bugs need to investigate.
New Bugs Breaking the Automation smoketests
- [Investigating] test_clock_create_new_alarm.py - Bug 1138278- [Clock] After running test_clock_create_new_alarm.py, it should set the alarm time to 1 min more than the current time
- [Investigating] [Not reproduced manually] test_ftu_skip_tour.py - Bug 1138278 - test_ftu_skip_tour should reflect new Send Data initial state and separate SIM- and network-available conditions
- [Investigating] test_marketplace_packaged_app.py - Bug 1151172 - [marketplace-tests-gaia] "test_marketplace_search_and_install_app" is failing on b2g v2.2
- [Not reproduced] test_sms_with_attachments.py - Bug 1151696 - [v2.2] Investigate test_sms_with_attachments.py failure . (Cannot reproduce locally)
- [Not reproduced manually] test_music_artist_mp3.py - Bug 1149886 - using to replace at Music app results in unable to locate <span.list-song-index> error

Automation reports - are available on Github.IO & fxosqa-report-watchlist.

Nexus-5-L 5.1 Red

Testing on PVT build from 2015/04/14
Cases blocking manual test
- OTA not working
Manual - are available on Google Drive
Known issues - no FM radio support, no DSDS support and no UMS support.

Regression Testing

2nd Run (2015/04/14)

NEXUS-5-L 2nd Run (focus on essential functionality) Done
- Test cases http://mzl.la/1D76gXG

Result
- 1119(total), 1090(passed), 8(failed), 21(invalid)
- 0.007% fail rate

bugs
- Total 5
- http://mzl.la/1cYZZau (Bugs related to software home button, privacy or ui polish will be skipped for now)

FLAME-KK 2nd Run (focus on dsds and gaia apps) Done
- Test cases http://mzl.la/1aKYIC8

Result
- 342(total), 334(passed), 5(failed), 3(invalid)
- 0.01% fail rate

bugs
- Total 5
- http://mzl.la/1GifEOz

1st Run (2015/03/03)

RTL/V2.2 Flame-KK Done
- Moztrap result - 76 fail, 326 pass, check with Josh for detail.

Full Regression/V2.2 Flame-KK Done
- Moztrap result - 55 fail, 3426 pass, 100% complete

Partial Regression/V2.2 Nexus-5-L Done
- Moztrap result - 30 fail, 1216 pass, 100% complete
- Bug list

Bluetooth PTS/V2.2 Flame-KK Done
- Bug Meta dependency tree - 0 open bugs

Functional Testing

Red

General
- Details:
- 2nd regression, NEXUS-5-L 1119(total), 8(failed) bugs http://mzl.la/1cYZZau
- FLAME-KK 342(total), 5(failed) bugs http://mzl.la/1GifEOz
- NEXUS-5-L MTBF Average: 29.18 hrs
- Flame-L, RD has just received complete code, but building fail.

RTL (Marigold)
- Health Status: Green
- Details: 2 blockers http://mzl.la/1Juz6oH

Cut, Copy, and Paste (Gerry, Edward, Teri)
- Health Status:
- Details: 1 blocker
  - https://bugzilla.mozilla.org/show_bug.cgi?id=1021499

System Platform (Gerry)
- Health Status:
- Details: 7 blockers

OTA/FOTA (Gerry)
- Health Status:
- Details:
  - 2.0/2.1 can’t upgrade to 2.2 via FOTA - Bug 1155035
  - 2.0 can upgrade to 2.2 via OTA with some issues
    - Bluetooth issue - Bug 1132388 & Bug 1134821
    - Bug 1153787 & Bug 1153778
  - 2.1 can’t upgrade to 2.2 via OTA - Bug 1151006

NFC (Alison)
- Health Status: Yellow
- Details: 2 blockers
  - https://bugzilla.mozilla.org/show_bug.cgi?id=1155873
  - https://bugzilla.mozilla.org/show_bug.cgi?id=1156605 (r+, wait to land code)

Ringtones (Alison)
- Health Status: Green
- Details: No blockers

MTP/UMS (Alison)
- Health Status: Green
- Details: No blockers

System Frontend bugs (Hermes)
- Health Status: RED
- Details: 7 blockers, 2 nom

L-porting V5.1 (Hermes)
- Health Status: Green
- Details: no blockers

CAF bugs (Eric)
- Health Status: Red
- Details: 3 new bluetooth bugs opened this week, RDs are working on them, http://mzl.la/1HWrTQa 28 in total

COMMS, RIL bugs (Eric)
- Health Status: Yellow
- Details: 2WIP http://mzl.la/1HWsscJ

MUSIC (Eric)
- Health Status: Yellow
- Details: lots of regression recently http://mzl.la/1HWsSzS
  - dominic kuo/jim porter (cc Hema)

Bluetooth (Eric, Teri)
- Health Status: Yellow
- Details: https://bugzil.la/1143925 Patch in review

Metrics (Shing)
- Health Status: Yellow
- Details: WIP https://bugzil.la/1134998

256M RAM memory optimization for WFVGA, 5MP. (Teri)
- Health Status: Green
- Details:

Exploratory Test

Flame-KK & NEXUS-5-L

268 bugs found
- UNCONFIRMED, NEW, ASSIGNED, REOPENED - 253 bugs found.
- UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - 12 bugs found.
- RESOLVED, VERIFIED, CLOSED - 191 bugs found.
- RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - 19 bugs found.

Regression bugs
- UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - 17 bugs found.
- RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - 256 bugs found.

Stability

NEXUS-5-L 5.1

Environment
- Display DIM
- Rest time with dummy case (30 sec)
- Running tests ** 14 case http://bit.ly/1aWkJyY
- NEXUS *4, ZTE *2

- Result

NEXUS-5-L 5.1
- Current run - NEXUS-5-L 5.1 *4: 38.5 hrs and counting = [55+93(running)+93(running) +67(running)] / (4 devices + 4 crashes) = 38.5
- Current run - ZTE-L 5.1 * 2: Just started
- Previous run - NEXUS-5-L 5.1 *4: MTBF 34.42 hrs = (41.3 + 53.2 + 43.8 + 6.1)/4
- NEXUS-5-L 5.1 Rest time: 10%
- [script issue] OSError: [Errno 24] Too many open files.
- [adb issue] https://bugzil.la/1154629

FLAME-KK 4.4

Environment
- Display DIM
- Rest time with dummy case (30 sec)
- FLAME *9

- Result

FLAME-KK 4.4
- LAME-KK 4.4 * 4: MTBF 25.81 hrs = 50.77/4.96/25.81 - max/min/avg

V2.2 Test Case Creation

On Track

Moztrap link of V2.2 New Features suite.

Test Case Creation WIP

[2.2-feature-qa+] & in-moztrap? (0 bugs) --> WORK DONE
feature-b2g=2.2+ & in-moztrap? (0 bugs) --> WORK DONE

Test Case Creation DONE

[2.2-feature-qa+] & in-moztrap+ (19 bugs) --> WORK DONE
feature-b2g=2.2+ & in-moztrap+ (78 bugs) --> WORK DONE

V2.2 Testing Verification Status

Overall Verification Summary

Need Verification

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1

Verification Done

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1

@@ Line 1: / Line 1: @@
 == V2.2 Highlight & Risk ==
 * Highlight
-** V2.2 FC moved to 2015/04/29
+** V2.2 FC moved to 2015/04/29.
-** NEXUS-5-L 5.1 PVT build MTBF 34.42 hrs
+** NEXUS-5-L 5.1 PVT build MTBF 38.5 hrs & counting.
-** NEXUS-5-L 5.1 Regression test started from 2015/04/14 for 2 wks
+** FLAME-KK 4.4 PVT build MTBF 34.42 hrs.
+** NEXUS-5-L 5.1 Regression test done.
 ** 48 bugs with blocking=2.2+ http://mzl.la/1CDEaqi
-** 22 bugs with blocking=2.2? http://mzl.la/1CDEKV4
+** 12 bugs with blocking=2.2? http://mzl.la/1CDEKV4
 == [[Release_Management/FirefoxOS/2_2_Schedule|V2.2 Milestone]] ==
@@ Line 12: / Line 13: @@
 * <strike>2015/04/06 V2.2 FC</strike>
 * 2015/02/23 V2.2 FL
-* QA Shared calendar
-** You can add it to your Google Calendar view by clicking on this [https://www.google.com/calendar/embed?src=bW96aWxsYS5jb21fMmQzNTMxMzYzMTMzMzUzNTM5MmQzNjMxMzBAcmVzb3VyY2UuY2FsZW5kYXIuZ29vZ2xlLmNvbQ link] and then clicking on the small blue "+ Google Calendar" button at the bottom, right hand of the screen.
-** If you are signed in with your @mozilla.com account, you can also access it this way:
-*** In Google Calendar, you should go to Other calendars > Browse Interesting  Calendars > More > Resources for mozilla.com > Public (9) >  Public - FxOS QA Schedule > Subscribe
 == V2.2 Testing Scope ==
@@ Line 22: / Line 18: @@
 * [[B2G/QA/2.2#V2.2_QA_Owners|B2G V2.2 Overview Page]] for more info.
-* NEW [http://mzl.la/1B95Yi8 blocking-b2g=2.2+] 49 bugs
+* Status NEW: [http://mzl.la/1B95Yi8 blocking-b2g=2.2+] 49 bugs
 <bugzilla>
      {
@@ Line 56: / Line 52: @@
 ==== Regression Testing ====
 == 2nd Run (2015/04/14) ==
-* NEXUS-5-L (80% done)
+* '''NEXUS-5-L 2nd Run (focus on essential functionality)'''  <span style="color:#ffffff; background:#008000"> Done </span>
-** http://mzl.la/1D76gXG
+** Test cases  http://mzl.la/1D76gXG
-* Result
-* 1076(total), 1049(passed), 6(failed), 21(invalid)
+* '''Result'''
-* 0.005% fail rate
+** 1119(total), 1090(passed), 8(failed), 21(invalid)
+** 0.007% fail rate
+* '''bugs'''
+** Total 5
+** http://mzl.la/1cYZZau (Bugs related to software home button, privacy or ui polish will be skipped for now)
+* '''FLAME-KK 2nd Run (focus on dsds and gaia apps)'''  <span style="color:#ffffff; background:#008000"> Done </span>
+** Test cases  http://mzl.la/1aKYIC8
-* FLAME-KK (done)
+* '''Result'''
-** http://mzl.la/1aKYIC8
+** 342(total), 334(passed), 5(failed), 3(invalid)
-* Result
+** 0.01% fail rate
-* 342(total), 334(passed), 5(failed), 3(invalid)
-* 0.01% fail rate
+* '''bugs'''
+** Total 5
+** http://mzl.la/1GifEOz
 == 1st Run (2015/03/03) ==
-* RTL/V2.2 Flame-KK <span style="color:#000000; background:#FFFF00"> Done </span>
+* RTL/V2.2 Flame-KK <span style="color:#ffffff; background:#008000"> Done </span>
 ** [https://moztrap.mozilla.org/results/runs/?&pagenumber=1&pagesize=20&sortfield=start&sortdirection=desc&filter-status=active&filter-name=rtl+full+test Moztrap result] - [https://moztrap.mozilla.org/results/cases/?&pagenumber=1&pagesize=100&sortfield=created_on&sortdirection=desc&filter-resultstatus=failed&filter-run=6626 76 fail], 326 pass, check with Josh for detail.
-* Full Regression/V2.2 Flame-KK <span style="color:#000000; background:#FFFF00"> Done </span>
+* Full Regression/V2.2 Flame-KK <span style="color:#ffffff; background:#008000"> Done </span>
 ** [http://mzl.la/1wtR60Y Moztrap result] - [https://moztrap.mozilla.org/results/cases/?&pagenumber=1&pagesize=100&sortfield=created_on&sortdirection=desc&filter-resultstatus=failed&filter-run=6551 55 fail], 3426 pass, 100% complete
-* Partial Regression/V2.2 Nexus-5-L <span style="color:#000000; background:#FFFF00"> Done </span>
+* Partial Regression/V2.2 Nexus-5-L <span style="color:#ffffff; background:#008000"> Done </span>
 ** [http://mzl.la/1EXfMQQ Moztrap result] - [https://moztrap.mozilla.org/results/cases/?&pagenumber=1&pagesize=100&sortfield=created_on&sortdirection=desc&filter-resultstatus=failed&filter-run=6552 30 fail], 1216 pass, 100% complete
 ** [http://mzl.la/18T1AMy Bug list]
-* Bluetooth PTS/V2.2 Flame-KK <span style="color:#000000; background:#FFFF00"> Done </span>
+* Bluetooth PTS/V2.2 Flame-KK <span style="color:#ffffff; background:#008000"> Done </span>
 ** [https://bugzilla.mozilla.org/showdependencytree.cgi?id=1139349&hide_resolved=1 Bug Meta dependency tree] - 0 open bugs
@@ Line 94: / Line 100: @@
 * RTL (Marigold)
 ** Health Status: Green
-** Details: 3 blockers http://mzl.la/1Juz6oH
+** Details: 2 blockers http://mzl.la/1Juz6oH
 * Cut, Copy, and Paste  (Gerry, Edward, Teri)
@@ Line 163: / Line 169: @@
 ==== Exploratory Test ====
 '''Flame-KK & NEXUS-5-L'''
-* [http://mzl.la/1GHbwWU 240 bugs found]
+* [http://mzl.la/1GHbwWU 268 bugs found]
-** UNCONFIRMED, NEW, ASSIGNED, REOPENED - [http://mzl.la/1IB4Ith 224 bugs found].
+** UNCONFIRMED, NEW, ASSIGNED, REOPENED - [http://mzl.la/1IB4Ith 253 bugs found].
-** UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - [http://mzl.la/1ATnw3C 14 bugs found].
+** UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - [http://mzl.la/1ATnw3C 12 bugs found].
-** RESOLVED, VERIFIED, CLOSED - [http://mzl.la/17KvaU8 184 bugs found].
+** RESOLVED, VERIFIED, CLOSED - [http://mzl.la/17KvaU8 191 bugs found].
-** RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - [http://mzl.la/14Dhet5 16 bugs found].
+** RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - [http://mzl.la/14Dhet5 19 bugs found].
 * Regression bugs
-** UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - [http://mzl.la/1Afjcx9 15 bugs found].
+** UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - [http://mzl.la/1Afjcx9 17 bugs found].
-** RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - [http://mzl.la/1GHbDC1 '''249''' bugs found].
+** RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - [http://mzl.la/1GHbDC1 '''256''' bugs found].
 ==== Stability ====
 '''NEXUS-5-L 5.1'''
-*Environment
+* Environment
 ** Display DIM
 ** Rest time with dummy case (30 sec)
-** Running tests
+** Running tests ** 14 case http://bit.ly/1aWkJyY
-** 14 case http://bit.ly/1aWkJyY
+** NEXUS *4, ZTE *2
-** NEXUS *3
 - Result
 * NEXUS-5-L 5.1
-** NEXUS-5-L 5.1 * 4: MTBF 34.42 hrs = (41.3 + 53.2 + 43.8 + 6.1)/4
+** Current run -  NEXUS-5-L 5.1 *4: '''38.5 hrs and counting''' = [55+93(running)+93(running) +67(running)] / (4 devices + 4 crashes) = 38.5
+** Current run -  ZTE-L 5.1 * 2: Just started
+** Previous run - NEXUS-5-L 5.1 *4: MTBF 34.42 hrs = (41.3 + 53.2 + 43.8 + 6.1)/4
 ** NEXUS-5-L 5.1 Rest time: 10%
 ** [script issue] OSError: [Errno 24] Too many open files.
 ** [adb issue] https://bugzil.la/1154629
+'''FLAME-KK 4.4'''
+* Environment
+** Display DIM
+** Rest time with dummy case (30 sec)
+** FLAME *9
+- Result
+* FLAME-KK 4.4
+** LAME-KK 4.4  * 4: MTBF 25.81 hrs = 50.77/4.96/25.81 - max/min/avg
 == V2.2 Test Case Creation ==

B2G/QA/2.2/2015-04-27Status: Difference between revisions

Latest revision as of 14:56, 27 April 2015

Contents

V2.2 Highlight & Risk

V2.2 Milestone

V2.2 Testing Scope

Bugzilla query error

V2.2 Testing by Type

Smoke Testing

Regression Testing

2nd Run (2015/04/14)

1st Run (2015/03/03)

Functional Testing

Exploratory Test

Stability

V2.2 Test Case Creation

Test Case Creation WIP

Test Case Creation DONE

V2.2 Testing Verification Status

Overall Verification Summary

Bugzilla query error

Bugzilla query error

Navigation menu

B2G/QA/2.2/2015-04-27Status: Difference between revisions

Latest revision as of 14:56, 27 April 2015

V2.2 Highlight & Risk

V2.2 Milestone

V2.2 Testing Scope

Bugzilla query error

V2.2 Testing by Type

Smoke Testing

Regression Testing

2nd Run (2015/04/14)

1st Run (2015/03/03)

Functional Testing

Exploratory Test

Stability

V2.2 Test Case Creation

Test Case Creation WIP

Test Case Creation DONE

V2.2 Testing Verification Status

Overall Verification Summary

Bugzilla query error

Bugzilla query error

Navigation menu

Search