B2G/QA/2.2/2015-05-04Status

From MozillaWiki
< B2G‎ | QA‎ | 2.2
Jump to navigation Jump to search

V2.2 Highlight & Risk

V2.2 Milestone

  • 2015/06/08 V2.2 CC
  • 2015/04/29 V2.2 Updated FC
  • 2015/04/06 V2.2 FC
  • 2015/02/23 V2.2 FL

V2.2 Testing Scope

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


V2.2 Testing by Type

Smoke Testing

Nexus-5-L 5.1 Yellow

  • Testing on PVT build from 2015/04/14
  • Cases blocking manual test
    • OTA not working, but the update channel is fine for 2.2->2.2.
  • Manual - are available on Google Drive
  • Known issues - no FM radio support, no DSDS support and no UMS support.

Flame-KK Green

  • No bug found blocking manual smoketests but we have couple automation bugs need to investigate, and the removal of 'Mozilla Mobile' has huge impact on our testing framework, we are currently trying to resolve this.
  • New Bugs Breaking the Automation smoketests
    • [Investigating] test_clock_create_new_alarm.py - Bug 1138278- [Clock] After running test_clock_create_new_alarm.py, it should set the alarm time to 1 min more than the current time
    • [Investigating] [Not reproduced manually] test_ftu_skip_tour.py - Bug 1138278 - test_ftu_skip_tour should reflect new Send Data initial state and separate SIM- and network-available conditions
    • [Investigating] test_marketplace_packaged_app.py - Bug 1151172 - [marketplace-tests-gaia] "test_marketplace_search_and_install_app" is failing on b2g v2.2
    • [Not reproduced] test_sms_with_attachments.py - Bug 1151696 - [v2.2] Investigate test_sms_with_attachments.py failure . (Cannot reproduce locally)
    • [Not reproduced manually] test_music_artist_mp3.py - Bug 1149886 - using to replace at Music app results in unable to locate <span.list-song-index> error

Regression Testing

2nd Run (2015/04/14)

  • Result
    • 1119(total), 1090(passed), 8(failed), 21(invalid)
    • 0.007% fail rate
  • bugs
    • Total 5
    • http://mzl.la/1cYZZau (Bugs related to software home button, privacy or ui polish will be skipped for now)
  • Result
    • ​342(total), 334(passed), 5(failed), 3(invalid)
    • ​0.01% fail rate

1st Run (2015/03/03)

Functional Testing

Red

  • General
    • Details:
    • NEXUS-5-L, FLAME-KK - will start another exploratory test from next week in TPE and in China
    • FLAME-L - RD needs time to make it work, currently it fails to boot into home screen.
    • ZTE 8909 - just for reference, DSDS, FM okay, SD not working, running MTBF now.
  • Cut, Copy, and Paste (Gerry, Edward, Teri)
    • Health Status:
    • Details: 1 blocker
  • System Platform (Gerry)
    • Health Status:
    • Details: 4 blockers
  • OTA/FOTA (Gerry)
    • Health Status:
    • Details:
      • 2.0/2.1 can’t upgrade to 2.2 via FOTA - Bug 1155035 - patch ready
      • (Don't support) 2.0 can upgrade to 2.2 via OTA with some issues
        • Bluetooth issue (resolved) - Bug 1132388 & Bug 1134821
        • Bug 1153787 & Bug 1153778
      • (Don't support) 2.1 can’t upgrade to 2.2 via OTA - Bug 1151006
      • Best way to do FOTA - comment #55 in BZ1136638
  • NFC (Alison)
    • Health Status:
    • Details:
      • 2 blockers:

https://bugzilla.mozilla.org/show_bug.cgi?id=1155873 (patch r+, wait to land code) https://bugzilla.mozilla.org/show_bug.cgi?id=1156605 (RESOLVED FIXED, wait uplift to v2.2)

      • 2 nominators:

https://bugzilla.mozilla.org/show_bug.cgi?id=1158379 https://bugzilla.mozilla.org/show_bug.cgi?id=1158388


Exploratory Test

Flame-KK & NEXUS-5-L

  • Regression bugs

Stability

NEXUS-5-L 5.1

  • Environment
    • Display DIM
    • Rest time with dummy case (30 sec)
    • Running tests ** 14 case http://bit.ly/1aWkJyY
    • NEXUS *4, ZTE *2

- Result

  • NEXUS-5-L 5.1
    • Current run - NEXUS-5-L 5.1 *4: 13.77 hrs = [17.94+17.39+10.03+51.05] / (4 devices + 3 crashes) = 13.77
    • Current run - ZTE-L 5.1 * 2: Just started
    • Previous run - NEXUS-5-L 5.1 *4: MTBF 34.42 hrs = (41.3 + 53.2 + 43.8 + 6.1)/4
    • NEXUS-5-L 5.1 Rest time: 10%
    • [AudioSystem issue] https://bugzil.la/1155490
    • [adb issue] https://bugzil.la/1154629


FLAME-KK 4.4

  • Environment
    • Display DIM
    • Rest time with dummy case (30 sec)
    • FLAME *9

- Result

  • FLAME-KK 4.4
    • FLAME-KK 4.4 * 4: MTBF 34.42 hrs

V2.2 Test Case Creation

On Track

Test Case Creation WIP

Test Case Creation DONE

  • [2.2-feature-qa+] & in-moztrap+ (19 bugs) --> WORK DONE
  • feature-b2g=2.2+ & in-moztrap+ (78 bugs) --> WORK DONE

V2.2 Testing Verification Status

Overall Verification Summary

Need Verification

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Verification Done

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1

Retrieved from "https://wiki.mozilla.org/index.php?title=B2G/QA/2.2/2015-05-04Status&oldid=1071938"