B2G/QA/2.2/2015-06-15Status

V2.2 Highlight & Risk

• Highlight
  • NEXUS-5-L 5.1 PVT build MTBF 104.81 hrs
  • FLAME-KK 4.4 PVT build MTBF 21 hrs
  • 4 bugs with blocking=2.2+ http://mzl.la/1CDEaqi
  • 17 bugs with blocking=2.2? http://mzl.la/1CDEKV4
  • 5 bugs with CAF-v2.2-metabug http://mzl.la/1ELeOXg
  • FLAME-L 3.0 local build is ready, bugs here http://mzl.la/1KPpOoc.

V2.2 Milestone

• 2015/06/08 V2.2 CC
• 2015/04/29 V2.2 Updated FC
• 2015/02/23 V2.2 FL

V2.2 Testing Scope

• Focus on bugs in agreed list.
• B2G V2.2 Overview Page for more info.

• Status NEW: blocking-b2g=2.2+ 4 bugs

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1

V2.2 Testing by Type

Smoke Testing

Nexus-5-L 5.1 Green

• Testing on PVT build from 2015/04/14
• Manual test results - are available on Google Drive
  • Green for 3+ weeks
• Known issues
  • No FM radio support, no DSDS support and no UMS support.
  • OTA not working, but the update channel is fine for 2.2->2.2.

Flame-KK Green

• Manual test results - are available on Google Drive
• Automation reports - are available on Github.IO & fxosqa-report-watchlist.
  • Green for 3+ weeks
• Known issues
  • Bug 1165195 - [2.1][2.2][Flame][Settings][OTA] Can't detect update.

3rd Run (2015/05/19)

• Exploratory tests

2nd Run (2015/04/14)

• NEXUS-5-L 2nd Run (focus on essential functionality) Done
  • Test cases http://mzl.la/1D76gXG

• Result
  • 1119(total), 1090(passed), 8(failed), 21(invalid)
  • 0.007% fail rate

• bugs
  • Total 5
  • http://mzl.la/1cYZZau (Bugs related to software home button, privacy or ui polish will be skipped for now)

• FLAME-KK 2nd Run (focus on dsds and gaia apps) Done
  • ​Test cases http://mzl.la/1aKYIC8

• Result
  • ​342(total), 334(passed), 5(failed), 3(invalid)
  • ​0.01% fail rate

• bugs
  • ​Total 5
  • ​http://mzl.la/1GifEOz

1st Run (2015/03/03)

• RTL/V2.2 Flame-KK Done
  • Moztrap result - 76 fail, 326 pass, check with Josh for detail.

• Full Regression/V2.2 Flame-KK Done
  • Moztrap result - 55 fail, 3426 pass, 100% complete

• Partial Regression/V2.2 Nexus-5-L Done
  • Moztrap result - 30 fail, 1216 pass, 100% complete
  • Bug list

• Bluetooth PTS/V2.2 Flame-KK Done
  • Bug Meta dependency tree - 0 open bugs

Functional Testing

• General
  • Health Status: Yellow
  • Details: 2.2+ http://mzl.la/1CDEaqi 13 bugs

• RTL (Josh)
  • Health Status: Green
  • Details: 0 blocker

• Cut, Copy, and Paste (Gerry, Edward, Teri)
  • Health Status: Green
  • Details: 0 blocker

• System Platform (Gerry)
  • Health Status:
  • Details: 1 blocker
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1161364 - review+

• NFC (Alison)
  • Health Status: Green
  • Details: 0 blocker

• MTP/UMS (Alison)
  • Health Status: Green
  • Details: 0 blocker

• System Frontend bugs (Hermes)
  • Health Status: Green
  • Details: 0 blocker

• CAF bugs (Eric)
  • Health Status: Yellow
  • Details: http://mzl.la/1ELeOXg 5 Blockers

• COMMS, RIL bugs (Eric)
  • Health Status: Yellow
  • Details: http://mzl.la/1Jk9tuH 1 Blocker

• Bluetooth (Eric, Teri)
  • Health Status: Green
  • Details: 0 blocker

• Productivity (Teri)
  • Health Status: Green
  • Details: 0 blocker

• Metrics (Shing)
  • Health Status: Green
  • Details: 0 blocker

• Video (eric)
  • Health Status: Green
  • Details: 0 blocker

• Music (eric)
  • Health Status: Green
  • Details: 0 blocker

• Gallery (eric)
  • Health Status: Green
  • Details: 0 blocker

Exploratory Test

Flame-KK & NEXUS-5-L

• 321 bugs found
  • UNCONFIRMED, NEW, ASSIGNED, REOPENED - 296 bugs found.
  • UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - 2 bugs found.
  • RESOLVED, VERIFIED, CLOSED - 242 bugs found.
  • RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - 21 bugs found.

• Regression bugs
  • UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - 2 bugs found.
  • RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - 292 bugs found.

Stability

NEXUS-5-L 5.1

• Environment
  • Display DIM
  • Rest time with dummy case (30 sec)
  • Running tests ** 14 case http://bit.ly/1aWkJyY
  • NEXUS *5 with patch for [AudioSystem issue] https://bugzil.la/1155490

- Result

• NEXUS-5-L 5.1
  • Current run - NEXUS-5-L 5.1 * 4: 104.81 hrs

V2.2 Test Case Creation

On Track

• Moztrap link of V2.2 New Features suite.

Test Case Creation WIP

• [2.2-feature-qa+] & in-moztrap? (0 bugs) --> WORK DONE
• feature-b2g=2.2+ & in-moztrap? (0 bugs) --> WORK DONE

Test Case Creation DONE

• [2.2-feature-qa+] & in-moztrap+ (19 bugs) --> WORK DONE
• feature-b2g=2.2+ & in-moztrap+ (78 bugs) --> WORK DONE

V2.2 Testing Verification Status

Overall Verification Summary

Need Verification

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1

Verification Done

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1