B2G/QA/Gaia Test Plan/Crash

General Information

Area	Details
Team:	Developers: Designer: QA: Naoki Hirata, Marcia Knous

Test Plan

Design Documentation/Background Research

• Design Documentation
  • can be found : http://people.mozilla.com/~lco/Crash_Reporting_B2G/R1_Crash%20Reports%20v1.pdf

How to force a crash

• Plugin Crash
  • adb shell ps|grep plugin-container
  • app_0 409 109 55512 16760 ffffffff 40064330 S /system/b2g/plugin-container
  • adb shell kill -11 409
• b2g OS crash
  • adb shell ps |grep b2g
  • root 109 1 191120 66024 ffffffff 400fa330 S /system/b2g/b2g
  • adb shell kill -11 109
• browser tab crash
  • go to http://www.crashmybrowser.com/

How to check crash:

• checking crash:
• root@android:/data/b2g/mozilla/Crash Reports/
• /data/b2g/mozilla/Crash Reports/pending folder.

Kairo's reports:

• https://crash-analysis.mozilla.com/rkaiser/0000.overview.html#latest
• https://crash-analysis.mozilla.com/rkaiser/2012-11-28/2012-11-28.b2g.crashes.html

Socorro

• https://crash-stats.mozilla.com/products/B2G
• https://wiki.mozilla.org/CrashKill
• https://wiki.mozilla.org/Socorro

Mobile OS

• B2G - Firefox OS 18

Mobile Device

Test devices :

• Leo
• Inari

Backup test devices:

• Otoro
• Unagi

Strategy

• Exploratory Manual testing using outline; see Test Outline section
• Regression testing based on Test Cases
• Eventual automation testing

Signoff Criteria

• All tef+ blockers are closed
• All basecamp blockers are closed
• All smoke tests and basic functional tests are ran without finding any basecamp blockers

Test Cases and Results

• Test cases will be tracked in MozTrap under "crash reporting"

Bugs

• bug 761905 - (b2g-crash-reporting) Tracking: Crash reporting for b2g (gonk)
  • Implementation Meta Tracker
• bug 833574 - (b2g-frontend-crash-reporting) [Meta] Other Crash Reporting Bugs
  • Bugs after implementation Meta Tracker

Test Outline

Note : This test out line is a brain storm of all possible conditions to consider for testing. The scope of the testing will be narrowed from this list based on design information, risks of breakage, etc.

• collaborative effort by marcia and nhirata
• Test Case mapping

Positive Smoke/Alpha Tests

UI

• General
  • FTE Crash settings
    • Send
    • Don't send
  • OS Crash
    • Confirm System Crash Message
    • Confirm "Always send" is checked by default
    • Confirm "Don't Send" is inactive when "Always Send" is checked
    • Confirm What's in a Crash Report link
    • Subsequent OS crashes
  • App Crash
    • Confirm System Crash Message
    • Confirm "Always send" is checked by default
    • Confirm "Don't Send" is inactive when "Always Send" is checked
    • Confirm What's in a Crash Report link
    • Subsequent App crashes
  • Settings - send versus do not send
    • Pref on "Never send a report" in the Settings Preferences
    • Pref on "Always send a report" in the Settings Preferences
    • Pref on "Ask me when a crash occurs" in the Settings Preferences
  • Content Crash
    • Browser content crash
• about:Crashes

Functional

• UI Based Functionality
  • FTE Crash settings
    • Send
    • Don't send
  • OS Crash
    • Confirm System Crash Message
    • Confirm "Always send" is checked by default
    • Confirm "Don't Send" is inactive when "Always Send" is checked
    • Confirm What's in a Crash Report link
    • Subsequent OS crashes
  • App Crash
    • Confirm System Crash Message
    • Confirm "Always send" is checked by default
    • Confirm "Don't Send" is inactive when "Always Send" is checked
    • Confirm What's in a Crash Report link
    • Subsequent App crashes
  • Settings - send versus do not send
    • Pref on "Never send a report" in the Settings Preferences
    • Pref on "Always send a report" in the Settings Preferences
    • Pref on "Ask me when a crash occurs" in the Settings Preferences
  • Content Crash
    • Browser content crash

Beta Conditional

• Virtual Conditions

• Content Conditions
  • check Pending/Sent in /data/b2g/mozilla/Crash Reports/

• Physical Conditions
  • While Charged
  • While not Charged
  • Power Consumption
  • Performance
    • Speed of functions
    • Processor utilization
    • Memory utilization
      • Memory leaks?
      • Memory efficiency
  • Storage
    • Disk
      • near full SD card and crash
      • full SD card and crash

• Boundary testing
  • How many submissions can be sent at a time?
  • Sending of pending submissions

• Negative (Tests to Break) Tests

• Network Conditions
  • WiFi
    • Send only through wifi
    • Secured Wifi
      • WPA
      • Portal
    • Open Wifi
  • Mobile Data
    • 3G
    • 4G?
  • No connections

• Race Conditions
  • Battery
    • Battery Low
    • No Battery
  • Calls
    • Incoming Phone Call

• Device/Platform Specific Issues
  • Portrait/Landscape Orientation
  • Speaker(sound and crash)
  • Phone call and crash

• Localization
  • Language/Dialog check
  • Date Format (Euro/Japanese/Chinese)
  • Time Format (Chinese/Japanese)
  • Change of IME on Mobile OS

• Help?

Usability

• Usability
• Accessibility (508 compliance)