QA/Client Crash Id: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
 
(17 intermediate revisions by 2 users not shown)
Line 9: Line 9:
! Date !! Version !! Author !! Description  
! Date !! Version !! Author !! Description  
|-
|-
| 02/24/2017 || 1.0 || Abe Masresha || Created first draft
| 02/27/2017 || 1.0 || Abe Masresha || Created first draft
|-
|-
|- || ||   ||   
| 05/03/2017 || 1.0 || Petruta Rasa  ||  Added tests and results
|-
|-
|}
|}
Line 17: Line 17:
= Overview =
= Overview =
== Purpose ==
== Purpose ==
The goal of this feature is to align the breakpad-generated crash id with a crash report.
The goal of this feature is to align the break pad-generated crash id with a crash report.


== Scope ==
== Scope ==
* Testing will focus on:  
* Testing will focus on:  


::: Stability of the browser with this work.
::: Functionality and stability of Crash Reporter Client.
::: Functionality and stability of Crash Reporter Client.
::: Validating the crash report.
::: Comparing contents of pings from ping sender and restart.
::: Validating the content crash ping.
::: Comparing values of minidumpSha256Hash of ping sender ping and restart ping
::: Comparing browser crash pings from the minidump file and ping received after a restart.
::: Comparing SHA256 hash of the minidump file and value of minidumpSha256Hash from one of the crash pings.
::: Comparing content crash pings from the minidump file and ping received after a restart.
::: Comparing SHA256 hash of the minidump file and value of minidumpSha256Hash from the Content crash ping.
::: Stability of the browser with this work.


* How testing will be performed:
* How testing will be performed:
Line 47: Line 47:
[mailto:gwimberly@softvision.com Grover Wimberly IV - Tester]<br />
[mailto:gwimberly@softvision.com Grover Wimberly IV - Tester]<br />
[mailto:kkumari@softvision.com Kanchan Kumari - Tester]<br />
[mailto:kkumari@softvision.com Kanchan Kumari - Tester]<br />
[mailto:petruta.rasa@softvision.com Petruta Rasa - Tester]<br />


= Testing summary =  
= Testing summary =  
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
This feature will be tested manually on Windows, Mac, Linux operating system and Android devices.
This feature will be tested manually on Windows, Mac and Linux operating system.


=== Out of Scope ===
=== Out of Scope ===
Line 68: Line 69:


=== Build ===
=== Build ===
Firefox Nightly 54.0a1
Firefox Nightly 55.0a1


= Test Strategy =  
= Test Strategy =  
== Test Objectives ==
== Test Objectives ==
Verify Firefox is stable with this feature<br>
Verify Firefox is stable with this feature<br>
Verify SHA256 hashed minidump file and the crash ping after restart are identical.<br />
Verify two crash pings are received. One from the ping sender (first ping) and the other after the browser restarts (second ping).
Verify the hashed minidump file and the restart crash ping contain minidumpSha256Hash field in the payload element.<br />
Verify the first and the second ping contain minidumpSha256Hash field in the payload element.<br />
Verify the hashed minidump and restart ping have identical contents for the minidumpSha256Hash field<br>
Verify the first and the second ping have identical value for the minidumpSha256Hash field<br>
Verify the hashed minidump file and the restart crash ping contain minidumpSha256Hash field in the payload element for Content Process Crash. <br />
Verify SHA256 hash of the minidump file and the value of minidumpSha256Hash in crash ping are identical.<br />
Verify the hashed minidump and restart ping have identical contents for the minidumpSha256Hash field for Content Process Crash.<br>
Verify content crash has only one ping<br/>
Verify value of minidumpSha256Hash of the content crash is identical to the hash of the minidump file<br>
Verify Firefox restarts successfully after crash<br>
Verify Firefox restarts successfully after crash<br>
Verify crash reporter client is responsive and functional after or before the first ping<br>
Verify crash reporter client is responsive and functional after or before the first ping<br>
Line 113: Line 115:
| Test case creation || [https://docs.google.com/a/mozilla.com/spreadsheets/d/1O9cf70gxWkmmHTW3FIY_u8C-tc8LuAwDU0ai2TSdm64/edit?usp=sharing Test Cases GDoc]
| Test case creation || [https://docs.google.com/a/mozilla.com/spreadsheets/d/1O9cf70gxWkmmHTW3FIY_u8C-tc8LuAwDU0ai2TSdm64/edit?usp=sharing Test Cases GDoc]
|-
|-
| Test case execution || [Test Cases TestRail]
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/suites/view/925&group_by=cases:section_id&group_order=asc]
|-
|-
| Bugs management || Bugzilla
| Bugs management || Bugzilla
Line 122: Line 124:
== Overview ==
== Overview ==
= References =
= References =
Test cases: [ TestRail] or [https://docs.google.com/a/mozilla.com/spreadsheets/d/1O9cf70gxWkmmHTW3FIY_u8C-tc8LuAwDU0ai2TSdm64/edit?usp=sharing Test Cases GDoc]
Test cases: [https://testrail.stage.mozaws.net/index.php?/suites/view/925&group_by=cases:section_id&group_order=asc Test Cases TestRail] or [https://docs.google.com/a/mozilla.com/spreadsheets/d/1O9cf70gxWkmmHTW3FIY_u8C-tc8LuAwDU0ai2TSdm64/edit?usp=sharing Test Cases GDoc]
            
            
Bugs:  [https://bugzilla.mozilla.org/show_bug.cgi?id=1322611  Bug 1322611]<br />
Bugs:  [https://bugzilla.mozilla.org/show_bug.cgi?id=1322611  Bug 1322611]<br />
Line 157: Line 159:
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" | Yes || feature will send data to a telemetry server and this functionality will be tested
|style="text-align:center;" | Yes || Feature will send data to a telemetry server and this functionality will be tested
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
Line 179: Line 181:
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
|style="text-align:center;" | Yes ||  hashed minidump file and restart ping contents will be compared
|style="text-align:center;" | Yes ||  Hashed minidump file and restart ping contents will be compared
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
Line 186: Line 188:


== Test suite ==
== Test suite ==
  Full Test suite - [ Test Cases TestRail]/ [https://docs.google.com/a/mozilla.com/spreadsheets/d/1O9cf70gxWkmmHTW3FIY_u8C-tc8LuAwDU0ai2TSdm64/edit?usp=sharing  Test Cases GDoc]<br />
  Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/925&group_by=cases:section_id&group_order=asc  Test Cases TestRail]/ [https://docs.google.com/a/mozilla.com/spreadsheets/d/1O9cf70gxWkmmHTW3FIY_u8C-tc8LuAwDU0ai2TSdm64/edit?usp=sharing  Test Cases GDoc]<br />


= Bug Work =
= Bug Work =
Tracking bugs – [https://bugzilla.mozilla.org/show_bug.cgi?id=1310703   bug]<br />
Tracking bugs – [https://bugzilla.mozilla.org/show_bug.cgi?id=1322611   bug]<br />


====== Bugs ======
====== Bugs ======
<!--
{| class="wikitable" style="width:50%"
{| class="wikitable" style="width:50%"
|-
|-
Line 202: Line 205:


|}
|}
-->


= Sign off =
= Sign off =
Line 226: Line 230:
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" | {{Inprogress|}}
|style="text-align:center;" | {{mdone|}}
|  
|  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" | {{Inprogress|}}
| style="text-align:center;" | {{mdone|}}
|  
|  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" | {{Inprogress|}}
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" | [Test Cases TestRail]/ [https://docs.google.com/a/mozilla.com/spreadsheets/d/1O9cf70gxWkmmHTW3FIY_u8C-tc8LuAwDU0ai2TSdm64/edit?usp=sharing Test Cases GDoc]
 
|style="text-align:center;" | [https://testrail.stage.mozaws.net/index.php?/suites/view/925&group_by=cases:section_id&group_order=asc Test Cases TestRail]/ [https://docs.google.com/a/mozilla.com/spreadsheets/d/1O9cf70gxWkmmHTW3FIY_u8C-tc8LuAwDU0ai2TSdm64/edit?usp=sharing Test Cases GDoc]
|-
|-
|  Full Functional Tests Execution  
|  Full Functional Tests Execution  
Line 242: Line 247:
|-
|-
|  Automation Coverage ||
|  Automation Coverage ||
|style="text-align:center;" | will covered by developer if needed
|style="text-align:center;" | Will be covered by developer if needed
|-
|-
|  Performance Testing ||
|  Performance Testing ||
|style="text-align:center;" |  will covered by developer if needed
|style="text-align:center;" |  Will be covered by developer if needed
|-
|-
|  Load Testing ||
|  Load Testing ||
|style="text-align:center;" |  will covered by developer if needed
|style="text-align:center;" |  Will be covered by developer if needed
|-
|-
|  All Defects Logged || ||  
|  All Defects Logged || ||  
Line 257: Line 262:
|style="text-align:center;" |  
|style="text-align:center;" |  
|-
|-
|  QA Signoff  - Nightly Release||  
|  QA Signoff  - mid-Nightly || {{mdone|}} 
|style="text-align:center;" |  
|style="text-align:center;" |  
|-
|-
|  QA Aurora - Full Testing
|  QA Signoff  - Nightly Release || {{mdone|}} 
|style="text-align:center;" |  ||
|-
|  QA Signoff  - Aurora Release ||  
|style="text-align:center;"|
|style="text-align:center;"|
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing || {{mdone|}}
|style="text-align:center;" | ||
|style="text-align:center;" |  
|-
|-
|  QA Signoff  - Beta Release ||  
|  QA Signoff  - Beta Release || {{mdone|}} 
|style="text-align:center;" |   
|style="text-align:center;" |   
|}
|}

Latest revision as of 15:29, 17 July 2017

QA/Client Crash Id


Revision History


Date Version Author Description
02/27/2017 1.0 Abe Masresha Created first draft
05/03/2017 1.0 Petruta Rasa Added tests and results

Overview

Purpose

The goal of this feature is to align the break pad-generated crash id with a crash report.

Scope

  • Testing will focus on:
Stability of the browser with this work.
Functionality and stability of Crash Reporter Client.
Comparing contents of pings from ping sender and restart.
Comparing values of minidumpSha256Hash of ping sender ping and restart ping
Comparing SHA256 hash of the minidump file and value of minidumpSha256Hash from one of the crash pings.
Comparing SHA256 hash of the minidump file and value of minidumpSha256Hash from the Content crash ping.
  • How testing will be performed:
Manual

Risk analysis

This work includes modifications to both the crash reporter client and the code that handles content process crashes. So the risk will mainly base on those two:

  • First, it may break the crash reporter client in a way that prevents the user from sending crash reports or causes submitted crash reports to be malformed
  • Second, it may not send crash pings for the content crash, or send malformed crash pings.

Ownership

Mozilla Development

Gabriele Svelto

QA Eng. Softvision Las Vegas
Stefan Georgiev - Team Lead
Abe Masresha - Test Lead
Justin Williams - Tester
Grover Wimberly IV - Tester
Kanchan Kumari - Tester
Petruta Rasa - Tester

Testing summary

Scope of Testing

In Scope

This feature will be tested manually on Windows, Mac and Linux operating system.

Out of Scope

Automation will be handled by the developer (if needed)
iOS devices will not be tested

Requirements for testing

Environments: Operating Systems

Windows 10 64bits
Windows 7 32 bits
Ubuntu 16.04 32 and 64bits
MacOSX 10.12
Android devices

Firefox Channel and environment setups

Build

Firefox Nightly 55.0a1

Test Strategy

Test Objectives

Verify Firefox is stable with this feature
Verify two crash pings are received. One from the ping sender (first ping) and the other after the browser restarts (second ping). Verify the first and the second ping contain minidumpSha256Hash field in the payload element.
Verify the first and the second ping have identical value for the minidumpSha256Hash field
Verify SHA256 hash of the minidump file and the value of minidumpSha256Hash in crash ping are identical.
Verify content crash has only one ping
Verify value of minidumpSha256Hash of the content crash is identical to the hash of the minidump file
Verify Firefox restarts successfully after crash
Verify crash reporter client is responsive and functional after or before the first ping

Test Execution Schedule

Project phase Start Date End Date
Start project 02/24/2017 TBD
Study documentation/specs received from developers 02/24/2017 02/16/2017
QA - Test plan creation 02/27/2017 TBD
QA - Test cases/Env preparation 02/27/2017 TBD
QA - Nightly TBD TBD

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation Test Cases GDoc
Test case execution [1]
Bugs management Bugzilla

Status

Overview

References

Test cases: Test Cases TestRail or Test Cases GDoc

Bugs: Bug 1322611

Testcases

Overview

Summary of testing scenarios

Test Areas

Test Areas Covered Details
Private Window Yes Few tests will be made on private window as well
Non e10s Window Yes Few tests will be made with non e10s window as well
Install/Upgrade
Feature upgrades/downgrades data as expected Yes Feature will be tested against browser updates
Data Monitoring
Temporary or permanent telemetry monitoring Yes Data from Local telemetry server will be monitored
Telemetry correctness testing Yes Data correctness will be tested.
Server integration testing Yes Feature will send data to a telemetry server and this functionality will be tested
Offline and server failure testing Yes Server offline option will be included in testing
Add-ons
Testing with existing/popular add-ons Yes feature functionality will be tested with popular addons
Web Compatibility
Survey of popular sites for compatibility Yes Firefox will be tested for stability
Interoperability
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. Yes Hashed minidump file and restart ping contents will be compared
Interaction of this feature with other browser features Yes Firefox will be tested for stability across popular sites.

Test suite

Full Test suite - Test Cases TestRail/ Test Cases GDoc

Bug Work

Tracking bugs – bug

Bugs

Sign off

Criteria

Check list

  • All test cases should be executed
  • All blocker and critical bugs must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/developer/QA)

Results

Feature testing

  • Links for the tests run
    • Full Test suite: [ Test Cases runs TestRail]
    • Regression Test suite: [link] if needed/available

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE] Test Cases TestRail/ Test Cases GDoc
Full Functional Tests Execution
Automation Coverage Will be covered by developer if needed
Performance Testing Will be covered by developer if needed
Load Testing Will be covered by developer if needed
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - mid-Nightly [DONE]
QA Signoff - Nightly Release [DONE]
QA Beta - Full Testing [DONE]
QA Signoff - Beta Release [DONE]
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Client_Crash_Id&oldid=1175858"