QA/Crash Ping Sender

From MozillaWiki
< QA
Jump to: navigation, search

QA/Crash Ping Sender


Revision History


Date Version Author Description
02/16/2017 1.0 Abe Masresha Created first draft
02/22/2017 1.1 Abe Masresha added filed bug and links
03/03/2017 1.2 Abe Masresha updating project status

Overview

Purpose

The goal of this feature is to speed up the delivery of the crash pings to our telemetry servers by using a standalone executable, ping sender. It sends the ping immediately after the crash. It does not wait for the browser to restart.

Scope

  • Testing will focus on:
Sending valid Pings by ping sender
First Ping contents and URL. First ping is ping received from ping sender before a restart.
Second ping contents and URL. Second ping is received after Firefox restart.
Comparing first and second ping contents.
Functionality of ping sender across different platforms
Stability of the browser with this feature
Functionality of ping sender with browser update
Functionality of Crash Reporter Client with ping sender.
  • How testing will be performed:
Manual

Risk analysis

  • It may crash the crash reporter itself so that users may not be able to submit crash reports
  • May behave differently in different operating systems.
  • The browser update may have effect on the ping or the opposite.

Ownership

Mozilla Development
Gabriele Svelto

QA Eng. Softvision Las Vegas
Stefan Georgiev - Team Lead
Abe Masresha - Test Lead
Justin Williams - Tester
Grover Wimberly IV - Tester
Kanchan Kumari - Tester

Testing summary

Scope of Testing

In Scope

This feature will be tested manually on Windows, MacOSX, and Linux operating systems

Out of Scope

Mobile Devices will not be tested

Requirements for testing

Environments: Operating Systems

Windows 10 64bits
Windows 7 32 bits
Ubuntu 16.04 32 and 64bits
MacOSX 10.12

Firefox Channel and environment setups

Build

Firefox Nightly 54.0a1

Test Strategy

Test Objectives

Verify Firefox is stable with this feature
Verify the ping is sent immediately to telemetry server (local telemetry server will be used for testing)
Verify the ping has valid contents of payload, URL and ClientID
Verify a second ping is received by telemetry server after a browser restart
Verify the second ping has same payload, URL, and ClientID with the first ping
Verify browser update does not affect the ping
Verify Firefox restarts successfully after crash
Verify crash reporter client is responsive and functional after or before the first ping

Test Execution Schedule

Project phase Start Date End Date
Start project 02/16/2017 3/3/2017
Study documentation/specs received from developers 02/16/2017 02/16/2017
QA - Test plan creation 02/16/2017 3/3/2017
QA - Test cases/Env preparation 02/16/2017 3/3/2017
QA - Nightly 02/16/2017 3/3/2017

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Test Cases GDoc
Test case execution Test Cases TestRail
Bugs management Bugzilla

Status

Overview

References

Test cases: TestRail or Test Cases GDoc

Bugs: 1310703

Testcases

Overview

Summary of testing scenarios

Test Areas

Test Areas Covered Details
Private Window Yes Few tests will be made on private window as well
Non e10s Window Yes Few tests will be made with non e10s window as well
Install/Upgrade
Feature upgrades/downgrades data as expected Yes Feature will be tested against browser updates
Data Monitoring
Temporary or permanent telemetry monitoring Yes Data from Local telemetry server will be monitored
Telemetry correctness testing Yes Data correctness will be tested.
Server integration testing Yes feature will send data to a telemetry server and this functionality will be tested
Offline and server failure testing Yes Server offline option will be included in testing
Add-ons
Testing with existing/popular add-ons Yes feature functionality will be tested with popular addons
Web Compatibility
Survey of popular sites for compatibility Yes Firefox will be tested for stability
Interoperability
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. Yes first and second ping data will be compared
Interaction of this feature with other browser features Yes Firefox will be tested for stability across popular sites.

Test suite

Full Test suite - [ Test Cases TestRail]/ Test Cases GDoc

Bug Work

Tracking bugs – bug

Bugs
Bug No Summary Status Firefox Version
Bug 1341282 VERIFIED-FIXED The telemetry session id field of the crash ping sent by the crashreporter client is wrong 54

Sign off

Criteria

Check list

  • All test cases should be executed
  • All blocker and critical bugs must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/developer/QA)

Results

Feature testing

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE] Specs
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE] Test Cases TestRail/ Test Cases GDoc
Full Functional Tests Execution [DONE]
Automation Coverage will covered by developer if needed
Performance Testing will covered by developer if needed
Load Testing will covered by developer if needed
All Defects Logged [DONE]
Critical/Blockers Fixed and Verified [DONE]
Metrics/Telemetry [DONE]
QA Signoff - Nightly Release [DONE]
QA Aurora - Full Testing [DONE]
QA Signoff - Aurora Release [DONE]
QA Beta - Full Testing [DONE]
QA Signoff - Beta Release [DONE]