QA/Shutdown Ping

QA/Shutdown Ping

Revision History

Date	Version	Author	Description
02/28/2017	1.0	Abe Masresha	Created first draft

Overview

Purpose

The purpose of this work is use the ping sender,Bug 1310703, to send shutdown pings. Pings will be handed over to the ping sender. We do not have to wait for the next browser start to get shutdown pings.

Scope

• Testing will focus on:

Stability of the browser with this work.

If shutdown pings are being sent by ping sender

validating contents of the shutdown pings

Comparing contents of shutdown pings if received twice

Comparing shutdown ping Ids if received twice

• How testing will be performed:

Manual

Risk analysis

• A shutdown ping might be not sent: not through the PingSender nor on the next Firefox start.
• The shutdown ping might be sent twice: once from the PingSender, once on the next Firefox start (they must have the same ping-id). This can happen but must be not frequent.
• The shutdown ping might be sent twice, with different content and the same ping id. This must never happen.

Ownership

Mozilla Development

Alessio Placitelli

QA Eng. Softvision Las Vegas
Stefan Georgiev - Team Lead
Abe Masresha - Test Lead
Justin Williams - Tester
Grover Wimberly IV - Tester
Kanchan Kumari - Tester

Testing summary

Scope of Testing

In Scope

This feature will be tested manually on Windows, Mac, and Linux operating systems.

Out of Scope

Automation will be handled by the developer (if needed)
Mobile devices will not be tested.

Requirements for testing

Environments: Operating Systems

Windows 10 64bits
Windows 7 32 bits
Ubuntu 16.04 32 and 64bits
MacOSX 10.12

Firefox Channel and environment setups

Build

Firefox Nightly 54.0a1

Test Strategy

Test Objectives

Verify Firefox is stable with this feature
Verify shutdown ping is sent by ping sender
Verify shutdown ping is sent once (if a second ping is received after a restart, they must have the same pingId and content). This may happen but should be less frequently
Verify if two shutdown pings are received, they must have the same pingId
Verify if two shutdown pings are received, they must have the same content
Verify two different shutdown pings have different pingIds
Verify Firefox restarts successfully after clean shutdown

Test Execution Schedule

Project phase	Start Date	End Date
Start project	02/24/2017	TBD
Study documentation/specs received from developers	02/28/2017	02/16/2017
QA - Test plan creation	02/28/2017	TBD
QA - Test cases/Env preparation	02/28/2017	TBD
QA - Nightly	TBD	TBD

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	Test Cases GDoc
Test case execution	[Test Cases TestRail]
Bugs management	Bugzilla

References

Test cases: [ TestRail] or Test Cases GDoc

Bugs: Bug 1336360

Testcases

Overview

Summary of testing scenarios

Test Areas

Test Areas	Covered	Details
Private Window	Yes	Few tests will be made on private window as well
Non e10s Window	Yes	Few tests will be made with non e10s window as well
Install/Upgrade
Feature upgrades/downgrades data as expected	Yes	Feature will be tested against browser updates
Data Monitoring
Temporary or permanent telemetry monitoring	Yes	Data from Local telemetry server will be monitored
Telemetry correctness testing	Yes	Data correctness will be tested.
Server integration testing	Yes	feature will send data to a telemetry server and this functionality will be tested
Offline and server failure testing	Yes	Server offline option will be included in testing
Add-ons
Testing with existing/popular add-ons	Yes	feature functionality will be tested with popular addons
Web Compatibility
Survey of popular sites for compatibility	Yes	Firefox will be tested for stability
Interoperability
Interaction of this feature with other browser features	Yes	Firefox will be tested for stability across popular sites.

Test suite

Full Test suite - [ Test Cases TestRail]/ Test Cases GDoc

Bug Work

Tracking bugs – bug 1336360

Bugs

Bug No	Summary	Status	Firefox Version
			54
			54

Sign off

Criteria

Check list

• All test cases should be executed
• All blocker and critical bugs must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/developer/QA)

Results

Feature testing

• Links for the tests run
  • Full Test suite: [ Test Cases runs TestRail]
  • Regression Test suite: [link] if needed/available

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)	[DONE]
Testing Infrastructure setup	[IN PROGRESS]
Test Plan Creation	[IN PROGRESS]
Test Cases Creation	[IN PROGRESS]	[Test Cases TestRail]/ Test Cases GDoc
Full Functional Tests Execution
Automation Coverage		will covered by developer if needed
Performance Testing		will covered by developer if needed
Load Testing		will covered by developer if needed
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release
QA Aurora - Full Testing
QA Signoff - Aurora Release
QA Beta - Full Testing
QA Signoff - Beta Release