QA/Shutdown Ping
QA/Shutdown Ping
Revision History
| Date | Version | Author | Description |
|---|---|---|---|
| 03/01/2017 | 1.0 | Abe Masresha | Created first draft |
| 04/06/2017 | 1.1 | Abe Masresha | updated the test plan |
| 05/11/2017 | 1.2 | Abe Masresha | updated the test plan |
| 06/02/2017 | 1.3 | Abe Masresha | updated the test plan |
| 06/07/2017 | 1.4 | Abe Masresha | updated the test plan |
Contents
Overview
Purpose
The purpose of this work is to use the ping sender,Bug 1310703, to send shutdown pings. Pings will be handed over to the ping sender. We will not have to wait for the browser to be reopened to get a shutdown ping.
Scope
- Testing will focus on:
- Stability of the browser with this work.
- If shutdown pings are being sent by ping sender
- Validating contents of the shutdown pings
- Comparing contents, PingIDs and URLs of shutdown pings if received twice
- How testing will be performed:
- Manual
- Automation is covered by the developer
Risk analysis
- A shutdown ping might be not sent: not through the PingSender nor on the next Firefox start.
- The shutdown ping might be sent twice: once from the PingSender, once on the next Firefox start (if this happens, pings must have identical Contents, pingId and URL). This can happen but must be not frequent.
- The shutdown ping might be sent twice, with different contents but identical pingId. This must never happen.
- The shutdown ping must be correctly sent on the next Firefox restart if the server returned an error when sending using the PingSender.
- No shutdown ping must be sent if Telemetry data upload is disabled (i.e. datareporting.healthreport.uploadEnabled is false)
- If datareporting.healthreport.uploadEnabled is set to true but no data choices bar was displayed (on a new profile), no ping must be sent when shutting down.
- If it is started with upload enabled (datareporting.healthreport.uploadEnabled) and then disabled before the shutting down, no ping must be sent.
Ownership
Mozilla Development
Alessio Placitelli
QA Eng. Softvision Las Vegas
Stefan Georgiev - Team Lead
Abe Masresha - Test Lead
Justin Williams - Tester
Grover Wimberly IV - Tester
Kanchan Kumari - Tester
Testing summary
Scope of Testing
In Scope
This feature will be tested manually on Windows, Mac, and Linux operating systems.
Out of Scope
Automation will be handled by the developer.
Mobile devices will not be tested.
Requirements for testing
Environments: Operating Systems
Windows 10 64bits Windows 7 32bits Ubuntu 16.04 32 and 64bits MacOSX 10.12
Firefox Channel and environment setups
Build
Firefox Nightly 54.0a1
Firefox Nightly 55.0a1
Test Strategy
Test Objectives
Verify Firefox is stable with this feature
Verify Firefox restarts successfully after clean shutdown
Verify shutdown ping is sent by ping sender
Verify the ping contains the correct headers
Verify shutdown ping is sent once (if a second ping is received after a restart, both pings must have an identical pingId, URL and contents). If this happens, it should be less frequently
Verify shutdown pings of different profiles have different contents, PingIds and URLs
Verify shutdown ping is correctly sent on the next Firefox restart if the server returned an error when sending using the PingSender
Verify shutdown ping is not sent when Telemetry data upload is disabled (i.e. datareporting.healthreport.uploadEnabled is false)
Verify no ping is sent when upload is enabled and no data choices bar is displayed (on a new profile)
Verify no ping is sent when an enabled upload is disabled before shutdown
Test Execution Schedule
| Project phase | Start Date | End Date |
|---|---|---|
| Start project | 02/24/2017 | 4/6/2017 |
| Study documentation/specs received from developers | 02/28/2017 | 02/16/2017 |
| QA - Test plan creation | 03/01/2017 | 4/6/2017 |
| QA - Test cases/Env preparation | 03/01/2017 | 4/6/2017 |
| QA - Nightly | 4/4/2017 | 4/6/2017 |
| QA - Mid-Nightly | 5/4/2017 | 5/11/2017 |
| QA - Pre-Beta | 5/29/2017 | 06/02/2017 |
| QA - Pre-Release |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | Test Cases GDoc |
| Test case execution | Test Cases TestRail |
| Test case execution [Mid-Nightly] | Test Cases TestRail |
| Bugs management | Bugzilla |
Status
Overview
References
Test cases: TestRail or Test Cases GDoc
Bugs:
Meta 1343277
Tracking Bug 1336360
Testcases
Overview
Summary of testing scenarios
Test Areas
| Test Areas | Covered | Details |
|---|---|---|
| Private Window | Yes | Few tests will be made on private window as well |
| Non e10s Window | Yes | Few tests will be made with non e10s window as well |
| Install/Upgrade | ||
| Feature upgrades/downgrades data as expected | Yes | Feature will be tested against browser updates |
| Data Monitoring | ||
| Temporary or permanent telemetry monitoring | Yes | Data from Local telemetry server will be monitored |
| Telemetry correctness testing | Yes | Data correctness will be tested. |
| Offline and server failure testing | Yes | Server offline option will be included in testing |
| Add-ons | ||
| Testing with existing/popular add-ons | Yes | Feature functionality will be tested with popular addons |
| Web Compatibility | ||
| Survey of popular sites for compatibility | Yes | Firefox will be tested for stability |
| Interoperability | ||
| Interaction of this feature with other browser features | Yes | Feature will be tested for its interaction with other browser features |
Test suite
Full Test suite - [ Test Cases TestRail]/ Test Cases GDoc
Bug Work
Tracking bugs – bug 1336360
Bugs
| Bug No | Summary | Status | Firefox Version |
|---|---|---|---|
| 1360819 | On Windows Nightly, PingSender shows terminal while browser is shutting down | VERIFIED FIXED | 55 |
| 1356673 | Crash in TelemetryImpl::RunPingSender on shutdown | VERIFIED FIXED | 55 |
| 1362024 | "Unable to start the application" error dialog for pingsender.exe when shutting down windows while Firefox is open | VERIFIED FIXED | 55 |
| 1360818 | "Unable to start the application" error dialog for pingsender.exe when shutting down windows while Firefox is open | RESOLVED DUPLICATE of bug 1360819 | 55 |
| 1364673 | "Unable to start the application" error dialog for pingsender.exe when shutting down windows while Firefox is open | VERIFIED FIXED | 55 |
| 1362500 | "Profile In Use" message appears when nightly is reopened from the previous profile | RESOLVED WORKSFORME | 55 |
Sign off
Criteria
Check list
- All test cases should be executed
- All blocker and critical bugs must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/developer/QA)
Results
Feature testing
- Links for the tests run
- Full Test suite: [ https://testrail.stage.mozaws.net/index.php?/plans/view/3485 Test Cases runs TestRail]
- Regression Test suite: [link] if needed/available
Checklist
| Exit Criteria | Status | Notes/Details | |
|---|---|---|---|
| Testing Prerequisites (specs, use cases) | [DONE] | ||
| Testing Infrastructure setup | [DONE] | ||
| Test Plan Creation | [DONE] | ||
| Test Cases Creation | [DONE] | Test Cases TestRail/ Test Cases GDoc | |
| Full Functional Tests Execution | [DONE] | ||
| Automation Coverage | Covered by developer | ||
| Performance Testing | Covered by developer | ||
| Load Testing | Covered by developer | ||
| All Defects Logged | All found bugs are logged and fixed | ||
| Critical/Blockers Fixed and Verified | All found bugs are fixed | ||
| QA Signoff - Mid-Nightly | [DONE] | ||
| QA Signoff - Pre-Beta | [DONE] | ||
| QA Signoff - Pre-Release | [DONE] |
