QA/Crash Ping Sender
QA/Crash Ping Sender
Revision History
Date | Version | Author | Description |
---|---|---|---|
02/16/2017 | 1.0 | Abe Masresha | Created first draft |
02/22/2017 | 1.1 | Abe Masresha | added filed bug and links |
03/03/2017 | 1.2 | Abe Masresha | updating project status |
Contents
Overview
Purpose
The goal of this feature is to speed up the delivery of the crash pings to our telemetry servers by using a standalone executable, ping sender. It sends the ping immediately after the crash. It does not wait for the browser to restart.
Scope
- Testing will focus on:
- Sending valid Pings by ping sender
- First Ping contents and URL. First ping is ping received from ping sender before a restart.
- Second ping contents and URL. Second ping is received after Firefox restart.
- Comparing first and second ping contents.
- Functionality of ping sender across different platforms
- Stability of the browser with this feature
- Functionality of ping sender with browser update
- Functionality of Crash Reporter Client with ping sender.
- How testing will be performed:
- Manual
Risk analysis
- It may crash the crash reporter itself so that users may not be able to submit crash reports
- May behave differently in different operating systems.
- The browser update may have effect on the ping or the opposite.
Ownership
Mozilla Development
Gabriele Svelto
QA Eng. Softvision Las Vegas
Stefan Georgiev - Team Lead
Abe Masresha - Test Lead
Justin Williams - Tester
Grover Wimberly IV - Tester
Kanchan Kumari - Tester
Testing summary
Scope of Testing
In Scope
This feature will be tested manually on Windows, MacOSX, and Linux operating systems
Out of Scope
Mobile Devices will not be tested
Requirements for testing
Environments: Operating Systems
Windows 10 64bits Windows 7 32 bits Ubuntu 16.04 32 and 64bits MacOSX 10.12
Firefox Channel and environment setups
Build
Firefox Nightly 54.0a1
Test Strategy
Test Objectives
Verify Firefox is stable with this feature
Verify the ping is sent immediately to telemetry server (local telemetry server will be used for testing)
Verify the ping has valid contents of payload, URL and ClientID
Verify a second ping is received by telemetry server after a browser restart
Verify the second ping has same payload, URL, and ClientID with the first ping
Verify browser update does not affect the ping
Verify Firefox restarts successfully after crash
Verify crash reporter client is responsive and functional after or before the first ping
Test Execution Schedule
Project phase | Start Date | End Date |
---|---|---|
Start project | 02/16/2017 | 3/3/2017 |
Study documentation/specs received from developers | 02/16/2017 | 02/16/2017 |
QA - Test plan creation | 02/16/2017 | 3/3/2017 |
QA - Test cases/Env preparation | 02/16/2017 | 3/3/2017 |
QA - Nightly | 02/16/2017 | 3/3/2017 |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | TestRail/ Test Cases GDoc |
Test case execution | Test Cases TestRail |
Bugs management | Bugzilla |
Status
Overview
References
Test cases: TestRail or Test Cases GDoc
Bugs: 1310703
Testcases
Overview
Summary of testing scenarios
Test Areas
Test Areas | Covered | Details |
---|---|---|
Private Window | Yes | Few tests will be made on private window as well |
Non e10s Window | Yes | Few tests will be made with non e10s window as well |
Install/Upgrade | ||
Feature upgrades/downgrades data as expected | Yes | Feature will be tested against browser updates |
Data Monitoring | ||
Temporary or permanent telemetry monitoring | Yes | Data from Local telemetry server will be monitored |
Telemetry correctness testing | Yes | Data correctness will be tested. |
Server integration testing | Yes | feature will send data to a telemetry server and this functionality will be tested |
Offline and server failure testing | Yes | Server offline option will be included in testing |
Add-ons | ||
Testing with existing/popular add-ons | Yes | feature functionality will be tested with popular addons |
Web Compatibility | ||
Survey of popular sites for compatibility | Yes | Firefox will be tested for stability |
Interoperability | ||
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | Yes | first and second ping data will be compared |
Interaction of this feature with other browser features | Yes | Firefox will be tested for stability across popular sites. |
Test suite
Full Test suite - [ Test Cases TestRail]/ Test Cases GDoc
Bug Work
Tracking bugs – bug
Bugs
Bug No | Summary | Status | Firefox Version |
---|---|---|---|
Bug 1341282 | VERIFIED-FIXED | The telemetry session id field of the crash ping sent by the crashreporter client is wrong | 54 |
Sign off
Criteria
Check list
- All test cases should be executed
- All blocker and critical bugs must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/developer/QA)
Results
Feature testing
- Links for the tests run
- Full Test suite: [TestRail https://testrail.stage.mozaws.net/index.php?/plans/view/4228]
- Regression Test suite: [link] if needed/available
Checklist
Exit Criteria | Status | Notes/Details | |
---|---|---|---|
Testing Prerequisites (specs, use cases) | [DONE] | Specs | |
Testing Infrastructure setup | [DONE] | ||
Test Plan Creation | [DONE] | ||
Test Cases Creation | [DONE] | Test Cases TestRail/ Test Cases GDoc | |
Full Functional Tests Execution | [DONE] | ||
Automation Coverage | will covered by developer if needed | ||
Performance Testing | will covered by developer if needed | ||
Load Testing | will covered by developer if needed | ||
All Defects Logged | [DONE] | ||
Critical/Blockers Fixed and Verified | [DONE] | ||
Metrics/Telemetry | [DONE] | ||
QA Signoff - Nightly Release | [DONE] | ||
QA Aurora - Full Testing | [DONE] | ||
QA Signoff - Aurora Release | [DONE] | ||
QA Beta - Full Testing | [DONE] | ||
QA Signoff - Beta Release | [DONE] |