Symbol Exporting
QA/Symbols Exporting
Revision History
| Date | Version | Author | Description |
|---|---|---|---|
| 11/10/2016 | 1.0 | Abe Masresha | Created first draft |
| 11/10/2016 | 1.1 | Grover Wimberly IV | Fixed some wording in several sub-sections |
| 1/13/2017 | 1.2 | Abe Masresha | Updated some links and added info on testing on android. |
Contents
Overview
Purpose
The goal of this work is to make Firefox crash less for release users.
Scope
This wiki details the testing that will be performed by the project team for the Symbol Exporting project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
- Testing will focus on:
- Firefox launch and shutdown
- Firefox child processes.
- Sandboxing (content, GMP, plugin).
- Exploratory testing for existing 3rd-party addons
- Android basic functionality testing
- How testing will be performed
- Manual
Ownership
Mozilla Development
Benjamin Smedberg
QA Eng. Softvision Las Vegas
Michelle ->Stefan - Team Lead ]
Abe Masresha - Test Lead
Grover Wimberly IV - Tester
Kanchan Kumari - Tester
Justin Williams - Tester
Testing summary
Scope of Testing
In Scope
This feature will be tested manually on Windows, OSX, Linux operating systems and Android devices.
Out of Scope
iOS Devices (Not affected by this work)
Automation will be handled by the developer (if needed)
Requirements for testing
Environments: Operating Systems
Windows 8.1 Windows 10 Windows 7 Ubuntu/Linux OSX 10.12 Android
Firefox Channel and environment setups
Build
Test Strategy
Test Objectives
Verify feature works correctly with Firefox launch and shutdowns
Verify feature works correctly with Firefox child processes
Verify feature works correctly with sandboxing (content, GMP, plugin)
Verify feature works fine with existing 3rd-party addons
Verify basic functionalities are working properly on Android
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
| Project phase | Start Date | End Date |
|---|---|---|
| Start project | ||
| Study documentation/specs received from developers | 11/10/2016 | 11/10/2016 |
| QA - Test plan creation | 11/10/2016 | 11/10/2016 |
| QA - Test cases/Env preparation | 11/10/2016 | 11/16/2016 |
| QA - tryBuild | 11/15/2016 | |
| QA - Nightly | 1/20/2017 | 1/20/2017 |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | TestRail/ Google docs |
| Test case execution | TestRail |
| Bugs management | Bugzilla |
Status
Overview
Try Builds:
Testing on Android, retesting and bug verification: Nightly
Risk analysis
References
Meta bug: Bug-1299187
Testcases
Overview
Summary of testing scenarios
Test Areas
| Test Areas | Covered | Details |
|---|---|---|
| Private Window | Certain tests are performed in a private window. | |
| Multi-Process Enabled | Enabled by default | |
| Multi-process Disabled | To be tested | |
| Theme (high contrast) | Not applicable | |
| UI | ||
| Mouse-only operation | To be tested | |
| Keyboard-only operation | To be tested | |
| Display (HiDPI) | Not applicable | |
| Interraction (scroll, zoom) | To be tested | |
| Usable with a screen reader | Not applicable | |
| Usability and/or discoverability testing | Is this feature user friendly | |
| RTL build testing | To be tested | |
| Help/Support | ||
| Help/support interface required | Make sure link to support/help page exist and is easy reachable. | |
| Support documents planned(written) | Make sure support documents are written and are correct. | |
| Install/Upgrade | ||
| Feature installs as expected | To be tested | |
| Does sync work across Install | To be tested | |
| Requires install testing | To be tested | |
| Affects first-run or onboarding | Not applicable | |
| Does this affect partner builds? Partner build testing | Not applicable | |
| Enterprise | ||
| Enterprise administration | Not applicable | |
| Network proxies/autoconfig | Not applicable | |
| ESR behavior changes | Not applicable | |
| Locked preferences | Not applicable | |
| Data Monitoring | ||
| Temporary or permanent telemetry monitoring | List of error conditions to monitor | |
| Telemetry correctness testing | Not applicable | |
| Server integration testing | Not applicable | |
| Offline and server failure testing | Not applicable | |
| Load testing | Not applicable | |
| Add-ons | ||
| Addon API required? | Not applicable | |
| Comprehensive API testing | Not applicable | |
| Permissions | Not applicable | |
| Testing with existing/popular addons | To be tested | |
| Security | ||
| 3rd-party security review | Not applicable | |
| Privilege escalation testing | Not applicable | |
| Fuzzing | Not applicable | |
| Web Compatibility | depends on the feature | |
| Testing against target sites | To be Tested | |
| Survey of many sites for compatibility | Not applicable | |
| Interoperability | depends on the feature | |
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | Not Applicable | |
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | Desktop only | |
| Interaction of this feature with other browser features | Not Applicable |
Test suite
Full Test suite - TestRail/ Google docs
Bug Work
Tracking bug – meta bug
Bugs
| Bug No | Summary | Status | Firefox Verion |
|---|---|---|---|
| Bug 1318149 | Crash | Works for Me | TryBuild/Nightly |
| Bug 1318175 | Crash | Works for Me | TryBuild/Nightly |
| Bug 1317196 | Netflix not working on Linux | Known Issue | Nightly |
| Bug 1318430 | Norton Security Toolbar does not install in Firefox | Known Issue | Nightly |
| Bug 1317995 | Firefox 50 64-bit Stable channel is unable to play Farmville 2 Facebook Flash game | Known Issue | Nightly |
| Bug 1308677 | Flash warnings in Nightly are out of control | Known Issue | Nightly |
Sign off
Criteria
Check list
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Feature testing
- Links for the tests run
- Full Test suite: Link
- Regression Test suite: [link] if needed/available
Checklist
| Exit Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | ||
| Testing Infrastructure setup | ||
| Test Plan Creation | ||
| Test Cases Creation | ||
| Full Functional Tests Execution | ||
| Automation Coverage | ||
| Performance Testing | ||
| All Defects Logged | ||
| Critical/Blockers Fixed and Verified | ||
| Metrics/Telemetry | ||
| QA Signoff - Nightly Release | ||
| QA Aurora - Full Testing | ||
| QA Signoff - Aurora Release | Email sent | |
| QA Beta - Full Testing | ||
| QA Signoff - Beta Release |
