Symbol Exporting
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
| Date | Version | Author | Description |
|---|---|---|---|
| 11/10/2016 | 1.0 | Abe Masresha | Created first draft |
| 11/10/2016 | 1.1 | Grover Wimberly IV | Fixed some wording in several sub-sections |
Overview
Purpose
The goal of this work is to make Firefox crash less for release users.
Scope
This wiki details the testing that will be performed by the project team for the Symbol Exporting project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
- Testing will focus on:
- Firefox launch and shutdown
- Firefox child processes.
- Sandboxing (content, GMP, plugin).
- Exploratory testing for existing 3rd-party addons
- How testing will be performed
- Manual
Ownership
Mozilla Development
Benjamin Smedberg
QA Eng. Softvision Las Vegas
Michelle Funches - Team Lead ]
Abe Masresha - Test Lead
Grover Wimberly IV - Tester
Kanchan Kumari - Tester
Justin Williams - Tester
Testing summary
Scope of Testing
In Scope
This feature will be tested manually on Windows, OSX and Linux operating systems.
Out of Scope
Android & iOS Devices
Automation will be handled by the developer
Requirements for testing
Environments: Operating Systems
Windows 8.1 Windows 10 Windows 7 Ubuntu/Linux OSX 10.12
Firefox Channel and environment setups
Build
- trybuild will be used to test this feature.
- Build Link: [link]
Test Strategy
Test Objectives
Verify feature works correctly with Firefox launch and shutdowns
Verify feature works correctly with Firefox child processes
Verify feature works correctly with sandboxing (content, GMP, plugin)
Verify feature works fine with existing 3rd-party addons
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
| Project phase | Start Date | End Date |
|---|---|---|
| Start project | ||
| Study documentation/specs received from developers | 11/10/2016 | |
| QA - Test plan creation | 11/10/2016 | |
| QA - Test cases/Env preparation | 11/10/2016 | |
| QA - tryBuild | when ready |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool | |
|---|---|---|
| Test plan creation | Mozilla wiki | |
| Test case creation | TestRail/ Google docs | |
| Test case execution | TestRail | |
| Bugs management | Bugzilla |
Status
Overview
Track the dates and build number where feature was released TBD
Risk analysis
References
Test cases: [link]
Meta bug: Bug-1299187
Testcases
Overview
Summary of testing scenarios
Test Areas
| Test Areas | Covered | Details |
|---|---|---|
| Private Window | Certain tests are performed in a private window. | |
| Multi-Process Enabled | Enabled by default | |
| Multi-process Disabled | To be tested | |
| Theme (high contrast) | Not applicable | |
| UI | ||
| Mouse-only operation | To be tested | |
| Keyboard-only operation | To be tested | |
| Display (HiDPI) | Not applicable | |
| Interraction (scroll, zoom) | To be tested | |
| Usable with a screen reader | Not applicable | |
| Usability and/or discoverability testing | Is this feature user friendly | |
| RTL build testing | To be tested | |
| Help/Support | ||
| Help/support interface required | Make sure link to support/help page exist and is easy reachable. | |
| Support documents planned(written) | Make sure support documents are written and are correct. | |
| Install/Upgrade | ||
| Feature installs as expected | To be tested | |
| Does sync work across Install | To be tested | |
| Requires install testing | To be tested | |
| Affects first-run or onboarding | Not applicable | |
| Does this affect partner builds? Partner build testing | Not applicable | |
| Enterprise | ||
| Enterprise administration | Not applicable | |
| Network proxies/autoconfig | Not applicable | |
| ESR behavior changes | Not applicable | |
| Locked preferences | Not applicable | |
| Data Monitoring | ||
| Temporary or permanent telemetry monitoring | List of error conditions to monitor | |
| Telemetry correctness testing | Not applicable | |
| Server integration testing | Not applicable | |
| Offline and server failure testing | Not applicable | |
| Load testing | Not applicable | |
| Add-ons | ||
| Addon API required? | Not applicable | |
| Comprehensive API testing | Not applicable | |
| Permissions | Not applicable | |
| Testing with existing/popular addons | To be tested | |
| Security | ||
| 3rd-party security review | Not applicable | |
| Privilege escalation testing | Not applicable | |
| Fuzzing | Not applicable | |
| Web Compatibility | depends on the feature | |
| Testing against target sites | To be Tested | |
| Survey of many sites for compatibility | Not applicable | |
| Interoperability | depends on the feature | |
| Not Applicable | ||
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | Desktop only | |
| Interaction of this feature with other browser features | Not Applicable |
Test suite
Full Test suite - TestRail/ Google docs
Bug Work
Tracking bug – meta bug
Bug fix verification
| Bug No | Summary | Status | Firefox Verion |
|---|---|---|---|
| Bug 1318149 | Crash | New | TryBuild |
| Bug 1318175 | Crash | New | TryBuild |
| Bug 1317196 | Netflix not working on Linux | Known Issue | Nightly |
| Bug 1318430 | Norton Security Toolbar does not install in Firefox | Known Issue | Nightly |
| Bug 1317995 | Firefox 50 64-bit Stable channel is unable to play Farmville 2 Facebook Flash game | Known Issue | Nightly |
| Bug 1308677 | Flash warnings in Nightly are out of control | Known Issue | Nightly |
Sign off
Criteria
Check list
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Feature testing
- Links for the tests run
- Daily Smoke: [link]
- Full Test suite: [Link]
- Regression Test suite: [link] if needed/available
Checklist
| Exit Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | ||
| Testing Infrastructure setup | ||
| Test Plan Creation | ||
| Test Cases Creation | ||
| Full Functional Tests Execution | ||
| Automation Coverage | ||
| Performance Testing | ||
| All Defects Logged | ||
| Critical/Blockers Fixed and Verified | ||
| Metrics/Telemetry | ||
| QA Signoff - Nightly Release | Email to be sent | |
| QA Aurora - Full Testing | ||
| QA Signoff - Aurora Release | Email to be sent | |
| QA Beta - Full Testing | ||
| QA Signoff - Beta Release | Email to be sent |