QA/Flash Click to Play enabled by default
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
| Date | Version | Author | Description |
|---|---|---|---|
| 5/10/2017 | 1.0 | Stefan Georgiev | Created first draft |
Contents
Overview
Purpose
Firefox is planning to move to click-to-activate (CTA). Firefox wants to safe the user from attackers that are using the security flaws in Flash to run malicious software on your computer and gain access to your system.
Scope
This wiki details the testing that will be performed by the project team for the Flash Click to Play enabled by default feature. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
Goals:
Block Flash by default
Keep users secure from Flash exploits
How testing will be performed:
Manual
Ownership
Product Manager: Benjamin Smedberg Engineering Lead: Felipe Gomes Engineering Manager: David Durst Test Lead: Stefan Georgiev
Testing summary
Scope of Testing
In Scope
This feature will be tested manually on Windows, OS X and Linux operating system.
Out of Scope
Android & iOS devices
Requirements for testing
Environments
Windows 7
Windows 8.1
Windows 10
Linux/ Ubuntu 16.04
OS X 10.12 (Sierra)
Channel dependent settings (configs) and environment setups
Nightly
text
Beta
text
Post Beta / Release
text
Test Strategy
Risk analysis
- Breaking websites:
- Top destinations websites: Facebook, Yahoo, Google, Gmail, Google Docs, etc.
- Top media websites: Youtube, Vimeo, Spotify, some news websites with video
- Top Flash-based game websites: Kongregate, Facebook, etc.
- No site gets a broken or "transparent" ad on top of content that never loads
- No site gets in a forever-loading state due to this change
- No top-site hangs or significantly slows down Firefox due to this change
- Check that heuristics are correctly applied
- Check that whitelist/blocklist works correctly
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
| Ref | Function | Test Objective | Evaluation Criteria | Test Type | Owners |
|---|---|---|---|---|---|
| 1 | Flash is set to CTA | Flash content is CTA | All flash objects are Click to Activate, Flash is not starting automatically | Manual | Eng Team |
| 2 | Flash ads are properly shown | No site gets a broken or transparent ads | All flash ads are shown properly | Manual | Eng Team |
| 3 | Top sites are loading correctly | No sites gets in a forever-loading state | All top sites loads correctly | Manual | Eng Team |
| 4 | Top sites does not hangs or slows down Firefox | No performance issue on Top sites | No performance issues observed | Manual | Eng Team |
| 5 | Flash games are running correctly | No broken flash games | Flash games loads and are playable | Manual | Eng Team |
Builds
This section should contain links for builds with the feature -
- Links for Nightly builds
- Links for Beta builds
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
| Project phase | Start Date | End Date |
|---|---|---|
| Start project | ||
| Study documentation/specs received from developers | ||
| QA - Test plan creation | 5/10/2017 | 5/12/2017 |
| QA - Test cases/Env preparation | ||
| QA - Nightly Testing | ||
| QA - Beta Testing | ||
| Release Date |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | TestRail/ Google docs |
| Test case execution | TestRail |
| Bugs management | Bugzilla |
Status
Overview
Track the dates and build number where feature was released to Nightly Track the dates and build number where feature was merged to Release/Beta
References
- List and links for specs
List and links for available specs - documents, user stories, specifications
- Meta bug 1340934
Testcases
Test Areas
| Test Areas | Covered | Details |
|---|---|---|
| Private Window | Certain test are performed in a private window | |
| Multi-Process Enabled | Enabled by default | |
| Multi-process Disabled | To be tested | |
| UI | ||
| Mouse-only operation | To be tested | |
| Keyboard-only operation | To be tested | |
| Display (HiDPI) | To be tested | |
| Interaction (scroll, zoom) | To be tested | |
| Help/Support | ||
| Help/support interface required | Make sure link to support/help page exist and is easy reachable. | |
| Support documents planned(written) | Make sure support documents are written and are correct. | |
| Add-ons | If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on. | |
| Permissions | To be tested | |
| Web Compatibility | depends on the feature | |
| Testing against target sites | To be tested | |
| Survey of many sites for compatibility | To be tested |
Test suite
Full Test suite - Test suite for Flash CTP Regression Test suite - Regression suite for Flash CTP
Bug Work
Meta bug: 1340934 - bug summary
Sign off
Criteria
Checklist
- All test cases should be executed
- All blocker and critical bugs must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite, link to TestRail - Tests Runs and Results link
- Daily Smoke, if needed/available
- Regression Test suite, if needed/available
Merge to Beta Sign-off
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite
Checklist
| Exit Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | ||
| Testing Infrastructure setup | ||
| Test Plan Creation | Done | 5/10/2017 |
| Test Cases Creation | ||
| Automation Coverage | ||
| Performance Testing | ||
| All Defects Logged | ||
| Critical/Blockers Fixed and Verified | ||
| Metrics/Telemetry | ||
| Basic/Core functionality Nightly testing | ||
| QA mid-Nightly Signoff | Email to be sent | |
| QA Nightly - Full Testing | ||
| QA pre-Beta Signoff | Email to be sent | |
| QA Beta - Full Testing | ||
| QA pre-Release Signoff | Email to be sent |
