Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date	Version	Author	Description
12/08/2016	1.0	Stefan Georgiev	Created first draft
02/16/2018	1.1	Stefan Georgiev	Updated first draft

Overview

Purpose

Most of the browsers are planning to move to click-to-activate (CTA). But before Firefox prepare for this shift, we want to understand the impact on the user and the approach we should take.

Scope

This wiki details the testing that will be performed by the project team for the Flash SHIELD Experiment project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

• Goals

Block Flash by default
Keep users secure from Flash exploits

• How testing will be performed

Manual

Ownership

Product Manager: Benjamin Smedberg Engineering Lead: Felipe Gomes Engineering Manager: David Durst Test Lead: Stefan Georgiev

Testing summary

Scope of Testing

In Scope

This feature will be tested manually on Windows, OSX and Linux operating systems.

Out of Scope

Android & iOS Devices
Exclude users without Flash
Exclude users who don’t have the targeted gecko version

Requirements for testing

Environments

Windows 7
Windows 8.1
Windows 10
Ubuntu/Linux
Macintosh 10.12

Channel dependent settings (configs) and environment setups

Test Strategy

Test Objectives

URLs
Lists
Heuristics

Builds

• Links for Test builds:

1. Latest Mozilla Test Builds [[]]

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project
Study documentation/specs received from developers	12/06/2016
QA - Test plan creation	12/08/2016
QA - Test cases/Env preparation
QA - Nightly Testing
QA - Aurora Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	TestRail/ Google docs
Test case execution	TestRail
Bugs management	Bugzilla

Status

Overview

Track the dates and build number where feature was released to Nightly
TBD

Risk analysis

• Breaking websites:
  • Top destinations websites: Facebook, Yahoo, Google, Gmail, Google Docs, etc.
  • Top media websites: Youtube, Vimeo, Spotify, some news websites with video
  • Top Flash-based game websites: Kongregate, etc.

• Verify that no site gets a broken or "transparent" ad on top of content that never loads

• Verify that no site gets in a forever-loading state due to this change

• Verify that no top-site hangs or significantly slows down Firefox due to this change

• Verify that no data from the experiment is missing or incorrect

• Check that heuristics are correctly applied

• Check that whitelist/blocklist works correctly

References

• List and links for specs - Specs [1]

• Meta bug - Bug 1335232 [2]

Testcases

Overview

Summary of testing scenarios

1. URL

• Verify allowed list is applied - Flash is included in the navigator.plugins, Flash instances are activated immediately

• Verify blocked list is applied - Flash is not included in the navigator.plugins, any attempt to use Flash use fallback content

• Verify for all essential Flash, fallback (non-Flash) format is used to shown the content

• Verify Flash content is set to Click-to-active

• Verify allowed/denied list is set to geography-specific

• User is able to enable the Flash content

• User is able to block the Flash content

• Browsers send correct telemetry data to the server

• User has explicitly set "Disallow Flash" for this site - Flash content should be "click to activate" / Flash is present in the navigator.plugins

• User has explicitly set to "Allow Flash" for this site - Flash is present in the navigator.plugins / Flash elements are automatically activated

2. Lists

• Check whitelisted domain - page loads; Flash is not set to CTA - check iframes against the list

• Check blacklisted domain - page loads; Flash is not loaded - don't check iframes against the list

• Check non-whitelisted/non-blacklisted domain page loads - check all iframes against the list

3. Heuristics

• Flash has acceptable fallback content - Show the fallback content

• Flash doesn't have acceptable fallback content
  • If it pass the heuristics - set Flash to CTA
  • If it doesn't pass the heuristics - don't load the Flash content

Test Areas

Test Areas	Covered	Details
Private Window		Certain tests are performed in a private window.
Multi-Process Enabled		Enabled by default
Multi-process Disabled		To be tested
Theme (high contrast)		Not applicable
UI
Mouse-only operation		To be tested
Keyboard-only operation		To be tested
Display (HiDPI)		Not applicable
Interraction (scroll, zoom)		Not applicable
Usable with a screen reader		Not applicable
Usability and/or discoverability testing		Is this feature user friendly
RTL build testing		Not applicable
Help/Support
Help/support interface required		Make sure link to support/help page exist and is easy reachable.
Support documents planned(written)		Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected		Not applicable
Does sync work across upgrades		Not applicable
Requires install testing		Default installation with Test Build
Affects first-run or onboarding		To be tested
Enterprise
Enterprise administration		Not applicable
Network proxies/autoconfig		Not applicable
ESR behavior changes		Not applicable
Locked preferences		Not applicable
Data Monitoring
Temporary or permanent telemetry monitoring		List of error conditions to monitor
Telemetry correctness testing		To be tested
Server integration testing		To be tested
Offline and server failure testing		To be tested
Load testing		Not applicable
Add-ons
Addon API required?		Not applicable
Comprehensive API testing		Not applicable
Permissions		Not applicable
Testing with existing/popular addons		To be tested
Security
3rd-party security review		Not applicable
Privilege escalation testing		Not applicable
Fuzzing		Not applicable
Web Compatibility
Testing against target sites		To be tested
Survey of many sites for compatibility		To be tested
Interoperability
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS		Desktop only
Interaction of this feature with other browser features		Possible testing with other browsers

Test suite

Full Test suite - [TestRail]

Bug Work

Tracking bug – meta bug 1277346 [3]
Tracking bug - bug 1282484 [4]

Sign off

Criteria

Check list

• All test cases should be executed
• All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/QA)

Results

• Link for the tests run
  • Daily Smoke, use template from link
  • Full Test suite, use template from link
  • Regression Test suite, if needed/available

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release		Email to be sent
QA Aurora - Full Testing
QA Signoff - Aurora Release		Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release		Email to be sent