QA/Widevine CDM

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date	Version	Author	Description
04/12/2016	1.0	Karla Merza & Vlad Bacia	Created first draft

Overview

Purpose

Widevine support is an alternative solution for streaming services that currently rely on Silverlight for playback of DRM-protected video content. It will allow websites to show DRM-protected video content in Firefox without the use of NPAPI plugins. The Widevine CDM runs in an open-source CDM sandbox in Firefox providing better user security than NPAPI plugins.

Scope

This wiki details the testing that will be performed for Widevine CDM project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

• What will be tested
• How testing will be performed

Ownership

Technical Program Manager contact: TBD

Engineering contact: TBD

Production contact: TBD

User Experience contact: TBD

QA:
Rares Bologa - PM for QA team

Brindusa Tot - leading QA efforts

Cosmin Muntean - QA

Karla Merza - QA

Vlad Bacia - QA

Simona Badau - QA

Feature email address: TBD

Testing summary

Scope of Testing

In Scope

Widevine Content Decryption Modules (CDMs) are how streaming services protect content using HTML5 video to web browsers without relying on an NPAPI plugin like Flash or Silverlight. Currently we support Adobe Primetime CDM in Firefox and we are enabling the Google Widevine CDM starting in Nightly.

The scope of our testing is to test Widevine CDM functionality . The testing effort for Widevine CDM will be invested on the following areas:

• integration: verify the plug-in integration with the current browser functionalities and UI;
• functionality: basic and advanced functionality to be verified according to the existing requirements;
• usability: the streaming services work correctly;

Out of Scope

Following areas/features are considered out of scope and will not be considered as testing zones to be handled in this test plan:

- Linux testing

- Mobile testing

Requirements for testing

Environments

Testing will be performed on the following OSes (both x86 & x64 infrastructures):

• Windows XP
• Windows 7
• Windows 8.1
• Windows 10
• Mac OS X 10.9
• Mac OS X 10.10
• Mac OS X 10.11

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered.

Ref	Function	Test Objective	Evaluation Criteria	Test Type	Owners
1	Chose Widevine CDM to stream videos	Verify that the plug-in is working correctly	Videos are played with HTML 5 video player through Widevine DRM	Manual	Eng Team
2	Widevine functionality	Verify that the streaming is made through Widevine CDM	1. Play a single video 2. Play multiple videos from the same site simultaneously in different tabs 3. Play multiple Widevine videos from different sites simultaneously in different tabs	Manual	Eng Team
3	Player controls	Verify that all player controls are working as expected	1. Check basic controls (Play/Pause/Stop/Volume) 2. Quick seek forward and backward in a long video 3. Change video resolution (?) 4. Full Screen / Normal mode	Manual	Eng Team
4	Refresh	Verify that the video content is reloaded correctly	After refresh, video is streamed as expected	Manual	Eng Team
5	Private Browsing	Verify the functionality of video streaming	TBD	Manual	Eng Team
6	Disable Widevine CDM	Verify that Widevine CDM can be disabled and enabled without any issues	Video with Widevine CDM is not playing/playing	Manual	Eng Team
7	Disable DRM	Verify that DRM can be disabled and enabled without any issues	You will still be able to view DRM-controlled videos that require Silverlight or Flash plugins if you've enabled them. The Play DRM content setting only controls HTML5 DRM playback, not plugins	Manual	Eng Team

Builds

• Nightly builds with Widevine CDM are available at link?
• Links for Aurora builds ?
• Links for Beta builds ?

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project	04/11/2016	-
Study documentation/specs received	04/11/2016	-
QA - Test plan creation	04/12/2016	-
QA - Test cases/Env preparation	04/12/2016	-
QA - Nightly Testing	-	-
QA - Aurora Testing	-	-
QA - Beta Testing	-	-
Release Date	-	-

Testing Tools

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	Google docs
Test case execution	Google docs
Bugs management	Bugzilla

Status

Overview

• Track the dates and build number where feature was released to Nightly
  
• Track the dates and build number where feature was merged to Aurora
  
• Track the dates and build number where feature was merged to Release/Beta
  

Risk analysis

Risk areas	Mitigation
Browser widevine plugin compatibility and usability with content specific websites	- Websites shouldn't offer other video plugins for playing Widevine video compatible content. - If multiple video plugins are installed, Widevine should be prefered when playing supported content.

References

• Watch DRM content on Firefox
• Test Widevine CDM in Firefox Nightly

Testcases

Overview

• Summary of testing scenarios

Test Areas

Test Areas	Covered	Details
Private Window	Yes
Multi-Process Enabled	Yes
Multi-process Disabled	Yes
Theme (high contrast)	Yes
UI
Mouse-only operation	Yes
Keyboard-only operation	?
Display (HiDPI)	Yes
Interraction (scroll, zoom)	Yes
Usable with a screen reader	No	e.g. with NVDA
Usability and/or discoverability testing	?	Is this feature user friendly
Help/Support
Help/support interface required	TBD	Make sure link to support/help page exist and is easy reachable.
Support documents planned(written)	TBD	Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected	Yes
Does sync work across upgrades	?
Requires install testing	Yes	separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding	?	Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Does this affect partner builds? Partner build testing	?	yes/no options, add comment with details about who will lead testing
Enterprise		Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration
Network proxies/autoconfig
ESR behavior changes
Locked preferences
Data Monitoring
Temporary or permanent telemetry monitoring	?	List of error conditions to monitor
Telemetry correctness testing
Server integration testing
Offline and server failure testing
Load testing	?
Add-ons		If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required?
Comprehensive API testing
Permissions
Testing with existing/popular addons
Security		Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review
Privilege escalation testing
Fuzzing
Web Compatibility		depends on the feature
Testing against target sites	?
Survey of many sites for compatibility
Interoperability		depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS
Interaction of this feature with other browser features	Yes

Test suite

• Full Test suite - In progress
• Smoke Test suite -
• Regression Test suite -

Bug Work

Tracking bug – meta bug 1222845

Bug fix verification

Bug No	Summary	Status	Firefox Verion
-	-	-	-

Logged bugs

-

Sign off

Criteria

Check list

• All test cases should be executed
• Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
• All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

• Link for the tests run
  • Daily Smoke suite -
  • Full Test suite -
  • Regression Test suite, if needed/available

Merge to Aurora Sign-off List of OSes that will be covered by testing

• Link for the tests run -
  • Full Test suite -

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)	[IN PROGRESS]
Testing Infrastructure setup	No
Test Plan Creation	[IN PROGRESS]
Test Cases Creation	[IN PROGRESS]
Full Functional Tests Execution	[NOT STARTED]
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Daily Status Report (email/etherpad statuses/ gdoc with results)
Metrics/Telemetry		N/A
QA Signoff - Nightly Release		Email to be sent
QA Aurora - Full Testing
QA Signoff - Aurora Release		Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release		Email to be sent