QA/Widevine CDM

From MozillaWiki
< QA
Jump to: navigation, search

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
04/12/2016 1.0 Karla Merza & Vlad Bacia Created first draft



Widevine support is an alternative solution for streaming services that currently rely on Silverlight for playback of DRM-protected video content. It will allow websites to show DRM-protected video content in Firefox without the use of NPAPI plugins. The Widevine CDM runs in an open-source CDM sandbox in Firefox providing better user security than NPAPI plugins.


This wiki details the testing that will be performed for Widevine CDM project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed


Engineering Program Manager: Chris Peterson

Platform Engineer: Chris Pearce

Rares Bologa - PM for QA team

Brindusa Tot - leading QA efforts

Vlad Bacia - QA

Simona Badau - QA

Karla Merza - QA

Cosmin Muntean - QA

Feature email address: TBD

Testing summary

Scope of Testing

In Scope

Widevine Content Decryption Modules (CDMs) are how streaming services protect content using HTML5 video to web browsers without relying on an NPAPI plugin like Flash or Silverlight. Currently we support Adobe Primetime CDM in Firefox and we are enabling the Google Widevine CDM starting in Nightly.

The scope of our testing is to test Widevine CDM functionality . The testing effort for Widevine CDM will be invested on the following areas:

  • integration: verify the plug-in integration with the current browser functionalities and UI;
  • functionality: basic and advanced functionality to be verified according to the existing requirements;
  • usability: the streaming services work correctly;

Out of Scope

Following areas/features are considered out of scope and will not be considered as testing zones to be handled in this test plan:

- Linux testing

- Mobile testing

Requirements for testing


Testing will be performed on the following OSes (both x86 & x64 infrastructures):

  • Windows 7
  • Windows 8
  • Windows 10
  • Mac OS X 10.9
  • Mac OS X 10.10
  • Mac OS X 10.11

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered.

Ref Function Test Objective Evaluation Criteria Test Type Owners
1 Choose Widevine CDM to stream videos Verify that the plug-in is working correctly 1. Videos are played with HTML 5 video player through Widevine DRM
2. Firefox shows a "chain link" icon in the address bar if the video is using WidevineCDM
Manual Eng Team
2 Widevine functionality Verify that the streaming is made through Widevine CDM 1. Play a single video
2. Play multiple videos from the same site simultaneously in different tabs
3. Play multiple Widevine videos from different sites simultaneously in different tabs
Manual Eng Team
3 Player controls Verify that all player controls are working as expected 1. Check basic controls (Play/Pause/Stop/Volume)
2. Quick seek forward and backward in a long video
3. Change video resolution (?)
4. Full Screen / Normal mode
Manual Eng Team
4 Refresh Verify that the video content is reloaded correctly After refresh, video is streamed as expected Manual Eng Team
5 Private Browsing Verify the functionality of video streaming TBD Manual Eng Team
6 Disable Widevine CDM Verify that Widevine CDM can be disabled and enabled without any issues Video with Widevine CDM is not playing/playing Manual Eng Team
7 Disable DRM Verify that DRM can be disabled and enabled without any issues You will still be able to view DRM-controlled videos that require Silverlight or Flash plugins if you've enabled them. The Play DRM content setting only controls HTML5 DRM playback, not plugins Manual Eng Team


  • Links for Nightly builds - available since April 14th 2016 - link
  • Links for Aurora builds - available since April 21st 2016 - link
  • Links for Beta builds - available since April 26th 2016 - link

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 04/11/2016 -
Study documentation/specs received 04/11/2016 -
QA - Test plan creation 04/12/2016 -
QA - Test cases/Env preparation 04/12/2016 -
QA - Nightly Testing 04/15/2016 -
QA - Aurora Testing 04/22/2016 -
QA - Beta Testing 04/27/2016 -
Release Date - -

Testing Tools

Process Tool
Test plan creation Mozilla wiki
Test case creation Google docs
Test case execution Google docs
Bugs management Bugzilla



  • Track the dates and build number where feature was released to Nightly
  • Track the dates and build number where feature was merged to Aurora
  • Track the dates and build number where feature was merged to Release/Beta

Risk analysis

Risk areas Mitigation
Browser widevine plugin compatibility and usability with content specific websites - Websites shouldn't offer other video plugins for playing Widevine video compatible content.
- If multiple video plugins are installed, Widevine should be prefered when playing supported content.
- Websites should not serve Widevine to Firefox by default.




  • Summary of testing scenarios

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes
Theme (high contrast) Yes
Mouse-only operation Yes
Keyboard-only operation Yes This depends on the website
Display (HiDPI) Yes
Interraction (scroll, zoom) Yes
Usable with a screen reader No e.g. with NVDA
Usability and/or discoverability testing N/A Is this feature user friendly
Help/support interface required TBD Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) TBD Make sure support documents are written and are correct.
Feature upgrades/downgrades data as expected Yes
Does sync work across upgrades No Not relevant
Requires install testing Yes separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding Yes Not visibly to the user, but the Widevine and Adobe CDMs are downloaded in the background when Firefox first runs. The CDM downloads might make Firefox sluggish
Does this affect partner builds? Partner build testing Yes We won't have partner builds available for testing in the Nightly or Aurora channels
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration
Network proxies/autoconfig Yes Low priority
ESR behavior changes
Locked preferences
Data Monitoring
Temporary or permanent telemetry monitoring No
Telemetry correctness testing
Server integration testing
Offline and server failure testing
Load testing Yes Covered by testing videos in multiple tabs
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required?
Comprehensive API testing
Testing with existing/popular addons
Security Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review
Privilege escalation testing
Web Compatibility depends on the feature
Testing against target sites N/A
Survey of many sites for compatibility
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS
Interaction of this feature with other browser features Yes

Test suite

  • Full Test suite - In progress
  • Smoke Test suite -
  • Regression Test suite -

Bug Work

Tracking bug – meta bug 1222845

Bug fix verification
Bug No Summary Status Firefox Version
1265321 [Widevine] The Widevine plug-in can't be enabled from the "You must enable DRM..." bar Verified Fixed 47, 48
1264483 Hide Widevine CDM download on unsupported systems Verified Fixed 47, 48, 49
1265049 Shaka's "Axinom" Widevine videos do not play Verified Fixed 47, 48, 49
1270689 Widevine crash in non-virtual thunk to mozilla::WidevineDecryptor::OnSessionClosed when using DASH.js player Verified Fixed 47, 48, 49
1272627 [Widevine] Video stops when focus is changed Verified Fixed 49
Logged bugs

Bug 1265416 - [Widevine] Amazon videos are not always opening using the Widevine CDM plugin
Bug 1265375 - [Widevine] Different plugins are used when playing multiple Amazon videos in different tabs after Widevine was disabled
Bug 1265349 - [Widevine] After the "Widevine" plugin is re-enable, the video does not start
Bug 1265336 - [Widevine] After page refresh the user is instructed to install Microsoft Silverlight plug-in in order to play videos from Amazon Video
Bug 1265321 - [Widevine] The Widevine plug-in can't be enabled from the "You must enable DRM..." bar
Bug 1265320 - [Widevine] HBOGO.RO videos do not play through WidevineCDM
Bug 1267223 - [Widevine] A dotted rectangle is displayed around the volume bar on
Bug 1267582 - [Widevine] No clear message that Netflix videos are not running in Private browsing
Bug 1267590 - [Widevine] Reload is not properly done on Amazon videos in non-e10s
Bug 1272627 - [Widevine] Video stops when focus is changed
Bug 1272638 - [Widevine] Netflix - N8109 error is received when switching from Silverlight to Widevine
Bug 1272674 - [Widevine] On Netflix, if start movie from browse page black screen is displayed
Bug 1273525 - Videos are flickering when switching tabs playing with Widevine or Adobe Primetime CDM
Bug 1275562 - [Widevine] Netflix - M7353 error is received when trying to play any video

Sign off


Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)


Nightly testing

List of OSes that will be covered by testing

  • Link for the tests run
    • Daily Smoke suite -
    • Full Test suite -
    • Regression Test suite, if needed/available

Merge to Aurora Sign-off List of OSes that will be covered by testing

  • Link for the tests run -
    • Full Test suite -


Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [IN PROGRESS]
Testing Infrastructure setup No
Test Plan Creation [IN PROGRESS]
Test Cases Creation [IN PROGRESS]
Full Functional Tests Execution [NOT STARTED]
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Daily Status Report (email/etherpad statuses/ gdoc with results)
Metrics/Telemetry N/A
QA Signoff - Nightly Release Email to be sent
QA Aurora - Full Testing
QA Signoff - Aurora Release Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release Email to be sent