QA/Widevine CDM
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
Date | Version | Author | Description |
---|---|---|---|
04/12/2016 | 1.0 | Karla Merza & Vlad Bacia | Created first draft |
Contents
Overview
Purpose
Widevine support is an alternative solution for streaming services that currently rely on Silverlight for playback of DRM-protected video content. It will allow websites to show DRM-protected video content in Firefox without the use of NPAPI plugins. The Widevine CDM runs in an open-source CDM sandbox in Firefox providing better user security than NPAPI plugins.
Scope
This wiki details the testing that will be performed for Widevine CDM project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
- What will be tested
- How testing will be performed
Ownership
Engineering Program Manager: Chris Peterson
Platform Engineer: Chris Pearce
QA:
Rares Bologa - PM for QA team
Brindusa Tot - leading QA efforts
Vlad Bacia - QA
Simona Badau - QA
Karla Merza - QA
Cosmin Muntean - QA
Feature email address: TBD
Testing summary
Scope of Testing
In Scope
Widevine Content Decryption Modules (CDMs) are how streaming services protect content using HTML5 video to web browsers without relying on an NPAPI plugin like Flash or Silverlight. Currently we support Adobe Primetime CDM in Firefox and we are enabling the Google Widevine CDM starting in Nightly.
The scope of our testing is to test Widevine CDM functionality . The testing effort for Widevine CDM will be invested on the following areas:
- integration: verify the plug-in integration with the current browser functionalities and UI;
- functionality: basic and advanced functionality to be verified according to the existing requirements;
- usability: the streaming services work correctly;
Out of Scope
Following areas/features are considered out of scope and will not be considered as testing zones to be handled in this test plan:
- Linux testing
- Mobile testing
Requirements for testing
Environments
Testing will be performed on the following OSes (both x86 & x64 infrastructures):
- Windows 7
- Windows 8
- Windows 10
- Mac OS X 10.9
- Mac OS X 10.10
- Mac OS X 10.11
Test Strategy
Test Objectives
This section details the progression test objectives that will be covered.
Ref | Function | Test Objective | Evaluation Criteria | Test Type | Owners |
---|---|---|---|---|---|
1 | Choose Widevine CDM to stream videos | Verify that the plug-in is working correctly | 1. Videos are played with HTML 5 video player through Widevine DRM 2. Firefox shows a "chain link" icon in the address bar if the video is using WidevineCDM |
Manual | Eng Team |
2 | Widevine functionality | Verify that the streaming is made through Widevine CDM | 1. Play a single video 2. Play multiple videos from the same site simultaneously in different tabs 3. Play multiple Widevine videos from different sites simultaneously in different tabs |
Manual | Eng Team |
3 | Player controls | Verify that all player controls are working as expected | 1. Check basic controls (Play/Pause/Stop/Volume) 2. Quick seek forward and backward in a long video 3. Change video resolution (?) 4. Full Screen / Normal mode |
Manual | Eng Team |
4 | Refresh | Verify that the video content is reloaded correctly | After refresh, video is streamed as expected | Manual | Eng Team |
5 | Private Browsing | Verify the functionality of video streaming | TBD | Manual | Eng Team |
6 | Disable Widevine CDM | Verify that Widevine CDM can be disabled and enabled without any issues | Video with Widevine CDM is not playing/playing | Manual | Eng Team |
7 | Disable DRM | Verify that DRM can be disabled and enabled without any issues | You will still be able to view DRM-controlled videos that require Silverlight or Flash plugins if you've enabled them. The Play DRM content setting only controls HTML5 DRM playback, not plugins | Manual | Eng Team |
Builds
- Links for Nightly builds - available since April 14th 2016 - link
- Links for Aurora builds - available since April 21st 2016 - link
- Links for Beta builds - available since April 26th 2016 - link
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | 04/11/2016 | - |
Study documentation/specs received | 04/11/2016 | - |
QA - Test plan creation | 04/12/2016 | - |
QA - Test cases/Env preparation | 04/12/2016 | - |
QA - Nightly Testing | 04/15/2016 | - |
QA - Aurora Testing | 04/22/2016 | - |
QA - Beta Testing | 04/27/2016 | - |
Release Date | - | - |
Testing Tools
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | Google docs |
Test case execution | Google docs |
Bugs management | Bugzilla |
Status
Overview
- Track the dates and build number where feature was released to Nightly
- Track the dates and build number where feature was merged to Aurora
- Track the dates and build number where feature was merged to Release/Beta
Risk analysis
Risk areas | Mitigation |
---|---|
Browser widevine plugin compatibility and usability with content specific websites | - Websites shouldn't offer other video plugins for playing Widevine video compatible content. - If multiple video plugins are installed, Widevine should be prefered when playing supported content. - Websites should not serve Widevine to Firefox by default. |
References
Testcases
Overview
- Summary of testing scenarios
Test Areas
Test Areas | Covered | Details |
---|---|---|
Private Window | Yes | |
Multi-Process Enabled | Yes | |
Multi-process Disabled | Yes | |
Theme (high contrast) | Yes | |
UI | ||
Mouse-only operation | Yes | |
Keyboard-only operation | Yes | This depends on the website |
Display (HiDPI) | Yes | |
Interraction (scroll, zoom) | Yes | |
Usable with a screen reader | No | e.g. with NVDA |
Usability and/or discoverability testing | N/A | Is this feature user friendly |
Help/Support | ||
Help/support interface required | TBD | Make sure link to support/help page exist and is easy reachable. |
Support documents planned(written) | TBD | Make sure support documents are written and are correct. |
Install/Upgrade | ||
Feature upgrades/downgrades data as expected | Yes | |
Does sync work across upgrades | No | Not relevant |
Requires install testing | Yes | separate feature/application installation needed (not only Firefox) |
Affects first-run or onboarding | Yes | Not visibly to the user, but the Widevine and Adobe CDMs are downloaded in the background when Firefox first runs. The CDM downloads might make Firefox sluggish |
Does this affect partner builds? Partner build testing | Yes | We won't have partner builds available for testing in the Nightly or Aurora channels |
Enterprise | Raise up the topic to developers to see if they are expecting to work different on ESR builds | |
Enterprise administration | ||
Network proxies/autoconfig | Yes | Low priority |
ESR behavior changes | ||
Locked preferences | ||
Data Monitoring | ||
Temporary or permanent telemetry monitoring | No | |
Telemetry correctness testing | ||
Server integration testing | ||
Offline and server failure testing | ||
Load testing | Yes | Covered by testing videos in multiple tabs |
Add-ons | If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on. | |
Addon API required? | ||
Comprehensive API testing | ||
Permissions | ||
Testing with existing/popular addons | ||
Security | Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature. | |
3rd-party security review | ||
Privilege escalation testing | ||
Fuzzing | ||
Web Compatibility | depends on the feature | |
Testing against target sites | N/A | |
Survey of many sites for compatibility | ||
Interoperability | depends on the feature | |
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | ||
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | ||
Interaction of this feature with other browser features | Yes |
Test suite
- Full Test suite - In progress
- Smoke Test suite -
- Regression Test suite -
Bug Work
Tracking bug – meta bug 1222845
Bug fix verification
Bug No | Summary | Status | Firefox Version |
---|---|---|---|
1265321 | [Widevine] The Widevine plug-in can't be enabled from the "You must enable DRM..." bar | Verified Fixed | 47, 48 |
1264483 | Hide Widevine CDM download on unsupported systems | Verified Fixed | 47, 48, 49 |
1265049 | Shaka's "Axinom" Widevine videos do not play | Verified Fixed | 47, 48, 49 |
1270689 | Widevine crash in non-virtual thunk to mozilla::WidevineDecryptor::OnSessionClosed when using DASH.js player | Verified Fixed | 47, 48, 49 |
1272627 | [Widevine] Video stops when focus is changed | Verified Fixed | 49 |
Logged bugs
Bug 1265416 - [Widevine] Amazon videos are not always opening using the Widevine CDM plugin
Bug 1265375 - [Widevine] Different plugins are used when playing multiple Amazon videos in different tabs after Widevine was disabled
Bug 1265349 - [Widevine] After the "Widevine" plugin is re-enable, the video does not start
Bug 1265336 - [Widevine] After page refresh the user is instructed to install Microsoft Silverlight plug-in in order to play videos from Amazon Video
Bug 1265321 - [Widevine] The Widevine plug-in can't be enabled from the "You must enable DRM..." bar
Bug 1265320 - [Widevine] HBOGO.RO videos do not play through WidevineCDM
Bug 1267223 - [Widevine] A dotted rectangle is displayed around the volume bar on netflix.com
Bug 1267582 - [Widevine] No clear message that Netflix videos are not running in Private browsing
Bug 1267590 - [Widevine] Reload is not properly done on Amazon videos in non-e10s
Bug 1272627 - [Widevine] Video stops when focus is changed
Bug 1272638 - [Widevine] Netflix - N8109 error is received when switching from Silverlight to Widevine
Bug 1272674 - [Widevine] On Netflix, if start movie from browse page black screen is displayed
Bug 1273525 - Videos are flickering when switching tabs playing with Widevine or Adobe Primetime CDM
Bug 1275562 - [Widevine] Netflix - M7353 error is received when trying to play any video
Sign off
Criteria
Check list
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
List of OSes that will be covered by testing
- Link for the tests run
- Daily Smoke suite -
- Full Test suite -
- Regression Test suite, if needed/available
Merge to Aurora Sign-off
List of OSes that will be covered by testing
- Link for the tests run -
- Full Test suite -
Checklist
Exit Criteria | Status | Notes/Details |
---|---|---|
Testing Prerequisites (specs, use cases) | [IN PROGRESS] | |
Testing Infrastructure setup | No | |
Test Plan Creation | [IN PROGRESS] | |
Test Cases Creation | [IN PROGRESS] | |
Full Functional Tests Execution | [NOT STARTED] | |
Automation Coverage | ||
Performance Testing | ||
All Defects Logged | ||
Critical/Blockers Fixed and Verified | ||
Daily Status Report (email/etherpad statuses/ gdoc with results) | ||
Metrics/Telemetry | N/A | |
QA Signoff - Nightly Release | Email to be sent | |
QA Aurora - Full Testing | ||
QA Signoff - Aurora Release | Email to be sent | |
QA Beta - Full Testing | ||
QA Signoff - Beta Release | Email to be sent |