QA/Webm eme support widevine

From MozillaWiki
< QA
Jump to: navigation, search

WebM EME support for Widevine

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
2016-08-08 1.0 Ciprian Georgiu Created first draft

Overview

Purpose

Detail the purpose of this document. For example:

  • The test scope, focus areas and objectives
  • The test responsibilities
  • The test strategy for the levels and types of test for this release
  • The entry and exit criteria
  • The basis of the test estimates
  • Any risks, issues, assumptions and test dependencies
  • The test schedule and major milestones
  • The test deliverables

Scope

This wiki details the testing that will be performed by the project team for the WebM EME support for Widevine project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • what will be tested,
  • how testing will be performed.

Ownership

Engineering contact(s):

QA contact(s):

Testing summary

Scope of Testing

In Scope

Testing efforts will be focused on confirming that WebM EME is used by default for eligible video content that makes use of the Widevine DRM.

Out of Scope

Functionality of the Widevine Media Optimizer plugin with all available features is not included in the testing efforts. Only Firefox support for the WebM EME Widevine feature is tested.

Requirements for testing

Environments

What?

Encrypted Media Extensions (EME) is a JavaScript API for playing DRMed video content in HTML. A DRM component called a Content Decryption Module (CDM) decrypts, decodes, and displays the video. A JavaScript program coordinates the process using the EME API to pass messages between the CDM and a server that provides decryption keys and content.

Why?

Hollywood studios require companies that license movies for streaming use DRM between the streaming company and the end user. On the Web, this has traditionally been done by using the Microsoft PlayReady DRM component inside the Silverlight plug-in or the Adobe Access DRM component inside the Flash Player. As the Web platform gains more capabilities, general purpose plug-ins like Silverlight or Flash can be phased out.

The Widevine Media Optimizer is a browser plugin designed for the viewing of premium video content. The Widevine plugin is commonly used by premium content providers.


Full Testing will be performed on:

Windows 8
Windows 10
Mac OS X 10.6 or later


Exploratory/Smoke testing will be performed on:

Windows 7

Platform Support

Which streaming services use which DRM on which platforms?

Service Windows 7–10 Windows XP Mac OS X Linux
Amazon Video Widevine CDM Silverlight Widevine CDM Not supported yet

(Widevine CDM on Chrome)

Netflix Adobe CDM Silverlight Silverlight Not supported yet

(Widevine CDM on Chrome)

YouTube Widevine CDM Flash Access Widevine CDM Not supported yet

(Widevine CDM on Chrome)

Other configuration/environmental setup needed:

  • media.mediasource.mp4.enabled - set to false
  • media.mediasource.webm.enabled - set to true

Test Strategy

Test Objectives

Tbd.

Builds

The feature is available on Firefox 50 starting with 2016-07-31.

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 2016-08-08 2016-11-08
Study documentation/specs received from developers 2016-08-08 2016-08-09
QA - Test plan creation 2016-08-08 2016-08-10
QA - Test cases/Env preparation 2016-08-08 2016-08-08
QA - Nightly Testing N/A N/A
QA - Aurora Testing 2016-08-09 2016-09-14
QA - Beta Testing 2016-10-25 2016-10-28
Release Date 2016-11-08

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail
Test case execution TestRail
Bug management Bugzilla

Status

Overview

  • Landed in Nightly 50 on 2016-07-31.
  • Landed in Aurora 50 on 2016-08-03.
  • Landed in Beta 50 on 2016-09-20.
  • Shipped with Firefox 50 on Tbd.

Risk analysis

  • Identify the high-risk assumptions.
  • Identify existing bugs on the feature with high risk.
  • Identify if other areas are affected by the fix.

References

  • List and links for specs
 - https://en.wikipedia.org/wiki/Encrypted_Media_Extensions - Encrypted Media Extensions

Test cases

Overview

The support for the QA/Webm EME feature was added in Nightly with Bug 1279077 [1] which landed in 2016-07-31. First available in this build.
Track the dates and build number where feature was merged to Aurora.
Track the dates and build number where feature was merged to Release/Beta.

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes
UI
Mouse-only operation Yes
Keyboard-only operation Yes
Display (HiDPI) Yes
Interraction (scroll, zoom) Yes
Usability and/or discoverability testing Yes Is this feature user friendly
Does this affect partner builds? Partner build testing Yes yes/no options, add comment with details about who will lead testing
Web Compatibility depends on the feature
Testing against target sites Yes
Survey of many sites for compatibility Yes
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. Yes
Coordinated testing/interop across the Firefox products: Desktop, Android, iOS
Interaction of this feature with other browser features Yes

Test suite

Webm eme support widevine - Mid-Aurora/Pre-Beta Sign off (Fx50) testsuite in [2] (need Testrail account)

Bug Work

Tracking bug – meta bug.

Bug fix verification

Tbd.


Logged bugs
Full Query
ID Summary Status Resolution
1298027 [EME] Video playback fails when page with EME video is moved into new window RESOLVED FIXED
1298030 [EME] Video is stuck on Youtube when is moved into New window RESOLVED DUPLICATE
1298031 [W10] WebM is not being used via the 'Stats for nerds' on YouTube RESOLVED INVALID
1301423 [EME] Manually forwarding the video to it's end, gives an error which is thrown on top of the video NEW

4 Total; 1 Open (25%); 3 Resolved (75%); 0 Verified (0%);


Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

N/A.

Mid-Aurora Sign-off

List of OSes that will be covered by testing: Windows 7, Windows 10, Ubuntu 16.04 LTS, Mac OS 10.11.

Pre-Beta Sign-off

List of OSes that will be covered by testing: Windows 7, Windows 10, Ubuntu 16.04 LTS, macOS 10.12 Sierra (Beta).

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Full Functional Tests Execution [DONE]
Automation Coverage TBD
Performance Testing N/A
All Defects Logged [DONE]
Critical/Blockers Fixed and Verified [DONE]
Metrics/Telemetry N/A
QA Signoff - Nightly Release MISSED
QA Aurora - Full Testing [DONE] (Yellow) Sign off email sent on 26/08/2016
QA Signoff - Aurora Release [DONE] (Green) Sign off email sent on 14/09/2016
QA Beta - Full Testing [DONE]
QA Signoff - Beta Release [DONE] (Green) Sign off email sent on 01/11/2016
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Webm_eme_support_widevine&oldid=1154241"