QA/Block Playback

From MozillaWiki
< QA
Jump to: navigation, search

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
Andrei Vaida QA Manager Product Integrity n/a Email
Alastor Wu Software Engineer Engineering n/a Email
Blake Wu EPM Product Management n/a Email

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
10/17/2016 1.0 Adrian Florinescu Initial Draft
07/24/2017 1.1 Emil Ghitta Updates
08/14/2017 1.2 Emil Ghitta Updates

Overview

This feature's purpose is to block unexpected media from auto-playing in a background tab which hasn't been focused or visited before. The media would be automatically resumed when the tab becomes visible for the first time or when the user presses the Play Tab button (positioned on the opened tab).

Purpose

This document describes the following topics:

  • the test scope, focus areas and objectives
  • the test responsibilities
  • the test strategy for the levels and types of test for this release
  • the entry and exit criteria
  • the basis of the test estimates
  • any risks, issues, assumptions and test dependencies
  • the test schedule and major milestones
  • the test deliverables

Scope

This wiki section details the testing that will be performed by the project team for the Media Block Playback project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document what will be tested and how testing will be performed.

Ownership

Testing summary

Scope of Testing

In Scope

Testing will check the following items for popular websites providing media content:

  • eligible tabs are blocked from auto-playing media
  • video and sound are in sync when the tab is in focus
  • the Play Tab button has the intended behavior
  • the feature works as intended with and without e10s/e10s-multi
  • tabs recover their previous state after being restored (session restore)
  • the feature works as intended during private browsing sessions
  • the feature plays well with a11y
  • the feature plays well with High Contrast themes
  • the feature plays well with HiDPI displays
  • the feature is usable through mouse-only and keyboard-only operations
  • the feature plays well with RTL locales
  • associated Telemetry probes are gathering data
  • the feature "works well" from a user-experience point of view

Out of Scope

Testing will not check:

  • scroll and zoom effects
  • whether the feature works well with screen readers
  • telemetry data correctness
  • compatibility with add-ons

Requirements for testing

Environments

  • Operating systems:
    • Windows 7 x64, Windows 10 x64.
    • macOS 10.12
    • Ubuntu 16.04 x86
  • Devices: n/a
  • Other configuration/environmental setup needed:
    • media.block-autoplay-until-in-foreground needs to be set to true, but should be default-on

Channel dependent settings (configs) and environment setups

There are no special settings needed to test this feature, apart from its pref being set to true.

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 Mouse and keyboard actions are not recognized TO-1 2-Moderate 1-Unlikely 3-High 6
RAC-2 Feature performance affected by session restore TO-1 1- Low 2-Possible 3-High 6
RAC-3 Multiple tabs affect feature performance TO-4 2-Moderate 2-Possible 3-High 12
RAC-4 After visiting for the first time a tab containing a video, the video starts with a delay TO-1 3-High 3-Almost Certain 3-High 27

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

RAC: Refers to the "Risk Assessment and Coverage" item that is covering.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Evaluation Criteria Test Objective Test Type RAC Owners
TO-1 Video playback is not started until tab is visible for the first time Prevent Video playback until the tab is visible Verifying that playback does not start until the tab is focused for the first time Manual RAC-1, RAC-2, RAC-4 Eng Team
TO-2 Video resources are still pre-loaded even though the tab wasn't visited yet In the cases in which video playback is blocked, the resources are still being pre-loaded Verifying that video resources are pre-loaded even tho the autoplay is blocked Manual - Eng Team
TO-3 Notification sounds are blocked also for tabs not yet first visited The sounds from websites such Facebook, Gmail, Hangout, web-Skype are blocked until the tab is visited for first time Verifying that notification sounds are blocked for the newly opened background tabs Manual - Eng Team
TO-4 Performance is acceptable in case of multiple tabs and doesn't crash the browser or media container Media is blocked, Firefox does not crash Verifying that overall performance of the browser and multiple blocked tabs don't result in hangs or crashes Manual RAC-3 Eng Team

Builds

This section should contain links for builds with the feature:

  • Links for Nightly builds - available since November 11th 2016 link
  • Links for Aurora builds - available since March 3rd 2017 link
  • Links for Beta builds - available since August 7th 2017 link

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 10.03.2016
Study documentation/specs received from developers 10.11.2016 10.17.2016
QA - Test plan creation 10.18.2016 10.20.2016
QA - Test cases/Env preparation 10.20.2016 01.11.2017
QA - Nightly Testing 04.03.2017 07.28.2017
QA - Beta Testing 08.07.2017 09.26.2017
Release Date 09.26.2017

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla (bug 1308154)

Status

Overview

References

  • List and links for specs
  • bug 1308154 - [Meta] (block-autoplay-media) [meta] Block autoplay media until the tab is visible at first time

Test Cases

Full testsuites can be found on TestRail at link or in the PDF doc

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled No
Theme (high contrast) Yes
UI
Mouse-only operation Yes
Keyboard-only operation Yes
Display (HiDPI) Yes
Interraction (scroll, zoom) No
Usable with a screen reader No e.g. with NVDA
Usability and/or discoverability testing Yes Is this feature user friendly
RTL build testing Yes
Help/Support
Help/support interface required No Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) No Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected No
Does sync work across upgrades No
Requires install testing No separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding No
Does this affect partner builds? Partner build testing No yes/no options, add comment with details about who will lead testing
Enterprise
Enterprise administration No
Network proxies/autoconfig No
ESR behavior changes No
Locked preferences No
Data Monitoring
Temporary or permanent telemetry monitoring Yes TAB_AUDIO_INDICATOR_USED, TAB_MEDIA_BLOCKING_TIME_MS
Telemetry correctness testing No
Server integration testing No
Offline and server failure testing No
Load testing No
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? No
Comprehensive API testing No
Permissions No
Testing with existing/popular addons No
Security Security is in charge of Matt Wobensmith.
3rd-party security review No
Privilege escalation testing No
Fuzzing No
Web Compatibility depends on the feature
Testing against target sites No
Survey of many sites for compatibility No
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS No
Interaction of this feature with other browser features No

Test Suites

Bug Work

Meta bug: 1308154 - (block-autoplay-media) [meta Block autoplay media until the tab is visible at first time]

Logged bugs ( blocking 1308154 )
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1262053 P3 Audio/Video: Playback Alastor Wu [:alwu] Block playback of media until document is visible VERIFIED mozilla51
1302350 P3 Audio/Video: Playback Alastor Wu [:alwu] Refactor the block-media-element feature. RESOLVED mozilla52
1308153 P3 Audio/Video: Playback Alastor Wu [:alwu] Implement the new UX visual mechanism for blocking autoplay media RESOLVED mozilla52
1308399 P3 Audio/Video: Playback Mark Liang(:mark_liang) [UX] UX Spec for Delay Autoplay RESOLVED ---
1314220 -- Audio/Video: Playback Alastor Wu [:alwu] Add the Telemetry for blocking autoplay media VERIFIED mozilla53
1319526 -- Audio/Video: Playback Media autoplay blocking icon has the wrong tooltip RESOLVED ---
1319529 -- Audio/Video: Playback Alastor Wu [:alwu] The unblocking icon didn't be hided when resuming Flash RESOLVED mozilla53
1319771 -- Audio/Video: Playback Alastor Wu [:alwu] If the pinned tab is the first tab in the tab list, the media won't be blocked after browser restarts VERIFIED mozilla54
1320005 -- Audio/Video: Playback Alastor Wu [:alwu] Don't show 'play tab' icon for video without audio track VERIFIED mozilla53
1322505 P1 Audio/Video: Playback Alastor Wu [:alwu] Don't resume the video after calling video.pause() VERIFIED mozilla53
1328058 -- Audio/Video: Playback Alastor Wu [:alwu] Tab displays "play" button or sound indicator when the only media element is playing and has no sound VERIFIED mozilla54
1329170 P3 Audio/Video: Playback Alastor Wu [:alwu] Play Tab indicator looks pixelated in pinned tabs VERIFIED mozilla54
1329179 -- Audio/Video: Playback Alastor Wu [:alwu] Play Tab indicator is displayed on a foreground tab VERIFIED mozilla54
1330195 -- Audio/Video: Playback Alastor Wu [:alwu] The audio indicator doesn't work for flash on Linux RESOLVED ---
1330232 -- Audio/Video: Playback Alastor Wu [:alwu] Blocked media was resumed with sound when user click "mute tab" in the pop-up menu RESOLVED mozilla53
1330644 -- Audio/Video: Playback No indications that a video with no audio track is blocked RESOLVED ---
1330681 -- Audio/Video: Playback Alastor Wu [:alwu] Play tab indicator has no hover state in pinned tabs VERIFIED mozilla54
1330929 -- Audio/Video: Playback Play tab icon not displayed for certain plugins RESOLVED ---
1331317 -- Audio/Video: Playback Alastor Wu [:alwu] Only enable pref "media.block-autoplay-until-in-foreground" on Nightly VERIFIED mozilla53
1336309 -- Tabbed Browser Alastor Wu [:alwu] Modify the access key for "play tab" option VERIFIED Firefox 54
1337232 -- Audio/Video: Playback Alastor Wu [:alwu] Enable pref "media.block-autoplay-until-in-foreground" by default VERIFIED mozilla54
1337704 -- Audio/Video: Playback Flickery transition between tabs indicators (play tab -> sound) RESOLVED ---
1338137 P1 Audio/Video: Playback Alastor Wu [:alwu] If the first tab is pinned and muted - media is not blocked after browser restart VERIFIED mozilla54
1338145 -- Audio/Video: Playback Social media: notification inconsistent behavior (Play tab indicator + sound) RESOLVED ---
1343465 -- Audio/Video: Playback Alastor Wu [:alwu] The tab should show the "play tab" icon even the tab was muted. RESOLVED mozilla54
1345947 P1 Audio/Video: Playback Alastor Wu [:alwu] The blocked autoplay media icon appears for tabs that I've seen before RESOLVED ---
1347758 P1 Audio/Video: Playback Alastor Wu [:alwu] HTML5 video/audio doesn't play if node was removed in background tab RESOLVED mozilla55
1347791 -- Tabbed Browser Alastor Wu [:alwu] The pinned tab shows play tab icon every time when restarts browser VERIFIED Firefox 56
1348326 P1 Audio/Video: Playback Alastor Wu [:alwu] BBC video/audio fails to load after focusing the video’s tab RESOLVED ---
1348759 -- Audio/Video: Playback Alastor Wu [:alwu] The video/audio on be-at.tv is not working as expected after focusing the video's tab RESOLVED ---
1348859 -- Audio/Video: Playback Alastor Wu [:alwu] Focusing an opened Facebook game tab no longer outputs the audio RESOLVED ---
1348879 -- Audio/Video: Playback Playbutton not visible when opening a video from givemesport.com in a new tab RESOLVED ---
1348909 -- Audio/Video: Playback Focusing the Web Skype tab while being called does not output the call audio RESOLVED ---
1349202 -- Audio/Video: Playback Block Playback feature is not working as intended on several websites RESOLVED ---
1350947 -- Plug-ins Alastor Wu [:alwu] Postpone loading NPAPI Flash content until the tab is resumed RESOLVED ---
1352014 -- Audio/Video: Playback Alastor Wu [:alwu] Add the flag "RELEASE_OR_BETA" for the pref "media.block-autoplay-until-in-foreground" RESOLVED mozilla55
1358211 -- Audio/Video: Playback Blocking autoplay for background tabs (bug 1308154) is breaking twitter videos RESOLVED ---
1367356 -- Plug-ins Alastor Wu [:alwu] Add telemetry to measure how many Flash contents are blocked by blocking autoplay media RESOLVED ---
1378105 P3 Session Restore Media is not blocked for previously unfocused closed tabs after restoring them from the History menu REOPENED ---
1387917 -- Audio/Video: Playback Alastor Wu [:alwu] Turn on the pref "media.block-autoplay-until-in-foreground" by default RESOLVED mozilla57
1394455 P1 Audio/Video: Playback Mike Conley (:mconley) (:⚙️) [e10s] Media is unblocked while hovering an unfocused tab VERIFIED mozilla57
1572798 P2 DOM: Core & HTML Emilio Cobos Álvarez (:emilio) Video element created using document.open() doesn't work RESOLVED 86 Branch
1611748 P3 Audio/Video: Playback Alastor Wu [:alwu] video does not play in a dynamically created iframe (friendly) RESOLVED ---
1684876 -- Tabbed Browser kz04px "Mute Tab" sometimes starts playing the tab's video, with sound RESOLVED 95 Branch
1685163 P3 Audio/Video: Playback Alastor Wu [:alwu] MediaStreamTrack freezes when played in opened popup (used to work if visiting fullscreen) RESOLVED ---
1733902 P3 Audio/Video: Playback Alastor Wu [:alwu] Autoplay fails if the player is switched in a background tab VERIFIED 96 Branch

46 Total; 1 Open (2.17%); 29 Resolved (63.04%); 16 Verified (34.78%);


Verified bug fixes
Full Query
ID Priority Component Assigned to Summary Status Resolution Target milestone
1262053 P3 Audio/Video: Playback Alastor Wu [:alwu] Block playback of media until document is visible VERIFIED FIXED mozilla51
1302350 P3 Audio/Video: Playback Alastor Wu [:alwu] Refactor the block-media-element feature. RESOLVED FIXED mozilla52
1308153 P3 Audio/Video: Playback Alastor Wu [:alwu] Implement the new UX visual mechanism for blocking autoplay media RESOLVED FIXED mozilla52
1308399 P3 Audio/Video: Playback Mark Liang(:mark_liang) [UX] UX Spec for Delay Autoplay RESOLVED FIXED ---
1314220 -- Audio/Video: Playback Alastor Wu [:alwu] Add the Telemetry for blocking autoplay media VERIFIED FIXED mozilla53
1319529 -- Audio/Video: Playback Alastor Wu [:alwu] The unblocking icon didn't be hided when resuming Flash RESOLVED FIXED mozilla53
1319771 -- Audio/Video: Playback Alastor Wu [:alwu] If the pinned tab is the first tab in the tab list, the media won't be blocked after browser restarts VERIFIED FIXED mozilla54
1320005 -- Audio/Video: Playback Alastor Wu [:alwu] Don't show 'play tab' icon for video without audio track VERIFIED FIXED mozilla53
1322505 P1 Audio/Video: Playback Alastor Wu [:alwu] Don't resume the video after calling video.pause() VERIFIED FIXED mozilla53
1328058 -- Audio/Video: Playback Alastor Wu [:alwu] Tab displays "play" button or sound indicator when the only media element is playing and has no sound VERIFIED FIXED mozilla54
1329170 P3 Audio/Video: Playback Alastor Wu [:alwu] Play Tab indicator looks pixelated in pinned tabs VERIFIED FIXED mozilla54
1329179 -- Audio/Video: Playback Alastor Wu [:alwu] Play Tab indicator is displayed on a foreground tab VERIFIED FIXED mozilla54
1330232 -- Audio/Video: Playback Alastor Wu [:alwu] Blocked media was resumed with sound when user click "mute tab" in the pop-up menu RESOLVED FIXED mozilla53
1330681 -- Audio/Video: Playback Alastor Wu [:alwu] Play tab indicator has no hover state in pinned tabs VERIFIED FIXED mozilla54
1331317 -- Audio/Video: Playback Alastor Wu [:alwu] Only enable pref "media.block-autoplay-until-in-foreground" on Nightly VERIFIED FIXED mozilla53
1336309 -- Tabbed Browser Alastor Wu [:alwu] Modify the access key for "play tab" option VERIFIED FIXED Firefox 54
1337232 -- Audio/Video: Playback Alastor Wu [:alwu] Enable pref "media.block-autoplay-until-in-foreground" by default VERIFIED FIXED mozilla54
1338137 P1 Audio/Video: Playback Alastor Wu [:alwu] If the first tab is pinned and muted - media is not blocked after browser restart VERIFIED FIXED mozilla54
1343465 -- Audio/Video: Playback Alastor Wu [:alwu] The tab should show the "play tab" icon even the tab was muted. RESOLVED FIXED mozilla54
1347758 P1 Audio/Video: Playback Alastor Wu [:alwu] HTML5 video/audio doesn't play if node was removed in background tab RESOLVED FIXED mozilla55
1347791 -- Tabbed Browser Alastor Wu [:alwu] The pinned tab shows play tab icon every time when restarts browser VERIFIED FIXED Firefox 56
1352014 -- Audio/Video: Playback Alastor Wu [:alwu] Add the flag "RELEASE_OR_BETA" for the pref "media.block-autoplay-until-in-foreground" RESOLVED FIXED mozilla55
1387917 -- Audio/Video: Playback Alastor Wu [:alwu] Turn on the pref "media.block-autoplay-until-in-foreground" by default RESOLVED FIXED mozilla57
1394455 P1 Audio/Video: Playback Mike Conley (:mconley) (:⚙️) [e10s] Media is unblocked while hovering an unfocused tab VERIFIED FIXED mozilla57
1572798 P2 DOM: Core & HTML Emilio Cobos Álvarez (:emilio) Video element created using document.open() doesn't work RESOLVED FIXED 86 Branch
1684876 -- Tabbed Browser kz04px "Mute Tab" sometimes starts playing the tab's video, with sound RESOLVED FIXED 95 Branch
1733902 P3 Audio/Video: Playback Alastor Wu [:alwu] Autoplay fails if the player is switched in a background tab VERIFIED FIXED 96 Branch

27 Total; 0 Open (0%); 11 Resolved (40.74%); 16 Verified (59.26%);

Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

  • List of OSes that will be covered by testing:
    • Windows 10 64bit
    • Ubuntu 16.04 64bit
    • MacOS 10.11
  • Link for the tests run -link

Merge to Aurora Sign-Off

  • List of OSes that will be covered by testing
    • Windows 10 64bit
    • Windows 7 64bit
    • macOS 10.12
    • Ubuntu 16.04 32bit
  • Link for the tests run -link

Merge to Beta Sign-Off

  • List of OSes that will be covered by testing
    • Windows 10 64bit
    • Windows 7 64bit
    • macOS 10.11.6
    • Ubuntu 16.04 64 bit
  • Link for the tests run - link

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Full Functional Tests Execution [DONE]
Automation Coverage [IN PROGRESS]

http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_block_autoplay_media.js
http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_mediaPlayback_suspended_multipleAudio.js
http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_mediaPlayback_suspended.js

Performance Testing [NOT APPLICABLE]
All Defects Logged [DONE]
Critical/Blockers Fixed and Verified [DONE]
Metrics/Telemetry [DONE]
QA Signoff - Nightly Release [DONE] [Email sent in 2/17/2017]
QA Beta - Full Testing [DONE]
QA Signoff - Beta Release [DONE] [Email sent in 7/28/2017]
QA Signoff - Pre Release [DONE] [Email sent in 9/8/2017]