QA/Block Playback: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
No edit summary
Line 1: Line 1:
'''Approvals Required / Received'''
The following individuals are required to/have approved this Test Plan:
{| class="wikitable"
|-
! Name !! Title !! Department !! Approval Date !! Method
|-
| Andrei Vaida || QA Manager || Product Integrity || Date || [mailto:andrei.vaida@softvisioninc.eu Email]
|-
| Alastor Wu || Software Engineer || Engineering || Date || [mailto:alwu@mozilla.com Email]
|-
| Blake Wu || EPM || Product Management || Date || [mailto:bwu@mozilla.com Email]
|}
'''Revision History'''
'''Revision History'''


This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.


{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
Line 10: Line 25:
|-
|-
| 07/24/2017|| 1.1 || Emil Ghitta ||  Updates  
| 07/24/2017|| 1.1 || Emil Ghitta ||  Updates  
|-
| 08/14/2017|| 1.2 || Emil Ghitta ||  Updates
|}
|}


= Overview =
= Overview =
== Purpose ==
== Purpose ==
Detail the purpose of this document. For example:
This feature's main purpose is to block unexpected autoplay media from a newly opened media tab (opened in a background tab) which hasn't been focused or visited before. The media would be automatically resumed when the tab becomes visible for the first time or when the user presses the Play Tab button (positioned on the opened tab).
* The test scope, focus areas and objectives
 
* The test responsibilities
This changes will bring a benefit to the user by removing the possibility of having a discomfort generated by hearing an unexpected sound which comes from a newly opened media tab which was opened in a background tab (without being focused) or which hasn't been visited previously on.
* The test strategy for the levels and types of test for this release
* The entry and exit criteria
* The basis of the test estimates
* Any risks, issues, assumptions and test dependencies
* The test schedule and major milestones
* The test deliverables


== Scope ==
== Scope ==
This wiki details the testing that will be performed by the project team for the Block Playback project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
This wiki section details the testing that will be performed by the project team for the Media Block Playback project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
* What will be tested
* What will be tested:
The main testing focus is to assure that this feature works as intended by blocking the autplay media from media tabs that were opened in the background without being focused or if the weren't visited before.
The testing activities will also focus:
 
-Verifying if the video and sound are in sync when the tab is in focus.
 
-Verifying if the Play Tab button functionality is working as intended.
 
-Verifying if the tabs recovers their previous state after session restore.
 
-Verifying if this feature is "accessibly friendly".
 
* How testing will be performed
* How testing will be performed
Testing this feature will be done throw Manual Testing that will cover all known risk areas.
Each testing cycle will cover the following testing activities:
-Smoke testing
-Exploratory testing
-TC execution
-Regression testing


== Ownership ==
== Ownership ==
Line 59: Line 91:
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
The Block Playback feature is designed to block media with enabled autoplay if the tabs are opened in the background. Videos which autoplay in the background will now have their load deferred until the tab is visible for the first time -- this avoids autoplay during session restore and premature playback. Resources will still be preloaded but Firefox will delay the start of playback until the tab that contin a video with an enabled autoplay is visited. Once a tab / RenderFrame has ever played media before, it's allowed to continue to autoplay/autoload indefinitely; this is to support playlist type applications. The main advantages of the Block Playback feature is that it prevents obviously user annoyance but also conserves power as Firefox will only consume power once the tab is foregrounded.
Private Browsing -Check if this feature works as intended on private tabs.
 
Multi-Process Enabled/Disabled - Check if this feature works as intended with both e10s enabled and e10s disabled.
 
High Contrast themes -Check if this feature works and the Play Tab button is visible while using High Contrast themes.
 
Mouse Only operations-Check if this feature works as intended while using Mouse only operations to activate the Play Tab button or open/navigate throw tabs.
 
Keyboard-only operations-Check if this feature works as intended while using Keyboard only operations to navigate throw tabs.
 
Display(HiDPI)-Check if this feature works and the Play tab button is visible while using HiDPI.
 
Usability-Check if this feature is easy to use.
 
RTL build compat-Check if this feature works as intended and the Play Tab button is correctly displayed while using different RTL builds.
 
Telemetry monitoring-Monitoring the telemetry to check if it works as intended.


=== Out of Scope ===
=== Out of Scope ===
Detail what is out of scope from a testing perspective for the project team. Note: if usability testing is not in the scope of testing feature.
Interactions (scroll,zoom)
 
Usable with screen reader
 
Installation testing
 
Telemetry correctness testing
 
Comprehensive API testing
 
Testing with addons
 
Privilege escalation testing
 
Testing against target sites
 
 


= Requirements for testing =
= Requirements for testing =
Line 71: Line 135:
* Ubuntu 16.04 x86
* Ubuntu 16.04 x86


== Channel dependent settings (configs) and environment setups ==
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
=== Nightly ===
<div class="mw-collapsible-content">
This feature is enabled by default in Firefox Nightly 57
</div>
=== Beta ===
<div class="mw-collapsible-content">
This feature is enabled by default in Firefox Beta 56
</div>


Other configuration/environmental setup needed:
=== Post Beta / Release ===
* In Firefox configuration settings ''media.block-autoplay-until-in-foreground'' preference needs to be set to ''true''.
<div class="mw-collapsible-content">
In Firefox configuration settings ''media.block-autoplay-until-in-foreground'' preference needs to be set to ''true''.
</div>
</div>


= Test Strategy =  
= Test Strategy =  
== Risk Assessment and Coverage ==
{| class="wikitable"
|-
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score
|-
| RAC-1 || Mouse and keyboard actions may not be recognized || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
|-
| RAC-2 || Restoring after a crash or after a session may impact the performance of this feature || TO-1 || 1- Low || 2-Possible || 3-High || 6
|-
| RAC-3 || Possible performance issues with multiple tabs || TO-4 || 2-Moderate || 2-Possible || 3-High || 12
|-
| RAC-4 || After visiting for the first time a tab containing a video, the video starts with a delay || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
|-
|}
'''Values:'''
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''
'''Impact Score Breakdown:'''
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
'''RAC:''' Refers to the "Risk Assessment and Coverage" item that is covering.
== Test Objectives ==
== Test Objectives ==
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
Line 93: Line 204:
|-
|-
|}
|}
'''Impact Score Breakdown:'''
* An impact value of 1 or 2 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 4, 8 or 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
'''RAC:''' Refers to the "Risk Assessment and Coverage" item that is covering.


== Builds ==
== Builds ==
Line 106: Line 209:
* Links for Nightly builds - available since November 11th 2016 [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-12-03-02-03-mozilla-central/ link]  
* Links for Nightly builds - available since November 11th 2016 [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-12-03-02-03-mozilla-central/ link]  
* Links for Aurora builds - available since March 3rd 2017 [https://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-07-07-04-55-mozilla-aurora/ link]
* Links for Aurora builds - available since March 3rd 2017 [https://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-07-07-04-55-mozilla-aurora/ link]
* Links for Beta builds - TBD
* Links for Beta builds - available since August 7th 2017 [https://archive.mozilla.org/pub/firefox/candidates/56.0b1-candidates/build1/ link]


== Test Execution Schedule ==
== Test Execution Schedule ==
Line 130: Line 233:
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | TBD || TBD
|style="text-align:center;" | 08.07.2017 || 09.26.2017
|-
|-
| Release Date  
| Release Date  
Line 142: Line 245:
! Process !! Tool
! Process !! Tool
|-
|-
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki [https://wiki.mozilla.org/QA/Block_Playback]
|-
|-
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc TestRail]/ Google docs
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc TestRail]/ Google docs
Line 148: Line 251:
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc TestRail]
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc TestRail]
|-
|-
| Bugs management || Bugzilla
| Bugs management || Bugzilla [https://bugzilla.mozilla.org/show_bug.cgi?id=1308154]
|}
|}


= Status =  
= Status =  
(01.23.2017) - For the time being Block  Playback feature will be kept enabled only on Nightly - [https://bugzilla.mozilla.org/show_bug.cgi?id=1331317 Bug 1331317].
(03.07.2017) - Block Playback feature has reached Developer Edition 54.0a2.


== Overview ==
== Overview ==
  Track the dates and build number where feature was released to Nightly - [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-12-03-02-03-mozilla-central/-12) 52.0a1 (2016-11-12)]
  Firefox was released to Nightly - [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-12-03-02-03-mozilla-central/-12) 52.0a1 (2016-11-12)]
  Track the dates and build number where feature was merged to Aurora - [https://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-07-07-04-55-mozilla-aurora/  54.0a2 (2017-03-07)]
  Firefox was merged to Aurora - [https://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-07-07-04-55-mozilla-aurora/  54.0a2 (2017-03-07)]
  Track the dates and build number where feature was merged to Release/Beta
  Firefox was merged to Beta - [https://archive.mozilla.org/pub/firefox/candidates/56.0b1-candidates/build1/ 56.0b1 (2017-08-07)]
 
Firefox was merged to Release - TBD
== Risk Assessment and Coverage ==
 
{| class="wikitable"
|-
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score
|-
| RAC-1 || Mouse and keyboard actions may not be recognized || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
|-
| RAC-2 || Restoring after a crash or after a session may impact the performance of this feature || TO-1 || 1- Low || 2-Possible || 3-High || 6
|-
| RAC-3 || Possible performance issues with multiple tabs || TO-4 || 2-Moderate || 2-Possible || 3-High || 12
|-
| RAC-4 || After visiting for the first time a tab containing a video, the video starts with a delay || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
|-
|}
 
'''Values:'''
 
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
 
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
 
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''
 
'''Impact Score Breakdown:'''
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.


= References =
= References =
* List and links for specs
* List and links for specs
   https://wiki.mozilla.org/Firefox/Block_Playback
   [https://mozilla.invisionapp.com/share/K48PCVSEM#/screens/190748310 UX spec]
  https://mozilla.invisionapp.com/share/K48PCVSEM#/screens/190748310


*  {{Bug|1308154}} - [Meta] (block-autoplay-media) [meta] Block autoplay media until the tab is visible at first time
*  {{Bug|1308154}} - [Meta] (block-autoplay-media) [meta] Block autoplay media until the tab is visible at first time
Line 198: Line 270:
= Testcases =  
= Testcases =  
Full testsuites can be found on TestRail at [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc link] or in the [https://wiki.mozilla.org/images/0/05/Block_PLayback_-EngQA-_-_TestRail.pdf PDF doc]
Full testsuites can be found on TestRail at [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc link] or in the [https://wiki.mozilla.org/images/0/05/Block_PLayback_-EngQA-_-_TestRail.pdf PDF doc]
== Overview ==
Summary of testing scenarios


== Test Areas ==
== Test Areas ==
Line 266: Line 335:
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | TBD || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" | No ||  
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
Line 273: Line 342:
|-
|-
| ''' Enterprise '''  
| ''' Enterprise '''  
||  || Raise up the topic to developers to see if they are expecting to work different on ESR builds
||  ||  
|-
|-
| Enterprise administration   
| Enterprise administration   
Line 324: Line 393:
|-
|-
| ''' Security '''  
| ''' Security '''  
||  || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
||  || Security is in charge of Matt Wobensmith.
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" | TBD ||  
|style="text-align:center;" | No ||  
|-
|-
| Privilege escalation testing
| Privilege escalation testing
Line 340: Line 409:
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" | NO ||  
|style="text-align:center;" | No ||  
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" | NO ||  
|style="text-align:center;" | No ||  


|-
|-
Line 405: Line 474:
'''Nightly testing'''<br />
'''Nightly testing'''<br />


List of OSes that will be covered by testing<br />
List of OSes that will be covered by testing:<br />
 
-Windows 10 64bit
-Ubuntu 16.04 64bit
-MacOS 10.11
 
*Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/1688 link]
*Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/1688 link]


'''Merge to Aurora Sign-Off'''
'''Merge to Aurora Sign-Off'''
List of OSes that will be covered by testing<br />
List of OSes that will be covered by testing<br />
-Windows 10 64bit
-Windows 7 64bit
-macOS 10.12
-Ubuntu 16.04 32bit
*Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/3101 link]
*Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/3101 link]


'''Merge to Beta Sign-Off'''
'''Merge to Beta Sign-Off'''
List of OSes that will be covered by testing<br />
List of OSes that will be covered by testing<br />
*Link for the tests run - TBD
-Windows 10 64bit
-Windows 7 64bit
-macOS 10.11.6
-Ubuntu 16.04 64 bit
 
*Link for the tests run - [https://testrail.stage.mozaws.net/index.php?/reports/view/474 link]


== Checklist ==
== Checklist ==
Line 459: Line 543:
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" | {{mdone|}}
|| [[https://wiki.mozilla.org/images/b/b9/-desktop--feature-_Block_Playback_-_Nightly_54.0a1_Sign-off.pdf  Email sent in 2/17/2017]]
|| [[https://wiki.mozilla.org/images/b/b9/-desktop--feature-_Block_Playback_-_Nightly_54.0a1_Sign-off.pdf  Email sent in 2/17/2017]]
|-
|  QA Aurora - Full Testing
||  ||
|-
|  QA Signoff  - Aurora Release ||
|| Email to be sent
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
| | ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  QA Signoff  - Beta Release ||  
|  QA Signoff  - Beta Release ||  
| | Email to be sent  
  {{mdone|}}
|| [[https://drive.google.com/open?id=0B94iuIVMr-TEaDZmcnVqSGZwUDg  Email sent in 7/28/2017]]
|}
|}

Revision as of 11:39, 14 August 2017

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
Andrei Vaida QA Manager Product Integrity Date Email
Alastor Wu Software Engineer Engineering Date Email
Blake Wu EPM Product Management Date Email

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
10/17/2016 1.0 Adrian Florinescu Initial Draft
07/24/2017 1.1 Emil Ghitta Updates
08/14/2017 1.2 Emil Ghitta Updates

Overview

Purpose

This feature's main purpose is to block unexpected autoplay media from a newly opened media tab (opened in a background tab) which hasn't been focused or visited before. The media would be automatically resumed when the tab becomes visible for the first time or when the user presses the Play Tab button (positioned on the opened tab).

This changes will bring a benefit to the user by removing the possibility of having a discomfort generated by hearing an unexpected sound which comes from a newly opened media tab which was opened in a background tab (without being focused) or which hasn't been visited previously on.

Scope

This wiki section details the testing that will be performed by the project team for the Media Block Playback project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested:

The main testing focus is to assure that this feature works as intended by blocking the autplay media from media tabs that were opened in the background without being focused or if the weren't visited before. The testing activities will also focus:

-Verifying if the video and sound are in sync when the tab is in focus.

-Verifying if the Play Tab button functionality is working as intended.

-Verifying if the tabs recovers their previous state after session restore.

-Verifying if this feature is "accessibly friendly".

  • How testing will be performed

Testing this feature will be done throw Manual Testing that will cover all known risk areas. Each testing cycle will cover the following testing activities:

-Smoke testing

-Exploratory testing

-TC execution

-Regression testing

Ownership

 Engineering Manager: 
 Blake Wu

 Engineering Program Manager: 
 Bobby Chien

 Eng.:
 Alastor Wu
 JW Wang
 Tzuhao Kuo

 UX:
 Mark Liang
 Morpheus Chen

 QA Engineering Team - Nightly Testing
 Rares Bologa (irc: RaresB) - PM for QA team 
 Brindusa Tot (irc: brindusat) - leading QA efforts 
 Simona Badau (irc: simonab) - QA

 QA Release Team - Aurora/Beta/Release Testing
 Andrei Vaida (irc: avaida) - TL for QA team 
 Emil Ghitta (irc: emilghitta) - QA
 Bogdan Maris (irc: bogdan_maris) - QA peer

Testing summary

Scope of Testing

In Scope

Private Browsing -Check if this feature works as intended on private tabs.

Multi-Process Enabled/Disabled - Check if this feature works as intended with both e10s enabled and e10s disabled.

High Contrast themes -Check if this feature works and the Play Tab button is visible while using High Contrast themes.

Mouse Only operations-Check if this feature works as intended while using Mouse only operations to activate the Play Tab button or open/navigate throw tabs.

Keyboard-only operations-Check if this feature works as intended while using Keyboard only operations to navigate throw tabs.

Display(HiDPI)-Check if this feature works and the Play tab button is visible while using HiDPI.

Usability-Check if this feature is easy to use.

RTL build compat-Check if this feature works as intended and the Play Tab button is correctly displayed while using different RTL builds.

Telemetry monitoring-Monitoring the telemetry to check if it works as intended.

Out of Scope

Interactions (scroll,zoom)

Usable with screen reader

Installation testing

Telemetry correctness testing

Comprehensive API testing

Testing with addons

Privilege escalation testing

Testing against target sites


Requirements for testing

Environments

Full testing will be performed on the following OSes:

  • Windows 7 x64, Windows 10 x64.
  • macOS 10.12
  • Ubuntu 16.04 x86

Channel dependent settings (configs) and environment setups

Nightly

This feature is enabled by default in Firefox Nightly 57

Beta

This feature is enabled by default in Firefox Beta 56

Post Beta / Release

In Firefox configuration settings media.block-autoplay-until-in-foreground preference needs to be set to true.

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 Mouse and keyboard actions may not be recognized TO-1 2-Moderate 1-Unlikely 3-High 6
RAC-2 Restoring after a crash or after a session may impact the performance of this feature TO-1 1- Low 2-Possible 3-High 6
RAC-3 Possible performance issues with multiple tabs TO-4 2-Moderate 2-Possible 3-High 12
RAC-4 After visiting for the first time a tab containing a video, the video starts with a delay TO-1 3-High 3-Almost Certain 3-High 27

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

RAC: Refers to the "Risk Assessment and Coverage" item that is covering.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Test Type RAC Owners
TO-1 Video playback is not started until tab is visible for the first time Prevent Video playback until the tab is visible Manual RAC-1, RAC-2, RAC-4 Eng Team
TO-2 Video resources are still pre-loaded even though the tab wasn't visited yet In the cases in which video playback is blocked, the resources are still being pre-loaded Manual - Eng Team
TO-3 Notification sounds are blocked also for tabs not yet first visited The sounds from websites such Facebook, Gmail, Hangout, web-Skype are blocked until the tab is visited for first time Manual - Eng Team
TO-4 Performance is acceptable in case of multiple tabs and doesn't crash the browser or media container TBD what are the expectation in terms of "acceptable" Manual RAC-3 Eng Team

Builds

This section should contain links for builds with the feature:

  • Links for Nightly builds - available since November 11th 2016 link
  • Links for Aurora builds - available since March 3rd 2017 link
  • Links for Beta builds - available since August 7th 2017 link

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 10.03.2016
Study documentation/specs received from developers 10.11.2016 10.17.2016
QA - Test plan creation 10.18.2016 10.20.2016
QA - Test cases/Env preparation 10.20.2016 01.11.2017
QA - Nightly Testing 04.03.2017 07.28.2017
QA - Beta Testing 08.07.2017 09.26.2017
Release Date TBD TBD

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki [1]
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla [2]

Status

Overview

Firefox was released to Nightly - 52.0a1 (2016-11-12)
Firefox was merged to Aurora - 54.0a2 (2017-03-07)
Firefox was merged to Beta - 56.0b1 (2017-08-07)
Firefox was merged to Release - TBD

References

  • List and links for specs
 UX spec
  • bug 1308154 - [Meta] (block-autoplay-media) [meta] Block autoplay media until the tab is visible at first time

Testcases

Full testsuites can be found on TestRail at link or in the PDF doc

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled No
Theme (high contrast) Yes
UI
Mouse-only operation Yes
Keyboard-only operation Yes
Display (HiDPI) Yes
Interraction (scroll, zoom) No
Usable with a screen reader No e.g. with NVDA
Usability and/or discoverability testing Yes Is this feature user friendly
RTL build testing Yes
Help/Support
Help/support interface required No Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) No Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected No
Does sync work across upgrades No
Requires install testing No separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding No
Does this affect partner builds? Partner build testing No yes/no options, add comment with details about who will lead testing
Enterprise
Enterprise administration No
Network proxies/autoconfig No
ESR behavior changes No
Locked preferences No
Data Monitoring
Temporary or permanent telemetry monitoring Yes TAB_AUDIO_INDICATOR_USED, TAB_MEDIA_BLOCKING_TIME_MS
Telemetry correctness testing No
Server integration testing No
Offline and server failure testing No
Load testing No
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? No
Comprehensive API testing No
Permissions No
Testing with existing/popular addons No
Security Security is in charge of Matt Wobensmith.
3rd-party security review No
Privilege escalation testing No
Fuzzing No
Web Compatibility depends on the feature
Testing against target sites No
Survey of many sites for compatibility No
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS No
Interaction of this feature with other browser features No

Test suite

Full Test suite -  TestRail link
Smoke Test suite - Gdoc link
Regression Test suite - Link with the gdoc - if available/needed.

Bug Work

Meta bug: 1308154 - (block-autoplay-media) [meta Block autoplay media until the tab is visible at first time]

Logged bugs ( blocking 1308154 )
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1262053 P3 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Block playback of media until document is visible VERIFIED mozilla51
1302350 P3 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Refactor the block-media-element feature. RESOLVED mozilla52
1308153 P3 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Implement the new UX visual mechanism for blocking autoplay media RESOLVED mozilla52
1308399 P3 Audio/Video: Playback Mark Liang(:mark_liang) [UX] UX Spec for Delay Autoplay RESOLVED ---
1314220 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Add the Telemetry for blocking autoplay media VERIFIED mozilla53
1319526 -- Audio/Video: Playback Media autoplay blocking icon has the wrong tooltip RESOLVED ---
1319529 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] The unblocking icon didn't be hided when resuming Flash RESOLVED mozilla53
1319771 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] If the pinned tab is the first tab in the tab list, the media won't be blocked after browser restarts VERIFIED mozilla54
1320005 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Don't show 'play tab' icon for video without audio track VERIFIED mozilla53
1322505 P1 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Don't resume the video after calling video.pause() VERIFIED mozilla53
1328058 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Tab displays "play" button or sound indicator when the only media element is playing and has no sound VERIFIED mozilla54
1329170 P3 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Play Tab indicator looks pixelated in pinned tabs VERIFIED mozilla54
1329179 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Play Tab indicator is displayed on a foreground tab VERIFIED mozilla54
1330195 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] The audio indicator doesn't work for flash on Linux RESOLVED ---
1330232 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Blocked media was resumed with sound when user click "mute tab" in the pop-up menu RESOLVED mozilla53
1330644 -- Audio/Video: Playback No indications that a video with no audio track is blocked RESOLVED ---
1330681 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Play tab indicator has no hover state in pinned tabs VERIFIED mozilla54
1330929 -- Audio/Video: Playback Play tab icon not displayed for certain plugins RESOLVED ---
1331317 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Only enable pref "media.block-autoplay-until-in-foreground" on Nightly VERIFIED mozilla53
1336309 -- Tabbed Browser Alastor Wu [:alwu][PTO, back on 12/15] Modify the access key for "play tab" option VERIFIED Firefox 54
1337232 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Enable pref "media.block-autoplay-until-in-foreground" by default VERIFIED mozilla54
1337704 -- Audio/Video: Playback Flickery transition between tabs indicators (play tab -> sound) RESOLVED ---
1338137 P1 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] If the first tab is pinned and muted - media is not blocked after browser restart VERIFIED mozilla54
1338145 -- Audio/Video: Playback Social media: notification inconsistent behavior (Play tab indicator + sound) RESOLVED ---
1343465 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] The tab should show the "play tab" icon even the tab was muted. RESOLVED mozilla54
1345947 P1 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] The blocked autoplay media icon appears for tabs that I've seen before RESOLVED ---
1347758 P1 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] HTML5 video/audio doesn't play if node was removed in background tab RESOLVED mozilla55
1347791 -- Tabbed Browser Alastor Wu [:alwu][PTO, back on 12/15] The pinned tab shows play tab icon every time when restarts browser VERIFIED Firefox 56
1348326 P1 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] BBC video/audio fails to load after focusing the video’s tab RESOLVED ---
1348759 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] The video/audio on be-at.tv is not working as expected after focusing the video's tab RESOLVED ---
1348859 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Focusing an opened Facebook game tab no longer outputs the audio RESOLVED ---
1348879 -- Audio/Video: Playback Playbutton not visible when opening a video from givemesport.com in a new tab RESOLVED ---
1348909 -- Audio/Video: Playback Focusing the Web Skype tab while being called does not output the call audio RESOLVED ---
1349202 -- Audio/Video: Playback Block Playback feature is not working as intended on several websites RESOLVED ---
1350947 -- Plug-ins Alastor Wu [:alwu][PTO, back on 12/15] Postpone loading NPAPI Flash content until the tab is resumed RESOLVED ---
1352014 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Add the flag "RELEASE_OR_BETA" for the pref "media.block-autoplay-until-in-foreground" RESOLVED mozilla55
1358211 -- Audio/Video: Playback Blocking autoplay for background tabs (bug 1308154) is breaking twitter videos RESOLVED ---
1367356 -- Plug-ins Alastor Wu [:alwu][PTO, back on 12/15] Add telemetry to measure how many Flash contents are blocked by blocking autoplay media RESOLVED ---
1378105 P3 Session Restore Media is not blocked for previously unfocused closed tabs after restoring them from the History menu RESOLVED ---
1387917 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Turn on the pref "media.block-autoplay-until-in-foreground" by default RESOLVED mozilla57
1394455 P1 Audio/Video: Playback Mike Conley (:mconley) (:⚙️) [e10s] Media is unblocked while hovering an unfocused tab VERIFIED mozilla57
1572798 P2 DOM: Core & HTML Emilio Cobos Álvarez [:emilio] Video element created using document.open() doesn't work RESOLVED 86 Branch
1611748 P3 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] video does not play in a dynamically created iframe (friendly) RESOLVED ---
1684876 -- Tabbed Browser kz04px "Mute Tab" sometimes starts playing the tab's video, with sound RESOLVED 95 Branch
1685163 P3 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] MediaStreamTrack freezes when played in opened popup (used to work if visiting fullscreen) RESOLVED ---
1733902 P3 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Autoplay fails if the player is switched in a background tab VERIFIED 96 Branch

46 Total; 0 Open (0%); 30 Resolved (65.22%); 16 Verified (34.78%);


Bug fix verification
Full Query
ID Priority Component Assigned to Summary Status Resolution Target milestone
1262053 P3 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Block playback of media until document is visible VERIFIED FIXED mozilla51
1302350 P3 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Refactor the block-media-element feature. RESOLVED FIXED mozilla52
1308153 P3 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Implement the new UX visual mechanism for blocking autoplay media RESOLVED FIXED mozilla52
1308399 P3 Audio/Video: Playback Mark Liang(:mark_liang) [UX] UX Spec for Delay Autoplay RESOLVED FIXED ---
1314220 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Add the Telemetry for blocking autoplay media VERIFIED FIXED mozilla53
1319529 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] The unblocking icon didn't be hided when resuming Flash RESOLVED FIXED mozilla53
1319771 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] If the pinned tab is the first tab in the tab list, the media won't be blocked after browser restarts VERIFIED FIXED mozilla54
1320005 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Don't show 'play tab' icon for video without audio track VERIFIED FIXED mozilla53
1322505 P1 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Don't resume the video after calling video.pause() VERIFIED FIXED mozilla53
1328058 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Tab displays "play" button or sound indicator when the only media element is playing and has no sound VERIFIED FIXED mozilla54
1329170 P3 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Play Tab indicator looks pixelated in pinned tabs VERIFIED FIXED mozilla54
1329179 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Play Tab indicator is displayed on a foreground tab VERIFIED FIXED mozilla54
1330232 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Blocked media was resumed with sound when user click "mute tab" in the pop-up menu RESOLVED FIXED mozilla53
1330681 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Play tab indicator has no hover state in pinned tabs VERIFIED FIXED mozilla54
1331317 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Only enable pref "media.block-autoplay-until-in-foreground" on Nightly VERIFIED FIXED mozilla53
1336309 -- Tabbed Browser Alastor Wu [:alwu][PTO, back on 12/15] Modify the access key for "play tab" option VERIFIED FIXED Firefox 54
1337232 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Enable pref "media.block-autoplay-until-in-foreground" by default VERIFIED FIXED mozilla54
1338137 P1 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] If the first tab is pinned and muted - media is not blocked after browser restart VERIFIED FIXED mozilla54
1343465 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] The tab should show the "play tab" icon even the tab was muted. RESOLVED FIXED mozilla54
1347758 P1 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] HTML5 video/audio doesn't play if node was removed in background tab RESOLVED FIXED mozilla55
1347791 -- Tabbed Browser Alastor Wu [:alwu][PTO, back on 12/15] The pinned tab shows play tab icon every time when restarts browser VERIFIED FIXED Firefox 56
1352014 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Add the flag "RELEASE_OR_BETA" for the pref "media.block-autoplay-until-in-foreground" RESOLVED FIXED mozilla55
1387917 -- Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Turn on the pref "media.block-autoplay-until-in-foreground" by default RESOLVED FIXED mozilla57
1394455 P1 Audio/Video: Playback Mike Conley (:mconley) (:⚙️) [e10s] Media is unblocked while hovering an unfocused tab VERIFIED FIXED mozilla57
1572798 P2 DOM: Core & HTML Emilio Cobos Álvarez [:emilio] Video element created using document.open() doesn't work RESOLVED FIXED 86 Branch
1684876 -- Tabbed Browser kz04px "Mute Tab" sometimes starts playing the tab's video, with sound RESOLVED FIXED 95 Branch
1733902 P3 Audio/Video: Playback Alastor Wu [:alwu][PTO, back on 12/15] Autoplay fails if the player is switched in a background tab VERIFIED FIXED 96 Branch

27 Total; 0 Open (0%); 11 Resolved (40.74%); 16 Verified (59.26%);

Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing:

-Windows 10 64bit
-Ubuntu 16.04 64bit
-MacOS 10.11
  • Link for the tests run -link

Merge to Aurora Sign-Off List of OSes that will be covered by testing

-Windows 10 64bit
-Windows 7 64bit
-macOS 10.12
-Ubuntu 16.04 32bit
  • Link for the tests run -link

Merge to Beta Sign-Off List of OSes that will be covered by testing

-Windows 10 64bit
-Windows 7 64bit
-macOS 10.11.6
-Ubuntu 16.04 64 bit
  • Link for the tests run - link

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Full Functional Tests Execution [DONE]
Automation Coverage IN PROGRESS

http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_block_autoplay_media.js
http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_mediaPlayback_suspended_multipleAudio.js
http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_mediaPlayback_suspended.js

Performance Testing N/A
All Defects Logged [DONE]
Critical/Blockers Fixed and Verified [DONE]
Metrics/Telemetry [DONE]
QA Signoff - Nightly Release [DONE] [Email sent in 2/17/2017]
QA Beta - Full Testing [DONE]
QA Signoff - Beta Release 
 [DONE]
[Email sent in 7/28/2017]
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Block_Playback&oldid=1178085"