QA/Block Playback: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
(-)
 
(20 intermediate revisions by 5 users not shown)
Line 1: Line 1:
'''Approvals Required / Received'''
The following individuals are required to/have approved this Test Plan:
{| class="wikitable"
|-
! Name !! Title !! Department !! Approval Date !! Method
|-
| Andrei Vaida || QA Manager || Product Integrity || n/a || [mailto:andrei.vaida@softvisioninc.eu Email]
|-
| Alastor Wu || Software Engineer || Engineering || n/a || [mailto:alwu@mozilla.com Email]
|-
| Blake Wu || EPM || Product Management || n/a || [mailto:bwu@mozilla.com Email]
|}
'''Revision History'''
'''Revision History'''


This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.


{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
Line 8: Line 23:
|-
|-
| 10/17/2016|| 1.0 || Adrian Florinescu ||  Initial Draft
| 10/17/2016|| 1.0 || Adrian Florinescu ||  Initial Draft
|-
| 07/24/2017|| 1.1 || Emil Ghitta ||  Updates
|-
| 08/14/2017|| 1.2 || Emil Ghitta ||  Updates
|}
|}


= Overview =
= Overview =
This feature's purpose is to block unexpected media from auto-playing in a background tab which hasn't been focused or visited before. The media would be automatically resumed when the tab becomes visible for the first time or when the user presses the Play Tab button (positioned on the opened tab).
== Purpose ==
== Purpose ==
Detail the purpose of this document. For example:
This document describes the following topics:
* The test scope, focus areas and objectives
* the test scope, focus areas and objectives
* The test responsibilities
* the test responsibilities
* The test strategy for the levels and types of test for this release
* the test strategy for the levels and types of test for this release
* The entry and exit criteria
* the entry and exit criteria
* The basis of the test estimates
* the basis of the test estimates
* Any risks, issues, assumptions and test dependencies
* any risks, issues, assumptions and test dependencies
* The test schedule and major milestones
* the test schedule and major milestones
* The test deliverables
* the test deliverables


== Scope ==
== Scope ==
This wiki details the testing that will be performed by the project team for the Block Playback project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
This wiki section details the testing that will be performed by the project team for the '''Media Block Playback''' project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document <u>what will be tested</u> and <u>how testing will be performed</u>.
* What will be tested
* How testing will be performed


== Ownership ==
== Ownership ==
 
* Engineering Manager:  
• Program Management:  
** [mailto:bwu@mozilla.com  Blake Wu]
* [mailto:bwu@mozilla.com  Blake Wu]
* Engineering Program Manager:  
 
** [mailto:bchien@mozilla.com  Bobby Chien]
• Engineering Program Manager:  
* Engineering:
* [mailto:ryang@mozilla.com  Rachelle Yang]
** [mailto:alwu@mozilla.com Alastor Wu]
 
** [mailto:jwwang@mozilla.com JW Wang]
• Eng.:
** [mailto:kaku@mozilla.com Tzuhao Kuo]
 
* UX:
* [mailto:jwwang@mozilla.com JW Wang]
** [mailto:mliang@mozilla.com Mark Liang]
* [mailto:kaku@mozilla.com Tzuhao Kuo]
** [mailto:mochen@mozilla.com Morpheus Chen]
* [mailto:alwu@mozilla.com Alastor Wu]
* QA
 
** Engineering Team - Nightly Testing
• UX:
*** [mailto:rares.bologa@softvisioninc.eu Rares Bologa] (irc: RaresB) - PM for QA team  
* [mailto:mliang@mozilla.com Mark Liang]
*** [mailto:brindusa.tot@softvisioninc.eu Brindusa Tot] (irc: brindusat) - leading QA efforts  
* [mailto:mochen@mozilla.com Morpheus Chen]
*** [mailto:simona.marcu@softvision.ro Simona Badau] (irc: simonab) - QA
 
** Release Team - Beta/Release Testing
• QA Engineering Team - Nightly Testing
*** [mailto:andrei.vaida@softvisioninc.eu Andrei Vaida] (irc: avaida) - TL for QA team  
* [mailto:rares.bologa@softvisioninc.eu Rares Bologa] (irc: RaresB) - PM for QA team  
*** [mailto:emil.ghitta@softvision.ro Emil Ghitta] (irc: emilghitta) - QA
* [mailto:brindusa.tot@softvisioninc.eu Brindusa Tot] (irc: brindusat) - leading QA efforts  
*** [mailto:bogdan.maris@softvision.ro Bogdan Maris] (irc: bogdan_maris) - QA peer
* [mailto:simona.marcu@softvision.ro Simona Badau] (irc: simonab) - QA
 
• QA Release Team - Aurora/Beta/Release Testing
* [mailto:andrei.vaida@softvisioninc.eu Andrei Vaida] (irc: avaida) - TL for QA team  
* [mailto:emil.ghitta@softvision.ro Emil Ghitta] (irc: emilghitta) - QA
* [mailto:bogdan.maris@softvision.ro Bogdan Maris] (irc: bogdan_maris) - QA peer


= Testing summary =  
= Testing summary =  
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
The Block Playback feature is designed to block media with enabled autoplay if the tabs are opened in the background. Videos which autoplay in the background will now have their load deferred until the tab is visible for the first time -- this avoids autoplay during session restore and premature playback. Resources will still be preloaded but Firefox will delay the start of playback until the tab that contin a video with an enabled autoplay is visited. Once a tab / RenderFrame has ever played media before, it's allowed to continue to autoplay/autoload indefinitely; this is to support playlist type applications. The main advantages of the Block Playback feature is that it prevents obviously user annoyance but also conserves power as Firefox will only consume power once the tab is foregrounded.
Testing will check the following items for popular websites providing media content:
* eligible tabs are blocked from auto-playing media
* video and sound are in sync when the tab is in focus
* the Play Tab button has the intended behavior
* the feature works as intended with and without e10s/e10s-multi
* tabs recover their previous state after being restored (session restore)
* the feature works as intended during private browsing sessions
* the feature plays well with a11y
* the feature plays well with High Contrast themes
* the feature plays well with HiDPI displays
* the feature is usable through mouse-only and keyboard-only operations
* the feature plays well with RTL locales
* associated Telemetry probes are gathering data
* the feature "works well" from a user-experience point of view


=== Out of Scope ===
=== Out of Scope ===
Detail what is out of scope from a testing perspective for the project team. Note: if usability testing is not in the scope of testing feature.
Testing will <u>not</u> check:
* scroll and zoom effects
* whether the feature works well with screen readers
* telemetry data correctness
* compatibility with add-ons


= Requirements for testing =
= Requirements for testing =
== Environments ==
== Environments ==
Full testing will be performed on the following OSes:
* '''Operating systems''':
* Windows 7 x64, Windows 10 x64.  
** Windows 7 x64, Windows 10 x64.  
* macOS 10.12
** macOS 10.12
* Ubuntu 16.04 x86
** Ubuntu 16.04 x86
* '''Devices''': n/a
* '''Other configuration/environmental setup needed''':
** <code>media.block-autoplay-until-in-foreground</code> needs to be set to <code>true</code>, but should be default-on


== Channel dependent settings (configs) and environment setups ==
There are no special settings needed to test this feature, apart from its pref being set to true.


Other configuration/environmental setup needed:
= Test Strategy =
* In Firefox configuration settings ''media.block-autoplay-until-in-foreground'' preference needs to be set to ''true''.


= Test Strategy =  
== Risk Assessment and Coverage ==
== Test Objectives ==
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
This could be documented in bullet form or in a table similar to the one below.


{| class="wikitable"
{| class="wikitable"
|-
|-
! Ref !! Function !! Test Objective !! Test Type !! RAC !! Owners
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score
|-
|-
| TO-1 || Video playback is not started until tab is visible for the first time || Prevent Video playback until the tab is visible  || Manual || RAC-1, RAC-2, RAC-4 || Eng Team
| RAC-1 || Mouse and keyboard actions are not recognized || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
|-
|-
| TO-2 || Video resources are still pre-loaded even though the tab wasn't visited yet || In the cases in which video playback is blocked, the resources are still being pre-loaded || Manual || - || Eng Team
| RAC-2 || Feature performance affected by session restore || TO-1 || 1- Low || 2-Possible || 3-High || 6
|-
|-
| TO-3 || Notification sounds are blocked also for tabs not yet first visited  || The sounds from websites such Facebook, Gmail, Hangout, web-Skype are blocked until the tab is visited for first time  || Manual || - || Eng Team
| RAC-3 || Multiple tabs affect feature performance || TO-4 || 2-Moderate || 2-Possible || 3-High || 12
|-
|-
| TO-4 || Performance is acceptable in case of multiple tabs and doesn't crash the browser or media container  || TBD what are the expectation in terms of "acceptable"  || Manual || RAC-3 || Eng Team
| RAC-4 || After visiting for the first time a tab containing a video, the video starts with a delay || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
|-
|-
|}
|}


'''Values:'''
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''


'''Impact Score Breakdown:'''  
'''Impact Score Breakdown:'''  
* An impact value of 1 or 2 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 4, 8 or 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.


'''RAC:''' Refers to the "Risk Assessment and Coverage" item that is covering.
'''RAC:''' Refers to the "Risk Assessment and Coverage" item that is covering.
== Test Objectives ==
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
This could be documented in bullet form or in a table similar to the one below.
{| class="wikitable"
|-
! Ref !! Function !! Evaluation Criteria !! Test Objective !! Test Type !! RAC !!  Owners
|-
| TO-1 || Video playback is not started until tab is visible for the first time || Prevent Video playback until the tab is visible || Verifying that playback does not start until the tab is focused for the first time || Manual || RAC-1, RAC-2, RAC-4 ||  Eng Team
|-
| TO-2 || Video resources are still pre-loaded even though the tab wasn't visited yet || In the cases in which video playback is blocked, the resources are still being pre-loaded || Verifying that video resources are pre-loaded even tho the autoplay is blocked || Manual ||  - || Eng Team
|-
| TO-3 || Notification sounds are blocked also for tabs not yet first visited  || The sounds from websites such Facebook, Gmail, Hangout, web-Skype are blocked until the tab is visited for first time || Verifying that notification sounds are blocked for the newly opened background tabs || Manual || - ||  Eng Team
|-
| TO-4 || Performance is acceptable in case of multiple tabs and doesn't crash the browser or media container  || Media is blocked, Firefox does not crash || Verifying that overall performance of the browser and multiple blocked tabs don't result in hangs or crashes || Manual || RAC-3 ||  Eng Team
|-
|}


== Builds ==
== Builds ==
Line 105: Line 161:
* Links for Nightly builds - available since November 11th 2016 [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-12-03-02-03-mozilla-central/ link]  
* Links for Nightly builds - available since November 11th 2016 [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-12-03-02-03-mozilla-central/ link]  
* Links for Aurora builds - available since March 3rd 2017 [https://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-07-07-04-55-mozilla-aurora/ link]
* Links for Aurora builds - available since March 3rd 2017 [https://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-07-07-04-55-mozilla-aurora/ link]
* Links for Beta builds -
* Links for Beta builds - available since August 7th 2017 [https://archive.mozilla.org/pub/firefox/candidates/56.0b1-candidates/build1/ link]


== Test Execution Schedule ==
== Test Execution Schedule ==
The following table identifies the anticipated testing period available for test execution.
The following table identifies the anticipated testing period available for test execution.
{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
|-
|-
Line 114: Line 171:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | 10.03.2016 ||
| style="text-align:center;" colspan="2" | 10.03.2016  
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
Line 126: Line 183:
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | 11.22.2016 || 17.02.2017
|style="text-align:center;" | 04.03.2017 || 07.28.2017
|-
| QA - Aurora Testing
|style="text-align:center;" | 03.17.2017 || 03.21.2017  
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 08.07.2017 || 09.26.2017
|-
|-
| Release Date  
| Release Date  
|style="text-align:center;" | ||
|style="text-align:center;" colspan="2" | 09.26.2017
|}
|}


Line 144: Line 198:
! Process !! Tool
! Process !! Tool
|-
|-
| Test plan creation || Mozilla wiki
| Test plan creation || [https://wiki.mozilla.org/QA/Block_Playback Mozilla wiki]
|-
|-
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc TestRail]/ Google docs
|-
|-
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc TestRail]
|-
|-
| Bugs management || Bugzilla
| Bugs management || Bugzilla ({{bug|1308154}})
|}
|}


= Status =  
= Status =  
(01.23.2017) - For the time being Block  Playback feature will be kept enabled only on Nightly - [https://bugzilla.mozilla.org/show_bug.cgi?id=1331317 Bug 1331317].
(03.07.2017) - Block Playback feature has reached Developer Edition 54.0a2.


== Overview ==
== Overview ==
Track the dates and build number where feature was released to Nightly - [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-12-03-02-03-mozilla-central/-12) 52.0a1 (2016-11-12)]
* Feature was released to Nightly - [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-12-03-02-03-mozilla-central/-12) 52.0a1 (2016-11-12)]
Track the dates and build number where feature was merged to Aurora - [https://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-07-07-04-55-mozilla-aurora/  54.0a2 (2017-03-07)]
* Feature was merged to Aurora - [https://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-07-07-04-55-mozilla-aurora/  54.0a2 (2017-03-07)]
Track the dates and build number where feature was merged to Release/Beta
* Feature was merged to Beta - [https://archive.mozilla.org/pub/firefox/candidates/56.0b1-candidates/build1/ 56.0b1 (2017-08-07)]
 
* Feature was merged to Release - [https://archive.mozilla.org/pub/firefox/candidates/56.0-candidates/build1/ 56.0 (2017-09-18)]
== Risk Assessment and Coverage ==
 
{| class="wikitable"
|-
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score
|-
| RAC-1 || Mouse and keyboard actions may not be recognized || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
|-
| RAC-2 || Restoring after a crash or after a session may impact the performance of this feature || TO-1 || 1- Low || 2-Possible || 3-High || 6
|-
| RAC-3 || Possible performance issues with multiple tabs || TO-4 || 2-Moderate || 2-Possible || 3-High || 12
|-
| RAC-4 || After visiting for the first time a tab containing a video, the video starts with a delay || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
|-
|}
 
'''Values:'''
 
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
 
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
 
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''
 
'''Impact Score Breakdown:'''
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.


= References =
= References =
* List and links for specs
* List and links for specs
  https://wiki.mozilla.org/Firefox/Block_Playback
** [https://mozilla.invisionapp.com/share/K48PCVSEM#/screens/190748310 UX spec]
  https://mozilla.invisionapp.com/share/6T8UPZR8K#/screens
* {{Bug|1308154}} - [Meta] (block-autoplay-media) [meta] Block autoplay media until the tab is visible at first time
 
* {{Bug|1308154}} - [Meta] (block-autoplay-media) [meta] Block autoplay media until the tab is visible at first time


= Testcases =  
= Test Cases =  
Full testsuites can be found on TestRail at [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc link] or in the [https://wiki.mozilla.org/images/0/05/Block_PLayback_-EngQA-_-_TestRail.pdf PDF doc]
Full testsuites can be found on TestRail at [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc link] or in the [https://wiki.mozilla.org/images/0/05/Block_PLayback_-EngQA-_-_TestRail.pdf PDF doc]
== Overview ==
Summary of testing scenarios


== Test Areas ==
== Test Areas ==
Line 268: Line 287:
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | TBD || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" | No ||  
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
Line 275: Line 294:
|-
|-
| ''' Enterprise '''  
| ''' Enterprise '''  
||  || Raise up the topic to developers to see if they are expecting to work different on ESR builds
||  ||  
|-
|-
| Enterprise administration   
| Enterprise administration   
Line 326: Line 345:
|-
|-
| ''' Security '''  
| ''' Security '''  
||  || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
||  || Security is in charge of Matt Wobensmith.
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" | TBD ||  
|style="text-align:center;" | No ||  
|-
|-
| Privilege escalation testing
| Privilege escalation testing
Line 342: Line 361:
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" | NO ||  
|style="text-align:center;" | No ||  
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" | NO ||  
|style="text-align:center;" | No ||  


|-
|-
Line 361: Line 380:
|}
|}


== Test suite ==
== Test Suites ==
Full Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
* Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc TestRail link]
Smoke Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
* Smoke Test suite - [https://docs.google.com/spreadsheets/d/1O0fAjkBpTQlN9-n_gZp9gRbTCm6Xh2Ej-Jgf65uHw9U/edit#gid=229595553 Gdoc link]
Regression Test suite - Link with the gdoc - if available/needed.
* Regression Test suite - n/a.


= Bug Work =
= Bug Work =
Line 385: Line 404:
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">


====== Bug fix verification ======
====== Verified bug fixes ======
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
<bugzilla>
<bugzilla>
Line 407: Line 426:
'''Nightly testing'''<br />
'''Nightly testing'''<br />


List of OSes that will be covered by testing<br />
* List of OSes that will be covered by testing:<br />
*Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/1688 link]
** Windows 10 64bit
** Ubuntu 16.04 64bit
** MacOS 10.11
* Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/1688 link]


'''Merge to Aurora Sign-off'''
'''Merge to Aurora Sign-Off'''
List of OSes that will be covered by testing<br />
* List of OSes that will be covered by testing<br />
*Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/3101 link]
** Windows 10 64bit
** Windows 7 64bit
** macOS 10.12
** Ubuntu 16.04 32bit
* Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/3101 link]
 
'''Merge to Beta Sign-Off'''
* List of OSes that will be covered by testing<br />
** Windows 10 64bit
** Windows 7 64bit
** macOS 10.11.6
** Ubuntu 16.04 64 bit
* Link for the tests run - [https://testrail.stage.mozaws.net/index.php?/reports/view/474 link]


== Checklist ==
== Checklist ==
Line 419: Line 453:
! Exit Criteria !! Status !! Notes/Details
! Exit Criteria !! Status !! Notes/Details
|-
|-
| Testing Prerequisites (specs, use cases)  
| Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |  {{mdone|}}
| style="text-align:center;" |  {{mdone|}}
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
| Testing Infrastructure setup  
| Testing Infrastructure setup  
|style="text-align:center;" |  {{mdone|}} ||  
|style="text-align:center;" |  {{mdone|}} ||  
|-
|-
| Test Plan Creation  
| Test Plan Creation  
| style="text-align:center;" |  {{mdone|}} ||  
| style="text-align:center;" |  {{mdone|}} ||  
|-
|-
| Test Cases Creation  
| Test Cases Creation  
|style="text-align:center;" |  {{mdone|}} ||  
|style="text-align:center;" |  {{mdone|}} ||  
|-
|-
| Full Functional Tests Execution  
| Full Functional Tests Execution  
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
| Automation Coverage  
| Automation Coverage  
|style="text-align:center;" | IN PROGRESS||  
|style="text-align:center;" | {{mprog|}} ||  
http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_block_autoplay_media.js <br>
http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_block_autoplay_media.js <br>
http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_mediaPlayback_suspended_multipleAudio.js <br>
http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_mediaPlayback_suspended_multipleAudio.js <br>
http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_mediaPlayback_suspended.js
http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_mediaPlayback_suspended.js
|-
|-
| Performance Testing  
| Performance Testing  
|style="text-align:center;" | N/A||  
|style="text-align:center;" | <font style="font-weight:bold;font-size:80%;">[NOT APPLICABLE]</font> ||  
|-
|-
| All Defects Logged  
| All Defects Logged  
|style="text-align:center;" | {{mdone|}} ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
| Critical/Blockers Fixed and Verified  
| Critical/Blockers Fixed and Verified  
|style="text-align:center;" |  {{mdone|}}  ||  
|style="text-align:center;" |  {{mdone|}}  ||  
|-
|-
| Metrics/Telemetry  
| Metrics/Telemetry  
|style="text-align:center;" | {{mdone|}} ||
|style="text-align:center;" | {{mdone|}} ||
|-
|-
| QA Signoff  - Nightly Release
| QA Signoff  - Nightly Release
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" | {{mdone|}}
|| [[https://wiki.mozilla.org/images/b/b9/-desktop--feature-_Block_Playback_-_Nightly_54.0a1_Sign-off.pdf|Email sent]]
|| [[https://wiki.mozilla.org/images/b/b9/-desktop--feature-_Block_Playback_-_Nightly_54.0a1_Sign-off.pdf Email sent in 2/17/2017]]
|-
|-
| QA Aurora - Full Testing  
| QA Beta - Full Testing
||  ||
|style="text-align:center;" | {{mdone|}} ||  
|-
| QA Signoff  - Aurora Release ||
|| Email to be sent
|-
|-
|  QA Beta - Full Testing
| QA Signoff - Beta Release 
| | ||  
| style="text-align:center;" | {{mdone|}} || [[https://drive.google.com/open?id=0B94iuIVMr-TEaDZmcnVqSGZwUDg  Email sent in 7/28/2017]]
|-
|-
| QA Signoff - Beta Release ||  
| QA Signoff - Pre Release
| | Email to be sent  
| style="text-align:center;" | {{mdone|}} || [[https://drive.google.com/open?id=0B94iuIVMr-TEWXhkRjFaLVk0bzA Email sent in 9/8/2017]]
|}
|}

Latest revision as of 11:44, 29 September 2017

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
Andrei Vaida QA Manager Product Integrity n/a Email
Alastor Wu Software Engineer Engineering n/a Email
Blake Wu EPM Product Management n/a Email

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
10/17/2016 1.0 Adrian Florinescu Initial Draft
07/24/2017 1.1 Emil Ghitta Updates
08/14/2017 1.2 Emil Ghitta Updates

Overview

This feature's purpose is to block unexpected media from auto-playing in a background tab which hasn't been focused or visited before. The media would be automatically resumed when the tab becomes visible for the first time or when the user presses the Play Tab button (positioned on the opened tab).

Purpose

This document describes the following topics:

  • the test scope, focus areas and objectives
  • the test responsibilities
  • the test strategy for the levels and types of test for this release
  • the entry and exit criteria
  • the basis of the test estimates
  • any risks, issues, assumptions and test dependencies
  • the test schedule and major milestones
  • the test deliverables

Scope

This wiki section details the testing that will be performed by the project team for the Media Block Playback project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document what will be tested and how testing will be performed.

Ownership

Testing summary

Scope of Testing

In Scope

Testing will check the following items for popular websites providing media content:

  • eligible tabs are blocked from auto-playing media
  • video and sound are in sync when the tab is in focus
  • the Play Tab button has the intended behavior
  • the feature works as intended with and without e10s/e10s-multi
  • tabs recover their previous state after being restored (session restore)
  • the feature works as intended during private browsing sessions
  • the feature plays well with a11y
  • the feature plays well with High Contrast themes
  • the feature plays well with HiDPI displays
  • the feature is usable through mouse-only and keyboard-only operations
  • the feature plays well with RTL locales
  • associated Telemetry probes are gathering data
  • the feature "works well" from a user-experience point of view

Out of Scope

Testing will not check:

  • scroll and zoom effects
  • whether the feature works well with screen readers
  • telemetry data correctness
  • compatibility with add-ons

Requirements for testing

Environments

  • Operating systems:
    • Windows 7 x64, Windows 10 x64.
    • macOS 10.12
    • Ubuntu 16.04 x86
  • Devices: n/a
  • Other configuration/environmental setup needed:
    • media.block-autoplay-until-in-foreground needs to be set to true, but should be default-on

Channel dependent settings (configs) and environment setups

There are no special settings needed to test this feature, apart from its pref being set to true.

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 Mouse and keyboard actions are not recognized TO-1 2-Moderate 1-Unlikely 3-High 6
RAC-2 Feature performance affected by session restore TO-1 1- Low 2-Possible 3-High 6
RAC-3 Multiple tabs affect feature performance TO-4 2-Moderate 2-Possible 3-High 12
RAC-4 After visiting for the first time a tab containing a video, the video starts with a delay TO-1 3-High 3-Almost Certain 3-High 27

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

RAC: Refers to the "Risk Assessment and Coverage" item that is covering.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Evaluation Criteria Test Objective Test Type RAC Owners
TO-1 Video playback is not started until tab is visible for the first time Prevent Video playback until the tab is visible Verifying that playback does not start until the tab is focused for the first time Manual RAC-1, RAC-2, RAC-4 Eng Team
TO-2 Video resources are still pre-loaded even though the tab wasn't visited yet In the cases in which video playback is blocked, the resources are still being pre-loaded Verifying that video resources are pre-loaded even tho the autoplay is blocked Manual - Eng Team
TO-3 Notification sounds are blocked also for tabs not yet first visited The sounds from websites such Facebook, Gmail, Hangout, web-Skype are blocked until the tab is visited for first time Verifying that notification sounds are blocked for the newly opened background tabs Manual - Eng Team
TO-4 Performance is acceptable in case of multiple tabs and doesn't crash the browser or media container Media is blocked, Firefox does not crash Verifying that overall performance of the browser and multiple blocked tabs don't result in hangs or crashes Manual RAC-3 Eng Team

Builds

This section should contain links for builds with the feature:

  • Links for Nightly builds - available since November 11th 2016 link
  • Links for Aurora builds - available since March 3rd 2017 link
  • Links for Beta builds - available since August 7th 2017 link

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 10.03.2016
Study documentation/specs received from developers 10.11.2016 10.17.2016
QA - Test plan creation 10.18.2016 10.20.2016
QA - Test cases/Env preparation 10.20.2016 01.11.2017
QA - Nightly Testing 04.03.2017 07.28.2017
QA - Beta Testing 08.07.2017 09.26.2017
Release Date 09.26.2017

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla (bug 1308154)

Status

Overview

References

  • List and links for specs
  • bug 1308154 - [Meta] (block-autoplay-media) [meta] Block autoplay media until the tab is visible at first time

Test Cases

Full testsuites can be found on TestRail at link or in the PDF doc

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled No
Theme (high contrast) Yes
UI
Mouse-only operation Yes
Keyboard-only operation Yes
Display (HiDPI) Yes
Interraction (scroll, zoom) No
Usable with a screen reader No e.g. with NVDA
Usability and/or discoverability testing Yes Is this feature user friendly
RTL build testing Yes
Help/Support
Help/support interface required No Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) No Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected No
Does sync work across upgrades No
Requires install testing No separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding No
Does this affect partner builds? Partner build testing No yes/no options, add comment with details about who will lead testing
Enterprise
Enterprise administration No
Network proxies/autoconfig No
ESR behavior changes No
Locked preferences No
Data Monitoring
Temporary or permanent telemetry monitoring Yes TAB_AUDIO_INDICATOR_USED, TAB_MEDIA_BLOCKING_TIME_MS
Telemetry correctness testing No
Server integration testing No
Offline and server failure testing No
Load testing No
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? No
Comprehensive API testing No
Permissions No
Testing with existing/popular addons No
Security Security is in charge of Matt Wobensmith.
3rd-party security review No
Privilege escalation testing No
Fuzzing No
Web Compatibility depends on the feature
Testing against target sites No
Survey of many sites for compatibility No
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS No
Interaction of this feature with other browser features No

Test Suites

Bug Work

Meta bug: 1308154 - (block-autoplay-media) [meta Block autoplay media until the tab is visible at first time]

Logged bugs ( blocking 1308154 )
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1262053 P3 Audio/Video: Playback Alastor Wu [:alwu] Block playback of media until document is visible VERIFIED mozilla51
1302350 P3 Audio/Video: Playback Alastor Wu [:alwu] Refactor the block-media-element feature. RESOLVED mozilla52
1308153 P3 Audio/Video: Playback Alastor Wu [:alwu] Implement the new UX visual mechanism for blocking autoplay media RESOLVED mozilla52
1308399 P3 Audio/Video: Playback Mark Liang(:mark_liang) [UX] UX Spec for Delay Autoplay RESOLVED ---
1314220 -- Audio/Video: Playback Alastor Wu [:alwu] Add the Telemetry for blocking autoplay media VERIFIED mozilla53
1319526 -- Audio/Video: Playback Media autoplay blocking icon has the wrong tooltip RESOLVED ---
1319529 -- Audio/Video: Playback Alastor Wu [:alwu] The unblocking icon didn't be hided when resuming Flash RESOLVED mozilla53
1319771 -- Audio/Video: Playback Alastor Wu [:alwu] If the pinned tab is the first tab in the tab list, the media won't be blocked after browser restarts VERIFIED mozilla54
1320005 -- Audio/Video: Playback Alastor Wu [:alwu] Don't show 'play tab' icon for video without audio track VERIFIED mozilla53
1322505 P1 Audio/Video: Playback Alastor Wu [:alwu] Don't resume the video after calling video.pause() VERIFIED mozilla53
1328058 -- Audio/Video: Playback Alastor Wu [:alwu] Tab displays "play" button or sound indicator when the only media element is playing and has no sound VERIFIED mozilla54
1329170 P3 Audio/Video: Playback Alastor Wu [:alwu] Play Tab indicator looks pixelated in pinned tabs VERIFIED mozilla54
1329179 -- Audio/Video: Playback Alastor Wu [:alwu] Play Tab indicator is displayed on a foreground tab VERIFIED mozilla54
1330195 -- Audio/Video: Playback Alastor Wu [:alwu] The audio indicator doesn't work for flash on Linux RESOLVED ---
1330232 -- Audio/Video: Playback Alastor Wu [:alwu] Blocked media was resumed with sound when user click "mute tab" in the pop-up menu RESOLVED mozilla53
1330644 -- Audio/Video: Playback No indications that a video with no audio track is blocked RESOLVED ---
1330681 -- Audio/Video: Playback Alastor Wu [:alwu] Play tab indicator has no hover state in pinned tabs VERIFIED mozilla54
1330929 -- Audio/Video: Playback Play tab icon not displayed for certain plugins RESOLVED ---
1331317 -- Audio/Video: Playback Alastor Wu [:alwu] Only enable pref "media.block-autoplay-until-in-foreground" on Nightly VERIFIED mozilla53
1336309 -- Tabbed Browser Alastor Wu [:alwu] Modify the access key for "play tab" option VERIFIED Firefox 54
1337232 -- Audio/Video: Playback Alastor Wu [:alwu] Enable pref "media.block-autoplay-until-in-foreground" by default VERIFIED mozilla54
1337704 -- Audio/Video: Playback Flickery transition between tabs indicators (play tab -> sound) RESOLVED ---
1338137 P1 Audio/Video: Playback Alastor Wu [:alwu] If the first tab is pinned and muted - media is not blocked after browser restart VERIFIED mozilla54
1338145 -- Audio/Video: Playback Social media: notification inconsistent behavior (Play tab indicator + sound) RESOLVED ---
1343465 -- Audio/Video: Playback Alastor Wu [:alwu] The tab should show the "play tab" icon even the tab was muted. RESOLVED mozilla54
1345947 P1 Audio/Video: Playback Alastor Wu [:alwu] The blocked autoplay media icon appears for tabs that I've seen before RESOLVED ---
1347758 P1 Audio/Video: Playback Alastor Wu [:alwu] HTML5 video/audio doesn't play if node was removed in background tab RESOLVED mozilla55
1347791 -- Tabbed Browser Alastor Wu [:alwu] The pinned tab shows play tab icon every time when restarts browser VERIFIED Firefox 56
1348326 P1 Audio/Video: Playback Alastor Wu [:alwu] BBC video/audio fails to load after focusing the video’s tab RESOLVED ---
1348759 -- Audio/Video: Playback Alastor Wu [:alwu] The video/audio on be-at.tv is not working as expected after focusing the video's tab RESOLVED ---
1348859 -- Audio/Video: Playback Alastor Wu [:alwu] Focusing an opened Facebook game tab no longer outputs the audio RESOLVED ---
1348879 -- Audio/Video: Playback Playbutton not visible when opening a video from givemesport.com in a new tab RESOLVED ---
1348909 -- Audio/Video: Playback Focusing the Web Skype tab while being called does not output the call audio RESOLVED ---
1349202 -- Audio/Video: Playback Block Playback feature is not working as intended on several websites RESOLVED ---
1350947 -- Plug-ins Alastor Wu [:alwu] Postpone loading NPAPI Flash content until the tab is resumed RESOLVED ---
1352014 -- Audio/Video: Playback Alastor Wu [:alwu] Add the flag "RELEASE_OR_BETA" for the pref "media.block-autoplay-until-in-foreground" RESOLVED mozilla55
1358211 -- Audio/Video: Playback Blocking autoplay for background tabs (bug 1308154) is breaking twitter videos RESOLVED ---
1367356 -- Plug-ins Alastor Wu [:alwu] Add telemetry to measure how many Flash contents are blocked by blocking autoplay media RESOLVED ---
1378105 P3 Session Restore Media is not blocked for previously unfocused closed tabs after restoring them from the History menu RESOLVED ---
1387917 -- Audio/Video: Playback Alastor Wu [:alwu] Turn on the pref "media.block-autoplay-until-in-foreground" by default RESOLVED mozilla57
1394455 P1 Audio/Video: Playback Mike Conley (:mconley) (:⚙️) [e10s] Media is unblocked while hovering an unfocused tab VERIFIED mozilla57
1572798 P2 DOM: Core & HTML Emilio Cobos Álvarez [:emilio] Video element created using document.open() doesn't work RESOLVED 86 Branch
1611748 P3 Audio/Video: Playback Alastor Wu [:alwu] video does not play in a dynamically created iframe (friendly) RESOLVED ---
1684876 -- Tabbed Browser kz04px "Mute Tab" sometimes starts playing the tab's video, with sound RESOLVED 95 Branch
1685163 P3 Audio/Video: Playback Alastor Wu [:alwu] MediaStreamTrack freezes when played in opened popup (used to work if visiting fullscreen) RESOLVED ---
1733902 P3 Audio/Video: Playback Alastor Wu [:alwu] Autoplay fails if the player is switched in a background tab VERIFIED 96 Branch

46 Total; 0 Open (0%); 30 Resolved (65.22%); 16 Verified (34.78%);


Verified bug fixes
Full Query
ID Priority Component Assigned to Summary Status Resolution Target milestone
1262053 P3 Audio/Video: Playback Alastor Wu [:alwu] Block playback of media until document is visible VERIFIED FIXED mozilla51
1302350 P3 Audio/Video: Playback Alastor Wu [:alwu] Refactor the block-media-element feature. RESOLVED FIXED mozilla52
1308153 P3 Audio/Video: Playback Alastor Wu [:alwu] Implement the new UX visual mechanism for blocking autoplay media RESOLVED FIXED mozilla52
1308399 P3 Audio/Video: Playback Mark Liang(:mark_liang) [UX] UX Spec for Delay Autoplay RESOLVED FIXED ---
1314220 -- Audio/Video: Playback Alastor Wu [:alwu] Add the Telemetry for blocking autoplay media VERIFIED FIXED mozilla53
1319529 -- Audio/Video: Playback Alastor Wu [:alwu] The unblocking icon didn't be hided when resuming Flash RESOLVED FIXED mozilla53
1319771 -- Audio/Video: Playback Alastor Wu [:alwu] If the pinned tab is the first tab in the tab list, the media won't be blocked after browser restarts VERIFIED FIXED mozilla54
1320005 -- Audio/Video: Playback Alastor Wu [:alwu] Don't show 'play tab' icon for video without audio track VERIFIED FIXED mozilla53
1322505 P1 Audio/Video: Playback Alastor Wu [:alwu] Don't resume the video after calling video.pause() VERIFIED FIXED mozilla53
1328058 -- Audio/Video: Playback Alastor Wu [:alwu] Tab displays "play" button or sound indicator when the only media element is playing and has no sound VERIFIED FIXED mozilla54
1329170 P3 Audio/Video: Playback Alastor Wu [:alwu] Play Tab indicator looks pixelated in pinned tabs VERIFIED FIXED mozilla54
1329179 -- Audio/Video: Playback Alastor Wu [:alwu] Play Tab indicator is displayed on a foreground tab VERIFIED FIXED mozilla54
1330232 -- Audio/Video: Playback Alastor Wu [:alwu] Blocked media was resumed with sound when user click "mute tab" in the pop-up menu RESOLVED FIXED mozilla53
1330681 -- Audio/Video: Playback Alastor Wu [:alwu] Play tab indicator has no hover state in pinned tabs VERIFIED FIXED mozilla54
1331317 -- Audio/Video: Playback Alastor Wu [:alwu] Only enable pref "media.block-autoplay-until-in-foreground" on Nightly VERIFIED FIXED mozilla53
1336309 -- Tabbed Browser Alastor Wu [:alwu] Modify the access key for "play tab" option VERIFIED FIXED Firefox 54
1337232 -- Audio/Video: Playback Alastor Wu [:alwu] Enable pref "media.block-autoplay-until-in-foreground" by default VERIFIED FIXED mozilla54
1338137 P1 Audio/Video: Playback Alastor Wu [:alwu] If the first tab is pinned and muted - media is not blocked after browser restart VERIFIED FIXED mozilla54
1343465 -- Audio/Video: Playback Alastor Wu [:alwu] The tab should show the "play tab" icon even the tab was muted. RESOLVED FIXED mozilla54
1347758 P1 Audio/Video: Playback Alastor Wu [:alwu] HTML5 video/audio doesn't play if node was removed in background tab RESOLVED FIXED mozilla55
1347791 -- Tabbed Browser Alastor Wu [:alwu] The pinned tab shows play tab icon every time when restarts browser VERIFIED FIXED Firefox 56
1352014 -- Audio/Video: Playback Alastor Wu [:alwu] Add the flag "RELEASE_OR_BETA" for the pref "media.block-autoplay-until-in-foreground" RESOLVED FIXED mozilla55
1387917 -- Audio/Video: Playback Alastor Wu [:alwu] Turn on the pref "media.block-autoplay-until-in-foreground" by default RESOLVED FIXED mozilla57
1394455 P1 Audio/Video: Playback Mike Conley (:mconley) (:⚙️) [e10s] Media is unblocked while hovering an unfocused tab VERIFIED FIXED mozilla57
1572798 P2 DOM: Core & HTML Emilio Cobos Álvarez [:emilio] Video element created using document.open() doesn't work RESOLVED FIXED 86 Branch
1684876 -- Tabbed Browser kz04px "Mute Tab" sometimes starts playing the tab's video, with sound RESOLVED FIXED 95 Branch
1733902 P3 Audio/Video: Playback Alastor Wu [:alwu] Autoplay fails if the player is switched in a background tab VERIFIED FIXED 96 Branch

27 Total; 0 Open (0%); 11 Resolved (40.74%); 16 Verified (59.26%);

Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

  • List of OSes that will be covered by testing:
    • Windows 10 64bit
    • Ubuntu 16.04 64bit
    • MacOS 10.11
  • Link for the tests run -link

Merge to Aurora Sign-Off

  • List of OSes that will be covered by testing
    • Windows 10 64bit
    • Windows 7 64bit
    • macOS 10.12
    • Ubuntu 16.04 32bit
  • Link for the tests run -link

Merge to Beta Sign-Off

  • List of OSes that will be covered by testing
    • Windows 10 64bit
    • Windows 7 64bit
    • macOS 10.11.6
    • Ubuntu 16.04 64 bit
  • Link for the tests run - link

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Full Functional Tests Execution [DONE]
Automation Coverage [IN PROGRESS]

http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_block_autoplay_media.js
http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_mediaPlayback_suspended_multipleAudio.js
http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_mediaPlayback_suspended.js

Performance Testing [NOT APPLICABLE]
All Defects Logged [DONE]
Critical/Blockers Fixed and Verified [DONE]
Metrics/Telemetry [DONE]
QA Signoff - Nightly Release [DONE] [Email sent in 2/17/2017]
QA Beta - Full Testing [DONE]
QA Signoff - Beta Release [DONE] [Email sent in 7/28/2017]
QA Signoff - Pre Release [DONE] [Email sent in 9/8/2017]
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Block_Playback&oldid=1181442"