MozillaWiki

QA/Block Playback: Difference between revisions

Page Discussion

QA/Block Playback (view source)

Revision as of 11:39, 14 August 2017

1,574 bytes added ,  14 August 2017
no edit summary
No edit summary
Line 1: Line 1:
'''Approvals Required / Received'''
The following individuals are required to/have approved this Test Plan:
{| class="wikitable"
|-
! Name !! Title !! Department !! Approval Date !! Method
|-
| Andrei Vaida || QA Manager || Product Integrity || Date || [mailto:andrei.vaida@softvisioninc.eu Email]
|-
| Alastor Wu || Software Engineer || Engineering || Date || [mailto:alwu@mozilla.com Email]
|-
| Blake Wu || EPM || Product Management || Date || [mailto:bwu@mozilla.com Email]
|}
'''Revision History'''
'''Revision History'''


This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.


{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
Line 10: Line 25:
|-
|-
| 07/24/2017|| 1.1 || Emil Ghitta ||  Updates  
| 07/24/2017|| 1.1 || Emil Ghitta ||  Updates  
|-
| 08/14/2017|| 1.2 || Emil Ghitta ||  Updates
|}
|}


= Overview =
= Overview =
== Purpose ==
== Purpose ==
Detail the purpose of this document. For example:
This feature's main purpose is to block unexpected autoplay media from a newly opened media tab (opened in a background tab) which hasn't been focused or visited before. The media would be automatically resumed when the tab becomes visible for the first time or when the user presses the Play Tab button (positioned on the opened tab).
* The test scope, focus areas and objectives
 
* The test responsibilities
This changes will bring a benefit to the user by removing the possibility of having a discomfort generated by hearing an unexpected sound which comes from a newly opened media tab which was opened in a background tab (without being focused) or which hasn't been visited previously on.
* The test strategy for the levels and types of test for this release
* The entry and exit criteria
* The basis of the test estimates
* Any risks, issues, assumptions and test dependencies
* The test schedule and major milestones
* The test deliverables


== Scope ==
== Scope ==
This wiki details the testing that will be performed by the project team for the Block Playback project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
This wiki section details the testing that will be performed by the project team for the Media Block Playback project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
* What will be tested
* What will be tested:
The main testing focus is to assure that this feature works as intended by blocking the autplay media from media tabs that were opened in the background without being focused or if the weren't visited before.
The testing activities will also focus:
 
-Verifying if the video and sound are in sync when the tab is in focus.
 
-Verifying if the Play Tab button functionality is working as intended.
 
-Verifying if the tabs recovers their previous state after session restore.
 
-Verifying if this feature is "accessibly friendly".
 
* How testing will be performed
* How testing will be performed
Testing this feature will be done throw Manual Testing that will cover all known risk areas.
Each testing cycle will cover the following testing activities:
-Smoke testing
-Exploratory testing
-TC execution
-Regression testing


== Ownership ==
== Ownership ==
Line 59: Line 91:
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
The Block Playback feature is designed to block media with enabled autoplay if the tabs are opened in the background. Videos which autoplay in the background will now have their load deferred until the tab is visible for the first time -- this avoids autoplay during session restore and premature playback. Resources will still be preloaded but Firefox will delay the start of playback until the tab that contin a video with an enabled autoplay is visited. Once a tab / RenderFrame has ever played media before, it's allowed to continue to autoplay/autoload indefinitely; this is to support playlist type applications. The main advantages of the Block Playback feature is that it prevents obviously user annoyance but also conserves power as Firefox will only consume power once the tab is foregrounded.
Private Browsing -Check if this feature works as intended on private tabs.
 
Multi-Process Enabled/Disabled - Check if this feature works as intended with both e10s enabled and e10s disabled.
 
High Contrast themes -Check if this feature works and the Play Tab button is visible while using High Contrast themes.
 
Mouse Only operations-Check if this feature works as intended while using Mouse only operations to activate the Play Tab button or open/navigate throw tabs.
 
Keyboard-only operations-Check if this feature works as intended while using Keyboard only operations to navigate throw tabs.
 
Display(HiDPI)-Check if this feature works and the Play tab button is visible while using HiDPI.
 
Usability-Check if this feature is easy to use.
 
RTL build compat-Check if this feature works as intended and the Play Tab button is correctly displayed while using different RTL builds.
 
Telemetry monitoring-Monitoring the telemetry to check if it works as intended.


=== Out of Scope ===
=== Out of Scope ===
Detail what is out of scope from a testing perspective for the project team. Note: if usability testing is not in the scope of testing feature.
Interactions (scroll,zoom)
 
Usable with screen reader
 
Installation testing
 
Telemetry correctness testing
 
Comprehensive API testing
 
Testing with addons
 
Privilege escalation testing
 
Testing against target sites
 
 


= Requirements for testing =
= Requirements for testing =
Line 71: Line 135:
* Ubuntu 16.04 x86
* Ubuntu 16.04 x86


== Channel dependent settings (configs) and environment setups ==
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
=== Nightly ===
<div class="mw-collapsible-content">
This feature is enabled by default in Firefox Nightly 57
</div>
=== Beta ===
<div class="mw-collapsible-content">
This feature is enabled by default in Firefox Beta 56
</div>


Other configuration/environmental setup needed:
=== Post Beta / Release ===
* In Firefox configuration settings ''media.block-autoplay-until-in-foreground'' preference needs to be set to ''true''.
<div class="mw-collapsible-content">
In Firefox configuration settings ''media.block-autoplay-until-in-foreground'' preference needs to be set to ''true''.
</div>
</div>


= Test Strategy =  
= Test Strategy =  
== Risk Assessment and Coverage ==
{| class="wikitable"
|-
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score
|-
| RAC-1 || Mouse and keyboard actions may not be recognized || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
|-
| RAC-2 || Restoring after a crash or after a session may impact the performance of this feature || TO-1 || 1- Low || 2-Possible || 3-High || 6
|-
| RAC-3 || Possible performance issues with multiple tabs || TO-4 || 2-Moderate || 2-Possible || 3-High || 12
|-
| RAC-4 || After visiting for the first time a tab containing a video, the video starts with a delay || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
|-
|}
'''Values:'''
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''
'''Impact Score Breakdown:'''
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
'''RAC:''' Refers to the "Risk Assessment and Coverage" item that is covering.
== Test Objectives ==
== Test Objectives ==
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
Line 93: Line 204:
|-
|-
|}
|}
'''Impact Score Breakdown:'''
* An impact value of 1 or 2 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 4, 8 or 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
'''RAC:''' Refers to the "Risk Assessment and Coverage" item that is covering.


== Builds ==
== Builds ==
Line 106: Line 209:
* Links for Nightly builds - available since November 11th 2016 [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-12-03-02-03-mozilla-central/ link]  
* Links for Nightly builds - available since November 11th 2016 [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-12-03-02-03-mozilla-central/ link]  
* Links for Aurora builds - available since March 3rd 2017 [https://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-07-07-04-55-mozilla-aurora/ link]
* Links for Aurora builds - available since March 3rd 2017 [https://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-07-07-04-55-mozilla-aurora/ link]
* Links for Beta builds - TBD
* Links for Beta builds - available since August 7th 2017 [https://archive.mozilla.org/pub/firefox/candidates/56.0b1-candidates/build1/ link]


== Test Execution Schedule ==
== Test Execution Schedule ==
Line 130: Line 233:
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | TBD || TBD
|style="text-align:center;" | 08.07.2017 || 09.26.2017
|-
|-
| Release Date  
| Release Date  
Line 142: Line 245:
! Process !! Tool
! Process !! Tool
|-
|-
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki [https://wiki.mozilla.org/QA/Block_Playback]
|-
|-
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc TestRail]/ Google docs
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc TestRail]/ Google docs
Line 148: Line 251:
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc TestRail]
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc TestRail]
|-
|-
| Bugs management || Bugzilla
| Bugs management || Bugzilla [https://bugzilla.mozilla.org/show_bug.cgi?id=1308154]
|}
|}


= Status =  
= Status =  
(01.23.2017) - For the time being Block  Playback feature will be kept enabled only on Nightly - [https://bugzilla.mozilla.org/show_bug.cgi?id=1331317 Bug 1331317].
(03.07.2017) - Block Playback feature has reached Developer Edition 54.0a2.


== Overview ==
== Overview ==
  Track the dates and build number where feature was released to Nightly - [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-12-03-02-03-mozilla-central/-12) 52.0a1 (2016-11-12)]
  Firefox was released to Nightly - [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-12-03-02-03-mozilla-central/-12) 52.0a1 (2016-11-12)]
  Track the dates and build number where feature was merged to Aurora - [https://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-07-07-04-55-mozilla-aurora/  54.0a2 (2017-03-07)]
  Firefox was merged to Aurora - [https://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-07-07-04-55-mozilla-aurora/  54.0a2 (2017-03-07)]
  Track the dates and build number where feature was merged to Release/Beta
  Firefox was merged to Beta - [https://archive.mozilla.org/pub/firefox/candidates/56.0b1-candidates/build1/ 56.0b1 (2017-08-07)]
 
Firefox was merged to Release - TBD
== Risk Assessment and Coverage ==
 
{| class="wikitable"
|-
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score
|-
| RAC-1 || Mouse and keyboard actions may not be recognized || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
|-
| RAC-2 || Restoring after a crash or after a session may impact the performance of this feature || TO-1 || 1- Low || 2-Possible || 3-High || 6
|-
| RAC-3 || Possible performance issues with multiple tabs || TO-4 || 2-Moderate || 2-Possible || 3-High || 12
|-
| RAC-4 || After visiting for the first time a tab containing a video, the video starts with a delay || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
|-
|}
 
'''Values:'''
 
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
 
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
 
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''
 
'''Impact Score Breakdown:'''
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.


= References =
= References =
* List and links for specs
* List and links for specs
   https://wiki.mozilla.org/Firefox/Block_Playback
   [https://mozilla.invisionapp.com/share/K48PCVSEM#/screens/190748310 UX spec]
  https://mozilla.invisionapp.com/share/K48PCVSEM#/screens/190748310


*  {{Bug|1308154}} - [Meta] (block-autoplay-media) [meta] Block autoplay media until the tab is visible at first time
*  {{Bug|1308154}} - [Meta] (block-autoplay-media) [meta] Block autoplay media until the tab is visible at first time
Line 198: Line 270:
= Testcases =  
= Testcases =  
Full testsuites can be found on TestRail at [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc link] or in the [https://wiki.mozilla.org/images/0/05/Block_PLayback_-EngQA-_-_TestRail.pdf PDF doc]
Full testsuites can be found on TestRail at [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc link] or in the [https://wiki.mozilla.org/images/0/05/Block_PLayback_-EngQA-_-_TestRail.pdf PDF doc]
== Overview ==
Summary of testing scenarios


== Test Areas ==
== Test Areas ==
Line 266: Line 335:
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | TBD || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" | No ||  
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
Line 273: Line 342:
|-
|-
| ''' Enterprise '''  
| ''' Enterprise '''  
||  || Raise up the topic to developers to see if they are expecting to work different on ESR builds
||  ||  
|-
|-
| Enterprise administration   
| Enterprise administration   
Line 324: Line 393:
|-
|-
| ''' Security '''  
| ''' Security '''  
||  || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
||  || Security is in charge of Matt Wobensmith.
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" | TBD ||  
|style="text-align:center;" | No ||  
|-
|-
| Privilege escalation testing
| Privilege escalation testing
Line 340: Line 409:
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" | NO ||  
|style="text-align:center;" | No ||  
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" | NO ||  
|style="text-align:center;" | No ||  


|-
|-
Line 405: Line 474:
'''Nightly testing'''<br />
'''Nightly testing'''<br />


List of OSes that will be covered by testing<br />
List of OSes that will be covered by testing:<br />
 
-Windows 10 64bit
-Ubuntu 16.04 64bit
-MacOS 10.11
 
*Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/1688 link]
*Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/1688 link]


'''Merge to Aurora Sign-Off'''
'''Merge to Aurora Sign-Off'''
List of OSes that will be covered by testing<br />
List of OSes that will be covered by testing<br />
-Windows 10 64bit
-Windows 7 64bit
-macOS 10.12
-Ubuntu 16.04 32bit
*Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/3101 link]
*Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/3101 link]


'''Merge to Beta Sign-Off'''
'''Merge to Beta Sign-Off'''
List of OSes that will be covered by testing<br />
List of OSes that will be covered by testing<br />
*Link for the tests run - TBD
-Windows 10 64bit
-Windows 7 64bit
-macOS 10.11.6
-Ubuntu 16.04 64 bit
 
*Link for the tests run - [https://testrail.stage.mozaws.net/index.php?/reports/view/474 link]


== Checklist ==
== Checklist ==
Line 459: Line 543:
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" | {{mdone|}}
|| [[https://wiki.mozilla.org/images/b/b9/-desktop--feature-_Block_Playback_-_Nightly_54.0a1_Sign-off.pdf  Email sent in 2/17/2017]]
|| [[https://wiki.mozilla.org/images/b/b9/-desktop--feature-_Block_Playback_-_Nightly_54.0a1_Sign-off.pdf  Email sent in 2/17/2017]]
|-
|  QA Aurora - Full Testing
||  ||
|-
|  QA Signoff  - Aurora Release ||
|| Email to be sent
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
| | ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  QA Signoff  - Beta Release ||  
|  QA Signoff  - Beta Release ||  
| | Email to be sent  
  {{mdone|}}
|| [[https://drive.google.com/open?id=0B94iuIVMr-TEaDZmcnVqSGZwUDg  Email sent in 7/28/2017]]
|}
|}
Eghitta
54

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1178085"