QA/Block Playback: Difference between revisions

Jump to navigation Jump to search

QA/Block Playback (view source)

Revision as of 12:54, 14 August 2017

1,321 bytes removed ,  14 August 2017
m
-
m (-)
m (-)
Line 71: Line 71:
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
* What will be tested:
Testing will check the following items for popular websites providing media content:
The main testing focus is to assure that this feature works as intended by blocking the autplay media from media tabs that were opened in the background without being focused or if the weren't visited before.
* eligible tabs are blocked from auto-playing media
The testing activities will also focus:
* video and sound are in sync when the tab is in focus
 
* the Play Tab button has the intended behavior
-Verifying if the video and sound are in sync when the tab is in focus.
* the feature works as intended with and without e10s/e10s-multi
 
* tabs recover their previous state after being restored (session restore)
-Verifying if the Play Tab button functionality is working as intended.
* the feature works as intended during private browsing sessions
 
* the feature plays well with a11y
-Verifying if the tabs recovers their previous state after session restore.
* the feature plays well with High Contrast themes
 
* the feature plays well with HiDPI displays
-Verifying if this feature is "accessibly friendly".
* the feature is usable through mouse-only and keyboard-only operations
 
* the feature plays well with RTL locales
* How testing will be performed
* associated Telemetry probes are gathering data
Testing this feature will be done throw Manual Testing that will cover all known risk areas.
* the feature "works well" from a user-experience point of view
Each testing cycle will cover the following testing activities:
 
-Smoke testing
 
-Exploratory testing
 
-TC execution
 
-Regression testing
 
---
Private Browsing -Check if this feature works as intended on private tabs.
 
Multi-Process Enabled/Disabled - Check if this feature works as intended with both e10s enabled and e10s disabled.
 
High Contrast themes -Check if this feature works and the Play Tab button is visible while using High Contrast themes.
 
Mouse Only operations-Check if this feature works as intended while using Mouse only operations to activate the Play Tab button or open/navigate throw tabs.
 
Keyboard-only operations-Check if this feature works as intended while using Keyboard only operations to navigate throw tabs.
 
Display(HiDPI)-Check if this feature works and the Play tab button is visible while using HiDPI.
 
Usability-Check if this feature is easy to use.
 
RTL build compat-Check if this feature works as intended and the Play Tab button is correctly displayed while using different RTL builds.
 
Telemetry monitoring-Monitoring the telemetry to check if it works as intended.


=== Out of Scope ===
=== Out of Scope ===
Interactions (scroll,zoom)
Testing will <u>not</u> check:
 
* scroll and zoom effects
Usable with screen reader
* whether the feature works well with screen readers
 
* telemetry data correctness
Installation testing
* compatibility with add-ons
 
Telemetry correctness testing
 
Comprehensive API testing
 
Testing with addons
 
Privilege escalation testing
 
Testing against target sites
 
 


= Requirements for testing =
= Requirements for testing =
== Environments ==
== Environments ==
Full testing will be performed on the following OSes:
* '''Operating systems''':
* Windows 7 x64, Windows 10 x64.  
** Windows 7 x64, Windows 10 x64.  
* macOS 10.12
** macOS 10.12
* Ubuntu 16.04 x86
** Ubuntu 16.04 x86
* '''Devices''': n/a
* '''Other configuration/environmental setup needed''':
** <code>media.block-autoplay-until-in-foreground</code> needs to be set to <code>true</code>, but should be default-on


== Channel dependent settings (configs) and environment setups ==
== Channel dependent settings (configs) and environment setups ==
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
There are no special settings needed to test this feature, apart from its pref being set to true.
 
=== Nightly ===
<div class="mw-collapsible-content">
This feature is enabled by default in Firefox Nightly 57
</div>
 
=== Beta ===
<div class="mw-collapsible-content">
This feature is enabled by default in Firefox Beta 56
</div>
 
=== Post Beta / Release ===
<div class="mw-collapsible-content">
In Firefox configuration settings ''media.block-autoplay-until-in-foreground'' preference needs to be set to ''true''.
</div>
</div>


= Test Strategy =  
= Test Strategy =  
Line 167: Line 114:
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score  
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score  
|-
|-
| RAC-1 || Mouse and keyboard actions may not be recognized || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
| RAC-1 || Mouse and keyboard actions are not be recognized || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
|-
|-
| RAC-2 || Restoring after a crash or after a session may impact the performance of this feature || TO-1 || 1- Low || 2-Possible || 3-High || 6
| RAC-2 || Feature performance affected by session restore || TO-1 || 1- Low || 2-Possible || 3-High || 6
|-
|-
| RAC-3 || Possible performance issues with multiple tabs || TO-4 || 2-Moderate || 2-Possible || 3-High || 12
| RAC-3 || Multiple tabs affect feature performance || TO-4 || 2-Moderate || 2-Possible || 3-High || 12
|-
|-
| RAC-4 || After visiting for the first time a tab containing a video, the video starts with a delay || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
| RAC-4 || After visiting for the first time a tab containing a video, the video starts with a delay || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
Line 218: Line 165:
== Test Execution Schedule ==
== Test Execution Schedule ==
The following table identifies the anticipated testing period available for test execution.
The following table identifies the anticipated testing period available for test execution.
{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
|-
|-
Line 223: Line 171:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | 10.03.2016 ||
| style="text-align:center;" colspan="2" | 10.03.2016  
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
Line 241: Line 189:
|-
|-
| Release Date  
| Release Date  
|style="text-align:center;" |TBD || TBD
|style="text-align:center;" colspan="2" | TBD  
|}
|}


Line 256: Line 204:
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc TestRail]
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc TestRail]
|-
|-
| Bugs management || Bugzilla [https://bugzilla.mozilla.org/show_bug.cgi?id=1308154]
| Bugs management || Bugzilla ({{bug|1308154}})
|}
|}


Line 262: Line 210:


== Overview ==
== Overview ==
Firefox was released to Nightly - [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-12-03-02-03-mozilla-central/-12) 52.0a1 (2016-11-12)]
* Firefox was released to Nightly - [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-12-03-02-03-mozilla-central/-12) 52.0a1 (2016-11-12)]
Firefox was merged to Aurora - [https://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-07-07-04-55-mozilla-aurora/  54.0a2 (2017-03-07)]
* Firefox was merged to Aurora - [https://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-07-07-04-55-mozilla-aurora/  54.0a2 (2017-03-07)]
Firefox was merged to Beta - [https://archive.mozilla.org/pub/firefox/candidates/56.0b1-candidates/build1/ 56.0b1 (2017-08-07)]
* Firefox was merged to Beta - [https://archive.mozilla.org/pub/firefox/candidates/56.0b1-candidates/build1/ 56.0b1 (2017-08-07)]
Firefox was merged to Release - TBD
* Firefox was merged to Release - TBD


= References =
= References =
* List and links for specs
* List and links for specs
  [https://mozilla.invisionapp.com/share/K48PCVSEM#/screens/190748310 UX spec]
** [https://mozilla.invisionapp.com/share/K48PCVSEM#/screens/190748310 UX spec]
* {{Bug|1308154}} - [Meta] (block-autoplay-media) [meta] Block autoplay media until the tab is visible at first time


*  {{Bug|1308154}} - [Meta] (block-autoplay-media) [meta] Block autoplay media until the tab is visible at first time
= Test Cases =  
 
= Testcases =  
Full testsuites can be found on TestRail at [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc link] or in the [https://wiki.mozilla.org/images/0/05/Block_PLayback_-EngQA-_-_TestRail.pdf PDF doc]
Full testsuites can be found on TestRail at [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc link] or in the [https://wiki.mozilla.org/images/0/05/Block_PLayback_-EngQA-_-_TestRail.pdf PDF doc]


Line 433: Line 380:
|}
|}


== Test suite ==
== Test Suites ==
Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc TestRail link]
* Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/356&group_by=cases:section_id&group_order=asc TestRail link]
Smoke Test suite - [https://docs.google.com/spreadsheets/d/1O0fAjkBpTQlN9-n_gZp9gRbTCm6Xh2Ej-Jgf65uHw9U/edit#gid=229595553 Gdoc link]
* Smoke Test suite - [https://docs.google.com/spreadsheets/d/1O0fAjkBpTQlN9-n_gZp9gRbTCm6Xh2Ej-Jgf65uHw9U/edit#gid=229595553 Gdoc link]
Regression Test suite - Link with the gdoc - if available/needed.
* Regression Test suite - n/a.


= Bug Work =
= Bug Work =
Line 457: Line 404:
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">


====== Bug fix verification ======
====== Verified bug fixes ======
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
<bugzilla>
<bugzilla>
Line 479: Line 426:
'''Nightly testing'''<br />
'''Nightly testing'''<br />


List of OSes that will be covered by testing:<br />
* List of OSes that will be covered by testing:<br />
 
** Windows 10 64bit
-Windows 10 64bit
** Ubuntu 16.04 64bit
-Ubuntu 16.04 64bit
** MacOS 10.11
-MacOS 10.11
* Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/1688 link]
 
*Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/1688 link]


'''Merge to Aurora Sign-Off'''
'''Merge to Aurora Sign-Off'''
List of OSes that will be covered by testing<br />
* List of OSes that will be covered by testing<br />
-Windows 10 64bit
** Windows 10 64bit
-Windows 7 64bit
** Windows 7 64bit
-macOS 10.12
** macOS 10.12
-Ubuntu 16.04 32bit
** Ubuntu 16.04 32bit
 
* Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/3101 link]
*Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/3101 link]


'''Merge to Beta Sign-Off'''
'''Merge to Beta Sign-Off'''
List of OSes that will be covered by testing<br />
* List of OSes that will be covered by testing<br />
-Windows 10 64bit
** Windows 10 64bit
-Windows 7 64bit
** Windows 7 64bit
-macOS 10.11.6
** macOS 10.11.6
-Ubuntu 16.04 64 bit
** Ubuntu 16.04 64 bit
 
* Link for the tests run - [https://testrail.stage.mozaws.net/index.php?/reports/view/474 link]
*Link for the tests run - [https://testrail.stage.mozaws.net/index.php?/reports/view/474 link]


== Checklist ==
== Checklist ==
Line 510: Line 453:
! Exit Criteria !! Status !! Notes/Details
! Exit Criteria !! Status !! Notes/Details
|-
|-
| Testing Prerequisites (specs, use cases)  
| Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |  {{mdone|}}
| style="text-align:center;" |  {{mdone|}}
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
| Testing Infrastructure setup  
| Testing Infrastructure setup  
|style="text-align:center;" |  {{mdone|}} ||  
|style="text-align:center;" |  {{mdone|}} ||  
|-
|-
| Test Plan Creation  
| Test Plan Creation  
| style="text-align:center;" |  {{mdone|}} ||  
| style="text-align:center;" |  {{mdone|}} ||  
|-
|-
| Test Cases Creation  
| Test Cases Creation  
|style="text-align:center;" |  {{mdone|}} ||  
|style="text-align:center;" |  {{mdone|}} ||  
|-
|-
| Full Functional Tests Execution  
| Full Functional Tests Execution  
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
| Automation Coverage  
| Automation Coverage  
|style="text-align:center;" | IN PROGRESS||  
|style="text-align:center;" | IN PROGRESS||  
http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_block_autoplay_media.js <br>
http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_block_autoplay_media.js <br>
Line 533: Line 476:
http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_mediaPlayback_suspended.js
http://searchfox.org/mozilla-central/source/toolkit/content/tests/browser/browser_mediaPlayback_suspended.js
|-
|-
| Performance Testing  
| Performance Testing  
|style="text-align:center;" | N/A||  
|style="text-align:center;" | N/A||  
|-
|-
| All Defects Logged  
| All Defects Logged  
|style="text-align:center;" | {{mdone|}} ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
| Critical/Blockers Fixed and Verified  
| Critical/Blockers Fixed and Verified  
|style="text-align:center;" |  {{mdone|}}  ||  
|style="text-align:center;" |  {{mdone|}}  ||  
|-
|-
| Metrics/Telemetry  
| Metrics/Telemetry  
|style="text-align:center;" | {{mdone|}} ||
|style="text-align:center;" | {{mdone|}} ||
|-
|-
| QA Signoff  - Nightly Release
| QA Signoff  - Nightly Release
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" | {{mdone|}}
|| [[https://wiki.mozilla.org/images/b/b9/-desktop--feature-_Block_Playback_-_Nightly_54.0a1_Sign-off.pdf  Email sent in 2/17/2017]]
|| [[https://wiki.mozilla.org/images/b/b9/-desktop--feature-_Block_Playback_-_Nightly_54.0a1_Sign-off.pdf  Email sent in 2/17/2017]]
|-
|-
| QA Beta - Full Testing
| QA Beta - Full Testing
|style="text-align:center;" | {{mdone|}} ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
| QA Signoff  - Beta Release ||  
| QA Signoff  - Beta Release
  {{mdone|}}  
| style="text-align:center;" | {{mdone|}} || [[https://drive.google.com/open?id=0B94iuIVMr-TEaDZmcnVqSGZwUDg  Email sent in 7/28/2017]]  
|| [[https://drive.google.com/open?id=0B94iuIVMr-TEaDZmcnVqSGZwUDg  Email sent in 7/28/2017]]
|}
|}
Andrei.vaida
Confirmed users
1,595

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1178087"

Navigation menu