MozillaWiki

QA/Video Play Visual Refresh: Difference between revisions

Page Discussion

QA/Video Play Visual Refresh (view source)

Revision as of 14:47, 16 March 2017

1,356 bytes added ,  16 March 2017
→‎Checklist
 
(39 intermediate revisions by 3 users not shown)
Line 39: Line 39:
[mailto:brindusa.tot@softvisioninc.eu Brindusa Tot] (irc: brindusat) - leading QA efforts <br />
[mailto:brindusa.tot@softvisioninc.eu Brindusa Tot] (irc: brindusat) - leading QA efforts <br />
[mailto:hani.yacoub@softvision.ro Hani Yacoub] (irc: hani) - QA <br />
[mailto:hani.yacoub@softvision.ro Hani Yacoub] (irc: hani) - QA <br />
'''<big>Release QA Team</big> - Aurora & Beta Testing''':<br />
[mailto:cornel.ionce@softvision.ro Ionce Cornel] (irc: cornel_ionce) <br />
[mailto:iulia.cristescu@softvision.ro Iulia Cristescu] (irc: IuliaC) <br />
[mailto:bogdan.maris@softvision.ro Bogdan Maris] (irc: bogdan_maris) <br />


= Testing summary =  
= Testing summary =  
Line 88: Line 93:
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
This could be documented in bullet form or in a table similar to the one below.
This could be documented in bullet form or in a table similar to the one below.


{| class="wikitable"
{| class="wikitable"
|-
|-
! Ref !! Function !! Test Objective !! Test Type !! RAC !! Owners  
! Ref !! Function !! Test Objective !! Test Type !! Risk Assessment and Coverage(RAC) !! Owners  
|-
|-
| 1 || Video play controllers display|| The new visual design is displayed and videos are played correctly  || Manual || || Eng Team
| TO-1 || Video play controllers display|| The new visual design is displayed and videos are played correctly  || Manual || RAC-1, RAC-4 || Eng Team
|-
|-
| 2 || Video play controllers functionality || The objective the test is trying to demonstrate that video controllers works correctly || Manual || || Eng Team
| TO-2 || Video play controllers functionality || The basic functions(play/stop/pause, drag the play progress bar, mute, volume control, full screen) all work properly
|| Manual || RAC-1 || Eng Team
|-
|-
| 3 || Different video types || The objective the test is trying to demonstrate that new videos player design works correctly on different videos types || Manual ||  || Eng Team
| TO-3 || Different video types || Validate that new video player design works correctly on different videos types || Manual || RAC-2 || Eng Team
|-
|-
| 4 || Performance is acceptable in case of multiple tabs and doesn't crash the browser || The objective the test is trying to demonstrate that new videos player design works correctly when videos are opened on multiple tabs or windows || Manual || || Eng Team
| TO-4 || Performance is acceptable in case of multiple tabs and doesn't crash the browser || The objective the test is trying to demonstrate that new videos player design works correctly when videos are opened on multiple tabs or windows || Manual || RAC-3 || Eng Team
|-
|-
| 5 || Accessibility || Usable from accessibility point of view || Testing for accessibility includes:  
| TO-5 || Accessibility || Testing for accessibility includes:  
- testing using high Contract themes <br />
- testing using high Contract themes <br />
- Screen Reader validation<br />
- Screen Reader validation<br />
- Keyboard-only
- Keyboard-only  
|| || Eng QA
|| Manual || RAC-1 || Eng Team
|}
|}


== Risk Assessment and Coverage ==
{| class="wikitable"
|-
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score
|-
| RAC-1 || Mouse and keyboard actions may not be recognized || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
|-
| RAC-2 || Some Video types are not recognized and played || TO-3 || 3- High|| 2-Possible || 3-High || 18
|-
| RAC-3 || Possible performance issues with multiple tabs || TO-4 || 2-Moderate || 2-Possible || 3-High || 12
|-
| RAC-4 || After visiting for the first time a tab containing a video, the video starts with a delay || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
|-
|}
'''Values:'''
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''


'''Impact Score Breakdown:'''  
'''Impact Score Breakdown:'''  
* An impact value of 1 or 2 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 4, 8 or 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
'''RAC:''' Refers to the "Risk Assessment and Coverage" item that is covering.


== Builds ==
== Builds ==
This section should contain links for builds with the feature -  
This section should contain links for builds with the feature -  
* Links for Nightly builds - available since November 20 2016 - [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-20-03-02-05-mozilla-central/ link]
* Links for Nightly builds - available since November 20 2016 - [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-20-03-02-05-mozilla-central/ link]
* Links for Aurora builds
* Links for Aurora builds - http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-aurora/ (53.0a2)
* Links for Beta builds
* Links for Beta builds


Line 138: Line 166:
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" | 2016.12.13 ||  
|style="text-align:center;" | 2016.12.13 || 2016.12.20
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | 2016.12.16 ||  
|style="text-align:center;" | 2016.12.16 || 2017.01.23
|-
|-
| QA - Aurora Testing
| QA - Aurora Testing
Line 171: Line 199:
== Overview ==
== Overview ==
  Landed on Nightly - [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-20-03-02-05-mozilla-central/ 53.0a1 (2016-11-20)]
  Landed on Nightly - [http://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-20-03-02-05-mozilla-central/ 53.0a1 (2016-11-20)]
  Track the dates and build number where feature was merged to Aurora
  Merged to Aurora on 2017-02-15
  Track the dates and build number where feature was merged to Release/Beta
  Track the dates and build number where feature was merged to Release/Beta
== Risk Assessment and Coverage ==
{| class="wikitable"
|-
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score
|-
| RAC-1 || Mouse and keyboard actions may not be recognized || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
|-
| RAC-2 || Restoring after a crash or after a session may impact the performance of this feature || TO-1 || 1- Low || 2-Possible || 3-High || 6
|-
| RAC-3 || Possible performance issues with multiple tabs || TO-4 || 2-Moderate || 2-Possible || 3-High || 12
|-
| RAC-4 || After visiting for the first time a tab containing a video, the video starts with a delay || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
|-
|}
'''Values:'''
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''
'''Impact Score Breakdown:'''
* An impact value of 1 or 2 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 4, 8 or 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.


= References =
= References =
* List and links for specs
* List and links for specs
   List and links for available specs - documents, user stories, specifications
   Media Control Specs - [https://bug1271765.bmoattachments.org/attachment.cgi?id=8842340 link]
* Meta bug
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1271765 1271765]
 
= Testcases =  
= Testcases =  
== Overview ==
Summary of testing scenarios


== Test Areas ==
== Test Areas ==
Line 218: Line 215:
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| '''UI'''  
| '''UI'''  
Line 233: Line 230:
|-
|-
| Mouse-only operation   
| Mouse-only operation   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Keyboard-only operation   
| Keyboard-only operation   
|style="text-align:center;" | ||
|style="text-align:center;" | Yes ||
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Interraction (scroll, zoom)  
| Interraction (scroll, zoom)  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" | || e.g. with NVDA
|style="text-align:center;" | Yes || e.g. with NVDA
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
|style="text-align:center;" | || Is this feature user friendly
|style="text-align:center;" | Yes || Is this feature user friendly
|-
|-
| RTL build testing   
| RTL build testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| '''Help/Support'''  
| '''Help/Support'''  
Line 257: Line 254:
|-
|-
| Help/support interface required   
| Help/support interface required   
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.
|style="text-align:center;" | No || Make sure link to support/help page exist and is easy reachable.
|-
|-
| Support documents planned(written)   
| Support documents planned(written)   
|style="text-align:center;" | || Make sure support documents are written and are correct.
|style="text-align:center;" | No || Make sure support documents are written and are correct.


|-
|-
Line 267: Line 264:
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature upgrades/downgrades data as expected   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Does sync work across upgrades   
| Does sync work across upgrades   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Requires install testing     
| Requires install testing     
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)
|style="text-align:center;" | No || separate feature/application installation needed (not only Firefox)
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" | No || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
|style="text-align:center;" | || yes/no options, add comment with details about who will lead testing
|style="text-align:center;" | No || yes/no options, add comment with details about who will lead testing


|-
|-
Line 286: Line 283:
|-
|-
| Enterprise administration   
| Enterprise administration   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Network proxies/autoconfig   
| Network proxies/autoconfig   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| ESR behavior changes   
| ESR behavior changes   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Locked preferences   
| Locked preferences   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||


|-
|-
Line 302: Line 299:
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | || List of error conditions to monitor
|style="text-align:center;" | No || List of error conditions to monitor
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||


|-
|-
Line 321: Line 318:
|-
|-
| Addon API required?   
| Addon API required?   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Comprehensive API testing   
| Comprehensive API testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 337: Line 334:
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Privilege escalation testing
| Privilege escalation testing
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Fuzzing   
| Fuzzing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 350: Line 347:
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 360: Line 357:
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|}
|}


== Test suite ==
== Test suite ==
  Full Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
  Full Test suite - in TestRail at [https://testrail.stage.mozaws.net/index.php?/suites/view/360&group_by=cases:section_id&group_order=asc link]
Smoke Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
Regression Test suite - Link with the gdoc - if available/needed.


= Bug Work =
= Bug Work =
Tracking bug – meta bug
Meta bug - [https://bugzilla.mozilla.org/show_bug.cgi?id=1271765 1271765 - Visual refresh of media controls]


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1271765 1271765 ] )======


====== Bug fix verification ======
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
{| class="wikitable" style="width:50%"
<bugzilla>
|-
    {
! Bug No !! Summary !! Status !! Firefox Verion
        "blocks":[1271765],
|-
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
|  123 || bug summary || NEW || Nighly 45
    }
|-
</bugzilla>
| || || ||
 
|}
</div>
</div></div>
</div>


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
 
====== Bug fix verification ======
====== Logged bugs ======
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
Bug 111111<br/>
<bugzilla>
Bug 211111<br/>
    {
        "blocks":[1271765],
        "resolution":"FIXED",
        "include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"
    }
</bugzilla>
</div>
</div>
</div>
</div>


= Sign off =
= Sign off =
Line 405: Line 403:
Check list
Check list
* All test cases should be executed
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)


Line 413: Line 410:
List of OSes that will be covered by testing<br />
List of OSes that will be covered by testing<br />
*Link for the tests run
*Link for the tests run
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** List of OSes that will be covered by testing: Windows 10 x64, Ubuntu 16.04 x86 and Mac OS X 10.11 (El Capitan)
** Full Test suite, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Link for the test runs on Nightly - [https://testrail.stage.mozaws.net/index.php?/plans/view/1696 link] <br />
** Regression Test suite, if needed/available
** Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/360&group_by=cases:section_id&group_order=asc test suite link]


'''Merge to Aurora Sign-off'''
'''Aurora Testing'''
List of OSes that will be covered by testing<br />
*Mid-Aurora sign-off
*Link for the tests run
**List of OSes that will be covered by testing: Windows 10 x64, Windows 7 x64, Ubuntu 16.04 x86 and Mac OS X 10.12.3 Beta (Sierra) <br />
** Full Test suite
**Link for the tests run - https://testrail.stage.mozaws.net/index.php?/plans/view/2414
**Full Test suite - https://testrail.stage.mozaws.net/index.php?/suites/view/360&group_by=cases:section_id&group_order=asc
 
*Pre-Beta sign-off
**List of OSes that will be covered by testing: Windows 10 x64, Windows 7 x64, Ubuntu 16.04 x64 and Mac OS X 10.12.3 Beta (Sierra) <br />
**Link for the tests run - https://testrail.stage.mozaws.net/index.php?/plans/view/2856
**Full Test suite - https://testrail.stage.mozaws.net/index.php?/suites/view/360&group_by=cases:section_id&group_order=asc


== Checklist ==
== Checklist ==
{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:80%"
|-
|-
! Exit Criteria !! Status !! Notes/Details
! Exit Criteria !! Status !! Notes/Details
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |   
| style="text-align:center;" |  {{mdone|}}
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |   ||  
| style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Full Functional Tests Execution  
|  Full Functional Tests Execution  
|style="text-align:center;" |   
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
|  Automation Coverage ||
|  Automation Coverage
|style="text-align:center;" |  
|style="text-align:center;" | {{mdone|}} || Available test for Video Play:<br/> https://dxr.mozilla.org/mozilla-central/source/toolkit/content/tests/widgets<br/>https://dxr.mozilla.org/mozilla-central/source/accessible/tests/mochitest/actions/test_media.html
|-
|-
|  Performance Testing  
|  Performance Testing  
|style="text-align:center;" |  ||  
|style="text-align:center;" |  N/A ||  
|-
|-
|  All Defects Logged || ||  
|  All Defects Logged  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Critical/Blockers Fixed and Verified || ||  
|  Critical/Blockers Fixed and Verified  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Metrics/Telemetry||
|  Metrics/Telemetry  
|style="text-align:center;" |  
|style="text-align:center;" | N/A ||
|-
|-
|  QA Signoff  - Nightly Release||  
|  QA Signoff  - Nightly Release
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" | Email sent 01-23-2017 ([https://wiki.mozilla.org/images/e/e6/-desktop--feature-_Video_Play_Visual_Refresh_Aurora_Conditional_Sign-off.pdf GREEN])
|-
|-
|  QA Aurora - Full Testing
|  QA Aurora - Mid-Aurora Sign off
|style="text-align:center;" | ||  
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" | Email sent 02-20-2017 (GREEN])
|-
|-
|  QA Signoff  - Aurora Release ||  
|  QA Signoff  - Pre-Beta Sign off
|style="text-align:center;"| Email to be sent  
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;"| Email sent 03-03-2017 ([https://drive.google.com/file/d/0B0nYKG6PRiCcemNtbXkwOXVubGc/view?usp=sharing GREEN])
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
Brindusa.tot
533

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1157228...1165859"