MozillaWiki

QA/Pointer Lock API Improvement: Difference between revisions

Page Discussion

QA/Pointer Lock API Improvement (view source)

Revision as of 13:20, 9 August 2016

3,038 bytes added ,  9 August 2016
→‎Environments
 
(24 intermediate revisions by 2 users not shown)
Line 26: Line 26:
== Scope ==
== Scope ==
This wiki details the testing that will be performed by the project team for the new implementation of Pointer Lock API with permission requirement removed. As the permission requirement for Fullscreen API was removed since several versions before, it make sense to not require additional permission for PointerLock when the document is in fullscreen.  
This wiki details the testing that will be performed by the project team for the new implementation of Pointer Lock API with permission requirement removed. As the permission requirement for Fullscreen API was removed since several versions before, it make sense to not require additional permission for PointerLock when the document is in fullscreen.  
With this feature the permission requirement for Pointer Lock is removed, and a warning bar is displayed with message "<site> has control of your pointer. Press Esc to take back control." went pointer lock is activated. If full screen is requested as well as pointerlock then only the full screen warning is shown.
With this feature the permission requirement for Pointer Lock is removed and when pointer lock is activated, a warning bar is displayed with message "<site> has control of your pointer. Press Esc to take back control.". If full screen is requested as well as pointerlock then only the full screen warning is shown.


Following in this document is presented:
Following in this document is presented:
Line 60: Line 60:


=== Out of Scope ===
=== Out of Scope ===
Detail what is out of scope from a testing perspective for the project team. Note: if usability testing is not in the scope of testing feature.
The full functionality of Pointer Lock is not in the scope, as this feature's updates were done in the permissions area and notifications. Although, a smoke will be run on the functionality part.


= Requirements for testing =
= Requirements for testing =
== Environments ==
== Environments ==
Full testing will be done on following OSes:
Full testing will be done on following OSes:
* Windows 7 x64
* Windows  
* Windows 10 x32
* Ubuntu
* Ububtu 15.04
* Mac OS X
* Mac OS X 10.10


= Test Strategy =  
= Test Strategy =  
Line 79: Line 78:
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! Owners  
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! Owners  
|-
|-
| 1 || Name of the feature or sub-function being tested || The objective the test is trying to demonstrate || The criteria that will be evaluated to demonstrate the test is successful || Manual/ Automation/ Regression/ Performance/ Usability/ Security/ Telemetry || Eng Team
| 1 || Permission requirement for Pointer lock API was disabled || The test objective is to demonstrate that Pointer lock API is enabled without Permission request || Permission approval is not requested when try to access Pointer Lock API  || Manual || Eng Team
|-
|-
| 2 || Repeat for each feature/sub-function || || || ||
| 2 || Warning bar is displayed when pointer lock is activated || The test objective is to demonstrate that when Pointer lock API is activated, a warning message: "<site> has control of your pointer. Press Esc to take back control." is displayed || Correct warning messages is displayed when Pointer Lock API is activated || Manual || Eng Team
|-
|-
| 3 || || || || ||
| 3 || Pointer lock functionality without full screen || The test objective is to demonstrate that when Pointer lock API functionality is correct without full screen enabled || Pointer lock API functions correctly || Manual || Eng Team
|-
| 4 || Pointer lock functionality with full screen enabled || The test objective is to demonstrate that when Pointer lock API functionality is correct with full screen enabled. Warning message for Pointer lock is not displayed, only the full screen warning is displayed|| Pointer lock API functions correctly under full screen || Manual || Eng Team
|-
| 5 || Pointer lock accessibility || The objective of these test suites is to verify that the application is accessible. || Testing for accessibility includes: <br />
- testing using high Contract themes<br />
- Mouse-only <br />
- Keyboard-only
|| Manual || Eng Team
|}
|}


== Builds ==
== Builds ==
Line 107: Line 113:
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | 07.14.2016 ||  
|style="text-align:center;" | 07.14.2016  
|style="text-align:center;" | 07.16.2016
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" | 07.14.2016 ||  
|style="text-align:center;" | 07.14.2016  
|style="text-align:center;" | 07.15.2016 
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | 07.15.2016 ||  
|style="text-align:center;" | 07.15.2016  
|style="text-align:center;" | 07.26.2016 
|-
|-
| QA - Aurora Testing
| QA - Aurora Testing
Line 133: Line 142:
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki
|-
|-
| Test case creation || Google docs
| Test case creation || Google docs/ Test Rail
|-
|-
| Test case execution || Google docs
| Test case execution || TestRail
|-
|-
| Bugs management || Bugzilla
| Bugs management || Bugzilla
Line 147: Line 156:


== Risk analysis ==
== Risk analysis ==
Identify the high-risk assumptions
- Will be added during testing
Identify existing bugs on the feature with high risk
 
Identify if other areas are affected by the fix
= References =
= References =
* List and links for specs
* List and links for specs
   - http://www.html5rocks.com/en/tutorials/pointerlock/intro/ - Pointer Lock API introduction
   - http://www.html5rocks.com/en/tutorials/pointerlock/intro/ - Pointer Lock API introduction
  - https://developer.mozilla.org/en-US/docs/Web/API/Pointer_Lock_API - Pointer Lock API introduction
  - http://mdn.github.io/pointer-lock-demo/
   - http://threejs.org/examples/misc_controls_pointerlock.html  
   - http://threejs.org/examples/misc_controls_pointerlock.html  
  - https://kripken.github.io/BananaBread/wasm-demo/index.html
  - http://substack.net/projects/voxel-forest/
  - http://media.tojicode.com/q3bsp/
* Meta bug: Bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1273351 1273351]
* Meta bug: Bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1273351 1273351]


= Testcases =  
= Testcases =  
== Overview ==
Test cases can be found here - gdoc [https://docs.google.com/spreadsheets/d/1qjA6Qy17vxAeihsRFg96UUf5sOCP8Y8J5vvyts3hO2s/edit#gid=898586100 link]<br />
Summary of testing scenarios
 
Test cases can be found also on TestRail at: [https://testrail.stage.mozaws.net/index.php?/suites/view/53&group_by=cases:section_id&group_order=asc link]
 


== Test Areas ==
== Test Areas ==
Line 167: Line 181:
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| '''UI'''  
| '''UI'''  
Line 182: Line 196:
|-
|-
| Mouse-only operation   
| Mouse-only operation   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Keyboard-only operation   
| Keyboard-only operation   
|style="text-align:center;" | ||
|style="text-align:center;" | Yes ||
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Interraction (scroll, zoom)  
| Interraction (scroll, zoom)  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" | || e.g. with NVDA
|style="text-align:center;" | No || e.g. with NVDA
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
|style="text-align:center;" | || Is this feature user friendly
|style="text-align:center;" | Yes || Is this feature user friendly


|-
|-
Line 204: Line 218:
|-
|-
| Help/support interface required   
| Help/support interface required   
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.
|style="text-align:center;" | No || Make sure link to support/help page exist and is easy reachable.
|-
|-
| Support documents planned(written)   
| Support documents planned(written)   
|style="text-align:center;" | || Make sure support documents are written and are correct.
|style="text-align:center;" | No || Make sure support documents are written and are correct.


|-
|-
Line 214: Line 228:
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature upgrades/downgrades data as expected   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Does sync work across upgrades   
| Does sync work across upgrades   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Requires install testing     
| Requires install testing     
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)
|style="text-align:center;" | No || separate feature/application installation needed (not only Firefox)
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" | No || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
|style="text-align:center;" | || yes/no options, add comment with details about who will lead testing
|style="text-align:center;" | No || yes/no options, add comment with details about who will lead testing


|-
|-
Line 233: Line 247:
|-
|-
| Enterprise administration   
| Enterprise administration   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Network proxies/autoconfig   
| Network proxies/autoconfig   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| ESR behavior changes   
| ESR behavior changes   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Locked preferences   
| Locked preferences   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||


|-
|-
Line 249: Line 263:
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | || List of error conditions to monitor
|style="text-align:center;" | No || List of error conditions to monitor
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||


|-
|-
Line 268: Line 282:
|-
|-
| Addon API required?   
| Addon API required?   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Comprehensive API testing   
| Comprehensive API testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 284: Line 298:
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Privilege escalation testing
| Privilege escalation testing
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Fuzzing   
| Fuzzing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 297: Line 311:
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 307: Line 321:
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes || Interaction with full screen feature
|}
|}


== Test suite ==
== Test suite ==
  Full Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
  First Run of Testcases can be found at: [https://testrail.stage.mozaws.net/index.php?/plans/view/65 link]
  Smoke Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
  Full Test Run, after fix of bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1287408 1287408]  - can be found at [https://testrail.stage.mozaws.net/index.php?/plans/view/90 link]
  Regression Test suite - Link with the gdoc - if available/needed.


= Bug Work =
= Bug Work =
Tracking bug – meta bug
Tracking bug – meta [https://bugzilla.mozilla.org/show_bug.cgi?id=1273351 bug 1273351]


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
Line 332: Line 345:
! Bug No !! Summary !! Status !! Firefox Verion
! Bug No !! Summary !! Status !! Firefox Verion
|-
|-
| 123 || bug summary || NEW || Nighly 45
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1273351 1273351] || Get rid of permission requirement for PointerLock API || Verified-Fixed || Nighly 50
|-
|-
| || || ||  
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1287408 1287408] || Geolocation preferences not applied properly in Nightly || Verified-Fixed || Nighly 50
|}
|}
</div></div>
</div></div>
Line 342: Line 355:
====== Logged bugs ======
====== Logged bugs ======
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
Bug 111111<br/>
Bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1287104 1287104] - PointerLock demo can't be enabled after changing focus<br/>
Bug 211111<br/>
Bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1288692 1288692] - After Web Console is opened - full screen is entered on a PointerLock demo instead of just entering PointerLock <br/>
Bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1289403 1289403] - [e10s] PointerLock works very slow on Nightly and Aurora with this site http://mdn.github.io/pointer-lock-demo/ <br/>
Bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1287408 1287408] - Geolocation preferences not applied properly in Nightly <br/>
</div>
</div>
</div>
</div>


= Sign off =
= Sign off =
Line 352: Line 366:
Check list
Check list
* All test cases should be executed
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
 
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
{| class="wikitable"
 
 
|+ SignOff Criteria - Small feature
 
! rowspan="6" | Nightly to Aurora
! Open Bugs Severity
! Number of Open Bugs
|-
| style="text-align: left;font-weight: bold;" | Blocker 
| style="text-align: center; font-weight: bold;" | 0
|-
| style="text-align: left;font-weight: bold;" | Critical
| style="text-align: center;font-weight: bold;" | 2
|-
| style="text-align: left;font-weight: bold;" | Major
| style="text-align: center;font-weight: bold;" | 4
|-
| style="text-align: left ;" | Normal
| style="text-align: center;" | 12
|-
| style="text-align: left;" | Minor+Enh
| style="text-align: center;" | 15
|-
 
|}


== Results ==
== Results ==
Line 360: Line 399:
List of OSes that will be covered by testing<br />
List of OSes that will be covered by testing<br />
*Link for the tests run
*Link for the tests run
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Full Test suite if testcase ca be found on TestRail at [https://testrail.stage.mozaws.net/index.php?/plans/view/90 link]
** Full Test suite, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Regression Test suite, if needed/available


'''Merge to Aurora Sign-off'''
'''Merge to Aurora Sign-off'''
Line 375: Line 412:
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |  
| style="text-align:center;" | {{mdone|}}
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |   ||  
| style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Full Functional Tests Execution  
|  Full Functional Tests Execution  
|style="text-align:center;" |   
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
Line 395: Line 432:
|-
|-
|  Performance Testing  
|  Performance Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A ||  
|-
|-
|  All Defects Logged || ||  
|  All Defects Logged  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Critical/Blockers Fixed and Verified || ||  
|  Critical/Blockers Fixed and Verified  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Daily Status Report (email/etherpad statuses/ gdoc with results) || ||
|  Metrics/Telemetry
|-
|style="text-align:center;" | N/A ||
|  Metrics/Telemetry||
|style="text-align:center;" |  
|-
|-
|  QA Signoff  - Nightly Release||  
|  QA Signoff  - Nightly Release
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" | [https://wiki.mozilla.org/images/c/c9/Firefox_Nighty_50_build_-_Pointer_Lock_API_improvement_Sign_Off.pdf Sign Off email] sent on 
|-
|-
|  QA Aurora - Full Testing  
|  QA Aurora - Full Testing  
Simona Badau
Confirmed users
1,333

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1140082...1143278"