QA/WebGL2: Difference between revisions

Jump to navigation Jump to search

QA/WebGL2 (view source)

Revision as of 16:11, 23 November 2016

7,084 bytes added ,  23 November 2016
no edit summary
No edit summary
Line 1: Line 1:
''Draft testplan''
'''Revision History'''
 
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded).  The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
 
{| class="wikitable" style="width:60%"
|-
! Date !! Version !! Author !! Description
|-
| 2016/11/23 || 1.0 || Bogdan Maris || Created first draft
|}
 
= Overview =
== Purpose ==
WebGL 2 is based on OpenGL ES 3.0, and brings with it many improvements and additions to help developers create stunning visuals on the Web. WebGL 2 will raise many restrictions and add new capabilities compared to WebGL 1. For example, while WebGL 1 only required support for being able to render using 8 textures at a time, WebGL 2 raises this minimum limit to 32.
 
<small>(quote from Martin Best from Mozilla)</small>
 
== Scope ==
This wiki details the testing that will be performed for the WebGL 2.0 feature. It defines the overall testing requirements and provides an integrated view of the project test activities.


= WebGL2 =
== Ownership ==
== Ownership ==
Developer contacts: Jeff Gilbert [mailto:jgilbert@mozilla.com mail] , :jgilbert on IRC  
Developer contact: [mailto:jgilbert@mozilla.com Jeff Gilbert] (:jgilbert on IRC) <br />
 
QA Lead: [https://mozillians.org/en-US/u/bogdan.maris/ Bogdan Maris] [mailto:bogdan.maris@softvision.ro mail], (:bogdan_maris on IRC)


QA Lead: [https://mozillians.org/en-US/u/bogdan.maris/ Bogdan Maris] [mailto:bogdan.maris@softvision.ro mail] , :bogdan_maris on IRC
= Testing summary =
== Scope of Testing ==
=== In Scope ===
The goal is for Firefox to display the content for WebGL 2.0.


== Status ==
=== Out of Scope ===
===Overview===
- n/a
<small>'''<font color="green">[LANDED]</font>'''</small> Nightly 46: first landed in '''2015-12-17'''


{{ok|}} Aurora 46: tbd
= Requirements for testing =
== Environments ==
Testing will be performed on the following OSes (both x86 & x64 infrastructures):


{{ok|}} Beta 46: tbd
* Windows 7
* Windows 10
* Ubuntu 16.04
* Mac OS X 10.11
* Mac OS X 10.12


{{ok|}} Release 46: tbd
= Test Strategy =
== Test Objectives ==
This section details the progression test objectives that will be covered.


===Risk analysis===
===== ''This is a draft'' =====
{| class="wikitable"
{| class="wikitable"
|-
|-
! Event !! Probability !! Impact !! Current situation !! Mitigation steps
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type
|-
| 1 || WebGL 2.0 is enabled  || Verify that the “webgl.enable-webgl2” is set to true in about:config || Observe the pref value || Manual
|-
| 2 || WebGL 2.0 functionality || Verify that WebGL 2.0 demos work || Visit [http://toji.github.io/webgl2-particles/ demo] || Manual
|-
| 3 || Private Browsing || Verify that the feature works while in private browsing || Visit [http://toji.github.io/webgl2-particles/ demo] || Manual
|-
|-
| Firefox crashes is a user shares tabs/windows  || probable || blocker || Firefox will crash intermitently if a user tries to share a tab/window  || This issue is tracked in [https://bugzilla.mozilla.org/show_bug.cgi?id=1137614 Bug 1137614]
| 4 || WebGL 2.0 is disabled || Set “webgl.enable-webgl2” to false || [http://toji.github.io/webgl2-particles/ demo] displays WebGL only and not WebGL 2.0 || Manual || Eng Team 
|-
|-
| Firefox fails to display shared tabs/windows  || probable || high || Sometimes starting tab/window sharing before the call will not show the tab/window when someone joins conversation  || This issue is tracked in [https://bugzilla.mozilla.org/show_bug.cgi?id=1146882 Bug 1146882] and covered by regression testing
|}
 
== Builds ==
This section should contain links for builds with the feature -
* [https://nightly.mozilla.org Link for Nightly builds]
 
== Test Execution Schedule ==
The following table identifies the anticipated testing period available for test execution.
{| class="wikitable" style="width:60%"
|-
|-
| Shared window is displayed propperly || probable|| high || There may be lots of windows that will not be propperly displayed || Periodical regression testing covers this area
! Project phase !! Start Date !! End Date
|-
|-
| Various graphical issues while sharing || probable || medium || Some graphical issues can appear during a conversation or with the Hello controls || Periodical regression testing covers this area
| Start project
|style="text-align:center;" | 01/03/2016 || -
|-
|-
| Sharing options displayed in unsuported OS`s || improbable || medium || Some buttons/options can be available on platforms that don`t support Hello screensharing (Windows XP and Mac OS X 10.6) || Periodical regression testing covers this area
| Study documentation/specs received
|style="text-align:center;" | 05/07/2016 || 06/07/2016
|-
|-
| QA - Test plan creation
|style="text-align:center;" | 13/07/2016 || 14/07/2016
|-
| QA - Test cases/Env preparation
|style="text-align:center;" | 14/07/2016 || 15/07/2016
|-
| QA - Nightly Testing
|style="text-align:center;" | 06/07/2016 || 11/07/2016
|-
| QA - Aurora Testing
|style="text-align:center;" | || -
|-
| QA - Beta Testing
|style="text-align:center;" | 28/09/2016 || 29/09/2016
|-
| Release Date
|style="text-align:center;" | - || -
|}
|}


== Summary ==
== Testing Tools ==
===Context=== 
{| class="wikitable" style="width:50%"
|-
! Process !! Tool
|-
| Test plan creation || Mozilla wiki
|-
| Test case creation || Google Docs
|-
| Test case execution || Google Docs
|-
| Bugs management || Bugzilla
|}


===== ''This is a draft'' =====
= Status =
== Overview ==
* Track the dates and build number where feature was released to Nightly<br />
* Track the dates and build number where feature was merged to Aurora<br />
* Track the dates and build number where feature was merged to Release/Beta<br />


WebGL 2 is based on OpenGL ES 3.0, and brings with it many improvements and additions to help developers create stunning visuals on the Web.
== Risk analysis ==
WebGL 2 will raise many restrictions and add new capabilities compared to WebGL 1. For example, while WebGL 1 only required support for being able to render using 8 textures at a time, WebGL 2 raises this minimum limit to 32.


<small>(quote from Martin Best from Mozilla)</small>
{| class="wikitable" style="width:60%"
|-
! Risk areas !! Mitigation
|-
| External dependencies on the Print feature || Simplify Page is dependent on the ability to print<br />https://bugzilla.mozilla.org/show_bug.cgi?id=1285547
|}
 
= References =
* [https://bugzilla.mozilla.org/show_bug.cgi?id=962433 Use Reader Mode for printing articles]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1285607 Enable Simplify Print Feature by default]


===Enabling the feature===
= Testcases =
*Enabled by default in Nightly:
== Overview ==
**Build from '''2015-12-17''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1232864 bug 1232864]
* Summary of testing scenarios


Go to about:config, create new boolean preference with the name
== Test Areas ==
{| class="wikitable" style="width:80%"
|-
! Test Areas !! Covered !! Details
|-
| Private Window
|style="text-align:center;" | Yes ||
|-
| Multi-Process Enabled
|style="text-align:center;" | Yes ||
|-
| Multi-process Disabled
|style="text-align:center;" | Yes ||
|-
| Theme (high contrast)
|style="text-align:center;" | Yes ||
|-
| '''UI'''
||  ||
|-
| Mouse-only operation 
|style="text-align:center;" | Yes ||
|-
| Keyboard-only operation 
|style="text-align:center;" | No ||
|-
| Display (HiDPI)
|style="text-align:center;" | Yes ||
|-
| Interaction (scroll, zoom)
|style="text-align:center;" | Yes ||
|-
| Usable with a screen reader 
|style="text-align:center;" | No || e.g. with NVDA
|-
| Usability and/or discoverability testing 
|style="text-align:center;" | Yes || Is this feature user friendly
 
|-
| '''Help/Support'''
||  ||
|-
| Help/support interface required 
|style="text-align:center;" | TBD || Make sure link to support/help page exist and is easy reachable.
|-
| Support documents planned(written) 
|style="text-align:center;" | TBD || Make sure support documents are written and are correct.


     webgl.enable-prototype-webgl2
|-
| '''Install/Upgrade'''
||  ||
|-
| Feature upgrades/downgrades data as expected 
|style="text-align:center;" | N/A ||
|-
| Does sync work across upgrades 
|style="text-align:center;" | No || Not relevant
|-
| Requires install testing   
|style="text-align:center;" | No ||
|-
| Affects first-run or onboarding      
|style="text-align:center;" | No ||
|-
| Does this affect partner builds? Partner build testing 
|style="text-align:center;" | No || We won't have partner builds available for testing in the Nightly or Aurora channels


and the value:
|-
| ''' Enterprise '''
||  ||  Raise up the topic to developers to see if they are expecting to work different on ESR builds
|-
| Enterprise administration 
|style="text-align:center;" |  ||
|-
| Network proxies/autoconfig 
|style="text-align:center;" | ||
|-
| ESR behavior changes 
|style="text-align:center;" |  ||
|-
| Locked preferences 
|style="text-align:center;" |  ||


    true
|-
| ''' Data Monitoring '''
||  ||
|-
| Temporary or permanent telemetry monitoring 
|style="text-align:center;" | No ||
|-
| Telemetry correctness testing 
|style="text-align:center;" | No ||
|-
| Server integration testing 
|style="text-align:center;" | No ||
|-
| Offline and server failure testing 
|style="text-align:center;" | No ||
|-
| Load testing 
|style="text-align:center;" | No ||


== References ==
|-
* WebGL2 demo in Firefox can be seen [https://www.youtube.com/watch?v=2v6iLpY7j5M here].
| ''' Add-ons '''
* Test if your browser has WebGL2: [http://toji.github.io/webgl2-particles/ Here] (If it says WebGL2 it's working with WebGL2).
||  || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
* Main Wiki page for WebGL2 development [https://wiki.mozilla.org/Platform/GFX/WebGL2 wiki]
|-
* Original Firefox blogpost announcement of WebGL2 [https://blog.mozilla.org/futurereleases/2015/03/03/an-early-look-at-webgl-2/ blog]
| Addon API required?  
|style="text-align:center;" |  ||
|-
| Comprehensive API testing 
|style="text-align:center;" |  ||
|-
| Permissions 
|style="text-align:center;" |  ||
|-
| Testing with existing/popular addons
|style="text-align:center;" | Yes ||


== Test cases ==
|-
* {{ok|}} Test suite: '''[<URL> here]'''.
| ''' Security '''
||  || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
|-
| 3rd-party security review 
|style="text-align:center;" | ? ||
|-
| Privilege escalation testing
|style="text-align:center;" | ? ||
|-
| Fuzzing 
|style="text-align:center;" | ? ||


== Bug work ==
|-
====== Tracking bug ======
| ''' Web Compatibility '''
'''meta''': [https://bugzilla.mozilla.org/show_bug.cgi?id=889977 Bug 889977] - (webgl2) [meta] WebGL2 Support
||  || depends on the feature
* blocks:
|-
**[https://bugzilla.mozilla.org/buglist.cgi?id=710398 Bug 710398] - (gecko-games) Tracking bug for making Gecko ready as a games platform
| Testing against target sites 
**[https://bugzilla.mozilla.org/buglist.cgi?id=802882 Bug 802882] - (html5test) improve score on html5test.com
|style="text-align:center;" | Yes ||
|-
| Survey of many sites for compatibility 
|style="text-align:center;" | No ||


* dependency tree: [https://bugzilla.mozilla.org/showdependencytree.cgi?id=889977&hide_resolved=0 Bugzilla query]
|-
| ''' Interoperability  '''
||  || depends on the feature
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.  
|style="text-align:center;" |  ||
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS 
|style="text-align:center;" |  ||
|-
| Interaction of this feature with other browser features 
|style="text-align:center;" | Yes ||
|}


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
== Test suite ==
* Full Test suite - [https://docs.google.com/spreadsheets/d/16SsP_igSdp-H0q03AHeHuEFqstbk0FfsooA0_pch4_g/edit Google doc with test cases]
* Smoke Test suite -
* Regression Test suite -


====== Bug triage +/- for verification ======
= Bug Work =
<div class="mw-collapsible-content">
  Meta bug - [https://bugzilla.mozilla.org/show_bug.cgi?id=962433 Use Reader Mode for printing articles]
TBD


</div>
</div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">


====== Bug fix verification ======
====== Bug fix verification ======
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
<font color="green">'''[VERIFIED]'''</font> [https://bugzilla.mozilla.org/show_bug.cgi?id=1176278 Bug 1176278] - Long strings cause text-chat to be horizontally scrollable
{| class="wikitable" style="width:50%"
:↳ <small>'''2015-07-06'''</small>: verified fixed on 41.0a2, across platforms
|-
 
! Bug No !! Summary !! Status !! Firefox Version
 
|-
</div>
|  123 || bug summary || NEW || Nightly 45
</div>
|-
| || || ||
|}
</div></div>


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
Line 98: Line 313:
====== Logged bugs ======
====== Logged bugs ======
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
[https://bugzilla.mozilla.org/show_bug.cgi?id=1179784 Bug 1179784] - If re-joining the conversation - the chat area will be active but typing area can`t be accessed and invite/share buttons should be over video window still
{| class="wikitable" style="width:50%"
:↳ <small>'''2015-07-02'''</small>:
|-
 
! Bug No !! Summary !! Status !! Firefox Version
 
|-
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1285543 Bug 1285543] ||  Different Yahoo News article title and content is displayed when using Reader Mode || NEW || Trunk
|-
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1285544 Bug 1285544] ||  Pages printed with Simplify Page show "about:blank" in the upper right corner instead of the page URL || RESOLVED FIXED || Trunk
|-
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1285545 Bug 1285545] ||  When printed, pages from National Geographic show a much larger logo instead of the normal one in Reader Mode || RESOLVED WONTFIX || Trunk
|-
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1285547 Bug 1285547] ||  National Geographic articles that have videos break the Print Preview functionality || NEW || Trunk
|-
| || || ||
|}
</div>
</div>
</div>
</div>


== Sign off ==
= Sign off =
===Criteria===
== Criteria ==
* All the test cases were executed.
Check list
* All the blocker, critical, major bugs have been fixed.
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)


===Results===
== Results ==
'''Nightly testing'''<br />


'''Aurora Sign-off''' - ''(xxxx-xx-xx)''
*Link for the tests run
* {{ok|}} Aurora 46
** Daily Smoke suite -
** Full Test suite -
** Regression Test suite, if needed/available


'''Beta Sign-off''' - ''(xxxx-xx-xx)''
'''Merge to Aurora Sign-off'''<br />
* {{ok|}} Beta 46


'''Release Sign-off''' ''(xxxx-xx-xx)''
*Link for the tests run -
* {{ok|}} RC 46.0
** Full Test suite -
 
== Checklist ==
{| class="wikitable" style="width:60%"
|-
! Exit Criteria !! Status !! Notes/Details
|-
|  Testing Prerequisites (specs, use cases)
| style="text-align:center;" | {{mdone|}} 
| style="text-align:center;" |
|-
|  Testing Infrastructure setup
|style="text-align:center;" | No ||
|-
|  Test Plan Creation
| style="text-align:center;" | {{mdone|}} ||
|-
|  Test Cases Creation
|style="text-align:center;" | {{mdone|}} ||
|-
|  Full Functional Tests Execution
|style="text-align:center;" | {{mdone}}
|style="text-align:center;" | 
|-
|  Automation Coverage ||
|style="text-align:center;" |
|-
|  Performance Testing
|style="text-align:center;" |  ||
|-
|  All Defects Logged || ||
|-
|  Critical/Blockers Fixed and Verified || ||
|-
|  Daily Status Report (email/etherpad statuses/ gdoc with results) || ||
|-
|  Metrics/Telemetry||
|style="text-align:center;" | N/A
|-
|  QA Signoff  - Nightly Release||
|style="text-align:center;" | Email to be sent
|-
|  QA Aurora - Full Testing
|style="text-align:center;" |  ||
|-
|  QA Signoff  - Aurora Release ||
|style="text-align:center;"| Email to be sent
|-
|  QA Beta - Full Testing
|style="text-align:center;" | {{mdone|}} ||
|-
|  QA Signoff  - Beta Release ||
|style="text-align:center;" | Email to be sent
|}
Bogdan.maris
Confirmed users
1,904

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1155668"

Navigation menu