QA/Platform/Graphics/Features/GPU Process: Difference between revisions

From MozillaWiki
< QA‎ | Platform‎ | Graphics
Jump to navigation Jump to search
(dvander is lead)
 
(110 intermediate revisions by the same user not shown)
Line 1: Line 1:
'''Revision History'''
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded).  The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
{| class="wikitable" style="width:60%"
|-
! Date !! Version !! Author !! Description
|-
| 11/02/2016 || 1.0 || Anthony Hughes || Created first draft
|-
|}
= Overview =
= Overview =
== Purpose ==
== Purpose ==
Line 22: Line 10:
* The test schedule and major milestones
* The test schedule and major milestones
* The test deliverables
* The test deliverables
== Ownership ==
{| class="wikitable sortable"
!Responsibility
!Role
!Person
|-
| Accountable
| Engineering Manager
| [mailto:msreckovic@mozilla.com Milan Sreckovic (:milan)]
|-
| Responsible
| Tech Lead
| [mailto:danderson@mozilla.com David Anderson (:dvander)]
|-
| Responsible
| QA Lead
| [mailto:ahughes@mozilla.com Anthony Hughes (:ashughes)]
|-
| Supporting
| Engineer
| [mailto:gwright@mozilla.com George Wright (:gw280)]
|-
| Supporting
| Engineer
| [mailto:rhunt@mozilla.com Ryan Hunt (:rhunt)]
|-
|}


== Scope ==
== Scope ==
Line 39: Line 55:
* ''to be determined''
* ''to be determined''


== Ownership ==
== Schedule ==
; Developers
{| class="wikitable" style="text-align:center"
* [mailto:danderson@mozilla.com David Anderson (:dvander)] (Lead)
|-
* [mailto:gwright@mozilla.com George Wright (:gw280)]
! Project phase !! Date Started !! Date Completed
* [mailto:rhunt@mozilla.com Ryan Hunt (:rhunt)]
|-
 
| style="text-align:left" | '''Planning''' || ||
; QA
|-
* [mailto:ahughes@mozilla.com Anthony Hughes (:ashughes)] (Lead)
| style="text-align:left" |
 
::Development planning & Spec documentation
; Management
|
* [mailto:msreckovic@mozilla.com Milan Sreckovic (:milan)]
|
|-
| style="text-align:left" |
::Test planning & Spec review
|
|
|-
| style="text-align:left" | '''Development''' || ||
|-
| style="text-align:left" |
::Code lands in Nightly
|
|
|-
| style="text-align:left" |
::Test case development
|
|
|-
| style="text-align:left" |
::Test environment preparation
|
|
|-
| style="text-align:left" | '''QA''' || ||
|-
| style="text-align:left" |
::Nightly testing & sign-off
|
|
|-
| style="text-align:left" |
::Aurora testing & sign-off
|
|
|-
| style="text-align:left" |
::Beta testing & sign-off
|
|
|-
| style="text-align:left" |
::RC testing & sign-off
|
|
|-
| style="text-align:left" | '''Release''' || ||
|-
|}


== Outstanding Questions ==
== Outstanding Questions ==
* What testing is out of scope for this feature?
* What platforms, hardware, drivers, renderers, etc need coverage?


= Coverage Requirements =
= Test Coverage =  
== Environments ==
== Environments ==
=== Platform Coverage ===
* Windows versions...
* Mac versions...
* Linux versions...
* Mobile versions...
=== Device Coverage ===
* AMD, NVIDIA, Intel chipsets
* Single, Multi, and Hybrid GPU setups
* External GPU setups
* Current, Beta, and Legacy drivers
=== Other Configuration Details ===
* Third party software...
== Channel dependent settings (configs) and environment setups ==
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
=== Nightly ===
<div class="mw-collapsible-content">
text
</div>
=== Aurora ===
<div class="mw-collapsible-content">
text
</div>
=== Beta ===
<div class="mw-collapsible-content">
text
</div>
=== Post Beta / Release ===
<div class="mw-collapsible-content">
text
</div>
</div>
= Test Strategy =
== Test Objectives ==
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
This could be documented in bullet form or in a table similar to the one below.
{| class="wikitable"
{| class="wikitable"
! Coverage
! In Scope for MVP
! Out of Scope for MVP
|-
| Platforms
| Windows 7 Platform Update and later
| Windows 7 without Platform Update and earlier, Mac OS, Linux, Android, iOS
|-
| Devices
| AMD, Intel, and NVIDIA hardware with Direct3D 11 and later support
| AMD, Intel, and NVIDIA hardware with Direct3D 10 and earlier support
|-
|-
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! Owners
| Configurations
| ''TBD - Single/Multi/Hybrid/etc, Internal/External, Current/Beta/Legacy drivers''
| ''TBD - Single/Multi/Hybrid/etc, Internal/External, Current/Beta/Legacy drivers''
|-
|-
| 1 || Name of the feature or sub-function being tested || The objective the test is trying to demonstrate || The criteria that will be evaluated to demonstrate the test is successful || Manual/ Automation/ Regression/ Performance/ Usability/ Security/ Telemetry || Eng Team
| Other
| ''TBD - 3rd-party software (Antivirus, Security, WindowsBlinds, etc), Preference changes''
| ''TBD - 3rd-party software (Antivirus, Security, WindowsBlinds, etc), Preference changes''
|-
|-
| 2 || Repeat for each feature/sub-function || || || ||
| Channel Specific
| ''TBD - Unique coverage requirements for Nightly, Aurora, Beta, Release''
| ''TBD - Unique coverage requirements for Nightly, Aurora, Beta, Release''
|-
|-
| 3 || || || || ||
|}
|}


== Risks ==
Be sure we have test coverage to ensure:
* GPU Process does not to regress acceleration coverage, performance, startup time, rendering quality, and correctness for platforms without GPU Process and platforms with GPU Process where GPU Process is disabled.
* GPU Process recovers more gracefully from a crash than legacy
* GPU Process handles device resets more gracefully than legacy
* GPU Process returns with hardware (not software) compositor following device reset
* GPU Process does not regress font rendering in chrome and content (eg. Skia fonts)
* GPU Process does not fail in some way that results in a lot of users not getting Direct3D11 acceleration
* GPU Process does not break existing "new" technologies (eg. WebVR, WebGL2)
* GPU Process does not break legacy behaviours (eg. PDF viewing, Printing, Video playback, WebGL1, UI controls, about pages, etc)
* Legacy behavior/quality/performance (GPU Process disabled) remains on-par with the Release prior to GPU Process shipping
Acceptable Risks:
* GPU Process causes a moderate spike in acceleration crashes [''specific number for acceptable spike to be determined'']


== Builds ==
== Builds ==
This section should contain links for builds with the feature -
This section should contain links for builds with the feature  
* Links for Nightly builds
* [https://nightly.mozilla.org/ Links for Nightly builds]
* Links for Aurora builds
* [https://aurora.mozilla.org/ Links for Aurora builds]
* Links for Beta builds
* Links for Beta builds


== Test Execution Schedule ==
== Tools ==
The following table identifies the anticipated testing period available for test execution.
{| class="wikitable" style="width:60%"
|-
! Project phase !! Start Date !! End Date
|-
| Start project
|style="text-align:center;" |  ||
|-
| Study documentation/specs received from developers
|style="text-align:center;" |  ||
|-
| QA - Test plan creation
|style="text-align:center;" | 2016-11-02 ||
|-
| QA - Test cases/Env preparation
|style="text-align:center;" |  ||
|-
| QA - Nightly Testing
|style="text-align:center;" |  ||
|-
| QA - Aurora Testing
|style="text-align:center;" |  || 
|-
| QA - Beta Testing
|style="text-align:center;" |  ||
|-
| Release Date
|style="text-align:center;" | ||
|}
 
== Testing Tools ==
Detail the tools to be used for testing, for example see the following table:
Detail the tools to be used for testing, for example see the following table:
{| class="wikitable" style="width:50%"
{| class="wikitable" style="width:50%"
Line 165: Line 182:
|-
|-
| Bugs management || Bugzilla
| Bugs management || Bugzilla
|-
| Experimentation || Telemetry Experiments
|-
|}
|}


= Status =  
== Use Cases ==
== Overview ==
{| class="wikitable" style="width:100%"
{| class="wikitable" style="width:60%"
|-
! Use Cases !! Smoketest Coverage !! BFT Coverage !! FFT Coverage !! Notes
|-
| colspan="5" | ''The following need to be verified with GPU Process enabled, GPU Process disabled, and across a process reset''
|-
| Printing || || || || Verify previewing and printing various content types is not regressed by GPU Process
|-
| Private Window  || || ||  || Verify browser usage inside a Private Browsing window
|-
|-
! Project phase !! Date !! Build ID
| Multi-Process Enabled || || ||  || Verify GPU process is created when multi-process is enabled
|-
|-
| Landing on Nightly
| Multi-process Disabled || || || || Verify no GPU process is created when multi-process is disabled
|
|
|-
|-
| Merged to Aurora
| Single-process Window || || || || Verify no GPU process when using a non-e10s window
|
|
|-
|-
| Merged to Beta
| Offline Content || || || || Verify accelerated content continues to render when in Offline mode
|
|
|-
|-
| Released
| Video || || || || Verify HTML5 and Flash video playback acceleration
|
|
|-
|-
|}
| Tab/Window Management || || || || Verify arranging and management of tabs/windows
 
== Risk analysis ==
=== High-risk Assumptions ===
=== High-risk Bugs ===
=== Other Areas of Risk ===
 
= References =
* List and links for available specs - documents, user stories, specifications
* Meta bug: {{bug|1264543}}
* https://public.etherpad-mozilla.org/p/GPU_Process_Testing
 
= Testcases =
== Overview ==
Summary of testing scenarios
 
== Test Areas ==
{| class="wikitable" style="width:80%"
|-
|-
! Test Areas !! Covered !! Details
| Theme Management || || || || Verify management of various Firefox and system-level themes
|-
|-
| Private Window
| WebVR || || || || Verify WebVR demos continue to work at parity with the current Release
|style="text-align:center;" | ||  
|-
|-
| Multi-Process Enabled
| WebGL || || || || Verify WebGL demos continue to work at parity with the current Release
|style="text-align:center;" | ||  
|-
|-
| Multi-process Disabled
| Dev Tools || || || || Verify Developer Tools continue to work at parity with the current Release
|style="text-align:center;" | ||  
|-
|-
| Theme (high contrast)
| Custom fonts || || || || Verify content rendering, printing, scrolling, etc with common font customizations
|style="text-align:center;" | ||  
|-
|-
| '''UI'''  
| 2D Canvas || || || || Verify content rendering and performance with 2D Canvas and SVG demos
||  ||
|-
| colspan="5" | '''Accessibility'''  
|-
|-
| Mouse-only operation
| Mouse-only operation || || || ||  
|style="text-align:center;" | ||  
|-
|-
| Keyboard-only operation
| Keyboard-only operation || || || ||  
|style="text-align:center;" | ||
|-
|-
| Display (HiDPI)  
| Display (HiDPI) || || || ||  
|style="text-align:center;" | ||  
|-
|-
| Interraction (scroll, zoom)  
| Interraction (scroll, zoom) || || || ||  
|style="text-align:center;" | ||  
|-
|-
| Usable with a screen reader
| Usable with a screen reader || || || ||
|style="text-align:center;" | || e.g. with NVDA
|-
|-
| Usability and/or discoverability testing  
| Usability and/or discoverability testing || || || ||
|style="text-align:center;" | || Is this feature user friendly
|-
|-
| RTL build testing  
| RTL build testing || || || ||  
|style="text-align:center;" | ||
|-
|-
| '''Help/Support'''  
| colspan="5" | '''Help/Support'''  
||  ||
|-
|-
| Help/support interface required  
| Help/support interface required || || || || Make sure link to support/help page exist and is easy reachable.
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.
|-
|-
| Support documents planned(written)
| Support documents planned(written) || || || || Make sure support documents are written and are correct.
|style="text-align:center;" | || Make sure support documents are written and are correct.


|-
|-
| '''Install/Upgrade'''  
| colspan="5" | '''Install/Upgrade'''  
||  ||
|-
|-
| Feature upgrades/downgrades data as expected  
| Feature upgrades/downgrades data as expected || || || ||  
|style="text-align:center;" | ||
|-
|-
| Does sync work across upgrades  
| Does sync work across upgrades || || || || 
|style="text-align:center;" | ||  
|-
|-
| Requires install testing  
| Requires install testing || || || || Separate feature/application installation needed (not only Firefox)
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)
|-
|-
| Affects first-run or onboarding    
| Affects first-run or onboarding || || || || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|-
|-
| Does this affect partner builds? Partner build testing
| Partner build testing || || || || Does this affect partner builds? Yes/No options, add comment with details about who will lead testing
|style="text-align:center;" | || yes/no options, add comment with details about who will lead testing
 
|-
|-
| ''' Enterprise '''  
| colspan="5" | ''' Enterprise '''  
||  ||  Raise up the topic to developers to see if they are expecting to work different on ESR builds
''Raise up the topic to developers to see if they are expecting to work different on ESR builds''
|-
|-
| Enterprise administration   
| Enterprise administration   
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 
|-
|-
| Network proxies/autoconfig   
| Network proxies/autoconfig   
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 
|-
|-
| ESR behavior changes   
| ESR behavior changes   
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 
|-
|-
| Locked preferences   
| Locked preferences   
|style="text-align:center;" |  ||
|style="text-align:center;" |  || || ||  


|-
|-
| ''' Data Monitoring '''  
| colspan="5" | ''' Data Monitoring '''  
||  ||
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" |  || List of error conditions to monitor
|style="text-align:center;" |  || || ||  List of error conditions to monitor
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" |  ||
|style="text-align:center;" |  || || ||  
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" |  ||
|style="text-align:center;" |  || || ||  


|-
|-
| ''' Add-ons '''  
| colspan="5" | ''' Add-ons '''  
||  || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
''If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.''
|-
|-
| Addon API required?   
| Addon API required?   
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 
|-
|-
| Comprehensive API testing   
| Comprehensive API testing   
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 


|-
|-
| ''' Security '''  
| colspan="5" | ''' Security '''
||  || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
''Contact Matt Wobensmith to see if security testing is necessary for current feature''
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 
|-
|-
| Privilege escalation testing
| Privilege escalation testing
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 
|-
|-
| Fuzzing   
| Fuzzing   
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 


|-
|-
| ''' Web Compatibility '''  
| ''' Web Compatibility '''  
||  || depends on the feature
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 


|-
|-
| ''' Interoperability  '''  
| ''' Interoperability  '''  
||  || depends on the feature
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 
|-
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
|style="text-align:center;" |  ||  
|style="text-align:center;" |  || || || 
|-
| Interaction of this feature with third-party software (Antivirus, Security, Games, Theming, etc)
|style="text-align:center;" |  || || || 
|-
|}
|}


== Test suite ==
= Experiments =
Full Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
{| class="wikitable" style="width:100%"
Smoke Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
|-
Regression Test suite - Link with the gdoc - if available/needed.
! Experiment !! Contact !! Tracking Bug !! Status !! Start Date !! End Date
 
= Bug Work =
Tracking bug – meta bug
 
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
 
====== Bug fix verification ======
<div class="mw-collapsible-content">
{| class="wikitable" style="width:50%"
|-
|-
! Bug No !! Summary !! Status !! Firefox Verion
| A/B test layers.gpu-process.dev.enabled in Nightly 53 || [mailto:ahughes@mozilla.com Anthony Hughes] (:ashughes) || {{bug|1319206}} || Done [https://ashughes.com/?p=374] || <strike>2016-11-28</strike> 2016-12-24 || <strike>2016-12-12</strike> 2017-01-11
|-
|-
|  123 || bug summary || NEW || Nighly 45
| A/B test media.gpu-process-decoder in Nightly 53 || [mailto:ahughes@mozilla.com Anthony Hughes] (:ashughes) ||  || || ||  
|-
|-
| || || ||
|}
|}
</div></div>


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
== Nightly 53 A/B Test layers.gpu-process.dev.enabled ==
Purpose:
* Compare crash rates for users with GPU process enabled vs users with GPU process disabled
 
Dependencies:
* How do we deal with the fact that the pref is set early during startup? (emailed felipeg)
* Telemetry probes to measure crash rates
* Telemetry probe to measure proportion of users in each cohort


====== Logged bugs ======
Measurements:
<div class="mw-collapsible-content">
* [https://telemetry.mozilla.org/new-pipeline/evo.html#!aggregates=median!mean!5th-percentile!25th-percentile!75th-percentile!95th-percentile&cumulative=0&end_date=null&keys=&max_channel_version=nightly%252F53&measure=GPU_PROCESS_LAUNCH_TIME_MS_2&min_channel_version=nightly%252F50&product=Firefox&sanitize=1&sort_keys=submissions&start_date=null&trim=1&use_submission_date=0 Evolution for GPU_PROCESS_LAUNCH_TIME]
Bug 111111<br/>
* [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-12-19&keys=__none__!__none__!__none__&max_channel_version=nightly%252F53&measure=GPU_PROCESS_LAUNCH_TIME_MS_2&min_channel_version=nightly%252F50&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-12-10&table=0&trim=1&use_submission_date=0 Histogram for GPU_PROCESS_LAUNCH_TIME]
Bug 211111<br/>
* [https://docs.google.com/spreadsheets/d/1MmdtWVQMuuwO7fH59ckXWTWfQjGE3Smo2aBQqjlSkLQ/edit#gid=0 Crash Stats]
</div>
* MOZ30726-MINI#psd-gpu-process-nightly53 (see :ashughes to request access)
</div>
* SQL Telemetry
** [https://sql.telemetry.mozilla.org/queries/2089/source Overall w/GPU Process] (Romain's)
** [https://sql.telemetry.mozilla.org/queries/2070/source Experiment by architecture]
** [https://sql.telemetry.mozilla.org/queries/2068/source Experiment by compositor]
** [https://sql.telemetry.mozilla.org/queries/2067/source Experiment by platform]
** [https://sql.telemetry.mozilla.org/queries/2069/source Experiment by process model]
** [https://sql.telemetry.mozilla.org/queries/2065/source Experiment by process type]


= Bugs =
; <big>[https://bugzil.la/1264543 Meta Bug] - [https://bugzilla.mozilla.org/showdependencytree.cgi?id=1264543&hide_resolved=0 Dependency Tree]</big>
<bugzilla>
{
"blocks":"1264543",
"include_fields":"id,summary,status,resolution,depends_on"
}
</bugzilla>


= Sign off =
= Sign off =
== Criteria ==
== Nightly ==
Check list
* Metrics:
* All test cases should be executed
** Overall crash volume for graphics related issues is unchanged
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
**  
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)


== Results ==
* Testing:
'''Nightly testing'''<br />
** {{miss|}} Testrail regression tests passed by Softvision [https://testrail.stage.mozaws.net/index.php?/runs/view/2143]
* Automation:


List of OSes that will be covered by testing<br />
* Areas of Risk
*Link for the tests run
** Automation: No unit or functional test coverage
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Full Test suite, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Regression Test suite, if needed/available


'''Merge to Aurora Sign-off'''
{| class="wikitable"
List of OSes that will be covered by testing<br />
! Category
*Link for the tests run
! Criteria
** Full Test suite
! Status
|-
| rowspan="5" style="vertical-align: top" | Metrics
| [https://mzl.la/2kTYeTE GPU_PROCESS_LAUNCH_TIME_MS_2] is ...
| ''pending review''
<pre>
5th Percentile: 0.192s
25th Percentile: 0.296s
        Median: 0.462s
75th Percentile: 0.914s
95th Percentile: 3.440s
</pre>
|-
| [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-23&keys=__none__!__none__!__none__&max_channel_version=nightly%252F53&measure=GPU_PROCESS_LAUNCH_TIME_MS_2&min_channel_version=nightly%252F50&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-12-10&table=0&trim=1&use_submission_date=0 GPU_PROCESS_LAUNCH_TIME_MS_2 Histogram] is ...
| Pending developer review
|-
| MG: Main/Content/Plugin/GPU crashes [http://macmini/index.htm#psd-gpu-process]
|-
| [https://sql.telemetry.mozilla.org/dashboard/gfx-gpu-process-crashes GPU Process crash rate]
|-
| [https://telemetry.mozilla.org/new-pipeline/evo.html#!aggregates=median!mean!5th-percentile!25th-percentile!75th-percentile!95th-percentile&cumulative=0&end_date=null&keys=&max_channel_version=nightly%252F53&measure=SIMPLE_MEASURES_FIRSTPAINT&min_channel_version=nightly%252F50&os=Windows_NT%252C6.1!Windows_NT%252C10.0!Windows_NT%252C6.2&product=Firefox&sanitize=1&sort_keys=submissions&start_date=null&trim=1&use_submission_date=0 SIMPLE_MEASURES_FIRSTPAINT] does not regress
|
|-
| Experiments
| A/B stability does not regress
| -4% crashes overall, +5% plugin crashes (needs investigation) [https://ashughes.com/?p=374]
|-
| Testrail
| [https://testrail.stage.mozaws.net/index.php?/runs/view/2143 Regression test suite] passed on all platforms
| Pending review
|-
| Automation
| TBD
|
|-
| Bugzilla
| 0 blocking regressions
| [https://bugzilla.mozilla.org/buglist.cgi?list_id=13411501&o1=notequals&o2=equals&v1=---%2C%20unaffected&v2=%2B&f1=cf_status_firefox53&query_format=advanced&f2=cf_tracking_firefox53&component=Graphics&component=Graphics%3A%20Layers&component=Graphics%3A%20Text&product=Core tracked for 53]
|-
| Socorro
| 0 blocking signatures
|
|-
| rowspan="3" style="vertical-align: top" | Migration
| Okay to merge to Aurora
|-
| Aurora migration verified
|-
| Aurora release notes verified
|-
! Risks
! Description
! Mitigation
|-
| Automation
| No unit/function test coverage beyond a process startup test
| Manual testing of functionality and process termination/recovery
|}


== Checklist ==
{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
|-
|-
! Exit Criteria !! Status !! Notes/Details
! Criteria !! Status !! Notes/Details
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
Line 450: Line 496:
|style="text-align:center;" |  
|style="text-align:center;" |  
|-
|-
|  QA Signoff - Nightly Release||  
|  QA Signoff - Okay to Merge ||  
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" | Email to be sent  
|-
|-
|  QA Aurora - Full Testing  
| QA Signoff - Aurora Migration ||
|style="text-align:center;" |  ||  
|style="text-align:center;" | Email to be sent
|-
|  QA Signoff - Verify Aurora Release Notes ||
|style="text-align:center;" | Email to be sent
|}
 
== Aurora ==
{| class="wikitable" style="width:100%;"
|-
! style="width:30%" | Criteria
! style="width:10%" | Status
! style="width:60%" | Notes
|-
! colspan="3" style="text-align:left" | Testing
|-
| style="vertical-align:top" | Manual Test Coverage
| style="text-align:center; vertical-align:top" | {{done|}}
|
* {{done|}} Softvision mid-Aurora sign-off via [https://testrail.stage.mozaws.net/index.php?/suites/overview/46 Testrail]
* {{done|}} Aurora testday
* {{skip|}} Betabreakers regression
|-
| style="vertical-align:top" | Automated Test Coverage
| style="text-align:center; vertical-align:top" | {{done|}}
| Coverage limited to process instantiation. Risks to be mitigated via manual testing.
|-
| Performance Test Coverage
| style="text-align:center; vertical-align:top" | {{skip|}}
|
* {{skip|}} Softvision, Betabreakers, and/or Community testing Passmark test
* {{skip|}} Softvision, Betabreakers, and/or Community testing WebGL and HWA Video benchmarks
|-
! colspan="3" style="text-align:left" | Bug Reports
|-
| style="vertical-align:top" | [https://mzl.la/2mGuyaD No unresolved blocker bugs] <br /> [https://mzl.la/2mGmR4k No untriaged bugs blocking Fx53]
| style="text-align:center; vertical-align:top" | {{done|}}
| <u>Bugs pending Firefox 53 status review:</u><br />
<bugzilla display="list">
{
"blocks":"1264543",
"f1":"cf_status_firefox53",
"o1":"anywordssubstr",
"v1":"affected, ?",
"include_fields":"id,summary,status"
}
</bugzilla>
|-
| style="vertical-align:top" | [https://mzl.la/2lA94Lt No unverified fixed bugs]
| style="text-align:center; vertical-align:top" | {{done|}}
|<u>Bugs pending Firefox 53 status review:</u><br />
<bugzilla display="list">
{
"blocks":"1264543",
"resolution":"FIXED",
"f1":"cf_status_firefox53",
"o1":"notequals",
"v1":"fixed",
"f2":"target_milestone",
"o2":"anywordssubstr",
"v2":"---, mozilla53",
"include_fields":"id,summary,status"
}
</bugzilla>
|-
! colspan="3" style="text-align:left" | OKRs & Metrics
|- 
| style="vertical-align:top" | [https://mzl.la/2kU1sq9 GPU_PROCESS_LAUNCH_TIME_MS_2 distribution <= 3s]
| style="vertical-align:top; text-align:center" | {{done|}}
| ''Note: A value larger than 3s likely represents users failing to get GPU Process''
{| class="wikitable"
! Percentile || 2017-01-30 || 2017-02-06 || 2017-02-13 || 2017-02-20 || 2017-02-27 || 2017-03-06 || Net
|-
| 5th || 0.222s || 0.232s || 0.234s || 0.235s || 0.236s || 0.236s || +0.014s (+6.3%)
|-
| 25th || 0.360s || 0.366s || 0.365s || 0.364s || 0.365s || 0.365s || +0.005s (+1.4%)
|-
| 50th || 0.574s || 0.574s || 0.572s || 0.564s || 0.567s || 0.567s || -0.007s (-1.2%)
|-
| 75th || 1.150s || 1.140s || 1.130s || 1.100s || 1.110s || 1.110s || -0.040s (-3.5%)
|-
| 95th || 4.240s || 4.100s || 4.060s || 3.940s || 3.950s || 3.960s || -0.280s (-6.6%)
|-
! 3s Threshold || 91.94% || 92.20% || 92.32% || 92.41% || 92.57% || 92.55% || +1.01%
|-
! 1s Threshold || 70.65% || 71.05% || 71.22% || 71.28% || 71.47% || 71.79% || +1.14%
|}
|-
| style="vertical-align:top" | [https://mzl.la/2kXKjvM GPU_PROCESS_LAUNCH_TIME_MS_2 evolution doesn't regress]
| style="vertical-align:top; text-align:center" | {{done|}}
| ''Note: A value larger than 3s likely represents users failing to get GPU Process''
{| class="wikitable"
! Percentile || 2017-01-30 || 2017-02-06 || 2017-02-13 || 2017-02-20 || 2017-02-27 || 2017-03-06 || Net
|-
| 5th || 0.226s || 0.236s || 0.241s || 0.242s || 0.223s || 0.227s || +0.001s (+0.4%)
|-
| 25th || 0.358s || 0.361s || 0.365s || 0.365s || 0.339s || 0.345s || -0.013s (-3.6%)
|-
| 50th || 0.560s || 0.561s || 0.561s || 0.560s || 0.498s || 0.515s || -0.045s (-8.0%)
|-
| 75th || 1.120s || 1.100s || 1.090s || 1.090s || 0.834s || 0.935s || -0.185s (-16.5%)
|-
| 95th || 4.030s || 3.930s || 3.890s || 3.800s || 2.530s || 3.050s || -0.980s (-24.3%)
|}
|-
| style="vertical-align:top" | [https://sql.telemetry.mozilla.org/dashboard/gfx-gpu-process-crashes Telemetry graphics crash count/rate largely unchanged]
| style="text-align:center; vertical-align:top" | {{done|}}
|
{| class="wikitable"
! || 2017-01-30 || 2017-02-06 || 2017-02-13 || 2017-02-20 || 2017-02-27 || 2017-03-06 || Net
|-
| Crash Rate Average || 0.224 || 0.166 || 0.164 || 0.150 || 0.153 || 0.139 || -0.085 (-37.9%)
|-
| Crash Count Average || 64 || 73 || 87 || 86 || 88 || 74 || +10 (+15.6%)
|-
| Crash Count Percent || 1.17% || 1.07% || 1.18% || 1.14% || 1.17% || 1.14% || -0.03% (-2.5%)
|}
|-
| style="vertical-align:top" | [https://crash-stats.mozilla.com/topcrashers/?product=Firefox&version=53.0a2&days=7&process_type=gpu No unresolved topcrashes correlated to GPU Process]
| style="text-align:center; vertical-align:top" | {{done|}}
|
<u>GPU crashes rank 5th overall @ 0.82% with a combined 1071 reports</u><br />
1st-ranked is #47 overall @ 0.10% (mozilla::layers::CompositorD3D11::BeginFrame - {{bug|1306168}})<br />
2nd-ranked is #31 overall @ 0.13% (UMDevice::SetShaderResources - {{bug|1267997}})<br />
3rd-ranked is #51 overall @ 0.09% (nvwgf2um.dll | TCLSWrappers<T>::CLSDestroy - {{bug|1325933}})<br />
4th-ranked is #52 overall @ 0.09% (OOM | unknown | ... | mozilla::BufferList<T>::AllocateSegment)<br />
5th-ranked is #172 overall @ 0.03% (nvwgf2umx.dll | NDXGI::CDevice::DestroySynchronizationObjectCB - {{bug|1328082}})
 
|-
| style="vertical-align:top" | Telemetry Experiment reveals no regressions in A/B testing
| style="text-align:center; vertical-align:top" | {{skip|}}
|
* Compare SIMPLE_MEASURES_FIRSTPAINT as a proxy for performance
* Compare crash counts/rates of cohorts in Telemetry
* Compare crash counts/rates of cohorts in Socorro
* Compare crash signatures of cohorts in Socorro
|-
| style="vertical-align:top" | Percentage of users w/GPU Process and hardware profiles
| style="text-align:center; vertical-align:top" | {{skip|}}
|
* Measured via David Anderson's telemetry dashboard?
* GPU_PROCESS_STATUS probe added similar to E10S_STATUS? See {{bug|1338694}}.
|-
| style="vertical-align:top" | [https://people-mozilla.org/~danderson/moz-gfx-telemetry/www/#view=windows-features Insignificant change in hardware acceleration distribution]
| style="text-align:center; vertical-align:top" | {{done|}}
|
GPU Process Status:<br />
<pre>
Date        Available  Unavailable  Failed  Blocked
2017-02-23      71.2%        28.4%    0.3%      0.1%
2017-03-06      71.1%        28.5%    0.3%      0.1%
</pre>
D3D11 Status:<br />
<pre>
Date        Available  Unavailable  Failed  Blocked  Blacklisted
2017-02-23      67.4%        12.2%    4.3%      1.5%        14.6%
2017-03-06      67.5%        12.3%    4.3%      1.4%        14.5%
</pre>
Media Decoder Status:<br />
<pre>
Date        DXVA D3D9  DXVA D3D11  WMF Software
2017-02-23      25.3%        15.0%          59.7%
2017-03-06      25.2%        14.3%          60.5%
</pre>
|-
| style="vertical-align:top" | [https://mzl.la/2lnuYCz SIMPLE_MEASURES_FIRSTPAINT does not regress more than 5%]
| style="text-align:center; vertical-align:top" | {{done|}}
|
<pre>
Median Improvement
Aurora 53 2.760s n/a
Aurora 52 3.040s - 9.21%
Aurora 51 2.880s - 4.17%
Aurora 50 3.100s -10.97%
</pre>
''Note: even if it does regress it'll be hard to prove that it's GPU Process related and not some other change(s)''
|-
! colspan="3" style="text-align:left" | QA Signoffs
|-
| Okay to Merge to Beta
| style="text-align:center; vertical-align:top" | {{done|}}
| Signed off by Softvision on March 3, 2017 ([https://testrail.stage.mozaws.net/index.php?/reports/view/211 report])
|-
| Beta migration signed-off
| style="text-align:center; vertical-align:top" | {{done|}}
| Signed off by Softvision on March 8, 2017 ([https://wiki.mozilla.org/Releases/Firefox_53/Test_Plan/Beta/1 report])
|-
| Beta release notes signed-off
| style="text-align:center; vertical-align:top" | {{done|}}
| Signed off by Softvision on March 8, 2017
|-
|}
 
== Beta ==
{| class="wikitable" style="width:100%;"
|-
! style="width:30%" | Criteria
! style="width:10%" | Status
! style="width:60%" | Notes
|-
! colspan="3" style="text-align:left" | Testing
|-
| style="vertical-align:top" | Manual Test Coverage
| style="text-align:center; vertical-align:top" | {{done|}}
|
* Softvision mid-Beta sign-off via Testrail
* Softvision pre-Release sign-off via Testrail
* Beta testday
|-
| style="vertical-align:top" | Automated Test Coverage
| style="text-align:center; vertical-align:top" | {{done|}}
| Coverage limited to process instantiation. Risks to be mitigated via manual testing.
|-
| Performance Test Coverage
| style="text-align:center; vertical-align:top" | {{done|}}
|
* Passmark benchmark completed by Softvision
* WebGL benchmark completed by Softvision
* HWA Video benchmark completed by Softvision
|-
! colspan="3" style="text-align:left" | Bug Reports
|-
| style="vertical-align:top" | [https://mzl.la/2mGuyaD No unresolved blocker bugs] <br /> [https://mzl.la/2mGmR4k No untriaged bugs blocking Fx53]
| style="text-align:center; vertical-align:top" | {{done|}}
| <u>Bugs pending Firefox 53 status review:</u><br />
<bugzilla display="list">
{
  "blocks":"1264543",
"f1":"cf_status_firefox53",
"o1":"anywordssubstr",
"v1":"affected, ?",
"include_fields":"id,summary,status"
}
</bugzilla>
|-
| style="vertical-align:top" | [https://mzl.la/2lA94Lt No unverified fixed bugs]
| style="text-align:center; vertical-align:top" | {{done|}}
|<u>Bugs pending Firefox 53 status review:</u><br />
<bugzilla display="list">
{
"blocks":"1264543",
"resolution":"FIXED",
"f1":"cf_status_firefox53",
"o1":"notequals",
"v1":"fixed",
"f2":"target_milestone",
"o2":"anywordssubstr",
"v2":"---, mozilla53",
"include_fields":"id,summary,status"
}
</bugzilla>
|-
! colspan="3" style="text-align:left" | Metrics
|- 
| style="vertical-align:top" | [https://mzl.la/2lY5Jdt GPU_PROCESS_LAUNCH_TIME_MS_2 distribution <= 3s]
| style="text-align:center; vertical-align:top" | {{done|}}
| ''Note: A value larger than 3s likely represents users failing to get GPU Process''
* 5th Percentile: 0.213s
* 25th Percentile: 0.342s
* 50th Percentile: 0.605s
* 75th Percentile: 1.250s
* 95th Percentile: 4.620s
* Thresholds: 90.67% @ 3s, 68.17% @ 1s
|-
| style="vertical-align:top" | [https://mzl.la/2lXWWIA GPU_PROCESS_LAUNCH_TIME_MS_2 evolution doesn't regress]
| style="text-align:center; vertical-align:top" | {{done|}}
| ''Note: A value larger than 3s likely represents users failing to get GPU Process''
* 5th Percentile: 0.219s
* 25th Percentile: 0.325s
* 50th Percentile: 0.496s
* 75th Percentile: 0.936s
* 95th Percentile: 3.210s
|-
| style="vertical-align:top" | [https://sql.telemetry.mozilla.org/dashboard/gfx-gpu-process-crashes Telemetry graphics crash count/rate largely unchanged]
| style="text-align:center; vertical-align:top" | {{done|}}
|
* Crash Rate Average: 0.067 crashes per 1,000 usage hours
* Crash Count Average: 603 crashes / day
* Crash Count Percent: 0.389%
|-
| style="vertical-align:top" | [https://crash-stats.mozilla.com/topcrashers/?product=Firefox&version=53.0b&days=28&process_type=gpu No unresolved topcrashes correlated to GPU Process]
| style="text-align:center; vertical-align:top" | {{done|}}
|
<u>GPU crashes rank 13th overall @ 0.34% with a combined 2,295 reports</u><br />
1st-ranked is #115 overall @ 0.04% (mozilla::layers::CompositorD3D11::BeginFrame - {{bug|1306168}})<br />
2nd-ranked is #49 overall @ 0.09% (igd10iumd32.dll | CContext::EmptyOutAllDDIBindPoints - {{bug|}})<br />
3rd-ranked is #95 overall @ 0.05% (UMDevice::SetShaderResources - {{bug|1267997}})
<br />
<b>Also track [https://goo.gl/9b1rb0 53 vs 52 crash data for Ryan Pollock]</b>
|-
| style="vertical-align:top" | Telemetry Experiment reveals no regressions in A/B testing
| style="text-align:center; vertical-align:top" | {{done|}}
| DONE - https://ashughes.com/?p=426
* [https://bsmedberg.github.io/telemetry-experiments-dashboard/ Experiment stats]
* Compare SIMPLE_MEASURES_FIRSTPAINT as a proxy for performance
* [https://sql.telemetry.mozilla.org/dashboard/gfx-gpu-process-beta-53-experiment Compare crash counts/rates of cohorts in Telemetry]
* [https://docs.google.com/spreadsheets/d/1KDESBrehRcJVpUCfEouRmElL8q2z-I_bPkVe54Arpqs/edit#gid=0 Compare crash counts/rates of cohorts in Socorro]
* Compare crash signatures of cohorts in Socorro
|-
| style="vertical-align:top" | [https://people-mozilla.org/~danderson/moz-gfx-telemetry/www/#view=windows-features Insignificant change in hardware acceleration distribution]
| style="text-align:center; vertical-align:top" | {{done|}}
|
GPU Process Status:
* 47.3% Unavailable
* 33.2% Unused
* 19.4% Available
D3D11 Status:
* 71.7% D3D11
* 26.7% Basic (software)
* 1.60% D3D9
Media Decoder Status:
* 56.7% WMF (software)
* 25.6% D3D9
* 17.7% D3D11
|-
| style="vertical-align:top" | [https://mzl.la/2neFmgk SIMPLE_MEASURES_FIRSTPAINT does not regress more than 5%]
| style="text-align:center; vertical-align:top" | {{done|}}
|
* Median: 2.926s
* 15.9% improvement vs Firefox 52
* 18.4% improvement vs Firefox 51
* 2.46% improvement vs Firefox 50
''Note: even if it does regress it'll be hard to prove that it's GPU Process related and not some other change(s)''
|-
! colspan="3" style="text-align:left" | QA Signoffs
|-
| Okay to Merge to Release
| style="text-align:center; vertical-align:top" | {{done|}}
| DONE
|-
|-
| QA Signoff  - Aurora Release ||
| Release migration signed-off
|style="text-align:center;"| Email to be sent
| style="text-align:center; vertical-align:top" | {{done|}}
| DONE
|-
|-
| QA Beta - Full Testing
| Release notes signed-off
|style="text-align:center;" | ||  
| style="text-align:center; vertical-align:top" | {{done|}}
| DONE
|-
|-
|  QA Signoff  - Beta Release ||
|style="text-align:center;" | Email to be sent
|}
|}
== Release ==
* Verify release notes
* Verify feature enabled in Release
* Verify updating gives GPU Process
* Check for bugs during throttle period
= References =
* Meta bug: {{bug|1264543}}
* https://public.etherpad-mozilla.org/p/GPU_Process_Testing

Latest revision as of 23:20, 13 April 2017

Overview

Purpose

The purpose of this document is to describe the GPU Process feature from a testing perspective, including the following details:

  • Scope, focus areas and objectives of testing
  • Owners and points of contact for areas of responsibility
  • Strategy and types of testing
  • The entry and exit criteria
  • The basis of the test estimates
  • Any risks, issues, assumptions and test dependencies
  • The test schedule and major milestones
  • The test deliverables

Ownership

Responsibility Role Person
Accountable Engineering Manager Milan Sreckovic (:milan)
Responsible Tech Lead David Anderson (:dvander)
Responsible QA Lead Anthony Hughes (:ashughes)
Supporting Engineer George Wright (:gw280)
Supporting Engineer Ryan Hunt (:rhunt)

Scope

The following is in scope from a testing perspective for the project team.

  • GPU process startup time
  • Page load time
  • GPU process restart time (in case of process kill)
  • Real world performance with various common usecases (webgl, canvas2d, flash).
  • Plugin startup time
  • Antivirus conflicts
  • Memory footprint
  • Shutdown time
  • Overall product stability
  • Scrolling performance and correctness

The following is not in scope from a testing perspective for the project team.

  • to be determined

Schedule

Project phase Date Started Date Completed
Planning
Development planning & Spec documentation
Test planning & Spec review
Development
Code lands in Nightly
Test case development
Test environment preparation
QA
Nightly testing & sign-off
Aurora testing & sign-off
Beta testing & sign-off
RC testing & sign-off
Release

Outstanding Questions

Test Coverage

Environments

Coverage In Scope for MVP Out of Scope for MVP
Platforms Windows 7 Platform Update and later Windows 7 without Platform Update and earlier, Mac OS, Linux, Android, iOS
Devices AMD, Intel, and NVIDIA hardware with Direct3D 11 and later support AMD, Intel, and NVIDIA hardware with Direct3D 10 and earlier support
Configurations TBD - Single/Multi/Hybrid/etc, Internal/External, Current/Beta/Legacy drivers TBD - Single/Multi/Hybrid/etc, Internal/External, Current/Beta/Legacy drivers
Other TBD - 3rd-party software (Antivirus, Security, WindowsBlinds, etc), Preference changes TBD - 3rd-party software (Antivirus, Security, WindowsBlinds, etc), Preference changes
Channel Specific TBD - Unique coverage requirements for Nightly, Aurora, Beta, Release TBD - Unique coverage requirements for Nightly, Aurora, Beta, Release

Risks

Be sure we have test coverage to ensure:

  • GPU Process does not to regress acceleration coverage, performance, startup time, rendering quality, and correctness for platforms without GPU Process and platforms with GPU Process where GPU Process is disabled.
  • GPU Process recovers more gracefully from a crash than legacy
  • GPU Process handles device resets more gracefully than legacy
  • GPU Process returns with hardware (not software) compositor following device reset
  • GPU Process does not regress font rendering in chrome and content (eg. Skia fonts)
  • GPU Process does not fail in some way that results in a lot of users not getting Direct3D11 acceleration
  • GPU Process does not break existing "new" technologies (eg. WebVR, WebGL2)
  • GPU Process does not break legacy behaviours (eg. PDF viewing, Printing, Video playback, WebGL1, UI controls, about pages, etc)
  • Legacy behavior/quality/performance (GPU Process disabled) remains on-par with the Release prior to GPU Process shipping

Acceptable Risks:

  • GPU Process causes a moderate spike in acceleration crashes [specific number for acceptable spike to be determined]

Builds

This section should contain links for builds with the feature

Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Automated test execution Talos, etc...?
Bugs management Bugzilla
Experimentation Telemetry Experiments

Use Cases

Use Cases Smoketest Coverage BFT Coverage FFT Coverage Notes
The following need to be verified with GPU Process enabled, GPU Process disabled, and across a process reset
Printing Verify previewing and printing various content types is not regressed by GPU Process
Private Window Verify browser usage inside a Private Browsing window
Multi-Process Enabled Verify GPU process is created when multi-process is enabled
Multi-process Disabled Verify no GPU process is created when multi-process is disabled
Single-process Window Verify no GPU process when using a non-e10s window
Offline Content Verify accelerated content continues to render when in Offline mode
Video Verify HTML5 and Flash video playback acceleration
Tab/Window Management Verify arranging and management of tabs/windows
Theme Management Verify management of various Firefox and system-level themes
WebVR Verify WebVR demos continue to work at parity with the current Release
WebGL Verify WebGL demos continue to work at parity with the current Release
Dev Tools Verify Developer Tools continue to work at parity with the current Release
Custom fonts Verify content rendering, printing, scrolling, etc with common font customizations
2D Canvas Verify content rendering and performance with 2D Canvas and SVG demos
Accessibility
Mouse-only operation
Keyboard-only operation
Display (HiDPI)
Interraction (scroll, zoom)
Usable with a screen reader
Usability and/or discoverability testing
RTL build testing
Help/Support
Help/support interface required Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected
Does sync work across upgrades
Requires install testing Separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Partner build testing Does this affect partner builds? Yes/No options, add comment with details about who will lead testing
Enterprise

Raise up the topic to developers to see if they are expecting to work different on ESR builds

Enterprise administration
Network proxies/autoconfig
ESR behavior changes
Locked preferences
Data Monitoring
Temporary or permanent telemetry monitoring List of error conditions to monitor
Telemetry correctness testing
Server integration testing
Offline and server failure testing
Load testing
Add-ons

If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.

Addon API required?
Comprehensive API testing
Permissions
Testing with existing/popular addons
Security

Contact Matt Wobensmith to see if security testing is necessary for current feature

3rd-party security review
Privilege escalation testing
Fuzzing
Web Compatibility
Testing against target sites
Survey of many sites for compatibility
Interoperability
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS
Interaction of this feature with other browser features
Interaction of this feature with third-party software (Antivirus, Security, Games, Theming, etc)

Experiments

Experiment Contact Tracking Bug Status Start Date End Date
A/B test layers.gpu-process.dev.enabled in Nightly 53 Anthony Hughes (:ashughes) bug 1319206 Done [1] 2016-11-28 2016-12-24 2016-12-12 2017-01-11
A/B test media.gpu-process-decoder in Nightly 53 Anthony Hughes (:ashughes)

Nightly 53 A/B Test layers.gpu-process.dev.enabled

Purpose:

  • Compare crash rates for users with GPU process enabled vs users with GPU process disabled

Dependencies:

  • How do we deal with the fact that the pref is set early during startup? (emailed felipeg)
  • Telemetry probes to measure crash rates
  • Telemetry probe to measure proportion of users in each cohort

Measurements:

Bugs

Meta Bug - Dependency Tree
Full Query
ID Summary Status Resolution Depends on
1158076 Use light GTK theme for web content RESOLVED FIXED 1303310, 232227, 1216658, 1234279, 1282753, 1286000
1218576 Aggregate content process Telemetry in the main process VERIFIED FIXED 1468096, 1277504, 1282196, 1285006, 1286951, 1287101, 1290027, 1304009, 1304269, 1304519, 1304660, 1306856, 1306884, 1307031, 1307696, 1312778, 1356898
1255843 Develop a tool to measure resource usage of each subprocess RESOLVED FIXED 1277608, 1300850, 1300852, 1300853, 1300855, 1300856
1264545 Restrict Compositor access to nsIWidget RESOLVED FIXED
1265975 Remove compositor widget access on Windows RESOLVED FIXED 1264545, 1266536
1269037 Change CompositorVsyncScheduler to use widget proxies RESOLVED FIXED 1264545
1269653 Remove LayerManagerComposite access to nsIWidget RESOLVED FIXED 1271749
1271180 Introduce skeleton framework for the GPU process RESOLVED FIXED 1268559, 1277997
1271870 Remove compositor widget access on MacOS RESOLVED WONTFIX 1281686
1271872 Add process manager tool to developer tool. RESOLVED DUPLICATE
1272472 Isolate widget access to CompositorBridgeParent RESOLVED FIXED 1377415, 1273017
1274149 Restrict main-thread access to CompositorBridgeParent RESOLVED FIXED
1277068 Ignore SIGINT for all subprocesses RESOLVED FIXED
1277439 Add a non-Parent/Child protocol model RESOLVED WONTFIX
1280822 Make gfxPrefs usable in the GPU process RESOLVED FIXED
1281268 Measure overhead of GPU process on APZ RESOLVED FIXED
1281998 Implement IPDL version of CompositorWidgetProxy RESOLVED FIXED 1282343
1282348 Bootstrap out-of-process PCompositorBridge RESOLVED FIXED 1281575, 1281998, 1283744
1282364 Split up gfxWindowsPlatform RESOLVED FIXED 1288259, 1297182
1285625 Remote vsync notifications to the GPU process RESOLVED FIXED 1282348, 1285364
1287362 Don't use Preferences from CompositorLRU RESOLVED FIXED
1287366 Add ImageBridge support to the GPU process RESOLVED FIXED
1287597 Fix VRManager to work with the GPU process RESOLVED FIXED
1287917 Ensure ScrollMetadata::sNullMetadata is initialized in the GPU process RESOLVED FIXED
1287984 Content process cannot open GPU process DIBs RESOLVED FIXED
1288822 Create CompositorBridgeParent with a vsync rate RESOLVED FIXED
1289205 Fix typo in GPUChild::Init RESOLVED FIXED
1289251 Create a CompositorWidget for GTK+X11 Platform RESOLVED FIXED 1304199, 1304659
1289895 Add the GPU process as a broker peer RESOLVED FIXED
1294339 Don't use nsIGfxInfo in CompositorD3D11 RESOLVED FIXED
1294343 Remove fallback-to-WARP in the compositor RESOLVED FIXED
1294347 Always use DirectWrite if available RESOLVED FIXED
1294350 Fix GPUProcess startup issues RESOLVED FIXED
1294986 Disable DirectX in the UI process when using a GPU process RESOLVED FIXED
1294988 Enable Direct3D11 in the GPU process RESOLVED FIXED 1294986
1296749 Fix DirectX shutdown order RESOLVED FIXED
1297182 Add D3D9 support to the GPU process RESOLVED FIXED
1297251 [meta] Instant GPU Process Restarts NEW 1297260, 1337149, 1280715, 1297565, 1297567, 1297568, 1298938, 1300936, 1301780, 1302236, 1302611, 1303426, 1303451, 1303452, 1305628, 1305829, 1305897, 1305941, 1311715, 1311786, 1313770, 1314349, 1314375, 1314399, 1314408, 1314425, 1314427, 1314520, 1316419, 1316473, 1316476, 1316599, 1319213, 1319310, 1319557
1298507 Support tab-switching in the GPU process RESOLVED FIXED 1279086
1301266 Fix checkerboard reporting in the GPU process RESOLVED FIXED
1303820 Fix root scroll delta override when APZ is out of process RESOLVED FIXED
1304210 Fix crashes in ShadowLayerForwarder when the GPU process dies RESOLVED WONTFIX
1304494 Handle Telemetry::Accumulate in the GPU process RESOLVED FIXED
1305198 Fix null-deref when APZ is disabled in the compositor process RESOLVED FIXED
1307578 Let OOP compositor ride the trains for Windows RESOLVED FIXED 1294348, 1297790, 1297792, 1297822, 1297826, 1297828, 1297843, 1303451, 1307972, 1308398, 1309398, 1309714, 1309890, 1310110, 1310111, 1310397, 1310443, 1311716, 1311770, 1311786, 1312606, 1312835, 1313170, 1313199, 1313260, 1313499, 1313763, 1313978, 1314127, 1314133, 1314186, 1314349, 1314375, 1314399, 1314408, 1314425, 1314427, 1314520, 1314803, 1315699, 1315886, 1316419, 1316473, 1316481, 1316599, 1316632, 1317441, 1317796, 1318618, 1318845, 1319213, 1319310, 1319557, 1324095, 1325212, 1331005, 1339839, 1339852, 1343818
1314173 Add GPU process type to existing metrics RESOLVED FIXED 1314227, 1314231, 1314626
1314711 Killing the GPU process on Linux causes the parent process to terminate NEW
1314768 Tracking bug for all GPU process testing NEW 1297260, 1314426
1314804 D3D9DXVAManager still initializes in the UI process when the GPU process is enabled NEW
1317972 Work out what to do for crash reports with the GPU process NEW
1319206 Run a Telemetry experiment to vet GPU Process on Windows RESOLVED FIXED 1323304, 1325786
1321492 Add the GPU process to about:memory RESOLVED FIXED
1333880 D3D11 for h264 decoding fails since Firefox 53 when layers.gpu-process.enabled is true UNCONFIRMED
1338694 Add a telemetry probe for GPU_PROCESS_STATUS, similar to E10S_STATUS NEW
1343650 Annotate GPU Process in App Notes field RESOLVED FIXED
1350092 Run a Telemetry experiment to vet GPU Process on Windows prior to release in Firefox 53 RESOLVED FIXED
1352081 Google Maps experiences flickering when ending the GPU Process RESOLVED WONTFIX
1357744 No GPUProcessPid and GPIPRocess entries In the Graphics section of about:support in FIrefox 53 stable RESOLVED INCOMPLETE
1358304 Include GPUProcessPid in about:support for all release branches RESOLVED DUPLICATE
1362166 Enable IPDL timeout for communicating to the GPU process RESOLVED FIXED
1373088 Don't use a timeout for compositor process startup when debugging RESOLVED FIXED
1380144 Collect hang information from the compositor process RESOLVED DUPLICATE
1390488 GPU process crashes don't contain crash message (including rust panic messages) VERIFIED FIXED
1393703 Crash caused by Tab2QR add-on with GPU Process enabled NEW
1733680 [Linux/GPU] Crash at GPUProcessManager::SimulateDeviceReset() RESOLVED FIXED

65 Total; 8 Open (12.31%); 55 Resolved (84.62%); 2 Verified (3.08%);


Sign off

Nightly

  • Metrics:
    • Overall crash volume for graphics related issues is unchanged
  • Testing:
    • [MISSED] Testrail regression tests passed by Softvision [2]
  • Automation:
  • Areas of Risk
    • Automation: No unit or functional test coverage
Category Criteria Status
Metrics GPU_PROCESS_LAUNCH_TIME_MS_2 is ... pending review 
 5th Percentile: 0.192s
25th Percentile: 0.296s
         Median: 0.462s
75th Percentile: 0.914s
95th Percentile: 3.440s
GPU_PROCESS_LAUNCH_TIME_MS_2 Histogram is ... Pending developer review
MG: Main/Content/Plugin/GPU crashes [3]
GPU Process crash rate
SIMPLE_MEASURES_FIRSTPAINT does not regress
Experiments A/B stability does not regress -4% crashes overall, +5% plugin crashes (needs investigation) [4]
Testrail Regression test suite passed on all platforms Pending review
Automation TBD
Bugzilla 0 blocking regressions tracked for 53
Socorro 0 blocking signatures
Migration Okay to merge to Aurora
Aurora migration verified
Aurora release notes verified
Risks Description Mitigation
Automation No unit/function test coverage beyond a process startup test Manual testing of functionality and process termination/recovery
Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Okay to Merge Email to be sent
QA Signoff - Aurora Migration Email to be sent
QA Signoff - Verify Aurora Release Notes Email to be sent

Aurora

Criteria Status Notes
Testing
Manual Test Coverage [DONE]
  • [DONE] Softvision mid-Aurora sign-off via Testrail
  • [DONE] Aurora testday
  • [SKIPPED] Betabreakers regression
Automated Test Coverage [DONE] Coverage limited to process instantiation. Risks to be mitigated via manual testing.
Performance Test Coverage [SKIPPED]
  • [SKIPPED] Softvision, Betabreakers, and/or Community testing Passmark test
  • [SKIPPED] Softvision, Betabreakers, and/or Community testing WebGL and HWA Video benchmarks
Bug Reports
No unresolved blocker bugs
No untriaged bugs blocking Fx53 		[DONE] Bugs pending Firefox 53 status review:
No unverified fixed bugs [DONE] Bugs pending Firefox 53 status review:
OKRs & Metrics
GPU_PROCESS_LAUNCH_TIME_MS_2 distribution <= 3s [DONE] Note: A value larger than 3s likely represents users failing to get GPU Process
Percentile 2017-01-30 2017-02-06 2017-02-13 2017-02-20 2017-02-27 2017-03-06 Net
5th 0.222s 0.232s 0.234s 0.235s 0.236s 0.236s +0.014s (+6.3%)
25th 0.360s 0.366s 0.365s 0.364s 0.365s 0.365s +0.005s (+1.4%)
50th 0.574s 0.574s 0.572s 0.564s 0.567s 0.567s -0.007s (-1.2%)
75th 1.150s 1.140s 1.130s 1.100s 1.110s 1.110s -0.040s (-3.5%)
95th 4.240s 4.100s 4.060s 3.940s 3.950s 3.960s -0.280s (-6.6%)
3s Threshold 91.94% 92.20% 92.32% 92.41% 92.57% 92.55% +1.01%
1s Threshold 70.65% 71.05% 71.22% 71.28% 71.47% 71.79% +1.14%
GPU_PROCESS_LAUNCH_TIME_MS_2 evolution doesn't regress [DONE] Note: A value larger than 3s likely represents users failing to get GPU Process
Percentile 2017-01-30 2017-02-06 2017-02-13 2017-02-20 2017-02-27 2017-03-06 Net
5th 0.226s 0.236s 0.241s 0.242s 0.223s 0.227s +0.001s (+0.4%)
25th 0.358s 0.361s 0.365s 0.365s 0.339s 0.345s -0.013s (-3.6%)
50th 0.560s 0.561s 0.561s 0.560s 0.498s 0.515s -0.045s (-8.0%)
75th 1.120s 1.100s 1.090s 1.090s 0.834s 0.935s -0.185s (-16.5%)
95th 4.030s 3.930s 3.890s 3.800s 2.530s 3.050s -0.980s (-24.3%)
Telemetry graphics crash count/rate largely unchanged [DONE]
2017-01-30 2017-02-06 2017-02-13 2017-02-20 2017-02-27 2017-03-06 Net
Crash Rate Average 0.224 0.166 0.164 0.150 0.153 0.139 -0.085 (-37.9%)
Crash Count Average 64 73 87 86 88 74 +10 (+15.6%)
Crash Count Percent 1.17% 1.07% 1.18% 1.14% 1.17% 1.14% -0.03% (-2.5%)
No unresolved topcrashes correlated to GPU Process [DONE]

GPU crashes rank 5th overall @ 0.82% with a combined 1071 reports
1st-ranked is #47 overall @ 0.10% (mozilla::layers::CompositorD3D11::BeginFrame - bug 1306168)
2nd-ranked is #31 overall @ 0.13% (UMDevice::SetShaderResources - bug 1267997)
3rd-ranked is #51 overall @ 0.09% (nvwgf2um.dll | TCLSWrappers<T>::CLSDestroy - bug 1325933)
4th-ranked is #52 overall @ 0.09% (OOM | unknown | ... | mozilla::BufferList<T>::AllocateSegment)
5th-ranked is #172 overall @ 0.03% (nvwgf2umx.dll | NDXGI::CDevice::DestroySynchronizationObjectCB - bug 1328082)

Telemetry Experiment reveals no regressions in A/B testing [SKIPPED]
  • Compare SIMPLE_MEASURES_FIRSTPAINT as a proxy for performance
  • Compare crash counts/rates of cohorts in Telemetry
  • Compare crash counts/rates of cohorts in Socorro
  • Compare crash signatures of cohorts in Socorro
Percentage of users w/GPU Process and hardware profiles [SKIPPED]
  • Measured via David Anderson's telemetry dashboard?
  • GPU_PROCESS_STATUS probe added similar to E10S_STATUS? See bug 1338694.
Insignificant change in hardware acceleration distribution [DONE]

GPU Process Status:

Date        Available   Unavailable   Failed   Blocked
2017-02-23      71.2%         28.4%     0.3%      0.1%
2017-03-06      71.1%         28.5%     0.3%      0.1%

D3D11 Status:

Date        Available   Unavailable   Failed   Blocked   Blacklisted
2017-02-23      67.4%         12.2%     4.3%      1.5%         14.6%
2017-03-06      67.5%         12.3%     4.3%      1.4%         14.5%

Media Decoder Status:

Date         DXVA D3D9   DXVA D3D11   WMF Software
2017-02-23       25.3%        15.0%          59.7%
2017-03-06       25.2%        14.3%          60.5%
SIMPLE_MEASURES_FIRSTPAINT does not regress more than 5% [DONE] 
		Median	Improvement
Aurora 53	2.760s	 n/a
Aurora 52	3.040s	- 9.21%
Aurora 51	2.880s	- 4.17%
Aurora 50	3.100s	-10.97%

Note: even if it does regress it'll be hard to prove that it's GPU Process related and not some other change(s)

QA Signoffs
Okay to Merge to Beta [DONE] Signed off by Softvision on March 3, 2017 (report)
Beta migration signed-off [DONE] Signed off by Softvision on March 8, 2017 (report)
Beta release notes signed-off [DONE] Signed off by Softvision on March 8, 2017

Beta

Criteria Status Notes
Testing
Manual Test Coverage [DONE]
  • Softvision mid-Beta sign-off via Testrail
  • Softvision pre-Release sign-off via Testrail
  • Beta testday
Automated Test Coverage [DONE] Coverage limited to process instantiation. Risks to be mitigated via manual testing.
Performance Test Coverage [DONE]
  • Passmark benchmark completed by Softvision
  • WebGL benchmark completed by Softvision
  • HWA Video benchmark completed by Softvision
Bug Reports
No unresolved blocker bugs
No untriaged bugs blocking Fx53 		[DONE] Bugs pending Firefox 53 status review:
No unverified fixed bugs [DONE] Bugs pending Firefox 53 status review:
Metrics
GPU_PROCESS_LAUNCH_TIME_MS_2 distribution <= 3s [DONE] Note: A value larger than 3s likely represents users failing to get GPU Process
  • 5th Percentile: 0.213s
  • 25th Percentile: 0.342s
  • 50th Percentile: 0.605s
  • 75th Percentile: 1.250s
  • 95th Percentile: 4.620s
  • Thresholds: 90.67% @ 3s, 68.17% @ 1s
GPU_PROCESS_LAUNCH_TIME_MS_2 evolution doesn't regress [DONE] Note: A value larger than 3s likely represents users failing to get GPU Process
  • 5th Percentile: 0.219s
  • 25th Percentile: 0.325s
  • 50th Percentile: 0.496s
  • 75th Percentile: 0.936s
  • 95th Percentile: 3.210s
Telemetry graphics crash count/rate largely unchanged [DONE]
  • Crash Rate Average: 0.067 crashes per 1,000 usage hours
  • Crash Count Average: 603 crashes / day
  • Crash Count Percent: 0.389%
No unresolved topcrashes correlated to GPU Process [DONE]

GPU crashes rank 13th overall @ 0.34% with a combined 2,295 reports
1st-ranked is #115 overall @ 0.04% (mozilla::layers::CompositorD3D11::BeginFrame - bug 1306168)
2nd-ranked is #49 overall @ 0.09% (igd10iumd32.dll | CContext::EmptyOutAllDDIBindPoints - bug )
3rd-ranked is #95 overall @ 0.05% (UMDevice::SetShaderResources - bug 1267997)
Also track 53 vs 52 crash data for Ryan Pollock

Telemetry Experiment reveals no regressions in A/B testing [DONE] DONE - https://ashughes.com/?p=426
Insignificant change in hardware acceleration distribution [DONE]

GPU Process Status:

  • 47.3% Unavailable
  • 33.2% Unused
  • 19.4% Available

D3D11 Status:

  • 71.7% D3D11
  • 26.7% Basic (software)
  • 1.60% D3D9

Media Decoder Status:

  • 56.7% WMF (software)
  • 25.6% D3D9
  • 17.7% D3D11
SIMPLE_MEASURES_FIRSTPAINT does not regress more than 5% [DONE]
  • Median: 2.926s
  • 15.9% improvement vs Firefox 52
  • 18.4% improvement vs Firefox 51
  • 2.46% improvement vs Firefox 50

Note: even if it does regress it'll be hard to prove that it's GPU Process related and not some other change(s)

QA Signoffs
Okay to Merge to Release [DONE] DONE
Release migration signed-off [DONE] DONE
Release notes signed-off [DONE] DONE

Release

  • Verify release notes
  • Verify feature enabled in Release
  • Verify updating gives GPU Process
  • Check for bugs during throttle period

References

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Platform/Graphics/Features/GPU_Process&oldid=1168319"