MozillaWiki

QA/Platform/Graphics/Features/GPU Process: Difference between revisions

< QA‎ | Platform‎ | Graphics
 
(65 intermediate revisions by the same user not shown)
Line 344: Line 344:
|-
|-
|}
|}
== Test Suites ==
Full Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
Smoke Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
Regression Test suite - [https://testrail.stage.mozaws.net/index.php?/runs/view/2143]
Links for the tests run
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Full Test suite, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Regression Test suite, if needed/available


= Experiments =
= Experiments =
Line 387: Line 377:
** [https://sql.telemetry.mozilla.org/queries/2069/source Experiment by process model]
** [https://sql.telemetry.mozilla.org/queries/2069/source Experiment by process model]
** [https://sql.telemetry.mozilla.org/queries/2065/source Experiment by process type]
** [https://sql.telemetry.mozilla.org/queries/2065/source Experiment by process type]
== Nightly 53 A/B Test media.gpu-process-decoder ==
''TBD''


= Bugs =
= Bugs =
Line 402: Line 389:
= Sign off =
= Sign off =
== Nightly ==
== Nightly ==
* Metrics:
** Overall crash volume for graphics related issues is unchanged
**
* Testing:
** {{miss|}} Testrail regression tests passed by Softvision [https://testrail.stage.mozaws.net/index.php?/runs/view/2143]
* Automation:
* Areas of Risk
** Automation: No unit or functional test coverage
{| class="wikitable"
{| class="wikitable"
! Category
! Category
Line 407: Line 405:
! Status
! Status
|-
|-
| rowspan="2" style="vertical-align: top" | Telemetry
| rowspan="5" style="vertical-align: top" | Metrics
| [https://telemetry.mozilla.org/new-pipeline/evo.html#!aggregates=median!mean!5th-percentile!95th-percentile&cumulative=0&end_date=null&keys=&max_channel_version=nightly%252F53&measure=GPU_PROCESS_LAUNCH_TIME_MS_2&min_channel_version=nightly%252F50&product=Firefox&sanitize=1&sort_keys=submissions&start_date=null&trim=1&use_submission_date=0 GPU_PROCESS_LAUNCH_TIME_MS_2 Evolution] is ...
| [https://mzl.la/2kTYeTE GPU_PROCESS_LAUNCH_TIME_MS_2] is ...
| Pending developer review
| ''pending review''
<pre>
5th Percentile: 0.192s
25th Percentile: 0.296s
        Median: 0.462s
75th Percentile: 0.914s
95th Percentile: 3.440s
</pre>
|-
|-
| [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-23&keys=__none__!__none__!__none__&max_channel_version=nightly%252F53&measure=GPU_PROCESS_LAUNCH_TIME_MS_2&min_channel_version=nightly%252F50&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-12-10&table=0&trim=1&use_submission_date=0 GPU_PROCESS_LAUNCH_TIME_MS_2 Histogram] is ...
| [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-23&keys=__none__!__none__!__none__&max_channel_version=nightly%252F53&measure=GPU_PROCESS_LAUNCH_TIME_MS_2&min_channel_version=nightly%252F50&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-12-10&table=0&trim=1&use_submission_date=0 GPU_PROCESS_LAUNCH_TIME_MS_2 Histogram] is ...
| Pending developer review
| Pending developer review
|-
| MG: Main/Content/Plugin/GPU crashes [http://macmini/index.htm#psd-gpu-process]
|-
| [https://sql.telemetry.mozilla.org/dashboard/gfx-gpu-process-crashes GPU Process crash rate]
|-
| [https://telemetry.mozilla.org/new-pipeline/evo.html#!aggregates=median!mean!5th-percentile!25th-percentile!75th-percentile!95th-percentile&cumulative=0&end_date=null&keys=&max_channel_version=nightly%252F53&measure=SIMPLE_MEASURES_FIRSTPAINT&min_channel_version=nightly%252F50&os=Windows_NT%252C6.1!Windows_NT%252C10.0!Windows_NT%252C6.2&product=Firefox&sanitize=1&sort_keys=submissions&start_date=null&trim=1&use_submission_date=0 SIMPLE_MEASURES_FIRSTPAINT] does not regress
|
|-
| Experiments
| A/B stability does not regress
| -4% crashes overall, +5% plugin crashes (needs investigation) [https://ashughes.com/?p=374]
|-
|-
| Testrail
| Testrail
| Regression test suite passed on all platforms
| [https://testrail.stage.mozaws.net/index.php?/runs/view/2143 Regression test suite] passed on all platforms
| Pending review
| Pending review
|-
|-
Line 424: Line 440:
| Bugzilla
| Bugzilla
| 0 blocking regressions
| 0 blocking regressions
| [https://bugzilla.mozilla.org/buglist.cgi?list_id=13411501&o1=notequals&o2=equals&v1=---%2C%20unaffected&v2=%2B&f1=cf_status_firefox53&query_format=advanced&f2=cf_tracking_firefox53&component=Graphics&component=Graphics%3A%20Layers&component=Graphics%3A%20Text&product=Core tracked for 53]
|-
| Socorro
| 0 blocking signatures
|
|
|-
|-
Line 433: Line 453:
| Aurora release notes verified
| Aurora release notes verified
|-
|-
! Risks
! Description
! Mitigation
|-
| Automation
| No unit/function test coverage beyond a process startup test
| Manual testing of functionality and process termination/recovery
|}
|}


Line 480: Line 507:


== Aurora ==
== Aurora ==
{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:100%;"
|-
! style="width:30%" | Criteria
! style="width:10%" | Status
! style="width:60%" | Notes
|-
! colspan="3" style="text-align:left" | Testing
|-
| style="vertical-align:top" | Manual Test Coverage
| style="text-align:center; vertical-align:top" | {{done|}}
|
* {{done|}} Softvision mid-Aurora sign-off via [https://testrail.stage.mozaws.net/index.php?/suites/overview/46 Testrail]
* {{done|}} Aurora testday
* {{skip|}} Betabreakers regression
|-
| style="vertical-align:top" | Automated Test Coverage
| style="text-align:center; vertical-align:top" | {{done|}}
| Coverage limited to process instantiation. Risks to be mitigated via manual testing.
|-
| Performance Test Coverage
| style="text-align:center; vertical-align:top" | {{skip|}}
|
* {{skip|}} Softvision, Betabreakers, and/or Community testing Passmark test
* {{skip|}} Softvision, Betabreakers, and/or Community testing WebGL and HWA Video benchmarks
|-
! colspan="3" style="text-align:left" | Bug Reports
|-
| style="vertical-align:top" | [https://mzl.la/2mGuyaD No unresolved blocker bugs] <br /> [https://mzl.la/2mGmR4k No untriaged bugs blocking Fx53]
| style="text-align:center; vertical-align:top" | {{done|}}
| <u>Bugs pending Firefox 53 status review:</u><br />
<bugzilla display="list">
{
"blocks":"1264543",
"f1":"cf_status_firefox53",
"o1":"anywordssubstr",
"v1":"affected, ?",
"include_fields":"id,summary,status"
}
</bugzilla>
|-
| style="vertical-align:top" | [https://mzl.la/2lA94Lt No unverified fixed bugs]
| style="text-align:center; vertical-align:top" | {{done|}}
|<u>Bugs pending Firefox 53 status review:</u><br />
<bugzilla display="list">
{
"blocks":"1264543",
"resolution":"FIXED",
"f1":"cf_status_firefox53",
"o1":"notequals",
"v1":"fixed",
"f2":"target_milestone",
"o2":"anywordssubstr",
"v2":"---, mozilla53",
"include_fields":"id,summary,status"
}
</bugzilla>
|-
|-
! Criteria !! Status !! Notes/Details
! colspan="3" style="text-align:left" | OKRs & Metrics
|- 
| style="vertical-align:top" | [https://mzl.la/2kU1sq9 GPU_PROCESS_LAUNCH_TIME_MS_2 distribution <= 3s]
| style="vertical-align:top; text-align:center" | {{done|}}
| ''Note: A value larger than 3s likely represents users failing to get GPU Process''
{| class="wikitable"
! Percentile || 2017-01-30 || 2017-02-06 || 2017-02-13 || 2017-02-20 || 2017-02-27 || 2017-03-06 || Net
|-
|-
| Testing Prerequisites (specs, use cases)
| 5th || 0.222s || 0.232s || 0.234s || 0.235s || 0.236s || 0.236s || +0.014s (+6.3%)
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
| Testing Infrastructure setup
| 25th || 0.360s || 0.366s || 0.365s || 0.364s || 0.365s || 0.365s || +0.005s (+1.4%)
|style="text-align:center;" |   ||  
|-
|-
| Test Plan Creation
| 50th || 0.574s || 0.574s || 0.572s || 0.564s || 0.567s || 0.567s || -0.007s (-1.2%)
| style="text-align:center;" |   ||  
|-
|-
| Test Cases Creation
| 75th || 1.150s || 1.140s || 1.130s || 1.100s || 1.110s || 1.110s || -0.040s (-3.5%)
|style="text-align:center;" |   ||  
|-
|-
| Full Functional Tests Execution
| 95th || 4.240s || 4.100s || 4.060s || 3.940s || 3.950s || 3.960s || -0.280s (-6.6%)
|style="text-align:center;" |
|style="text-align:center;" | 
|-
|-
| Automation Coverage ||
! 3s Threshold || 91.94% || 92.20% || 92.32% || 92.41% || 92.57% || 92.55% || +1.01%
|style="text-align:center;" |  
|-
|-
| Performance Testing
! 1s Threshold || 70.65% || 71.05% || 71.22% || 71.28% || 71.47% || 71.79% || +1.14%
|style="text-align:center;" | ||  
|}
|-
|-
| All Defects Logged || ||  
| style="vertical-align:top" | [https://mzl.la/2kXKjvM GPU_PROCESS_LAUNCH_TIME_MS_2 evolution doesn't regress]
| style="vertical-align:top; text-align:center" | {{done|}}
| ''Note: A value larger than 3s likely represents users failing to get GPU Process''
{| class="wikitable"
! Percentile || 2017-01-30 || 2017-02-06 || 2017-02-13 || 2017-02-20 || 2017-02-27 || 2017-03-06 || Net
|-
|-
| Critical/Blockers Fixed and Verified || ||  
| 5th || 0.226s || 0.236s || 0.241s || 0.242s || 0.223s || 0.227s || +0.001s (+0.4%)
|-
|-
| Metrics/Telemetry||  
| 25th || 0.358s || 0.361s || 0.365s || 0.365s || 0.339s || 0.345s || -0.013s (-3.6%)
|style="text-align:center;" |
|-
|-
| QA Signoff - Okay to Merge ||  
| 50th || 0.560s || 0.561s || 0.561s || 0.560s || 0.498s || 0.515s || -0.045s (-8.0%)
|style="text-align:center;" | Email to be sent
|-
|-
| QA Signoff - Beta Migration ||
| 75th || 1.120s || 1.100s || 1.090s || 1.090s || 0.834s || 0.935s || -0.185s (-16.5%)
|style="text-align:center;" | Email to be sent
|-
|-
| QA Signoff - Verify Beta Release Notes ||  
| 95th || 4.030s || 3.930s || 3.890s || 3.800s || 2.530s || 3.050s || -0.980s (-24.3%)
|style="text-align:center;" | Email to be sent
|}
|}
== Beta ==
{| class="wikitable" style="width:60%"
|-
|-
! Criteria !! Status !! Notes/Details
| style="vertical-align:top" | [https://sql.telemetry.mozilla.org/dashboard/gfx-gpu-process-crashes Telemetry graphics crash count/rate largely unchanged]
| style="text-align:center; vertical-align:top" | {{done|}}
|
{| class="wikitable"
! || 2017-01-30 || 2017-02-06 || 2017-02-13 || 2017-02-20 || 2017-02-27 || 2017-03-06 || Net
|-
| Crash Rate Average || 0.224 || 0.166 || 0.164 || 0.150 || 0.153 || 0.139 || -0.085 (-37.9%)
|-
|-
| Testing Prerequisites (specs, use cases)
| Crash Count Average || 64 || 73 || 87 || 86 || 88 || 74 || +10 (+15.6%)
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
| Testing Infrastructure setup
| Crash Count Percent || 1.17% || 1.07% || 1.18% || 1.14% || 1.17% || 1.14% || -0.03% (-2.5%)
|style="text-align:center;" |   ||  
|}
|-
|-
| Test Plan Creation
| style="vertical-align:top" | [https://crash-stats.mozilla.com/topcrashers/?product=Firefox&version=53.0a2&days=7&process_type=gpu No unresolved topcrashes correlated to GPU Process]
| style="text-align:center;" |   ||  
| style="text-align:center; vertical-align:top" | {{done|}}
|  
<u>GPU crashes rank 5th overall @ 0.82% with a combined 1071 reports</u><br />
1st-ranked is #47 overall @ 0.10% (mozilla::layers::CompositorD3D11::BeginFrame - {{bug|1306168}})<br />
2nd-ranked is #31 overall @ 0.13% (UMDevice::SetShaderResources - {{bug|1267997}})<br />
3rd-ranked is #51 overall @ 0.09% (nvwgf2um.dll | TCLSWrappers<T>::CLSDestroy - {{bug|1325933}})<br />
4th-ranked is #52 overall @ 0.09% (OOM | unknown | ... | mozilla::BufferList<T>::AllocateSegment)<br />
5th-ranked is #172 overall @ 0.03% (nvwgf2umx.dll | NDXGI::CDevice::DestroySynchronizationObjectCB - {{bug|1328082}})
 
|-
|-
| Test Cases Creation
| style="vertical-align:top" | Telemetry Experiment reveals no regressions in A/B testing
|style="text-align:center;" |   ||  
| style="text-align:center; vertical-align:top" | {{skip|}}
|
* Compare SIMPLE_MEASURES_FIRSTPAINT as a proxy for performance
* Compare crash counts/rates of cohorts in Telemetry
* Compare crash counts/rates of cohorts in Socorro
* Compare crash signatures of cohorts in Socorro
|-
|-
|  Full Functional Tests Execution
| style="vertical-align:top" | Percentage of users w/GPU Process and hardware profiles
|style="text-align:center;" |
| style="text-align:center; vertical-align:top" | {{skip|}}
|style="text-align:center;" |
|
* Measured via David Anderson's telemetry dashboard?
* GPU_PROCESS_STATUS probe added similar to E10S_STATUS? See {{bug|1338694}}.
|-
|-
| Automation Coverage ||
| style="vertical-align:top" | [https://people-mozilla.org/~danderson/moz-gfx-telemetry/www/#view=windows-features Insignificant change in hardware acceleration distribution]
|style="text-align:center;" |  
| style="text-align:center; vertical-align:top" | {{done|}}
|
GPU Process Status:<br />
<pre>
Date        Available  Unavailable  Failed  Blocked
2017-02-23      71.2%        28.4%    0.3%      0.1%
2017-03-06      71.1%        28.5%    0.3%      0.1%
</pre>
D3D11 Status:<br />
<pre>
Date        Available  Unavailable  Failed  Blocked  Blacklisted
2017-02-23      67.4%        12.2%    4.3%      1.5%        14.6%
2017-03-06      67.5%        12.3%    4.3%      1.4%        14.5%
</pre>
Media Decoder Status:<br />
<pre>
Date        DXVA D3D9  DXVA D3D11  WMF Software
2017-02-23      25.3%        15.0%          59.7%
2017-03-06      25.2%        14.3%          60.5%
</pre>
|-
|-
| Performance Testing
| style="vertical-align:top" | [https://mzl.la/2lnuYCz SIMPLE_MEASURES_FIRSTPAINT does not regress more than 5%]
|style="text-align:center;" | ||  
| style="text-align:center; vertical-align:top" | {{done|}}
|  
<pre>
Median Improvement
Aurora 53 2.760s n/a
Aurora 52 3.040s - 9.21%
Aurora 51 2.880s - 4.17%
Aurora 50 3.100s -10.97%
</pre>
''Note: even if it does regress it'll be hard to prove that it's GPU Process related and not some other change(s)''
|-
|-
| All Defects Logged || ||
! colspan="3" style="text-align:left" | QA Signoffs
|-
|-
| Critical/Blockers Fixed and Verified || ||  
| Okay to Merge to Beta
| style="text-align:center; vertical-align:top" | {{done|}}
| Signed off by Softvision on March 3, 2017 ([https://testrail.stage.mozaws.net/index.php?/reports/view/211 report])
|-
|-
| Metrics/Telemetry||
| Beta migration signed-off
|style="text-align:center;" |  
| style="text-align:center; vertical-align:top" | {{done|}}
| Signed off by Softvision on March 8, 2017 ([https://wiki.mozilla.org/Releases/Firefox_53/Test_Plan/Beta/1 report])
|-
|-
| QA Signoff - Beta 1..n ||
| Beta release notes signed-off
|style="text-align:center;" | Email to be sent
| style="text-align:center; vertical-align:top" | {{done|}}
| Signed off by Softvision on March 8, 2017
|-
|-
| QA Signoff - Release Migration ||
|style="text-align:center;" | Email to be sent
|}
|}


== Release ==
== Beta ==
{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:100%;"
|-
! style="width:30%" | Criteria
! style="width:10%" | Status
! style="width:60%" | Notes
|-
! colspan="3" style="text-align:left" | Testing
|-
| style="vertical-align:top" | Manual Test Coverage
| style="text-align:center; vertical-align:top" | {{done|}}
|
* Softvision mid-Beta sign-off via Testrail
* Softvision pre-Release sign-off via Testrail
* Beta testday
|-
| style="vertical-align:top" | Automated Test Coverage
| style="text-align:center; vertical-align:top" | {{done|}}
| Coverage limited to process instantiation. Risks to be mitigated via manual testing.
|-
| Performance Test Coverage
| style="text-align:center; vertical-align:top" | {{done|}}
|
* Passmark benchmark completed by Softvision
* WebGL benchmark completed by Softvision
* HWA Video benchmark completed by Softvision
|-
! colspan="3" style="text-align:left" | Bug Reports
|-
|-
! Criteria !! Status !! Notes/Details
| style="vertical-align:top" | [https://mzl.la/2mGuyaD No unresolved blocker bugs] <br /> [https://mzl.la/2mGmR4k No untriaged bugs blocking Fx53]
| style="text-align:center; vertical-align:top" | {{done|}}
| <u>Bugs pending Firefox 53 status review:</u><br />
<bugzilla display="list">
{
"blocks":"1264543",
"f1":"cf_status_firefox53",
"o1":"anywordssubstr",
"v1":"affected, ?",
"include_fields":"id,summary,status"
}
</bugzilla>
|-
|-
|  Testing Prerequisites (specs, use cases)
| style="vertical-align:top" | [https://mzl.la/2lA94Lt No unverified fixed bugs]
| style="text-align:center;" |  
| style="text-align:center; vertical-align:top" | {{done|}}
| style="text-align:center;" |  
|<u>Bugs pending Firefox 53 status review:</u><br />
<bugzilla display="list">
{
"blocks":"1264543",
"resolution":"FIXED",
"f1":"cf_status_firefox53",
"o1":"notequals",
"v1":"fixed",
"f2":"target_milestone",
"o2":"anywordssubstr",
"v2":"---, mozilla53",
"include_fields":"id,summary,status"
}
</bugzilla>
|-
|-
|  Testing Infrastructure setup
! colspan="3" style="text-align:left" | Metrics
|style="text-align:center;" |   ||  
|-  
| style="vertical-align:top" | [https://mzl.la/2lY5Jdt GPU_PROCESS_LAUNCH_TIME_MS_2 distribution <= 3s]
| style="text-align:center; vertical-align:top" | {{done|}}
| ''Note: A value larger than 3s likely represents users failing to get GPU Process''
* 5th Percentile: 0.213s
* 25th Percentile: 0.342s
* 50th Percentile: 0.605s
* 75th Percentile: 1.250s
* 95th Percentile: 4.620s
* Thresholds: 90.67% @ 3s, 68.17% @ 1s
|-
|-
| Test Plan Creation
| style="vertical-align:top" | [https://mzl.la/2lXWWIA GPU_PROCESS_LAUNCH_TIME_MS_2 evolution doesn't regress]
| style="text-align:center;" |   ||  
| style="text-align:center; vertical-align:top" | {{done|}}
| ''Note: A value larger than 3s likely represents users failing to get GPU Process''
* 5th Percentile: 0.219s
* 25th Percentile: 0.325s
* 50th Percentile: 0.496s
* 75th Percentile: 0.936s
* 95th Percentile: 3.210s
|-
|-
| Test Cases Creation
| style="vertical-align:top" | [https://sql.telemetry.mozilla.org/dashboard/gfx-gpu-process-crashes Telemetry graphics crash count/rate largely unchanged]
|style="text-align:center;" |   ||  
| style="text-align:center; vertical-align:top" | {{done|}}
|
* Crash Rate Average: 0.067 crashes per 1,000 usage hours
* Crash Count Average: 603 crashes / day
* Crash Count Percent: 0.389%
|-
|-
|  Full Functional Tests Execution
| style="vertical-align:top" | [https://crash-stats.mozilla.com/topcrashers/?product=Firefox&version=53.0b&days=28&process_type=gpu No unresolved topcrashes correlated to GPU Process]
|style="text-align:center;" |
| style="text-align:center; vertical-align:top" | {{done|}}
|style="text-align:center;" |
|
<u>GPU crashes rank 13th overall @ 0.34% with a combined 2,295 reports</u><br />
1st-ranked is #115 overall @ 0.04% (mozilla::layers::CompositorD3D11::BeginFrame - {{bug|1306168}})<br />
2nd-ranked is #49 overall @ 0.09% (igd10iumd32.dll | CContext::EmptyOutAllDDIBindPoints - {{bug|}})<br />
3rd-ranked is #95 overall @ 0.05% (UMDevice::SetShaderResources - {{bug|1267997}})
<br />
<b>Also track [https://goo.gl/9b1rb0 53 vs 52 crash data for Ryan Pollock]</b>
|-
|-
| Automation Coverage ||
| style="vertical-align:top" | Telemetry Experiment reveals no regressions in A/B testing
|style="text-align:center;" |  
| style="text-align:center; vertical-align:top" | {{done|}}
| DONE - https://ashughes.com/?p=426
* [https://bsmedberg.github.io/telemetry-experiments-dashboard/ Experiment stats]
* Compare SIMPLE_MEASURES_FIRSTPAINT as a proxy for performance
* [https://sql.telemetry.mozilla.org/dashboard/gfx-gpu-process-beta-53-experiment Compare crash counts/rates of cohorts in Telemetry]
* [https://docs.google.com/spreadsheets/d/1KDESBrehRcJVpUCfEouRmElL8q2z-I_bPkVe54Arpqs/edit#gid=0 Compare crash counts/rates of cohorts in Socorro]
* Compare crash signatures of cohorts in Socorro
|-
|-
| Performance Testing
| style="vertical-align:top" | [https://people-mozilla.org/~danderson/moz-gfx-telemetry/www/#view=windows-features Insignificant change in hardware acceleration distribution]
|style="text-align:center;" | ||  
| style="text-align:center; vertical-align:top" | {{done|}}
|  
GPU Process Status:
* 47.3% Unavailable
* 33.2% Unused
* 19.4% Available
D3D11 Status:
* 71.7% D3D11
* 26.7% Basic (software)
* 1.60% D3D9
Media Decoder Status:
* 56.7% WMF (software)
* 25.6% D3D9
* 17.7% D3D11
|-
|-
| All Defects Logged || ||  
| style="vertical-align:top" | [https://mzl.la/2neFmgk SIMPLE_MEASURES_FIRSTPAINT does not regress more than 5%]
| style="text-align:center; vertical-align:top" | {{done|}}
|
* Median: 2.926s
* 15.9% improvement vs Firefox 52
* 18.4% improvement vs Firefox 51
* 2.46% improvement vs Firefox 50
''Note: even if it does regress it'll be hard to prove that it's GPU Process related and not some other change(s)''
|-
|-
| Critical/Blockers Fixed and Verified || ||
! colspan="3" style="text-align:left" | QA Signoffs
|-
|-
| Metrics/Telemetry||
| Okay to Merge to Release
|style="text-align:center;" |  
| style="text-align:center; vertical-align:top" | {{done|}}
| DONE
|-
|-
| QA Signoff - Okay to Release ||
| Release migration signed-off
|style="text-align:center;" | Email to be sent
| style="text-align:center; vertical-align:top" | {{done|}}
| DONE
|-
|-
| QA Signoff - Verify Release Notes ||
| Release notes signed-off
|style="text-align:center;" | Email to be sent
| style="text-align:center; vertical-align:top" | {{done|}}
| DONE
|-
|-
|}
|}
== Release ==
* Verify release notes
* Verify feature enabled in Release
* Verify updating gives GPU Process
* Check for bugs during throttle period


= References =
= References =
* List and links for available specs - documents, user stories, specifications
* Meta bug: {{bug|1264543}}
* Meta bug: {{bug|1264543}}
* https://public.etherpad-mozilla.org/p/GPU_Process_Testing
* https://public.etherpad-mozilla.org/p/GPU_Process_Testing
= Revision History =
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded).  The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
{| class="wikitable" style="text-align:center"
|-
! Date !! Version !! Author !! Description
|-
| 2016-11-02 || 1.0 || Anthony Hughes || Created first draft
|-
|}

Latest revision as of 23:20, 13 April 2017

Overview

Purpose

The purpose of this document is to describe the GPU Process feature from a testing perspective, including the following details:

  • Scope, focus areas and objectives of testing
  • Owners and points of contact for areas of responsibility
  • Strategy and types of testing
  • The entry and exit criteria
  • The basis of the test estimates
  • Any risks, issues, assumptions and test dependencies
  • The test schedule and major milestones
  • The test deliverables

Ownership

Responsibility Role Person
Accountable Engineering Manager Milan Sreckovic (:milan)
Responsible Tech Lead David Anderson (:dvander)
Responsible QA Lead Anthony Hughes (:ashughes)
Supporting Engineer George Wright (:gw280)
Supporting Engineer Ryan Hunt (:rhunt)

Scope

The following is in scope from a testing perspective for the project team.

  • GPU process startup time
  • Page load time
  • GPU process restart time (in case of process kill)
  • Real world performance with various common usecases (webgl, canvas2d, flash).
  • Plugin startup time
  • Antivirus conflicts
  • Memory footprint
  • Shutdown time
  • Overall product stability
  • Scrolling performance and correctness

The following is not in scope from a testing perspective for the project team.

  • to be determined

Schedule

Project phase Date Started Date Completed
Planning
Development planning & Spec documentation
Test planning & Spec review
Development
Code lands in Nightly
Test case development
Test environment preparation
QA
Nightly testing & sign-off
Aurora testing & sign-off
Beta testing & sign-off
RC testing & sign-off
Release

Outstanding Questions

Test Coverage

Environments

Coverage In Scope for MVP Out of Scope for MVP
Platforms Windows 7 Platform Update and later Windows 7 without Platform Update and earlier, Mac OS, Linux, Android, iOS
Devices AMD, Intel, and NVIDIA hardware with Direct3D 11 and later support AMD, Intel, and NVIDIA hardware with Direct3D 10 and earlier support
Configurations TBD - Single/Multi/Hybrid/etc, Internal/External, Current/Beta/Legacy drivers TBD - Single/Multi/Hybrid/etc, Internal/External, Current/Beta/Legacy drivers
Other TBD - 3rd-party software (Antivirus, Security, WindowsBlinds, etc), Preference changes TBD - 3rd-party software (Antivirus, Security, WindowsBlinds, etc), Preference changes
Channel Specific TBD - Unique coverage requirements for Nightly, Aurora, Beta, Release TBD - Unique coverage requirements for Nightly, Aurora, Beta, Release

Risks

Be sure we have test coverage to ensure:

  • GPU Process does not to regress acceleration coverage, performance, startup time, rendering quality, and correctness for platforms without GPU Process and platforms with GPU Process where GPU Process is disabled.
  • GPU Process recovers more gracefully from a crash than legacy
  • GPU Process handles device resets more gracefully than legacy
  • GPU Process returns with hardware (not software) compositor following device reset
  • GPU Process does not regress font rendering in chrome and content (eg. Skia fonts)
  • GPU Process does not fail in some way that results in a lot of users not getting Direct3D11 acceleration
  • GPU Process does not break existing "new" technologies (eg. WebVR, WebGL2)
  • GPU Process does not break legacy behaviours (eg. PDF viewing, Printing, Video playback, WebGL1, UI controls, about pages, etc)
  • Legacy behavior/quality/performance (GPU Process disabled) remains on-par with the Release prior to GPU Process shipping

Acceptable Risks:

  • GPU Process causes a moderate spike in acceleration crashes [specific number for acceptable spike to be determined]

Builds

This section should contain links for builds with the feature

Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Automated test execution Talos, etc...?
Bugs management Bugzilla
Experimentation Telemetry Experiments

Use Cases

Use Cases Smoketest Coverage BFT Coverage FFT Coverage Notes
The following need to be verified with GPU Process enabled, GPU Process disabled, and across a process reset
Printing Verify previewing and printing various content types is not regressed by GPU Process
Private Window Verify browser usage inside a Private Browsing window
Multi-Process Enabled Verify GPU process is created when multi-process is enabled
Multi-process Disabled Verify no GPU process is created when multi-process is disabled
Single-process Window Verify no GPU process when using a non-e10s window
Offline Content Verify accelerated content continues to render when in Offline mode
Video Verify HTML5 and Flash video playback acceleration
Tab/Window Management Verify arranging and management of tabs/windows
Theme Management Verify management of various Firefox and system-level themes
WebVR Verify WebVR demos continue to work at parity with the current Release
WebGL Verify WebGL demos continue to work at parity with the current Release
Dev Tools Verify Developer Tools continue to work at parity with the current Release
Custom fonts Verify content rendering, printing, scrolling, etc with common font customizations
2D Canvas Verify content rendering and performance with 2D Canvas and SVG demos
Accessibility
Mouse-only operation
Keyboard-only operation
Display (HiDPI)
Interraction (scroll, zoom)
Usable with a screen reader
Usability and/or discoverability testing
RTL build testing
Help/Support
Help/support interface required Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected
Does sync work across upgrades
Requires install testing Separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Partner build testing Does this affect partner builds? Yes/No options, add comment with details about who will lead testing
Enterprise

Raise up the topic to developers to see if they are expecting to work different on ESR builds

Enterprise administration
Network proxies/autoconfig
ESR behavior changes
Locked preferences
Data Monitoring
Temporary or permanent telemetry monitoring List of error conditions to monitor
Telemetry correctness testing
Server integration testing
Offline and server failure testing
Load testing
Add-ons

If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.

Addon API required?
Comprehensive API testing
Permissions
Testing with existing/popular addons
Security

Contact Matt Wobensmith to see if security testing is necessary for current feature

3rd-party security review
Privilege escalation testing
Fuzzing
Web Compatibility
Testing against target sites
Survey of many sites for compatibility
Interoperability
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS
Interaction of this feature with other browser features
Interaction of this feature with third-party software (Antivirus, Security, Games, Theming, etc)

Experiments

Experiment Contact Tracking Bug Status Start Date End Date
A/B test layers.gpu-process.dev.enabled in Nightly 53 Anthony Hughes (:ashughes) bug 1319206 Done [1] 2016-11-28 2016-12-24 2016-12-12 2017-01-11
A/B test media.gpu-process-decoder in Nightly 53 Anthony Hughes (:ashughes)

Nightly 53 A/B Test layers.gpu-process.dev.enabled

Purpose:

  • Compare crash rates for users with GPU process enabled vs users with GPU process disabled

Dependencies:

  • How do we deal with the fact that the pref is set early during startup? (emailed felipeg)
  • Telemetry probes to measure crash rates
  • Telemetry probe to measure proportion of users in each cohort

Measurements:

Bugs

Meta Bug - Dependency Tree
Full Query
ID Summary Status Resolution Depends on
1158076 Use light GTK theme for web content RESOLVED FIXED 1303310, 232227, 1216658, 1234279, 1282753, 1286000
1218576 Aggregate content process Telemetry in the main process VERIFIED FIXED 1468096, 1277504, 1282196, 1285006, 1286951, 1287101, 1290027, 1304009, 1304269, 1304519, 1304660, 1306856, 1306884, 1307031, 1307696, 1312778, 1356898
1255843 Develop a tool to measure resource usage of each subprocess RESOLVED FIXED 1277608, 1300850, 1300852, 1300853, 1300855, 1300856
1264545 Restrict Compositor access to nsIWidget RESOLVED FIXED
1265975 Remove compositor widget access on Windows RESOLVED FIXED 1264545, 1266536
1269037 Change CompositorVsyncScheduler to use widget proxies RESOLVED FIXED 1264545
1269653 Remove LayerManagerComposite access to nsIWidget RESOLVED FIXED 1271749
1271180 Introduce skeleton framework for the GPU process RESOLVED FIXED 1268559, 1277997
1271870 Remove compositor widget access on MacOS RESOLVED WONTFIX 1281686
1271872 Add process manager tool to developer tool. RESOLVED DUPLICATE
1272472 Isolate widget access to CompositorBridgeParent RESOLVED FIXED 1377415, 1273017
1274149 Restrict main-thread access to CompositorBridgeParent RESOLVED FIXED
1277068 Ignore SIGINT for all subprocesses RESOLVED FIXED
1277439 Add a non-Parent/Child protocol model RESOLVED WONTFIX
1280822 Make gfxPrefs usable in the GPU process RESOLVED FIXED
1281268 Measure overhead of GPU process on APZ RESOLVED FIXED
1281998 Implement IPDL version of CompositorWidgetProxy RESOLVED FIXED 1282343
1282348 Bootstrap out-of-process PCompositorBridge RESOLVED FIXED 1281575, 1281998, 1283744
1282364 Split up gfxWindowsPlatform RESOLVED FIXED 1288259, 1297182
1285625 Remote vsync notifications to the GPU process RESOLVED FIXED 1282348, 1285364
1287362 Don't use Preferences from CompositorLRU RESOLVED FIXED
1287366 Add ImageBridge support to the GPU process RESOLVED FIXED
1287597 Fix VRManager to work with the GPU process RESOLVED FIXED
1287917 Ensure ScrollMetadata::sNullMetadata is initialized in the GPU process RESOLVED FIXED
1287984 Content process cannot open GPU process DIBs RESOLVED FIXED
1288822 Create CompositorBridgeParent with a vsync rate RESOLVED FIXED
1289205 Fix typo in GPUChild::Init RESOLVED FIXED
1289251 Create a CompositorWidget for GTK+X11 Platform RESOLVED FIXED 1304199, 1304659
1289895 Add the GPU process as a broker peer RESOLVED FIXED
1294339 Don't use nsIGfxInfo in CompositorD3D11 RESOLVED FIXED
1294343 Remove fallback-to-WARP in the compositor RESOLVED FIXED
1294347 Always use DirectWrite if available RESOLVED FIXED
1294350 Fix GPUProcess startup issues RESOLVED FIXED
1294986 Disable DirectX in the UI process when using a GPU process RESOLVED FIXED
1294988 Enable Direct3D11 in the GPU process RESOLVED FIXED 1294986
1296749 Fix DirectX shutdown order RESOLVED FIXED
1297182 Add D3D9 support to the GPU process RESOLVED FIXED
1297251 [meta] Instant GPU Process Restarts NEW 1297260, 1337149, 1280715, 1297565, 1297567, 1297568, 1298938, 1300936, 1301780, 1302236, 1302611, 1303426, 1303451, 1303452, 1305628, 1305829, 1305897, 1305941, 1311715, 1311786, 1313770, 1314349, 1314375, 1314399, 1314408, 1314425, 1314427, 1314520, 1316419, 1316473, 1316476, 1316599, 1319213, 1319310, 1319557
1298507 Support tab-switching in the GPU process RESOLVED FIXED 1279086
1301266 Fix checkerboard reporting in the GPU process RESOLVED FIXED
1303820 Fix root scroll delta override when APZ is out of process RESOLVED FIXED
1304210 Fix crashes in ShadowLayerForwarder when the GPU process dies RESOLVED WONTFIX
1304494 Handle Telemetry::Accumulate in the GPU process RESOLVED FIXED
1305198 Fix null-deref when APZ is disabled in the compositor process RESOLVED FIXED
1307578 Let OOP compositor ride the trains for Windows RESOLVED FIXED 1294348, 1297790, 1297792, 1297822, 1297826, 1297828, 1297843, 1303451, 1307972, 1308398, 1309398, 1309714, 1309890, 1310110, 1310111, 1310397, 1310443, 1311716, 1311770, 1311786, 1312606, 1312835, 1313170, 1313199, 1313260, 1313499, 1313763, 1313978, 1314127, 1314133, 1314186, 1314349, 1314375, 1314399, 1314408, 1314425, 1314427, 1314520, 1314803, 1315699, 1315886, 1316419, 1316473, 1316481, 1316599, 1316632, 1317441, 1317796, 1318618, 1318845, 1319213, 1319310, 1319557, 1324095, 1325212, 1331005, 1339839, 1339852, 1343818
1314173 Add GPU process type to existing metrics RESOLVED FIXED 1314227, 1314231, 1314626
1314711 Killing the GPU process on Linux causes the parent process to terminate NEW
1314768 Tracking bug for all GPU process testing NEW 1297260, 1314426
1314804 D3D9DXVAManager still initializes in the UI process when the GPU process is enabled NEW
1317972 Work out what to do for crash reports with the GPU process NEW
1319206 Run a Telemetry experiment to vet GPU Process on Windows RESOLVED FIXED 1323304, 1325786
1321492 Add the GPU process to about:memory RESOLVED FIXED
1333880 D3D11 for h264 decoding fails since Firefox 53 when layers.gpu-process.enabled is true UNCONFIRMED
1338694 Add a telemetry probe for GPU_PROCESS_STATUS, similar to E10S_STATUS NEW
1343650 Annotate GPU Process in App Notes field RESOLVED FIXED
1350092 Run a Telemetry experiment to vet GPU Process on Windows prior to release in Firefox 53 RESOLVED FIXED
1352081 Google Maps experiences flickering when ending the GPU Process RESOLVED WONTFIX
1357744 No GPUProcessPid and GPIPRocess entries In the Graphics section of about:support in FIrefox 53 stable RESOLVED INCOMPLETE
1358304 Include GPUProcessPid in about:support for all release branches RESOLVED DUPLICATE
1362166 Enable IPDL timeout for communicating to the GPU process RESOLVED FIXED
1373088 Don't use a timeout for compositor process startup when debugging RESOLVED FIXED
1380144 Collect hang information from the compositor process RESOLVED DUPLICATE
1390488 GPU process crashes don't contain crash message (including rust panic messages) VERIFIED FIXED
1393703 Crash caused by Tab2QR add-on with GPU Process enabled NEW
1733680 [Linux/GPU] Crash at GPUProcessManager::SimulateDeviceReset() RESOLVED FIXED

65 Total; 8 Open (12.31%); 55 Resolved (84.62%); 2 Verified (3.08%);


Sign off

Nightly

  • Metrics:
    • Overall crash volume for graphics related issues is unchanged
  • Testing:
    • [MISSED] Testrail regression tests passed by Softvision [2]
  • Automation:
  • Areas of Risk
    • Automation: No unit or functional test coverage
Category Criteria Status
Metrics GPU_PROCESS_LAUNCH_TIME_MS_2 is ... pending review 
 5th Percentile: 0.192s
25th Percentile: 0.296s
         Median: 0.462s
75th Percentile: 0.914s
95th Percentile: 3.440s
GPU_PROCESS_LAUNCH_TIME_MS_2 Histogram is ... Pending developer review
MG: Main/Content/Plugin/GPU crashes [3]
GPU Process crash rate
SIMPLE_MEASURES_FIRSTPAINT does not regress
Experiments A/B stability does not regress -4% crashes overall, +5% plugin crashes (needs investigation) [4]
Testrail Regression test suite passed on all platforms Pending review
Automation TBD
Bugzilla 0 blocking regressions tracked for 53
Socorro 0 blocking signatures
Migration Okay to merge to Aurora
Aurora migration verified
Aurora release notes verified
Risks Description Mitigation
Automation No unit/function test coverage beyond a process startup test Manual testing of functionality and process termination/recovery
Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Okay to Merge Email to be sent
QA Signoff - Aurora Migration Email to be sent
QA Signoff - Verify Aurora Release Notes Email to be sent

Aurora

Criteria Status Notes
Testing
Manual Test Coverage [DONE]
  • [DONE] Softvision mid-Aurora sign-off via Testrail
  • [DONE] Aurora testday
  • [SKIPPED] Betabreakers regression
Automated Test Coverage [DONE] Coverage limited to process instantiation. Risks to be mitigated via manual testing.
Performance Test Coverage [SKIPPED]
  • [SKIPPED] Softvision, Betabreakers, and/or Community testing Passmark test
  • [SKIPPED] Softvision, Betabreakers, and/or Community testing WebGL and HWA Video benchmarks
Bug Reports
No unresolved blocker bugs
No untriaged bugs blocking Fx53 		[DONE] Bugs pending Firefox 53 status review:
No unverified fixed bugs [DONE] Bugs pending Firefox 53 status review:
OKRs & Metrics
GPU_PROCESS_LAUNCH_TIME_MS_2 distribution <= 3s [DONE] Note: A value larger than 3s likely represents users failing to get GPU Process
Percentile 2017-01-30 2017-02-06 2017-02-13 2017-02-20 2017-02-27 2017-03-06 Net
5th 0.222s 0.232s 0.234s 0.235s 0.236s 0.236s +0.014s (+6.3%)
25th 0.360s 0.366s 0.365s 0.364s 0.365s 0.365s +0.005s (+1.4%)
50th 0.574s 0.574s 0.572s 0.564s 0.567s 0.567s -0.007s (-1.2%)
75th 1.150s 1.140s 1.130s 1.100s 1.110s 1.110s -0.040s (-3.5%)
95th 4.240s 4.100s 4.060s 3.940s 3.950s 3.960s -0.280s (-6.6%)
3s Threshold 91.94% 92.20% 92.32% 92.41% 92.57% 92.55% +1.01%
1s Threshold 70.65% 71.05% 71.22% 71.28% 71.47% 71.79% +1.14%
GPU_PROCESS_LAUNCH_TIME_MS_2 evolution doesn't regress [DONE] Note: A value larger than 3s likely represents users failing to get GPU Process
Percentile 2017-01-30 2017-02-06 2017-02-13 2017-02-20 2017-02-27 2017-03-06 Net
5th 0.226s 0.236s 0.241s 0.242s 0.223s 0.227s +0.001s (+0.4%)
25th 0.358s 0.361s 0.365s 0.365s 0.339s 0.345s -0.013s (-3.6%)
50th 0.560s 0.561s 0.561s 0.560s 0.498s 0.515s -0.045s (-8.0%)
75th 1.120s 1.100s 1.090s 1.090s 0.834s 0.935s -0.185s (-16.5%)
95th 4.030s 3.930s 3.890s 3.800s 2.530s 3.050s -0.980s (-24.3%)
Telemetry graphics crash count/rate largely unchanged [DONE]
2017-01-30 2017-02-06 2017-02-13 2017-02-20 2017-02-27 2017-03-06 Net
Crash Rate Average 0.224 0.166 0.164 0.150 0.153 0.139 -0.085 (-37.9%)
Crash Count Average 64 73 87 86 88 74 +10 (+15.6%)
Crash Count Percent 1.17% 1.07% 1.18% 1.14% 1.17% 1.14% -0.03% (-2.5%)
No unresolved topcrashes correlated to GPU Process [DONE]

GPU crashes rank 5th overall @ 0.82% with a combined 1071 reports
1st-ranked is #47 overall @ 0.10% (mozilla::layers::CompositorD3D11::BeginFrame - bug 1306168)
2nd-ranked is #31 overall @ 0.13% (UMDevice::SetShaderResources - bug 1267997)
3rd-ranked is #51 overall @ 0.09% (nvwgf2um.dll | TCLSWrappers<T>::CLSDestroy - bug 1325933)
4th-ranked is #52 overall @ 0.09% (OOM | unknown | ... | mozilla::BufferList<T>::AllocateSegment)
5th-ranked is #172 overall @ 0.03% (nvwgf2umx.dll | NDXGI::CDevice::DestroySynchronizationObjectCB - bug 1328082)

Telemetry Experiment reveals no regressions in A/B testing [SKIPPED]
  • Compare SIMPLE_MEASURES_FIRSTPAINT as a proxy for performance
  • Compare crash counts/rates of cohorts in Telemetry
  • Compare crash counts/rates of cohorts in Socorro
  • Compare crash signatures of cohorts in Socorro
Percentage of users w/GPU Process and hardware profiles [SKIPPED]
  • Measured via David Anderson's telemetry dashboard?
  • GPU_PROCESS_STATUS probe added similar to E10S_STATUS? See bug 1338694.
Insignificant change in hardware acceleration distribution [DONE]

GPU Process Status:

Date        Available   Unavailable   Failed   Blocked
2017-02-23      71.2%         28.4%     0.3%      0.1%
2017-03-06      71.1%         28.5%     0.3%      0.1%

D3D11 Status:

Date        Available   Unavailable   Failed   Blocked   Blacklisted
2017-02-23      67.4%         12.2%     4.3%      1.5%         14.6%
2017-03-06      67.5%         12.3%     4.3%      1.4%         14.5%

Media Decoder Status:

Date         DXVA D3D9   DXVA D3D11   WMF Software
2017-02-23       25.3%        15.0%          59.7%
2017-03-06       25.2%        14.3%          60.5%
SIMPLE_MEASURES_FIRSTPAINT does not regress more than 5% [DONE] 
		Median	Improvement
Aurora 53	2.760s	 n/a
Aurora 52	3.040s	- 9.21%
Aurora 51	2.880s	- 4.17%
Aurora 50	3.100s	-10.97%

Note: even if it does regress it'll be hard to prove that it's GPU Process related and not some other change(s)

QA Signoffs
Okay to Merge to Beta [DONE] Signed off by Softvision on March 3, 2017 (report)
Beta migration signed-off [DONE] Signed off by Softvision on March 8, 2017 (report)
Beta release notes signed-off [DONE] Signed off by Softvision on March 8, 2017

Beta

Criteria Status Notes
Testing
Manual Test Coverage [DONE]
  • Softvision mid-Beta sign-off via Testrail
  • Softvision pre-Release sign-off via Testrail
  • Beta testday
Automated Test Coverage [DONE] Coverage limited to process instantiation. Risks to be mitigated via manual testing.
Performance Test Coverage [DONE]
  • Passmark benchmark completed by Softvision
  • WebGL benchmark completed by Softvision
  • HWA Video benchmark completed by Softvision
Bug Reports
No unresolved blocker bugs
No untriaged bugs blocking Fx53 		[DONE] Bugs pending Firefox 53 status review:
No unverified fixed bugs [DONE] Bugs pending Firefox 53 status review:
Metrics
GPU_PROCESS_LAUNCH_TIME_MS_2 distribution <= 3s [DONE] Note: A value larger than 3s likely represents users failing to get GPU Process
  • 5th Percentile: 0.213s
  • 25th Percentile: 0.342s
  • 50th Percentile: 0.605s
  • 75th Percentile: 1.250s
  • 95th Percentile: 4.620s
  • Thresholds: 90.67% @ 3s, 68.17% @ 1s
GPU_PROCESS_LAUNCH_TIME_MS_2 evolution doesn't regress [DONE] Note: A value larger than 3s likely represents users failing to get GPU Process
  • 5th Percentile: 0.219s
  • 25th Percentile: 0.325s
  • 50th Percentile: 0.496s
  • 75th Percentile: 0.936s
  • 95th Percentile: 3.210s
Telemetry graphics crash count/rate largely unchanged [DONE]
  • Crash Rate Average: 0.067 crashes per 1,000 usage hours
  • Crash Count Average: 603 crashes / day
  • Crash Count Percent: 0.389%
No unresolved topcrashes correlated to GPU Process [DONE]

GPU crashes rank 13th overall @ 0.34% with a combined 2,295 reports
1st-ranked is #115 overall @ 0.04% (mozilla::layers::CompositorD3D11::BeginFrame - bug 1306168)
2nd-ranked is #49 overall @ 0.09% (igd10iumd32.dll | CContext::EmptyOutAllDDIBindPoints - bug )
3rd-ranked is #95 overall @ 0.05% (UMDevice::SetShaderResources - bug 1267997)
Also track 53 vs 52 crash data for Ryan Pollock

Telemetry Experiment reveals no regressions in A/B testing [DONE] DONE - https://ashughes.com/?p=426
Insignificant change in hardware acceleration distribution [DONE]

GPU Process Status:

  • 47.3% Unavailable
  • 33.2% Unused
  • 19.4% Available

D3D11 Status:

  • 71.7% D3D11
  • 26.7% Basic (software)
  • 1.60% D3D9

Media Decoder Status:

  • 56.7% WMF (software)
  • 25.6% D3D9
  • 17.7% D3D11
SIMPLE_MEASURES_FIRSTPAINT does not regress more than 5% [DONE]
  • Median: 2.926s
  • 15.9% improvement vs Firefox 52
  • 18.4% improvement vs Firefox 51
  • 2.46% improvement vs Firefox 50

Note: even if it does regress it'll be hard to prove that it's GPU Process related and not some other change(s)

QA Signoffs
Okay to Merge to Release [DONE] DONE
Release migration signed-off [DONE] DONE
Release notes signed-off [DONE] DONE

Release

  • Verify release notes
  • Verify feature enabled in Release
  • Verify updating gives GPU Process
  • Check for bugs during throttle period

References

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Platform/Graphics/Features/GPU_Process&oldid=1168319"