QA/Platform/Graphics/Features/GPU Process
From MozillaWiki
Overview
Purpose
The purpose of this document is to describe the GPU Process feature from a testing perspective, including the following details:
- Scope, focus areas and objectives of testing
- Owners and points of contact for areas of responsibility
- Strategy and types of testing
- The entry and exit criteria
- The basis of the test estimates
- Any risks, issues, assumptions and test dependencies
- The test schedule and major milestones
- The test deliverables
Ownership
| Responsibility | Role | Person |
|---|---|---|
| Accountable | Engineering Manager | Milan Sreckovic (:milan) |
| Responsible | Tech Lead | David Anderson (:dvander) |
| Responsible | QA Lead | Anthony Hughes (:ashughes) |
| Supporting | Engineer | George Wright (:gw280) |
| Supporting | Engineer | Ryan Hunt (:rhunt) |
Scope
The following is in scope from a testing perspective for the project team.
- GPU process startup time
- Page load time
- GPU process restart time (in case of process kill)
- Real world performance with various common usecases (webgl, canvas2d, flash).
- Plugin startup time
- Antivirus conflicts
- Memory footprint
- Shutdown time
- Overall product stability
- Scrolling performance and correctness
The following is not in scope from a testing perspective for the project team.
- to be determined
Schedule
| Project phase | Date Started | Date Completed |
|---|---|---|
| Planning | ||
|
||
|
||
| Development | ||
|
||
|
||
|
||
| QA | ||
|
||
|
||
|
||
|
||
| Release |
Outstanding Questions
Test Coverage
Environments
| Coverage | In Scope for MVP | Out of Scope for MVP |
|---|---|---|
| Platforms | Windows 7 Platform Update and later | Windows 7 without Platform Update and earlier, Mac OS, Linux, Android, iOS |
| Devices | AMD, Intel, and NVIDIA hardware with Direct3D 11 and later support | AMD, Intel, and NVIDIA hardware with Direct3D 10 and earlier support |
| Configurations | TBD - Single/Multi/Hybrid/etc, Internal/External, Current/Beta/Legacy drivers | TBD - Single/Multi/Hybrid/etc, Internal/External, Current/Beta/Legacy drivers |
| Other | TBD - 3rd-party software (Antivirus, Security, WindowsBlinds, etc), Preference changes | TBD - 3rd-party software (Antivirus, Security, WindowsBlinds, etc), Preference changes |
| Channel Specific | TBD - Unique coverage requirements for Nightly, Aurora, Beta, Release | TBD - Unique coverage requirements for Nightly, Aurora, Beta, Release |
Risks
Be sure we have test coverage to ensure:
- GPU Process does not to regress acceleration coverage, performance, startup time, rendering quality, and correctness for platforms without GPU Process and platforms with GPU Process where GPU Process is disabled.
- GPU Process recovers more gracefully from a crash than legacy
- GPU Process handles device resets more gracefully than legacy
- GPU Process returns with hardware (not software) compositor following device reset
- GPU Process does not regress font rendering in chrome and content (eg. Skia fonts)
- GPU Process does not fail in some way that results in a lot of users not getting Direct3D11 acceleration
- GPU Process does not break existing "new" technologies (eg. WebVR, WebGL2)
- GPU Process does not break legacy behaviours (eg. PDF viewing, Printing, Video playback, WebGL1, UI controls, about pages, etc)
- Legacy behavior/quality/performance (GPU Process disabled) remains on-par with the Release prior to GPU Process shipping
Acceptable Risks:
- GPU Process causes a moderate spike in acceleration crashes [specific number for acceptable spike to be determined]
Builds
This section should contain links for builds with the feature
- Links for Nightly builds
- Links for Aurora builds
- Links for Beta builds
Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | TestRail/ Google docs |
| Test case execution | TestRail |
| Automated test execution | Talos, etc...? |
| Bugs management | Bugzilla |
| Experimentation | Telemetry Experiments |
Use Cases
| Use Cases | Smoketest Coverage | BFT Coverage | FFT Coverage | Notes |
|---|---|---|---|---|
| The following need to be verified with GPU Process enabled, GPU Process disabled, and across a process reset | ||||
| Printing | Verify previewing and printing various content types is not regressed by GPU Process | |||
| Private Window | Verify browser usage inside a Private Browsing window | |||
| Multi-Process Enabled | Verify GPU process is created when multi-process is enabled | |||
| Multi-process Disabled | Verify no GPU process is created when multi-process is disabled | |||
| Single-process Window | Verify no GPU process when using a non-e10s window | |||
| Offline Content | Verify accelerated content continues to render when in Offline mode | |||
| Video | Verify HTML5 and Flash video playback acceleration | |||
| Tab/Window Management | Verify arranging and management of tabs/windows | |||
| Theme Management | Verify management of various Firefox and system-level themes | |||
| WebVR | Verify WebVR demos continue to work at parity with the current Release | |||
| WebGL | Verify WebGL demos continue to work at parity with the current Release | |||
| Dev Tools | Verify Developer Tools continue to work at parity with the current Release | |||
| Custom fonts | Verify content rendering, printing, scrolling, etc with common font customizations | |||
| 2D Canvas | Verify content rendering and performance with 2D Canvas and SVG demos | |||
| Accessibility | ||||
| Mouse-only operation | ||||
| Keyboard-only operation | ||||
| Display (HiDPI) | ||||
| Interraction (scroll, zoom) | ||||
| Usable with a screen reader | ||||
| Usability and/or discoverability testing | ||||
| RTL build testing | ||||
| Help/Support | ||||
| Help/support interface required | Make sure link to support/help page exist and is easy reachable. | |||
| Support documents planned(written) | Make sure support documents are written and are correct. | |||
| Install/Upgrade | ||||
| Feature upgrades/downgrades data as expected | ||||
| Does sync work across upgrades | ||||
| Requires install testing | Separate feature/application installation needed (not only Firefox) | |||
| Affects first-run or onboarding | Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned. | |||
| Partner build testing | Does this affect partner builds? Yes/No options, add comment with details about who will lead testing | |||
| Enterprise
Raise up the topic to developers to see if they are expecting to work different on ESR builds | ||||
| Enterprise administration | ||||
| Network proxies/autoconfig | ||||
| ESR behavior changes | ||||
| Locked preferences | ||||
| Data Monitoring | ||||
| Temporary or permanent telemetry monitoring | List of error conditions to monitor | |||
| Telemetry correctness testing | ||||
| Server integration testing | ||||
| Offline and server failure testing | ||||
| Load testing | ||||
| Add-ons
If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on. | ||||
| Addon API required? | ||||
| Comprehensive API testing | ||||
| Permissions | ||||
| Testing with existing/popular addons | ||||
| Security
Contact Matt Wobensmith to see if security testing is necessary for current feature | ||||
| 3rd-party security review | ||||
| Privilege escalation testing | ||||
| Fuzzing | ||||
| Web Compatibility | ||||
| Testing against target sites | ||||
| Survey of many sites for compatibility | ||||
| Interoperability | ||||
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | ||||
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | ||||
| Interaction of this feature with other browser features | ||||
| Interaction of this feature with third-party software (Antivirus, Security, Games, Theming, etc) | ||||
Experiments
| Experiment | Contact | Tracking Bug | Status | Start Date | End Date |
|---|---|---|---|---|---|
| A/B test layers.gpu-process.dev.enabled in Nightly 53 | Anthony Hughes (:ashughes) | bug 1319206 | Done [1] | |
|
| A/B test media.gpu-process-decoder in Nightly 53 | Anthony Hughes (:ashughes) |
Nightly 53 A/B Test layers.gpu-process.dev.enabled
Purpose:
- Compare crash rates for users with GPU process enabled vs users with GPU process disabled
Dependencies:
- How do we deal with the fact that the pref is set early during startup? (emailed felipeg)
- Telemetry probes to measure crash rates
- Telemetry probe to measure proportion of users in each cohort
Measurements:
- Evolution for GPU_PROCESS_LAUNCH_TIME
- Histogram for GPU_PROCESS_LAUNCH_TIME
- Crash Stats
- MOZ30726-MINI#psd-gpu-process-nightly53 (see :ashughes to request access)
- SQL Telemetry
Bugs
65 Total; 8 Open (12.31%); 55 Resolved (84.62%); 2 Verified (3.08%);
Sign off
Nightly
- Metrics:
- Overall crash volume for graphics related issues is unchanged
- Testing:
- [MISSED] Testrail regression tests passed by Softvision [2]
- Automation:
- Areas of Risk
- Automation: No unit or functional test coverage
| Category | Criteria | Status |
|---|---|---|
| Metrics | GPU_PROCESS_LAUNCH_TIME_MS_2 is ... | pending review
5th Percentile: 0.192s
25th Percentile: 0.296s
Median: 0.462s
75th Percentile: 0.914s
95th Percentile: 3.440s
|
| GPU_PROCESS_LAUNCH_TIME_MS_2 Histogram is ... | Pending developer review | |
| MG: Main/Content/Plugin/GPU crashes [3] | ||
| GPU Process crash rate | ||
| SIMPLE_MEASURES_FIRSTPAINT does not regress | ||
| Experiments | A/B stability does not regress | -4% crashes overall, +5% plugin crashes (needs investigation) [4] |
| Testrail | Regression test suite passed on all platforms | Pending review |
| Automation | TBD | |
| Bugzilla | 0 blocking regressions | tracked for 53 |
| Socorro | 0 blocking signatures | |
| Migration | Okay to merge to Aurora | |
| Aurora migration verified | ||
| Aurora release notes verified | ||
| Risks | Description | Mitigation |
| Automation | No unit/function test coverage beyond a process startup test | Manual testing of functionality and process termination/recovery |
| Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | ||
| Testing Infrastructure setup | ||
| Test Plan Creation | ||
| Test Cases Creation | ||
| Full Functional Tests Execution | ||
| Automation Coverage | ||
| Performance Testing | ||
| All Defects Logged | ||
| Critical/Blockers Fixed and Verified | ||
| Metrics/Telemetry | ||
| QA Signoff - Okay to Merge | Email to be sent | |
| QA Signoff - Aurora Migration | Email to be sent | |
| QA Signoff - Verify Aurora Release Notes | Email to be sent |
Aurora
| Criteria | Status | Notes | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Testing | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Manual Test Coverage | [DONE] |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Automated Test Coverage | [DONE] | Coverage limited to process instantiation. Risks to be mitigated via manual testing. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Performance Test Coverage | [SKIPPED] |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Bug Reports | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| No unresolved blocker bugs No untriaged bugs blocking Fx53 |
[DONE] | Bugs pending Firefox 53 status review: | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| No unverified fixed bugs | [DONE] | Bugs pending Firefox 53 status review:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| OKRs & Metrics | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| GPU_PROCESS_LAUNCH_TIME_MS_2 distribution <= 3s | [DONE] | Note: A value larger than 3s likely represents users failing to get GPU Process
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| GPU_PROCESS_LAUNCH_TIME_MS_2 evolution doesn't regress | [DONE] | Note: A value larger than 3s likely represents users failing to get GPU Process
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Telemetry graphics crash count/rate largely unchanged | [DONE] |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| No unresolved topcrashes correlated to GPU Process | [DONE] |
GPU crashes rank 5th overall @ 0.82% with a combined 1071 reports | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Telemetry Experiment reveals no regressions in A/B testing | [SKIPPED] |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Percentage of users w/GPU Process and hardware profiles | [SKIPPED] |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Insignificant change in hardware acceleration distribution | [DONE] |
GPU Process Status: Date Available Unavailable Failed Blocked 2017-02-23 71.2% 28.4% 0.3% 0.1% 2017-03-06 71.1% 28.5% 0.3% 0.1% D3D11 Status: Date Available Unavailable Failed Blocked Blacklisted 2017-02-23 67.4% 12.2% 4.3% 1.5% 14.6% 2017-03-06 67.5% 12.3% 4.3% 1.4% 14.5% Media Decoder Status: Date DXVA D3D9 DXVA D3D11 WMF Software 2017-02-23 25.3% 15.0% 59.7% 2017-03-06 25.2% 14.3% 60.5% | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| SIMPLE_MEASURES_FIRSTPAINT does not regress more than 5% | [DONE] |
Median Improvement Aurora 53 2.760s n/a Aurora 52 3.040s - 9.21% Aurora 51 2.880s - 4.17% Aurora 50 3.100s -10.97% Note: even if it does regress it'll be hard to prove that it's GPU Process related and not some other change(s) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| QA Signoffs | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Okay to Merge to Beta | [DONE] | Signed off by Softvision on March 3, 2017 (report) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Beta migration signed-off | [DONE] | Signed off by Softvision on March 8, 2017 (report) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Beta release notes signed-off | [DONE] | Signed off by Softvision on March 8, 2017 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Beta
| Criteria | Status | Notes |
|---|---|---|
| Testing | ||
| Manual Test Coverage | [DONE] |
|
| Automated Test Coverage | [DONE] | Coverage limited to process instantiation. Risks to be mitigated via manual testing. |
| Performance Test Coverage | [DONE] |
|
| Bug Reports | ||
| No unresolved blocker bugs No untriaged bugs blocking Fx53 |
[DONE] | Bugs pending Firefox 53 status review: |
| No unverified fixed bugs | [DONE] | Bugs pending Firefox 53 status review:
|
| Metrics | ||
| GPU_PROCESS_LAUNCH_TIME_MS_2 distribution <= 3s | [DONE] | Note: A value larger than 3s likely represents users failing to get GPU Process
|
| GPU_PROCESS_LAUNCH_TIME_MS_2 evolution doesn't regress | [DONE] | Note: A value larger than 3s likely represents users failing to get GPU Process
|
| Telemetry graphics crash count/rate largely unchanged | [DONE] |
|
| No unresolved topcrashes correlated to GPU Process | [DONE] |
GPU crashes rank 13th overall @ 0.34% with a combined 2,295 reports |
| Telemetry Experiment reveals no regressions in A/B testing | [DONE] | DONE - https://ashughes.com/?p=426
|
| Insignificant change in hardware acceleration distribution | [DONE] |
GPU Process Status:
D3D11 Status:
Media Decoder Status:
|
| SIMPLE_MEASURES_FIRSTPAINT does not regress more than 5% | [DONE] |
Note: even if it does regress it'll be hard to prove that it's GPU Process related and not some other change(s) |
| QA Signoffs | ||
| Okay to Merge to Release | [DONE] | DONE |
| Release migration signed-off | [DONE] | DONE |
| Release notes signed-off | [DONE] | DONE |
Release
- Verify release notes
- Verify feature enabled in Release
- Verify updating gives GPU Process
- Check for bugs during throttle period
