Overview
Purpose
The purpose of this document is to describe the GPU Process feature from a testing perspective, including the following details:
- Scope, focus areas and objectives of testing
- Owners and points of contact for areas of responsibility
- Strategy and types of testing
- The entry and exit criteria
- The basis of the test estimates
- Any risks, issues, assumptions and test dependencies
- The test schedule and major milestones
- The test deliverables
Ownership
| Responsibility | Role | Person |
|---|---|---|
| Accountable | Engineering Manager | Milan Sreckovic (:milan) |
| Responsible | Tech Lead | David Anderson (:dvander) |
| Responsible | QA Lead | Anthony Hughes (:ashughes) |
| Supporting | Engineer | George Wright (:gw280) |
| Supporting | Engineer | Ryan Hunt (:rhunt) |
Scope
The following is in scope from a testing perspective for the project team.
- GPU process startup time
- Page load time
- GPU process restart time (in case of process kill)
- Real world performance with various common usecases (webgl, canvas2d, flash).
- Plugin startup time
- Antivirus conflicts
- Memory footprint
- Shutdown time
- Overall product stability
- Scrolling performance and correctness
The following is not in scope from a testing perspective for the project team.
- to be determined
Schedule
| Project phase | Date Started | Date Completed |
|---|---|---|
| Planning | ||
|
||
|
||
| Development | ||
|
||
|
||
|
||
| QA | ||
|
||
|
||
|
||
|
||
| Release |
Outstanding Questions
Test Coverage
Environments
| Coverage | In Scope for MVP | Out of Scope for MVP |
|---|---|---|
| Platforms | Windows 7 Platform Update and later | Windows 7 without Platform Update and earlier, Mac OS, Linux, Android, iOS |
| Devices | AMD, Intel, and NVIDIA hardware with Direct3D 11 and later support | AMD, Intel, and NVIDIA hardware with Direct3D 10 and earlier support |
| Configurations | TBD - Single/Multi/Hybrid/etc, Internal/External, Current/Beta/Legacy drivers | TBD - Single/Multi/Hybrid/etc, Internal/External, Current/Beta/Legacy drivers |
| Other | TBD - 3rd-party software (Antivirus, Security, WindowsBlinds, etc), Preference changes | TBD - 3rd-party software (Antivirus, Security, WindowsBlinds, etc), Preference changes |
| Channel Specific | TBD - Unique coverage requirements for Nightly, Aurora, Beta, Release | TBD - Unique coverage requirements for Nightly, Aurora, Beta, Release |
Risks
Be sure we have test coverage to ensure:
- GPU Process does not to regress acceleration coverage, performance, startup time, rendering quality, and correctness for platforms without GPU Process and platforms with GPU Process where GPU Process is disabled.
- GPU Process recovers more gracefully from a crash than legacy
- GPU Process handles device resets more gracefully than legacy
- GPU Process returns with hardware (not software) compositor following device reset
- GPU Process does not regress font rendering in chrome and content (eg. Skia fonts)
- GPU Process does not fail in some way that results in a lot of users not getting Direct3D11 acceleration
- GPU Process does not break existing "new" technologies (eg. WebVR, WebGL2)
- GPU Process does not break legacy behaviours (eg. PDF viewing, Printing, Video playback, WebGL1, UI controls, about pages, etc)
- Legacy behavior/quality/performance (GPU Process disabled) remains on-par with the Release prior to GPU Process shipping
Acceptable Risks:
- GPU Process causes a moderate spike in acceleration crashes [specific number for acceptable spike to be determined]
Builds
This section should contain links for builds with the feature
- Links for Nightly builds
- Links for Aurora builds
- Links for Beta builds
Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | TestRail/ Google docs |
| Test case execution | TestRail |
| Automated test execution | Talos, etc...? |
| Bugs management | Bugzilla |
| Experimentation | Telemetry Experiments |
Use Cases
| Use Cases | Smoketest Coverage | BFT Coverage | FFT Coverage | Notes |
|---|---|---|---|---|
| The following need to be verified with GPU Process enabled, GPU Process disabled, and across a process reset | ||||
| Printing | Verify previewing and printing various content types is not regressed by GPU Process | |||
| Private Window | Verify browser usage inside a Private Browsing window | |||
| Multi-Process Enabled | Verify GPU process is created when multi-process is enabled | |||
| Multi-process Disabled | Verify no GPU process is created when multi-process is disabled | |||
| Single-process Window | Verify no GPU process when using a non-e10s window | |||
| Offline Content | Verify accelerated content continues to render when in Offline mode | |||
| Video | Verify HTML5 and Flash video playback acceleration | |||
| Tab/Window Management | Verify arranging and management of tabs/windows | |||
| Theme Management | Verify management of various Firefox and system-level themes | |||
| WebVR | Verify WebVR demos continue to work at parity with the current Release | |||
| WebGL | Verify WebGL demos continue to work at parity with the current Release | |||
| Dev Tools | Verify Developer Tools continue to work at parity with the current Release | |||
| Custom fonts | Verify content rendering, printing, scrolling, etc with common font customizations | |||
| 2D Canvas | Verify content rendering and performance with 2D Canvas and SVG demos | |||
| Accessibility | ||||
| Mouse-only operation | ||||
| Keyboard-only operation | ||||
| Display (HiDPI) | ||||
| Interraction (scroll, zoom) | ||||
| Usable with a screen reader | ||||
| Usability and/or discoverability testing | ||||
| RTL build testing | ||||
| Help/Support | ||||
| Help/support interface required | Make sure link to support/help page exist and is easy reachable. | |||
| Support documents planned(written) | Make sure support documents are written and are correct. | |||
| Install/Upgrade | ||||
| Feature upgrades/downgrades data as expected | ||||
| Does sync work across upgrades | ||||
| Requires install testing | Separate feature/application installation needed (not only Firefox) | |||
| Affects first-run or onboarding | Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned. | |||
| Partner build testing | Does this affect partner builds? Yes/No options, add comment with details about who will lead testing | |||
| Enterprise
Raise up the topic to developers to see if they are expecting to work different on ESR builds | ||||
| Enterprise administration | ||||
| Network proxies/autoconfig | ||||
| ESR behavior changes | ||||
| Locked preferences | ||||
| Data Monitoring | ||||
| Temporary or permanent telemetry monitoring | List of error conditions to monitor | |||
| Telemetry correctness testing | ||||
| Server integration testing | ||||
| Offline and server failure testing | ||||
| Load testing | ||||
| Add-ons
If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on. | ||||
| Addon API required? | ||||
| Comprehensive API testing | ||||
| Permissions | ||||
| Testing with existing/popular addons | ||||
| Security
Contact Matt Wobensmith to see if security testing is necessary for current feature | ||||
| 3rd-party security review | ||||
| Privilege escalation testing | ||||
| Fuzzing | ||||
| Web Compatibility | ||||
| Testing against target sites | ||||
| Survey of many sites for compatibility | ||||
| Interoperability | ||||
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | ||||
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | ||||
| Interaction of this feature with other browser features | ||||
| Interaction of this feature with third-party software (Antivirus, Security, Games, Theming, etc) | ||||
Test Suites
Full Test suite - Link with the gdoc, follow the format from link Smoke Test suite - Link with the gdoc, follow the format from link Regression Test suite - [1]
Links for the tests run
Experiments
| Experiment | Contact | Tracking Bug | Status | Start Date | End Date |
|---|---|---|---|---|---|
| A/B test layers.gpu-process.dev.enabled in Nightly 53 | Anthony Hughes (:ashughes) | bug 1319206 | Done [2] | ||
| A/B test media.gpu-process-decoder in Nightly 53 | Anthony Hughes (:ashughes) |
Nightly 53 A/B Test layers.gpu-process.dev.enabled
Purpose:
- Compare crash rates for users with GPU process enabled vs users with GPU process disabled
Dependencies:
- How do we deal with the fact that the pref is set early during startup? (emailed felipeg)
- Telemetry probes to measure crash rates
- Telemetry probe to measure proportion of users in each cohort
Measurements:
- Evolution for GPU_PROCESS_LAUNCH_TIME
- Histogram for GPU_PROCESS_LAUNCH_TIME
- Crash Stats
- MOZ30726-MINI#psd-gpu-process-nightly53 (see :ashughes to request access)
- SQL Telemetry
Nightly 53 A/B Test media.gpu-process-decoder
TBD
Bugs
65 Total; 8 Open (12.31%); 55 Resolved (84.62%); 2 Verified (3.08%);
Sign off
Nightly
| Category | Criteria | Status |
|---|---|---|
| Telemetry | GPU_PROCESS_LAUNCH_TIME_MS_2 Evolution is ... | Pending developer review |
| GPU_PROCESS_LAUNCH_TIME_MS_2 Histogram is ... | Pending developer review | |
| Testrail | Regression test suite passed on all platforms | Pending review |
| Automation | TBD | |
| Bugzilla | 0 blocking regressions | |
| Migration | Okay to merge to Aurora | |
| Aurora migration verified | ||
| Aurora release notes verified |
| Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | ||
| Testing Infrastructure setup | ||
| Test Plan Creation | ||
| Test Cases Creation | ||
| Full Functional Tests Execution | ||
| Automation Coverage | ||
| Performance Testing | ||
| All Defects Logged | ||
| Critical/Blockers Fixed and Verified | ||
| Metrics/Telemetry | ||
| QA Signoff - Okay to Merge | Email to be sent | |
| QA Signoff - Aurora Migration | Email to be sent | |
| QA Signoff - Verify Aurora Release Notes | Email to be sent |
Aurora
| Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | ||
| Testing Infrastructure setup | ||
| Test Plan Creation | ||
| Test Cases Creation | ||
| Full Functional Tests Execution | ||
| Automation Coverage | ||
| Performance Testing | ||
| All Defects Logged | ||
| Critical/Blockers Fixed and Verified | ||
| Metrics/Telemetry | ||
| QA Signoff - Okay to Merge | Email to be sent | |
| QA Signoff - Beta Migration | Email to be sent | |
| QA Signoff - Verify Beta Release Notes | Email to be sent |
Beta
| Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | ||
| Testing Infrastructure setup | ||
| Test Plan Creation | ||
| Test Cases Creation | ||
| Full Functional Tests Execution | ||
| Automation Coverage | ||
| Performance Testing | ||
| All Defects Logged | ||
| Critical/Blockers Fixed and Verified | ||
| Metrics/Telemetry | ||
| QA Signoff - Beta 1..n | Email to be sent | |
| QA Signoff - Release Migration | Email to be sent |
Release
| Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | ||
| Testing Infrastructure setup | ||
| Test Plan Creation | ||
| Test Cases Creation | ||
| Full Functional Tests Execution | ||
| Automation Coverage | ||
| Performance Testing | ||
| All Defects Logged | ||
| Critical/Blockers Fixed and Verified | ||
| Metrics/Telemetry | ||
| QA Signoff - Okay to Release | Email to be sent | |
| QA Signoff - Verify Release Notes | Email to be sent |
References
- List and links for available specs - documents, user stories, specifications
- Meta bug: bug 1264543
- https://public.etherpad-mozilla.org/p/GPU_Process_Testing
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
| Date | Version | Author | Description |
|---|---|---|---|
| 2016-11-02 | 1.0 | Anthony Hughes | Created first draft |