QA/Platform/Graphics
Jump to navigation
Jump to search
Full Query
Consult this page for more information.
Metrics
- Crash Rates: STAGE - PROD
- Metrics: Code Quality, Quality Indices
- Telemetry: GFX, All
- Bugzilla: Bug Age
Documentation
- Team Page
- Device Inventory (moved to wiki)
- PCI Device ID List
- nVidia Codenames
- Graphics driver blocklisting
- Draft guides
Understanding the Problem Space
First order of business for my transition to the Graphics team is to understand the problem space so I can understand the immediate needs of the team and make the best impact I can in the shortest amount of time.
- What are the key problems/challenges facing the Graphics team in terms of quality?
- discrepancy in environments between testers and release users
- discoverability of bugs pre-release
- ?...
- Where can QA add value/support to the Graphics team?
- improving pre-release discoverability of bugs
- closing the gap between tester and release systems
- helping with bug triage, particularly with bugs hiding in general components
- representation in crashkill
- improving code coverage and/or identifying gaps in code coverage
- identifying ways to improve participation in the graphics team (events, projects, One & Done, etc)
- documentation of tools, testing processes, etc
- building out the lab in Toronto
- continuing to drive Betabreakers testing every 6 weeks
- verifying bug fixes (what does this look like)?
- profiling areas of risk (eg. troublesome configs)
- conducting root cause analysis for regressions
- understanding problems outside of our control (eg. driver resets)
- feature testing and upcoming priorities (e10s, Windows 10, El Capitain, Android, B2G, etc)
- What does QA need to know to be effective?
- key components of an actionable Graphics bug
- fundamentals/technologies that should be learned
- how to distinguish a graphics crash from a non-graphics crash with a graphics signature
- meetings, mailing lists, bugzilla components to watch, blogs, IRC channels to join, etc
- who is each member of the team (incl. contributors) and what do they do
- where does graphics code reside in the tree?
- what role does Unified Telemetry in graphics quality?
- what are the prefs to enable/disable different functionalities?
- we need a database of known-troublesome hardware/driver configurations to inform testing, hardware acquisitions, and blocklisting
Participation
- Sanity checking via One & Done
- Meetups to connect testers/users with devs
- Testdays to teach people about graphics testing
- Documentation and translation of documentation
- Engaging on community spaces (Discourse, Reddit, Facebook, Twitter, etc)
Telemetry
- COMPOSITE_TIME: time in CompositorParent::CompositeToTarget dispatching draw calls and calling SwapBuffers, but not texture upload (ie. complete composition)
Projects
Sanity Checking
| Project | Summary | Frequency | Status |
|---|---|---|---|
| Betabreakers | Per-version lab testing against Developer Edition | Every 6 weeks | 44.0a2 in progress, due 2015-12-09 |
| Boot2Gecko | Sanity checks on B2G phones at MozYYZ | Weekly | Last reviewed 2015-11-27 |
| One & Done | Crowd-sourced testing via One & Done (results) | Weekly | Last reviewed 2015-11-26 |
| Toronto Lab Testing | Sanity checks on computers at the MozYYZ lab | N/A | On hold as of 2015-08-20 |
Stability
| Project | Summary | Frequency | Status |
|---|---|---|---|
| Explosive Crashes | Crashes ranked by rate of growth | Weekly | Last reviewed 2015-11-27 |
| Top Crashes | Crashes ranked by volume | Weekly | Last reviewed 2015-11-27 |
Understanding the Problem
How do we identify a graphics crash?
- by signature: gfx, layers, D2D, D3D, ?...
- by topmost filename: gfx, ?...
- by driver (DLL, version, ?...)
- by device/vendor ID?...
- ?...
How do we prioritize graphics crashes?
- Overall topcrashes in release > beta > aurora > nightly
- Gfx crashes in release > beta > aurora > nightly
- Explosive crashes in release > beta > aurora > nightly
What tools do we have at our disposal to investigate crashes?
- Bughunter for investigating crashes correlated to a URL
- KaiRo's reports for identifying crashes that are new or escalating quickly
- Socorro for getting detailed information about crash reports
What information is needed to make a crash actionable by developers?
- Correlations to particular hardware, driver, add-on, 3rd-party software, or library
- ?...
Device Driver Crashes
Queries
- Top crashes by device
- Top crashes by driver
- AMD: amd*.dll, ati*.dll
- Intel: igd*.dll
- NVIDIA: nv*.dll
Top Crashes
5 Total; 5 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Triage
Bug Triage
| Category | Description | Frequency | Status |
|---|---|---|---|
| Cold Crashes | Crash bugs which have not been updated in more than 30 days | Monthly | Last reviewed 2015-11-09 |
| Cold Trackers | Tracked bugs which have not been updated since the previous release | Every 6 weeks | Last reviewed 2015-11-03 |
| Help Wanted | Bugs which lack information necessary to resolve | Monthly | Last reviewed 2015-11-09 |
| Incoming | Bugs reported but not triaged for more than one month | Monthly | Last reviewed 2015-11-09 |
Crash Triage
| Category | Description | Frequency | Status |
|---|---|---|---|
| Explosive Crashes | Check the explosiveness reports for new or spiking crashes | Weekly | Last reviewed 2015-12-02 |
| Top Crashes | Check the topcrash reports for new or spiking crashes | Weekly | Last reviewed 2015-12-02 |
Get Involved
Want to help the Graphics team? Here are some ways you can get involved.