QA/Platform/Graphics

From MozillaWiki
Jump to navigation Jump to search

Consult this page for more information.

Summary

Understanding the Problem Space

First order of business for my transition to the Graphics team is to understand the problem space so I can understand the immediate needs of the team and make the best impact I can in the shortest amount of time.

What are the key problems/challenges facing the Graphics team in terms of quality?
  • discrepancy in environments between testers and release users
  • discoverability of bugs pre-release
  • ?...
Where can QA add value/support to the Graphics team?
  • improving pre-release discoverability of bugs
  • closing the gap between tester and release systems
  • helping with bug triage, particularly with bugs hiding in general components
  • representation in crashkill
  • improving code coverage and/or identifying gaps in code coverage
  • identifying ways to improve participation in the graphics team (events, projects, One & Done, etc)
  • documentation of tools, testing processes, etc
  • building out the lab in Toronto
  • continuing to drive Betabreakers testing every 6 weeks
  • verifying bug fixes (what does this look like)?
  • profiling areas of risk (eg. troublesome configs)
  • conducting root cause analysis for regressions
  • understanding problems outside of our control (eg. driver resets)
  • feature testing and upcoming priorities (e10s, Windows 10, El Capitain, Android, B2G, etc)
What does QA need to know to be effective?
  • key components of an actionable Graphics bug
  • fundamentals/technologies that should be learned
  • how to distinguish a graphics crash from a non-graphics crash with a graphics signature
  • meetings, mailing lists, bugzilla components to watch, blogs, IRC channels to join, etc
  • who is each member of the team (incl. contributors) and what do they do
  • where does graphics code reside in the tree?
  • what role does Unified Telemetry in graphics quality?
  • what are the prefs to enable/disable different functionalities?
  • we need a database of known-troublesome hardware/driver configurations to inform testing, hardware acquisitions, and blocklisting

Sanity Checking

Stability

How do we identify a graphics crash?

  • by signature: gfx, layers, D2D, D3D, ?...
  • by topmost filename: gfx, ?...
  • ?...

How do we prioritize graphics crashes?

  • Overall topcrashes in release > beta > aurora > nightly
  • Gfx crashes in release > beta > aurora > nightly
  • Explosive crashes in release > beta > aurora > nightly

What tools do we have at our disposal to investigate crashes?

  • Bughunter for investigating crashes correlated to a URL
  • KaiRo's reports for identifying crashes that are new or escalating quickly
  • Socorro for getting detailed information about crash reports

What information is needed to make a crash actionable by developers?

  • Correlations to particular hardware, driver, add-on, 3rd-party software, or library
  • ?...

Weekly Triage

Top Graphics Driver Crashes

  1. Search for top crashes by driver filename in signature, facet on driver version
  2. Facet the top driver version by adapter device id
  3. Take the top device IDs and correlate to a specific device name using this database
  4. File a meta crasher bug for that card/driver
  5. See about acquiring that combination if not available in-house for testing
  6. Debug some of the higher-volume issues with that combination
Example
  • [1] shows top driver for nVidia driver crashes is 9.18.13.2702
  • [2] shows top device for that driver is 0x1049
  • [3] shows that device ID correlates to GeForce GT 620 OEM (GF119)
Drivers
Driver Description Weekly crashes
2015-07-07 		Top Crash
AMD
amdocl.dll OpenCL 1.1 AMD-APP 17
aticfx32.dll AMD Radeon DirectX 11 Driver 326
atidxx32.dll AMD Radeon DirectX 11 Driver 3,031
atioglxx.dll ATI OpenGL Driver 35
atiumdag.dll AMD Radeon DirectX Universal 2,320
Intel
igd10umd32.dll Intel Graphics LDDM User Mode Driver for Windows 8 11,436
igdumd32.dll Intel Graphics LDDM User Mode Driver for Windows Vista 1,843
igddxva32.dll Intel Graphics WDDM User Mode Driver for Windows 7 96
nVidia
igd10iumd32.dll nVidia D3D Shim Driver 5,097
nvapi.dll nVidia Windows Driver 435
nvd3dum.dll nVidia Windows WDDM D3D Driver 2,016
nvlsp.dll nVidia Application Filter 357
nvumdshim.dll nVidia D3D Shim Driver 2,048
nvwgf2um.dll nVidia D3D10 Driver 15,657 nvwgf2um.dll@0x39e5d (3.874%)
44,714

Features

Platform Roadmap:

Participation

  • Sanity checking via One & Done
  • Meetups to connect testers/users with devs
  • Testdays to teach people about graphics testing
  • Documentation and translation of documentation
  • Engaging on community spaces (Discourse, Reddit, Facebook, Twitter, etc)

Betabreakers

Testing:

Risks:

  • Betabreakers will not have Windows 10 deployed to machines until after it is officially released. However, they can deploy Preview to select machines upon request. We need to develop a set of requirements for Windows 10 testing, particularly machine specifications for any upcoming testrun that targets Windows 10.
  • We need to select hardware for testing based on data from past testruns and known-troublesome hardware
  • We need to identify gaps in test coverage and investigate whether they can fill these gaps for us

Guides

Interesting Initiatives

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Platform/Graphics&oldid=1083568"