QA/Platform/Graphics

From MozillaWiki
< QA‎ | Platform
Revision as of 18:21, 10 May 2018 by Msreckovic (talk | contribs) (→‎The Team: Not up to date)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Summary

This page holds important information related to testing and quality assurance for Gecko's graphics code.

Metrics

Tools

Device ID Catalogs:

Get Involved

Here are some ways you can help the Graphics team (and Mozilla):

Activities

One & Done Testing

DEPRECATION WARNING: One&Done is being decommissioned. In the meantime, all tasks have been archived to this wiki.


Betabreakers Lab Testing

Summary

There is a large discrepancy between the platform/hardware/driver configurations we see on Release versus what we see on our test branches. Additionally, the Graphics team and our testing community does not and can not provide sufficient test coverage to fill this gap. As a result we've historically been notified far too late about critical graphics issues triggering multiple chemspill releases and a loss of users.

In an effort to improve pre-release visibility of graphics regressions we have partnered with Betabreakers, a company that maintains a lab specifically designed to test various modern and legacy configurations of Windows versions, graphics hardware, and graphics drivers. We try to coordinate a graphics sanity testrun with them once per mozilla-aurora cycle, as detailed below.

More information about our relationship with Betabreakers can be found on mana.mozilla.org.

Test Runs

Bugs

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1

Triage

Triage currently focuses on two main areas: identifying critical stability regressions, and identifying issues which may have slipped through the cracks. We try to have a one hour triage session each weekday, usually starting around 10:00am Pacific. More information about this process can be found on the wiki and dashboard.

Retired Activities

Top Issues

Blockers

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Crashes

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Features

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


New Issues

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Understanding the Problem Space

First order of business for my transition to the Graphics team is to understand the problem space so I can understand the immediate needs of the team and make the best impact I can in the shortest amount of time.

What are the key problems/challenges facing the Graphics team in terms of quality?
  • discrepancy in environments between testers and release users
  • discoverability of bugs pre-release
  • ?...
Where can QA add value/support to the Graphics team?
  • improving pre-release discoverability of bugs
  • closing the gap between tester and release systems
  • helping with bug triage, particularly with bugs hiding in general components
  • representation in crashkill
  • improving code coverage and/or identifying gaps in code coverage
  • identifying ways to improve participation in the graphics team (events, projects, One & Done, etc)
  • documentation of tools, testing processes, etc
  • building out the lab in Toronto
  • continuing to drive Betabreakers testing every 6 weeks
  • verifying bug fixes (what does this look like)?
  • profiling areas of risk (eg. troublesome configs)
  • conducting root cause analysis for regressions
  • understanding problems outside of our control (eg. driver resets)
  • feature testing and upcoming priorities (e10s, Windows 10, El Capitain, Android, B2G, etc)
What does QA need to know to be effective?
  • key components of an actionable Graphics bug
  • fundamentals/technologies that should be learned
  • how to distinguish a graphics crash from a non-graphics crash with a graphics signature
  • meetings, mailing lists, bugzilla components to watch, blogs, IRC channels to join, etc
  • who is each member of the team (incl. contributors) and what do they do
  • where does graphics code reside in the tree?
  • what role does Unified Telemetry in graphics quality?
  • what are the prefs to enable/disable different functionalities?
  • we need a database of known-troublesome hardware/driver configurations to inform testing, hardware acquisitions, and blocklisting
Understanding the Stability
  • How do we identify a graphics crash?
    • by signature: gfx, layers, D2D, D3D, ?...
    • by topmost filename: gfx, ?...
    • by driver (DLL, version, ?...)
    • by device/vendor ID?...
    • ?...
  • How do we prioritize graphics crashes?
    • Overall topcrashes in release > beta > aurora > nightly
    • Gfx crashes in release > beta > aurora > nightly
    • Explosive crashes in release > beta > aurora > nightly
  • What tools do we have at our disposal to investigate crashes?
    • Bughunter for investigating crashes correlated to a URL
    • KaiRo's reports for identifying crashes that are new or escalating quickly
    • Socorro for getting detailed information about crash reports
  • What information is needed to make a crash actionable by developers?
    • Correlations to particular hardware, driver, add-on, 3rd-party software, or library
    • ?...
Participation
  • Sanity checking via One & Done
  • Meetups to connect testers/users with devs
  • Testdays to teach people about graphics testing
  • Documentation and translation of documentation
  • Engaging on community spaces (Discourse, Reddit, Facebook, Twitter, etc)
Telemetry
  • COMPOSITE_TIME: time in CompositorParent::CompositeToTarget dispatching draw calls and calling SwapBuffers, but not texture upload (ie. complete composition)
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Platform/Graphics&oldid=1193585"