MozillaWiki

QA/Platform/Graphics

< QA‎ | Platform
Revision as of 22:31, 20 November 2015 by Ashughes (talk | contribs) (→‎Projects)

Consult this page for more information.

Metrics

Documentation

Understanding the Problem Space

First order of business for my transition to the Graphics team is to understand the problem space so I can understand the immediate needs of the team and make the best impact I can in the shortest amount of time.

What are the key problems/challenges facing the Graphics team in terms of quality?
  • discrepancy in environments between testers and release users
  • discoverability of bugs pre-release
  • ?...
Where can QA add value/support to the Graphics team?
  • improving pre-release discoverability of bugs
  • closing the gap between tester and release systems
  • helping with bug triage, particularly with bugs hiding in general components
  • representation in crashkill
  • improving code coverage and/or identifying gaps in code coverage
  • identifying ways to improve participation in the graphics team (events, projects, One & Done, etc)
  • documentation of tools, testing processes, etc
  • building out the lab in Toronto
  • continuing to drive Betabreakers testing every 6 weeks
  • verifying bug fixes (what does this look like)?
  • profiling areas of risk (eg. troublesome configs)
  • conducting root cause analysis for regressions
  • understanding problems outside of our control (eg. driver resets)
  • feature testing and upcoming priorities (e10s, Windows 10, El Capitain, Android, B2G, etc)
What does QA need to know to be effective?
  • key components of an actionable Graphics bug
  • fundamentals/technologies that should be learned
  • how to distinguish a graphics crash from a non-graphics crash with a graphics signature
  • meetings, mailing lists, bugzilla components to watch, blogs, IRC channels to join, etc
  • who is each member of the team (incl. contributors) and what do they do
  • where does graphics code reside in the tree?
  • what role does Unified Telemetry in graphics quality?
  • what are the prefs to enable/disable different functionalities?
  • we need a database of known-troublesome hardware/driver configurations to inform testing, hardware acquisitions, and blocklisting

Participation

  • Sanity checking via One & Done
  • Meetups to connect testers/users with devs
  • Testdays to teach people about graphics testing
  • Documentation and translation of documentation
  • Engaging on community spaces (Discourse, Reddit, Facebook, Twitter, etc)

Telemetry

  • COMPOSITE_TIME: time in CompositorParent::CompositeToTarget dispatching draw calls and calling SwapBuffers, but not texture upload (ie. complete composition)

Projects

Sanity Checking

Project Summary Frequency Status
Betabreakers Per-version lab testing against Developer Edition Every 6 weeks 2015-11-20: 44.0a2 testrun in planning
Boot2Gecko Sanity checks on phones in the Toronto office Weekly 2015-11-20: 45.0a1 testrun in progress
One & Done Crowd-sourced sanity checking via One & Done (results) Daily 2015-10-15: Reviewed latest results
Toronto Lab Testing Sanity checks on systems in the Toronto office Daily 2015-08-20: On hold

Stability

The purpose of this project is to identify the most concerning graphics crashes and escalate them to developers.

Understanding the Problem

How do we identify a graphics crash?

  • by signature: gfx, layers, D2D, D3D, ?...
  • by topmost filename: gfx, ?...
  • by driver (DLL, version, ?...)
  • by device/vendor ID?...
  • ?...

How do we prioritize graphics crashes?

  • Overall topcrashes in release > beta > aurora > nightly
  • Gfx crashes in release > beta > aurora > nightly
  • Explosive crashes in release > beta > aurora > nightly

What tools do we have at our disposal to investigate crashes?

  • Bughunter for investigating crashes correlated to a URL
  • KaiRo's reports for identifying crashes that are new or escalating quickly
  • Socorro for getting detailed information about crash reports

What information is needed to make a crash actionable by developers?

  • Correlations to particular hardware, driver, add-on, 3rd-party software, or library
  • ?...

Device Driver Crashes

Queries

Top Crashes

Process
  1. Load the Firefox crash-data dashboard
    The first thing you'll see is a graph showing the crash rate for each of the branches.
    The second thing you'll see is a list of reports for each of the branches.
  2. Click the Top Crashers link for the version you want to look at
    From here you'll see a large table of crash signatures ranked by the number of reports in the last week.
    Typically you'll want to start by looking at Release (right-most) as that's the highest priority.
  3. Starting at the top of the table, scroll down until you find a graphics-related crash
    ** NEED A PRIMER ON IDENTIFYING A GRAPHICS CRASH **
Bugs


Triage

The goal of triage is to ensure issues are reviewed, prioritized, and escalated in a reasonable amount of time.

Bug Triage

Category Description Frequency Status
Cold Crashes Crash bugs which have not been updated in more than 30 days Monthly Last reviewed 2015-11-09
Cold Trackers Tracked bugs which have not been updated since the previous release Every 6 weeks Last reviewed 2015-11-03
Help Wanted Bugs which lack information necessary to resolve Monthly Last reviewed 2015-11-09
Incoming Bugs reported but not triaged for more than one month Monthly Last reviewed 2015-11-09

Crash Triage

Category Description Frequency Status
Explosive Crashes Check KaiRo's branch explosiveness reports for new or spiking crashes Weekly Last reviewed 2015-11-17
Top Crashes Check the Socorro topcrash reports for new or spiking crashes Weekly Last reviewed 2015-11-17

Get Involved

Want to help the Graphics team? Here are some ways you can get involved.

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Platform/Graphics&oldid=1106616"