QA/Platform/Graphics

From MozillaWiki
< QA‎ | Platform
Jump to: navigation, search

Summary

This page holds important information related to testing and quality assurance for Gecko's graphics code.

Metrics

Tools

Device ID Catalogs:

Get Involved

Here are some ways you can help the Graphics team (and Mozilla):

Activities

One & Done Testing

DEPRECATION WARNING: One&Done is being decommissioned. In the meantime, all tasks have been archived to this wiki.


Betabreakers Lab Testing

Summary

There is a large discrepancy between the platform/hardware/driver configurations we see on Release versus what we see on our test branches. Additionally, the Graphics team and our testing community does not and can not provide sufficient test coverage to fill this gap. As a result we've historically been notified far too late about critical graphics issues triggering multiple chemspill releases and a loss of users.

In an effort to improve pre-release visibility of graphics regressions we have partnered with Betabreakers, a company that maintains a lab specifically designed to test various modern and legacy configurations of Windows versions, graphics hardware, and graphics drivers. We try to coordinate a graphics sanity testrun with them once per mozilla-aurora cycle, as detailed below.

More information about our relationship with Betabreakers can be found on mana.mozilla.org.

Test Runs

Bugs

Full Query
ID Summary Status
1144199 Test # 4: Bioshock Infinite video goes to infinite loading screen when resuming to play after putting computer to sleep RESOLVED
1144237 During Test 3 (Stress test) multiple videos buffer forever or just stop playing RESOLVED
1144242 Videos do not resume playing when coming out of sleep RESOLVED
1144257 DXVA: Firefox crashed when opening right click menu RESOLVED
1144258 Bioshock Ininite command video is distorted when setting quality to 1080p RESOLVED
1144272 Test # 3 (Stress Test) Video playback stutters when setting video resolution to max RESOLVED
1144282 DXVA crash with 3 HD video tabs open RESOLVED
1144283 Test # 5: (Canvas2d demo): No snowflakes are displayed RESOLVED
1144378 Stress test: Severe lag when when having multiple tabs open RESOLVED
1144411 Video corruption in 1080p RESOLVED
1144754 Some videos wont play, videos only play in 360p on Windows XP RESOLVED
1167621 Dead Trigger 2 Demo becomes laggy when multiple assets (enemies) are on screen RESOLVED
1167656 Audio stuttering on Auralux menu RESOLVED
1167681 Audio stutters while playing and persists after game ends in Dead Trigger 2 RESOLVED
1167700 Polycraft Demo does not play since Firefox 38 RESOLVED
1167778 X-Type game suffers from low frame rate RESOLVED
1168451 BSOD when playing Banana Bread game RESOLVED
1168467 Tanx game suffers from low framerate spikes RESOLVED
1172561 Mini Dell: Browser becomes laggy when the user scrolls the page up/down RESOLVED
1172596 Videos will not play from CNN.com VERIFIED
1172628 Setesh (System): Firefox becomes unresponsive when right clicking link, and selecting open in new tab RESOLVED
1172634 (Setesh) Firefox becomes unresponsive when selecting to play any CNN.com videos RESOLVED
1173000 Rajh (System) All videos load to black screen even after Flash is updated/installed RESOLVED
1173012 CNN videos won't play as the file is corrupt VERIFIED
1180317 some image black displayed with Firefox 40.0* RESOLVED
1184193 Some Windows 7 systems running WARP when they shouldn't be RESOLVED
1184618 Some images on flickr.com show striping artifacts RESOLVED
1185017 Crash when resizing browser while entering Google Street View after updating driver with browser open RESOLVED
1201126 1080p 60 FPS video stuttering and occasionally stopping playback RESOLVED
1217124 Expand, Minimize and Close buttons do not display RESOLVED
1217170 YouTube serves 360p VP8 video instead of Flash video to FF 40+ VERIFIED
1217240 Youtube Video Resolution has maximum resolution of 360p RESOLVED
1217547 Windows shell buttons (minimize/maximize/close) disappear when Windowblinds theme is applied if OMTC is enabled RESOLVED
1217566 Enhance untrusted connection pages with basic troubleshooting tips UNCONFIRMED
1217587 Youtube Video Resolution has maximum resolution of 360p RESOLVED
1217936 Max Youtube Video Resolution 720p RESOLVED
1217999 "You can skip to video in 5" countdown timer not moving, stuck watching ads RESOLVED
1218546 sec_error_ocsp_future_response error when loading various pages RESOLVED
1219425 "Let It Snow" Microsoft Edge demo acquires the incorrect window height of 32px NEW
1219959 Youtube videos max resolution is 360p Mozilla Firefox 41.0.2 for Ubuntu RESOLVED
1229171 Problem loading page, "The connection has timed out" when accessing www.zamunda.net RESOLVED
1229190 Content does not load when a new window is called RESOLVED
1229193 New windows do not retain firefox/windows themes correctly UNCONFIRMED
1229207 Close, minimize, etc, buttons do not show with blinds and firefox theme RESOLVED
1229872 Videos on Twitter homepage display "This browser does not support video playback" RESOLVED
1229904 vid.me videos fail to fallback to Flash playback on Windows XP RESOLVED
1229914 Black corruption appears when opening/closing tabs UNCONFIRMED
1229917 Incredibly slow performance while simultaneously loading several tabs RESOLVED
1229922 Error with tv.digikala.com video player in Firefox RESOLVED
1229970 Crash in `anonymous namespace''::ScriptLoaderRunnable::OnStartRequest while dragging tabs to their own window RESOLVED
1230625 Google Icons/Buttons do not render RESOLVED
1265809 Choppy video and sound on youtube.com on Shadowsong computer. RESOLVED
1265820 Choppy Video and Sound on youtube.com on Anetheron computer. RESOLVED
1265870 Choppy video on Youtube 360 at 2k and 4k on Asaad computer. RESOLVED
1265879 No video or audio on Twitch on Asaad computer VERIFIED
1265894 Choppy video on Youtube 360 on 4k and 1440 videos on Azeroth computer RESOLVED
1265901 Choppy video on Vimeo whenever full-screened on Azeroth computer RESOLVED
1265910 Choppy on all video when full-screened on Azeroth RESOLVED
1265918 Doesn't load 360 portion of youtube 360 and is choppy on 4k and 1440 on Lothar RESOLVED
1265933 Embedded facebook video shows no picture when on HD on Lothar computer. RESOLVED
1265935 Video on Vimeo gets choppy in 1080p and even choppier/lagging in fullscreen 1080p on Lothar computer RESOLVED
1265937 Any and all ads loaded are extremely choppy on Lothar computer RESOLVED
1266111 Choppy video on Vimeo at 1080P when full-screened on Shadowsong RESOLVED
1266114 Doesn't load 360 controls for Youtube 360 and gets choppy at 720p and up on Shadowsong computer RESOLVED
1266141 All remaining issues on Shadowsong computer RESOLVED
1266186 All remaining issues on Anetheron RESOLVED
1266208 Minor video/sound sync with Twitch at high quality on Ming computer. RESOLVED
1266246 All issues on the Centurion computer RESOLVED
1266482 All issus with Frostwolf computer RESOLVED
1266504 All issues with Silvermoon computer RESOLVED
1266560 Youtube 360 has some choppiness with hardware acceleration off and no 4k option with it on on Dradis computer. RESOLVED
1266580 All problems on Palpatine computer RESOLVED
1266787 All issues on Warlock computer RESOLVED
1266814 All issues on Onyxia computer RESOLVED
1266861 All issues on Dragonmaw computer RESOLVED
1266881 All issues on Frostmane computer RESOLVED
1266902 All issues on Nefarian computer RESOLVED
1267304 All issues on Marwyn computer RESOLVED
1267341 All issues on Eomer computer RESOLVED
1267366 All issues on Skullcrusher computer RESOLVED
1267424 All issues on Kal-El computer RESOLVED
1267478 All issues on Horridon computer RESOLVED
1267667 Choppy video on youtube UNCONFIRMED
1267668 Choppy video on Youtube 360 RESOLVED
1267669 Choppy video on Vimeo UNCONFIRMED
1267672 Choppy video on Facebook UNCONFIRMED
1267673 Choppy video on DuckDuckGo UNCONFIRMED
1267674 Choppy video on Reddit UNCONFIRMED
1267676 Choppy video on Steam UNCONFIRMED
1267677 Choppy video on Twitch UNCONFIRMED
1267679 Choppy video on ABC News UNCONFIRMED
1267682 Choppy video on NFL.com UNCONFIRMED
1267700 360 controls on Youtube 360 don't load RESOLVED
1267714 Facebook HD video won't load RESOLVED
1267723 Audio/video sync issues on NFL.com RESOLVED
1267726 Audio/video sync issues on Twitch RESOLVED
1267728 Audio/video sync issues on ABC News RESOLVED
1269538 Vimeo not loading video on Talon RESOLVED
1269541 Some videos on Talon system require Flash to play RESOLVED

99 Total; 13 Open (13.13%); 82 Resolved (82.83%); 4 Verified (4.04%);

Triage

Triage currently focuses on two main areas: identifying critical stability regressions, and identifying issues which may have slipped through the cracks. We try to have a one hour triage session each weekday, usually starting around 10:00am Pacific. More information about this process can be found on the wiki and dashboard.

Retired Activities

Top Issues

Blockers

Full Query
ID Summary Status Last change time
988594 [Camera][Nexus4] Very slow to draw previews when swiping through Preview from Camera NEW 2022-05-16T12:18:27Z
1073873 Green screen when trying to use built-in camera with WebRTC on Flatfish NEW 2014-12-26T10:40:27Z
1342437 images with base64 src should be displayed immediately without using onload event in order to be drawn in HTML5 canvas (like on Chrome/IE/...) UNCONFIRMED 2018-02-15T23:36:10Z

3 Total; 3 Open (100%); 0 Resolved (0%); 0 Verified (0%);


Crashes

Full Query
ID Summary Status Last change time
1218607 [meta] Firefox on android crashing in [@ libGLESv2_adreno.so@ ...] NEW 2022-07-05T20:43:35Z
1761233 Crash in [@ webrender::render_api::RenderApi::send_transaction] NEW 2022-06-29T15:22:03Z
1769254 Crash in [@ mozilla::wr::RenderMacIOSurfaceTextureHost::GetSize] ASSIGNED 2022-08-10T12:39:38Z

3 Total; 3 Open (100%); 0 Resolved (0%); 0 Verified (0%);


Features

Full Query
ID Summary Status Last change time
703217 Implement image() from CSS Image Values and Replaced Content Module Level 4 NEW 2022-06-30T12:46:11Z
790640 add Media Fragments for images (spatial dimensions) NEW 2022-02-23T12:56:11Z
952539 WebGL should support efficiently uploading a subrect of a canvas to a texture NEW 2022-04-06T13:42:53Z
1152974 Enable by default DEAA (Anti-aliasing) for Linux NEW 2022-06-27T12:12:09Z
1200864 Skip DrawElements buffer validation when we have robust_buffer_access NEW 2019-04-22T23:40:45Z
1223880 Implement initial-letter (from CSS Inline Layout Module Level 3) NEW 2021-12-06T08:46:04Z
1233569 Deprecate RGB8 from being renderable in WebGL 1 NEW 2019-02-02T22:42:29Z
1237489 [webvr] Add basic MOZ_texture_from_element to enable creating a texture from an arbitrary element NEW 2019-04-17T00:58:12Z
1244513 webm videos should be color corrected UNCONFIRMED 2019-04-16T14:29:04Z
1264543 [meta] Out-of-process compositing NEW 2022-02-16T15:57:55Z
1282074 Support very large canvases NEW 2022-06-01T12:27:44Z
1302685 [meta] Support OpenType Font Variations (Variable fonts) NEW 2022-01-01T16:47:04Z
1309309 Report memory used by ANGLE in about:memory as its own block? NEW 2017-03-24T11:32:18Z
1350292 Memory allocated by WebGL in WebGLBuffer::BufferData is not tracked in about:memory (detected by DMD in heap-unclassified) NEW 2019-06-14T21:31:28Z
1367003 Page slow to load when scrolling up and down. NEW 2017-12-31T19:09:07Z
1371101 Run color, border-color, and background-color animations off the main thread NEW 2022-01-17T10:05:43Z
1371108 WebGL: Make default back buffer default to 8x8 AA when available UNCONFIRMED 2017-06-09T19:36:22Z
1371627 GL_EXT_draw_buffers extension not supported UNCONFIRMED 2017-06-09T19:23:55Z
1371836 Provide a performance object for the WebGL canvas NEW 2017-06-10T02:09:50Z
1374710 GLBlitHelper should cache the various shaders NEW 2017-09-11T15:51:48Z
1381098 imageBitmap should provide an api to pull out raw pixel data (unmodified) without having to draw it to a canvas or a renderBuffer. This should be able to run in a webworker UNCONFIRMED 2021-06-04T16:58:46Z
1391345 Some emoji on the emojione demo page don't show up NEW 2017-09-12T21:38:26Z
1392080 Feature request : ɴᴘᴀᴘɪ being phased out : please provide an alternative for direct Vulkan/Direct3ᴅ12 access for viewing stereoscopic videos ! UNCONFIRMED 2019-02-15T16:58:19Z
1402293 Implement support for High Efficiency Image File Format (HEIF) NEW 2022-06-26T05:57:12Z
1403299 Support rendering fonts via FreeType on Windows NEW 2022-06-19T12:26:29Z
1403933 Support standard swizzle in D3D11 texture NEW 2018-02-15T23:21:11Z
1412350 Support Color Management on Android NEW 2020-08-10T09:17:00Z
1428954 Support encoding .ico files with multiple sizes NEW 2021-12-09T05:07:04Z
1443863 Implement support for AV1 Still Image File Format (AVIF) NEW 2022-06-26T20:47:57Z
1461360 [meta] Cross-platform dependencies for pinch-zooming on desktop NEW 2021-04-19T21:33:07Z
1469281 Implement text rendering on OffscreenCanvas UNCONFIRMED 2018-09-05T07:22:37Z
1476861 Implement FillRoundedRect for DrawTargetSkia ASSIGNED 2021-09-10T04:57:41Z
1477078 Implement PushClipRoundedRect for DrawTargetSkia ASSIGNED 2021-09-10T04:58:01Z
1480894 Implement wide-gamut support on Canvas NEW 2018-10-31T09:17:41Z
1514957 offscreencanvas doesn't seem to work UNCONFIRMED 2018-12-19T06:37:58Z
1516362 Selecting text using background-clip: text should probably change the text color NEW 2020-08-03T19:32:47Z
1521723 Implement the hyphenate-limit-chars property NEW 2022-06-30T08:08:18Z
1522397 Displayed content in active tab doesn't adapt to screen resolution changes when connecting/disconnecting external monitor NEW 2019-02-04T18:42:01Z
1532721 Re-enable D2D on Qualcomm devices when able NEW 2019-03-06T18:48:20Z
1536809 Add support for Canvas 2D "desynchronized" construction attribute UNCONFIRMED 2020-06-04T17:09:19Z
1539075 Implement support for JPEG XL (image/jxl) ASSIGNED 2022-07-20T21:07:55Z
1547612 Use Lanczos method for upscaling images for different zoom levels NEW 2022-04-30T14:41:56Z
1605234 [meta] Support elastic overscroll on desktop NEW 2022-01-27T06:35:44Z

43 Total; 43 Open (100%); 0 Resolved (0%); 0 Verified (0%);


New Issues

Full Query
ID Summary Last change time
1783163 Sandbox: attempt to open unexpected file /sys/devices/system/cpu/cpu0/cache/index2/size 2022-08-08T17:39:13Z
1783183 Make amount of page scroll (page up / page down keypress) configurable 2022-08-10T16:33:35Z
1783512 BlockShutdown : ShutDownProcess Warning messages 2022-08-08T16:10:59Z
1783513 DeviceFamily::QualcommAll does not include SC7180 Adreno 618 GPUs 2022-08-09T13:47:24Z
1783525 Segfault in mozilla::DoCompressedTexImage running WebGL Conformance Tests 2022-08-08T21:51:14Z
1783575 Angle GPU type is wrong on YouTube in Firefox (correct in Chrome) 2022-08-09T00:18:04Z
1783691 Canvas ellipse precision 2022-08-09T02:00:15Z
1783707 Webpages will tear while scrolling. 2022-08-09T18:25:31Z
1783937 [css-values-4] changing ic unit character to 永 (U+6C38) 2022-08-10T14:00:38Z
1783965 Firefox crash when copying text 2022-08-10T01:00:51Z
1784022 Convert text-emphasis-position #defines to enum classes. 2022-08-11T04:38:33Z
1784093 Graphics driver crashes constantly on some sites after last update 103.0.2: Intel GFX driver 9.17.10.2932 / Windows 7 64 bit 2022-08-10T21:19:05Z
1784236 Firefox scrolling jumps in Facebook 2022-08-11T09:31:51Z

13 Total; 13 Open (100%); 0 Resolved (0%); 0 Verified (0%);


Understanding the Problem Space

First order of business for my transition to the Graphics team is to understand the problem space so I can understand the immediate needs of the team and make the best impact I can in the shortest amount of time.

What are the key problems/challenges facing the Graphics team in terms of quality?
  • discrepancy in environments between testers and release users
  • discoverability of bugs pre-release
  •  ?...
Where can QA add value/support to the Graphics team?
  • improving pre-release discoverability of bugs
  • closing the gap between tester and release systems
  • helping with bug triage, particularly with bugs hiding in general components
  • representation in crashkill
  • improving code coverage and/or identifying gaps in code coverage
  • identifying ways to improve participation in the graphics team (events, projects, One & Done, etc)
  • documentation of tools, testing processes, etc
  • building out the lab in Toronto
  • continuing to drive Betabreakers testing every 6 weeks
  • verifying bug fixes (what does this look like)?
  • profiling areas of risk (eg. troublesome configs)
  • conducting root cause analysis for regressions
  • understanding problems outside of our control (eg. driver resets)
  • feature testing and upcoming priorities (e10s, Windows 10, El Capitain, Android, B2G, etc)
What does QA need to know to be effective?
  • key components of an actionable Graphics bug
  • fundamentals/technologies that should be learned
  • how to distinguish a graphics crash from a non-graphics crash with a graphics signature
  • meetings, mailing lists, bugzilla components to watch, blogs, IRC channels to join, etc
  • who is each member of the team (incl. contributors) and what do they do
  • where does graphics code reside in the tree?
  • what role does Unified Telemetry in graphics quality?
  • what are the prefs to enable/disable different functionalities?
  • we need a database of known-troublesome hardware/driver configurations to inform testing, hardware acquisitions, and blocklisting
Understanding the Stability
  • How do we identify a graphics crash?
    • by signature: gfx, layers, D2D, D3D, ?...
    • by topmost filename: gfx, ?...
    • by driver (DLL, version, ?...)
    • by device/vendor ID?...
    •  ?...
  • How do we prioritize graphics crashes?
    • Overall topcrashes in release > beta > aurora > nightly
    • Gfx crashes in release > beta > aurora > nightly
    • Explosive crashes in release > beta > aurora > nightly
  • What tools do we have at our disposal to investigate crashes?
    • Bughunter for investigating crashes correlated to a URL
    • KaiRo's reports for identifying crashes that are new or escalating quickly
    • Socorro for getting detailed information about crash reports
  • What information is needed to make a crash actionable by developers?
    • Correlations to particular hardware, driver, add-on, 3rd-party software, or library
    •  ?...
Participation
  • Sanity checking via One & Done
  • Meetups to connect testers/users with devs
  • Testdays to teach people about graphics testing
  • Documentation and translation of documentation
  • Engaging on community spaces (Discourse, Reddit, Facebook, Twitter, etc)
Telemetry
  • COMPOSITE_TIME: time in CompositorParent::CompositeToTarget dispatching draw calls and calling SwapBuffers, but not texture upload (ie. complete composition)