QA/Platform/Graphics: Difference between revisions
Jump to navigation
Jump to search
Full Query
| Line 86: | Line 86: | ||
* [[Platform/GFX/TriageSchedule|Bug triage]] | * [[Platform/GFX/TriageSchedule|Bug triage]] | ||
== | == Sanity Checking == | ||
'''''Goal: identify graphics regressions in pre-release Firefox versions. | |||
=== <small>Betabreakers</small> === | |||
Betabreakers provides sanity testing of Firefox Developer Edition on a wide-array of systems. | |||
* {{done|}} [[QA/Platform/Graphics/Firefox_38|Firefox 38]]: MSE stress test | * {{done|}} [[QA/Platform/Graphics/Firefox_38|Firefox 38]]: MSE stress test | ||
* {{done|}} [[QA/Platform/Graphics/Firefox_39|Firefox 39]]: beta sanity check | * {{done|}} [[QA/Platform/Graphics/Firefox_39|Firefox 39]]: beta sanity check | ||
| Line 105: | Line 107: | ||
} | } | ||
</bugzilla> | </bugzilla> | ||
=== <small>Toronto Graphics Lab</small> === | |||
; <span style="background-color:#CD3333; color:#FFF; align:center">NOTE: Activity has stopped as of September 1, 2015 while we determine how best to resource this going forward.</span> | |||
The Graphics team hosts an ad-hoc lab of systems in Mozilla's Toronto office for testing and debugging. | |||
; Test ([https://docs.google.com/spreadsheets/d/147EhIRV-NveYpUkbjAdWtvLDv6yV_bP6hweXnzlitYg/edit#gid=1455079096 results]) | |||
* Start with a new profile (see above) | |||
* Resize the window so it's at least 1024x768 | |||
* Go to [http://www.nytimes.com/ nytimes.com], scroll up and down for ~15 seconds | |||
* Open another tab and go to [http://www.pinterest.com/ pinterest.com], scroll up and down for ~15 seconds. | |||
* Switch back and forth between the two tabs three times | |||
* In nytimes.com tab, go to youtube.com and play [https://www.youtube.com/watch?v=3UA9jaCOiTY this video], [https://www.youtube.com/watch?v=jofNR_WkoCE this video], and some other random video for 15 seconds | |||
** '''Note:''' If a video plays with Flash find another video to test. To check, right mouse click on the video and choose “Stats for nerds”. The video is valid if the mime type is ''video/mp4'' or ''video/webm''. | |||
* Go to [http://www.cnn.com/ cnn.com], scroll for ~15 seconds then find a video and play it | |||
* Update your graphics driver to the latest available version | |||
** '''Note:''' don't restart Firefox or your computer if it can be avoided. If Firefox crashes, report the crash, restart Firefox and continue on to step 9. | |||
* Repeat the test from step 3 through 7. | |||
** '''Note:''' if Firefox crashes, report the crash, restart Firefox and repeat steps 3 through 7. | |||
* Quit Firefox after all steps have been completed and [http://bit.ly/1JZqzMc submit your test results]. | |||
=== <small>Crowd Testing</small> === | |||
One & Done provides a task which enables volunteers to run a sanity test and report results to the team. | |||
* Task | |||
* Results | |||
Revision as of 18:57, 17 September 2015
Consult this page for more information.
Documentation
- Device Inventory (moved to wiki)
- PCI Device ID List
- nVidia Codenames
- Graphics driver blocklisting
- Draft guides
Understanding the Problem Space
First order of business for my transition to the Graphics team is to understand the problem space so I can understand the immediate needs of the team and make the best impact I can in the shortest amount of time.
- What are the key problems/challenges facing the Graphics team in terms of quality?
- discrepancy in environments between testers and release users
- discoverability of bugs pre-release
- ?...
- Where can QA add value/support to the Graphics team?
- improving pre-release discoverability of bugs
- closing the gap between tester and release systems
- helping with bug triage, particularly with bugs hiding in general components
- representation in crashkill
- improving code coverage and/or identifying gaps in code coverage
- identifying ways to improve participation in the graphics team (events, projects, One & Done, etc)
- documentation of tools, testing processes, etc
- building out the lab in Toronto
- continuing to drive Betabreakers testing every 6 weeks
- verifying bug fixes (what does this look like)?
- profiling areas of risk (eg. troublesome configs)
- conducting root cause analysis for regressions
- understanding problems outside of our control (eg. driver resets)
- feature testing and upcoming priorities (e10s, Windows 10, El Capitain, Android, B2G, etc)
- What does QA need to know to be effective?
- key components of an actionable Graphics bug
- fundamentals/technologies that should be learned
- how to distinguish a graphics crash from a non-graphics crash with a graphics signature
- meetings, mailing lists, bugzilla components to watch, blogs, IRC channels to join, etc
- who is each member of the team (incl. contributors) and what do they do
- where does graphics code reside in the tree?
- what role does Unified Telemetry in graphics quality?
- what are the prefs to enable/disable different functionalities?
- we need a database of known-troublesome hardware/driver configurations to inform testing, hardware acquisitions, and blocklisting
Sanity Checking
- Desktop
- Boot 2 Gecko (No-Jun Park)
- Android
- Telemetry
Stability
How do we identify a graphics crash?
- by signature: gfx, layers, D2D, D3D, ?...
- by topmost filename: gfx, ?...
- ?...
How do we prioritize graphics crashes?
- Overall topcrashes in release > beta > aurora > nightly
- Gfx crashes in release > beta > aurora > nightly
- Explosive crashes in release > beta > aurora > nightly
What tools do we have at our disposal to investigate crashes?
- Bughunter for investigating crashes correlated to a URL
- KaiRo's reports for identifying crashes that are new or escalating quickly
- Socorro for getting detailed information about crash reports
What information is needed to make a crash actionable by developers?
- Correlations to particular hardware, driver, add-on, 3rd-party software, or library
- ?...
Participation
- Sanity checking via One & Done
- Meetups to connect testers/users with devs
- Testdays to teach people about graphics testing
- Documentation and translation of documentation
- Engaging on community spaces (Discourse, Reddit, Facebook, Twitter, etc)
Telemetry
- COMPOSITE_TIME: time in CompositorParent::CompositeToTarget dispatching draw calls and calling SwapBuffers, but not texture upload (ie. complete composition)
Projects
Sanity Checking
Goal: identify graphics regressions in pre-release Firefox versions.
Betabreakers
Betabreakers provides sanity testing of Firefox Developer Edition on a wide-array of systems.
- [DONE] Firefox 38: MSE stress test
- [DONE] Firefox 39: beta sanity check
- [DONE] Firefox 40: WebGL with e10s
- [DONE] Firefox 41: WARP disabled on Windows 7 and earlier
- [DONE] Firefox 42: Windows 10 AMD/NVidia stability
- Firefox 43: to be determined
- Firefox 44: to be determined
99 Total; 5 Open (5.05%); 90 Resolved (90.91%); 4 Verified (4.04%);
Toronto Graphics Lab
- NOTE: Activity has stopped as of September 1, 2015 while we determine how best to resource this going forward.
The Graphics team hosts an ad-hoc lab of systems in Mozilla's Toronto office for testing and debugging.
- Test (results)
- Start with a new profile (see above)
- Resize the window so it's at least 1024x768
- Go to nytimes.com, scroll up and down for ~15 seconds
- Open another tab and go to pinterest.com, scroll up and down for ~15 seconds.
- Switch back and forth between the two tabs three times
- In nytimes.com tab, go to youtube.com and play this video, this video, and some other random video for 15 seconds
- Note: If a video plays with Flash find another video to test. To check, right mouse click on the video and choose “Stats for nerds”. The video is valid if the mime type is video/mp4 or video/webm.
- Go to cnn.com, scroll for ~15 seconds then find a video and play it
- Update your graphics driver to the latest available version
- Note: don't restart Firefox or your computer if it can be avoided. If Firefox crashes, report the crash, restart Firefox and continue on to step 9.
- Repeat the test from step 3 through 7.
- Note: if Firefox crashes, report the crash, restart Firefox and repeat steps 3 through 7.
- Quit Firefox after all steps have been completed and submit your test results.
Crowd Testing
One & Done provides a task which enables volunteers to run a sanity test and report results to the team.
- Task
- Results