QA/Platform/Graphics/Firefox 38

From MozillaWiki
< QA‎ | Platform‎ | Graphics
Jump to: navigation, search

The following documents testing of graphics in Firefox 38 across a wide range of test environments. For more information on the methodology please consult our wiki.

Suggested Systems

Run tests once with the latest GPU driver available and a second time with whichever older driver version you have access.
System Platform CPU GPU Bugs Reported
Centurion Windows 7 Ultimate 32-bit Intel Core 2 Duo 2.4 GHz nVidia GeForce 8800 GTS 640MB bug 1144282
bug 1144257
Frostmane Windows XP Professional SP3 32-bit AMD Turion64 X2 ATI Radeon Xpress 1100 128MB (shared)
Higgins Windows 8 Professional 32-bit Intel Core i7-2620M Intel HD Graphics 3000 1720MB (shared)
Jandice Windows 8 Consumer 64-bit AMD FX-8120 AMD Radeon HD7670 1024MB bug 1144242
bug 1144237
bug 1144199
Lothar Windows 8 Professional 64-bit Intel Core 2 Duo 2.13 GHz nVidia GeForce 7300LE 256MB
Mango Windows Vista Basic 64-bit Intel Core i7-920 ATI Radeon HD4850 512MB bug 1144411
Marwyn Windows 7 Home Premium 64-bit AMD Phenom II X4 ATI Radeon HD3450 256MB
Palpatine Windows XP Professional SP3 32-bit Intel Core 2 Duo nVidia GeForce 8800 GTX 768MB (x2 in SLI) bug 1144754
Setesh Windows 7 Home Premium 64-bit Intel Celeron 900 Intel GMA 4500 1700MB (shared)
Shadowsong Windows Vista Home Premium 32-bit Intel Pentium D 2.8 GHz Intel GMA 950 256MB (shared)
Silvermoon Windows XP Professional SP3 32-bit Intel Core 2 Duo Intel GMA 950 256MB (shared)
Zito Windows Vista Ultimate 32-bit Intel Core 2 Duo 2.5 GHz nVidia GeForce 8600M GT 256MB bug 1144272
bug 1144378
bug 1144283
bug 1144258

What to look for

Videos:

  • significant buffering pauses
  • black video
  • spikes in CPU and memory usage
  • crashes and/or hangs
  • tab rendering issues

WebGL/Canvas2d:

  • crashes
  • hangs
  • visible artifacts
  • black screen
  • complete inability to load the demo(s)
  • high cpu/memory usage

Instructions

Setting up the test environment

1. Install Firefox
2. Disable the automatic updater
  • Open C:\Aurora\defaults\pref\channel-prefs.js in a text editor
  • Change the line reading pref("app.update.channel", "release"); to pref("app.update.channel", "foobar");
  • Save and Close the text file
3. Launch Firefox with a clean profile
  • Run cmd.exe
  • Change directory to the folder where you installed Firefox: cd C:\Aurora
  • Run firefox.exe: firefox.exe -P -no-remote - This will bring up our profile selector window.
  • Use the "Create Profile" button to create a profile new profile
  • Once the profile is created, select it, and then click "Start Firefox"
4. Load some videos in the background [Windows Vista/7/8 only]

Reporting a crash

If you encounter a crash you will see a crash reporter dialog. Use this to report the crash to Mozilla.

  • Enter a comment to describe the circumstances of the crash
  • Click submit on the crash reporter dialog
  • Restart Firefox
  • Load the about:support page and copy all of the information on that page to a text file, upload it to the Google drive location
  • Load the about:crashes page and copy the link of the crash report you submitted (it will be the one at the top of the list)
  • Report the link to the about:support page in Google drive as well as the link to your crash report

Tests

Note: Systems running Windows XP do NOT run test 1-4, start on test 5

1. Crash test [Windows Vista/7/8 only]
  • Load each of these URLs in new tabs:
  • Resize the window smaller and larger to random sizes
  • Click a link on the page and resize the window some more
  • Switch into responsive design view (CTRL + Shift + M), change resolution, and refresh the page
  • If you encounter a crash then proceed with reporting a crash instructions
  • If you do NOT encounter a crash then this test is a PASS, move on to the next test
2. Basic test [Windows Vista/7/8 only]
  • Load the about:config and click I'll be careful.
  • Search for dxva and set media.windows-media-foundation.use-dxva to true then restart Firefox
  • Load https://www.youtube.com/watch?v=hicBgE6XndM
  • Use gear icon to set to highest quality (1080HD)
  • Open full screen and play video
  • Use the bottom bar of the video to reset different quality levels (using gear icon)
  • Use the arrow keys (or the slider at the bottom of the page) to seek forward and back through the video
  • Make at least three changes to quality level and seeking then stop the video
  • Close both tabs
3. Stress test [Windows Vista/7/8 only]
4. Test with system sleep [Windows Vista/7/8 only]
  • Load https://www.youtube.com/watch?v=hicBgE6XndM
  • Use gear icon to set to highest quality (1080HD)
  • Let the video play to about halfway
  • Put your system into sleep mode
  • Restore your system and resume watching the video
5. Canvas2d demo [All systems]
6. Once you've completed the test run... [All systems]
  • Save a copy of about:memory:
    • Open a tab to about:memory
    • Click measure and save, use the naming convention fx38_<machinename>_<driverlevel>_memory.json
    • Upload this file to the Graphics Test Reporting/Firefox 38/Memory Reports/ folder on Google drive
    • Click both the GC and CC buttons to free up memory
  • Report any bugs you encountered
  • Fill out this form to report your results

Results

Notes

  • bugs
  • test plan was well described
  • managed to get through the testing faster than planned
  • should we ask for regression range using mozregression out of the gate?
    • betabreakers could use better documentation on how to find a range with mozregression and finding the build date
  • finding median driver versions is a challenge depending on how discoverable they are on the web
  • remote access for system troubleshooting may be possible
  • similar failures should be filed in a single bug and then split out of they're found to be different issues through investigation
  • what is the expectation for engaging Betabreakers for follow-ups?
    • we should seek to get as much information upfront to minimize the follow-up cycle
    • follow-up is possible but may be delayed if the tester has moved on to another project
    • we should account for a percentage of follow-up time in future contracts
  • include systems with blacklisted drivers in future testing
  • may need a standard list of hardware that we always use
  • Is betabreakers set up with Windows 10?
    • they do but not in the general tester population
    • look into doing a testrun later in the summer
  • how representative of our Beta audience is Betabreakers?

Actions

  • Anthony to provide better documentation for mozregression and getting a build ID
  • Anthony to provide documentation on attaching about:support, fhr, and telemetry to bug reports
  • Jeremy will follow up with IT to figure out future Win10 testing capability

Correlations

Operating Systems:

  • 5 bugs against Windows Vista
  • 3 bugs against Windows 8.1
  • 2 bugs against Windows 7
  • 1 bug against Windows XP

GPUs:

  • 7 bugs against NVidia
  • 4 bugs against AMD
  • 0 bugs against Intel

Architectures:

  • 7 bugs against 32-bit
  • 4 bugs against 64-bit

Processors:

  • 8 bugs against Intel
  • 3 bugs against AMD