Latest revision as of 20:54, 5 May 2015

The following documents testing of graphics in Firefox 38 across a wide range of test environments. For more information on the methodology please consult our wiki.

Suggested Systems

Run tests once with the latest GPU driver available and a second time with whichever older driver version you have access.

System	Platform	CPU	GPU	Bugs Reported
Centurion	Windows 7 Ultimate 32-bit	Intel Core 2 Duo 2.4 GHz	nVidia GeForce 8800 GTS 640MB	bug 1144282 bug 1144257
Frostmane	Windows XP Professional SP3 32-bit	AMD Turion64 X2	ATI Radeon Xpress 1100 128MB (shared)
Higgins	Windows 8 Professional 32-bit	Intel Core i7-2620M	Intel HD Graphics 3000 1720MB (shared)
Jandice	Windows 8 Consumer 64-bit	AMD FX-8120	AMD Radeon HD7670 1024MB	bug 1144242 bug 1144237 bug 1144199
Lothar	Windows 8 Professional 64-bit	Intel Core 2 Duo 2.13 GHz	nVidia GeForce 7300LE 256MB
Mango	Windows Vista Basic 64-bit	Intel Core i7-920	ATI Radeon HD4850 512MB	bug 1144411
Marwyn	Windows 7 Home Premium 64-bit	AMD Phenom II X4	ATI Radeon HD3450 256MB
Palpatine	Windows XP Professional SP3 32-bit	Intel Core 2 Duo	nVidia GeForce 8800 GTX 768MB (x2 in SLI)	bug 1144754
Setesh	Windows 7 Home Premium 64-bit	Intel Celeron 900	Intel GMA 4500 1700MB (shared)
Shadowsong	Windows Vista Home Premium 32-bit	Intel Pentium D 2.8 GHz	Intel GMA 950 256MB (shared)
Silvermoon	Windows XP Professional SP3 32-bit	Intel Core 2 Duo	Intel GMA 950 256MB (shared)
Zito	Windows Vista Ultimate 32-bit	Intel Core 2 Duo 2.5 GHz	nVidia GeForce 8600M GT 256MB	bug 1144272 bug 1144378 bug 1144283 bug 1144258

What to look for

Videos:

significant buffering pauses
black video
spikes in CPU and memory usage
crashes and/or hangs
tab rendering issues

WebGL/Canvas2d:

crashes
hangs
visible artifacts
black screen
complete inability to load the demo(s)
high cpu/memory usage

Instructions

Setting up the test environment

1. Install Firefox

Download https://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-aurora/firefox-38.0a2.en-US.win32.installer.exe
Run installer and select customize
Put the install in a known directory (C:\Aurora for purposes of this example)
Do NOT allow it to launch Firefox at the end of the install

2. Disable the automatic updater

Open C:\Aurora\defaults\pref\channel-prefs.js in a text editor
Change the line reading pref("app.update.channel", "release"); to pref("app.update.channel", "foobar");
Save and Close the text file

3. Launch Firefox with a clean profile

Run cmd.exe
Change directory to the folder where you installed Firefox: cd C:\Aurora
Run firefox.exe: firefox.exe -P -no-remote - This will bring up our profile selector window.
Use the "Create Profile" button to create a profile new profile
Once the profile is created, select it, and then click "Start Firefox"

4. Load some videos in the background [Windows Vista/7/8 only]

Open each of the following videos in new tabs
Set the quality of these videos to the highest (1080p60 or 4K)
Make sure these videos run in the background for the duration of the testrun

Reporting a crash

If you encounter a crash you will see a crash reporter dialog. Use this to report the crash to Mozilla.

Enter a comment to describe the circumstances of the crash
Click submit on the crash reporter dialog
Restart Firefox
Load the about:support page and copy all of the information on that page to a text file, upload it to the Google drive location
Load the about:crashes page and copy the link of the crash report you submitted (it will be the one at the top of the list)
Report the link to the about:support page in Google drive as well as the link to your crash report

Tests

Note: Systems running Windows XP do NOT run test 1-4, start on test 5

1. Crash test [Windows Vista/7/8 only]

Load each of these URLs in new tabs:
Resize the window smaller and larger to random sizes
Click a link on the page and resize the window some more
Switch into responsive design view (CTRL + Shift + M), change resolution, and refresh the page
If you encounter a crash then proceed with reporting a crash instructions
If you do NOT encounter a crash then this test is a PASS, move on to the next test

2. Basic test [Windows Vista/7/8 only]

Load the about:config and click I'll be careful.
Search for dxva and set media.windows-media-foundation.use-dxva to true then restart Firefox
Load https://www.youtube.com/watch?v=hicBgE6XndM
Use gear icon to set to highest quality (1080HD)
Open full screen and play video
Use the bottom bar of the video to reset different quality levels (using gear icon)
Use the arrow keys (or the slider at the bottom of the page) to seek forward and back through the video
Make at least three changes to quality level and seeking then stop the video
Close both tabs

3. Stress test [Windows Vista/7/8 only]

Load the following URLs each in their own tab:
Cycle through the tabs, setting all videos to theater mode
Cycle through the tabs, setting all videos to highest bitrate playback mode (using the gear icon)
Cycle through the tabs, seeking all videos backward
Cycle through the tabs, setting all videos to a 480p video mode and seek them forward
Cycle back through the tabs, set to highest bitrate and seek backward far enough that they have at least 30s remaining on all videos
Let everything play at once for about 30s

4. Test with system sleep [Windows Vista/7/8 only]

Load https://www.youtube.com/watch?v=hicBgE6XndM
Use gear icon to set to highest quality (1080HD)
Let the video play to about halfway
Put your system into sleep mode
Restore your system and resume watching the video

5. Canvas2d demo [All systems]

Load http://ie.microsoft.com/testdrive/HTML5/Blizzard/Default.html
Check to see that snowflakes animate on the screen
NOTE: Slow performance is acceptable - we're just concerned if the flakes show or not

6. Once you've completed the test run... [All systems]

Save a copy of about:memory:
- Open a tab to about:memory
- Click measure and save, use the naming convention fx38_<machinename>_<driverlevel>_memory.json
- Upload this file to the Graphics Test Reporting/Firefox 38/Memory Reports/ folder on Google drive
- Click both the GC and CC buttons to free up memory
Report any bugs you encountered
Fill out this form to report your results

Results

Notes

bugs
test plan was well described
managed to get through the testing faster than planned
should we ask for regression range using mozregression out of the gate?
- betabreakers could use better documentation on how to find a range with mozregression and finding the build date
finding median driver versions is a challenge depending on how discoverable they are on the web
remote access for system troubleshooting may be possible
similar failures should be filed in a single bug and then split out of they're found to be different issues through investigation
what is the expectation for engaging Betabreakers for follow-ups?
- we should seek to get as much information upfront to minimize the follow-up cycle
- follow-up is possible but may be delayed if the tester has moved on to another project
- we should account for a percentage of follow-up time in future contracts
include systems with blacklisted drivers in future testing
may need a standard list of hardware that we always use
Is betabreakers set up with Windows 10?
- they do but not in the general tester population
- look into doing a testrun later in the summer
how representative of our Beta audience is Betabreakers?

Actions

Anthony to provide better documentation for mozregression and getting a build ID
Anthony to provide documentation on attaching about:support, fhr, and telemetry to bug reports
Jeremy will follow up with IT to figure out future Win10 testing capability

Correlations

Operating Systems:

5 bugs against Windows Vista
3 bugs against Windows 8.1
2 bugs against Windows 7
1 bug against Windows XP

GPUs:

7 bugs against NVidia
4 bugs against AMD
0 bugs against Intel

Architectures:

7 bugs against 32-bit
4 bugs against 64-bit

Processors:

8 bugs against Intel
3 bugs against AMD

QA/Platform/Graphics/Firefox 38: Difference between revisions