Auto-tools/Projects/Platform Quality/External Media Tests

From MozillaWiki
< Auto-tools‎ | Projects‎ | Platform Quality
Revision as of 21:02, 24 March 2016 by Sydpolk (talk | contribs) (Added links to other documentation.)
Jump to navigation Jump to search

External Media Tests is a suite of Marionette Python tests for media playback in Mozilla Firefox. Its aim is to test MSE (Media Source Extensions) and EME (Encrypted Media Extensions) on specific websites such as YouTube and Netflix.

Other documentation

General information about external-media-tests can be found on MDN. API documentation can be found on readthedocs.org.

Source

How are these tests run

There are five ways to run External Media Tests:

  • From <mozilla-central>, type
./mach external-media-tests

Automation Information

External Media Tests currently runs on buildbot (limited tests), and pf-jenkins. There are plans to migrate some or all of the pf-jenkins to QA/Automation/Projects/Mozmill_Automation/Mozmill_CI Mozmill_CI, and to taskcluster.

Results Reported to Treeherder

These are Tier-2 jobs, meaning they are visible by default but not sheriff-managed.

buildbot

There are two short test jobs that run in buildbot

  • Repos: trunk
  • Platforms: Linux x64 (opt only), Windows 7, Windows 8 x64, OS X 10.10
  • Group Name: VideoPuppeteer
  • Group Symbol: VP
  • Jobs:
    • b-m - "media-tests"
      • short tests using standard video API on YouTube; equivalent to 'm' below
    • b-y - "media-youtube-tests"
      • short tests using YouTube JS API on YouTube; equivalent to 'y' below

pf-jenkins

All available test jobs also run in our Jenkins instance, pf-jenkins.

  • Repos
    • mozilla-central
    • mozilla-aurora
  • Platforms:
    • Mac OS X 10.10 opt
    • Windows 7 opt
    • Windows 8 opt
  • Group Name: VideoPuppeteer
  • Group Symbol: VP
  • Jobs:
    • m - short tests using standard video API on YouTube
    • y - short tests using YouTube JS API
    • ym1-60 - hour-long tests using YouTube JS API
    • y-long - very long tests using YouTube JS API
    • e - short tests using standard video API to test EME
    • e-f - full tests using standard video API to test EME
    • *-e10s - job that is run with e10s enabled e.g. m-e10s, y-e10s
    • *-bw - tests that runs for various bandwidths

Debugging

You can RDC/VNC to individual machines in pf-jenkins. (User: mozilla). When there are problems, it often helps to trigger the job from pf-jenkins, and watch it run in your VNC window. You can also launch a mozilla-build shell and run scripts yourself. You should avoid running from the Jenkins workspace directories; you should be able to get the necessary assets to reproduce the tests on a directory on the local drives.

Command-lines for jobs can be found in the pf-jenkins GUI by selecting the project and either reading through the logs, or going into the configure pane, copying, and then pasting into your shell window.

Custom Profile

Netflix and network bandwidth limiting tests use a stored profile with custom logins and certificates installed.

  • Windows: c:\testing\eme_profile
  • Mac: /Users/mozilla/testing/eme_profile

How to disable an individual failing test on pf-jenkins.

Visit pf-jenkins (VPN required), open the summary page of the job you want to disable and click 'Disable Project'. When a job is reported to Treeherder, its Jenkins build url and build id are listed in the Job Details pane.

Command-lines for various tests

Notes

  • For all Netflix videos for anything thing built locally, you must copy plugin-container.exe and voucher.bin from a production Firefox installation into your <objdir>/dist/bin directory. You must also have a profile that has been logged in with a valid Netflix account.
  • Please see External Media Tests for information on setting up browsermob-proxy for network bandwidth limiting tests.
  • You must fill in values for $PROFILE and $BROWSERMOB-PROXY-SCRIPT

Default tests

./mach external-media-tests

Default tests with e10s

./mach external-media-tests --e10s

Basic YouTube API

./mach external-media-tests dom/media/test/external/external_media_tests/playback/youtube/manifest.ini

Basic YouTube API with e10s

./mach external-media-tests dom/media/test/external/external_media_tests/playback/youtube/manifest.ini --e10s

Basic YouTube API with medium length videos

./mach external-media-tests dom/media/test/external/external_media_tests/playback/youtube/manifest.ini --urls dom/media/test/external/external_media_tests/urls/youtube/medium1-60.ini

Basic YouTube API with long videos

./mach external-media-tests dom/media/test/external/external_media_tests/playback/youtube/manifest.ini --urls dom/media/test/external/external_media_tests/urls/youtube/long2-720.ini
  • Job symbol: y-long
  • Videos: Four very long videos, ranging from 58:50 to 6:00:00
  • What it tests: Full playback of long videos. Useful to check for memory leaks. Not run very often, if at all.
  • pf-jenkins:
    • None

Run videos under various network configuratons.

./mach external-media-tests --profile $PROFILE --browsermob-script $BROWSERMOB-PROXY-SCRIPT --browsermob-port 999 dom/media/test/external/external_media_tests/playback/test_playback_limiting_bandwidth.py

Test Netflix playback.

./mach external-media-tests dom/media/test/external/external_media_tests/playback/eme.ini --profile $PROFILE --urls=dom/media/test/external/external_media_tests/urls/netflix/default.ini

Test Netflix with bandwidth limits in place.

./mach external-media-tests dom/media/test/external/external_media_tests/playback/netflix_limiting_bandwidth.ini --browsermob-script=$BROWSERMOB-PROXY-SCRIPT --browsermob-port 999 --urls=dom/media/test/external/external_media_tests/urls/netflix/default.ini  --profile $PROFILE

Contacts

  • :SingingTree (primary contact)
  • :sydpolk

Where to file bugs

Test failures seen on Treeherder should be filed under Core::Audio/Video: Playback

Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Projects/Platform_Quality/External_Media_Tests&oldid=1124116"