Auto-tools/Projects/Platform Quality/External Media Tests

From MozillaWiki
Jump to: navigation, search

External Media Tests is a suite of Marionette Python tests for media playback in Mozilla Firefox. Its aim is to test MSE (Media Source Extensions) and EME (Encrypted Media Extensions) on specific websites such as YouTube and Netflix.

Other documentation

General information about external-media-tests can be found on MDN. API documentation can be found on readthedocs.org.

Source

How are these tests run

There are five ways to run External Media Tests:

  • From <mozilla-central>, type 
./mach external-media-tests

Automation Information

External Media Tests currently runs on buildbot (limited tests), and pf-jenkins. There are plans to migrate some or all of the pf-jenkins to Mozmill_CI and taskcluster.

Results Reported to Treeherder

These are Tier-2 jobs, meaning they are visible by default but not sheriff-managed.

buildbot

There are two short test jobs that run in buildbot

  • Repos: trunk
  • Platforms: Linux x64 (opt only), Windows 7, Windows 8 x64, OS X 10.10
  • Group Name: VideoPuppeteer
  • Group Symbol: VP
  • Jobs:
    • b-m - "media-tests"
      • short tests using standard video API on YouTube; equivalent to 'm' below
    • b-y - "media-youtube-tests"
      • short tests using YouTube JS API on YouTube; equivalent to 'y' below

taskcluster

  • Repos: trunk
  • Platforms: Linux x64 debug
  • Group Name: VideoPuppeteer
  • Group Sympol: VP
  • Jobs:
    • TC-m - "media-tests"
      • short tests using standard video API on YouTube; equivalent to 'm' below

pf-jenkins

All available test jobs also run in our Jenkins instance, pf-jenkins.

  • Repos
    • mozilla-central
    • mozilla-aurora
  • Platforms:
    • Mac OS X 10.10 opt
    • Windows 7 opt
    • Windows 8 opt
  • Group Name: VideoPuppeteer
  • Group Symbol: VP
  • Jobs:
    • m - short tests using standard video API on YouTube
    • y - short tests using YouTube JS API
    • ym1-60 - hour-long tests using YouTube JS API
    • y-long - very long tests using YouTube JS API
    • e - short tests using standard video API to test EME
    • e-f - full tests using standard video API to test EME
    • *-e10s - job that is run with e10s enabled e.g. m-e10s, y-e10s
    • *-bw - tests that runs for various bandwidths

Debugging

You can RDC/VNC to individual machines in pf-jenkins. (User: mozilla). When there are problems, it often helps to trigger the job from pf-jenkins, and watch it run in your VNC window. You can also launch a mozilla-build shell and run scripts yourself. You should avoid running from the Jenkins workspace directories; you should be able to get the necessary assets to reproduce the tests on a directory on the local drives.

Command-lines for jobs can be found in the pf-jenkins GUI by selecting the project and either reading through the logs, or going into the configure pane, copying, and then pasting into your shell window.

Custom Profile

Netflix and network bandwidth limiting tests use a stored profile with custom logins and certificates installed.

  • Windows: c:\testing\eme_profile
  • Mac: /Users/mozilla/testing/eme_profile

How to disable an individual failing test on pf-jenkins.

Visit pf-jenkins (VPN required), open the summary page of the job you want to disable and click 'Disable Project'. When a job is reported to Treeherder, its Jenkins build url and build id are listed in the Job Details pane.

Migrating tests from one release train to another

As Firefox 48 rides the trains, jobs for aurora, beta, release, and ESR will need to be setup in Jenkins.

  • git clone https://github.com/mozilla/mozplatformqa-jenkins-config
  • Create a virtualenv and activate it.
  • pip install python-jenkins
  • Run the following command to save off current job configurations on your hard drive. While there is no restore script right now, it would not be hard to cobble one together from the existing files:

mkdir jobs; python save_jobs.py --regexp "^mn-.*$" --jenkins-host http://pf-jenkins.qa.mtv2.mozilla.com:8080

  • Run the following command to copy all of the nightly jobs to aurora. Existing aurora jobs will be modified.

python copy_jobs_to_new_release.py --project-regexp "^mn-.*nightly.*$" --jenkins-host http://192.168.0.7:8080 --src-release nightly --dest-release aurora <note> This leaves all of the aurora jobs disabled just in case there is a mass of jobs created incorrectly and immediately starts spamming people with failure notifications.</note>

  • Enable a test job. mn-mse-youtube-default-aurora-mac is a good one. Run it and make sure it works. You may have to edit your command lines, or the code in copy_jobs_to_new_release.py does not work.
  • Once your trial job works, enable all of the others: python enable_jobs.py --regexp "^mn-.*aurora.*$"
Caveats
  • This always uses the mozilla-central form of mozharness. This is retrieved by a tool callled archiver_client.py by the Jenkins project mozharness-nightly. I could not get a reliable command-line to retrieve the current version of mozharness. This may cause breakage if mozharness makes incompatible command-line changes, such as removing <code>--e10s and adding --disable-e10s. Somebody needs to figure out how to retrieve the aurora version of mozharness using archiver_client.py, make a Jenkins project called mozharness-aurora, and update all of the aurora projects to use that version intead.
Other utilties

mozplatformqa-jenkins-config has several interesting python scripts to manipulate Jenkins projects.

  • copy_jobs_to_new_release.py - see above.
  • copy_to_second_jenkins.py - Take jobs from one jenkins instance, swap out all references to that instance with the second instance, and put those jobs into the second instance.
  • disable_jobs.py - Mass disable Jenkins jobs.
  • enable_jobs.py - Mass enable Jenkins jobs.
  • fix_projects.py - Do regular expression subsitutions inside of Jenkins jobs.
  • save_jobs.py - Save job configurations to local disk.

Command-lines for various tests

Notes

  • For all Netflix videos for anything thing built locally, you must copy plugin-container.exe and voucher.bin from a production Firefox installation into your <objdir>/dist/bin directory. You must also have a profile that has been logged in with a valid Netflix account.
  • Please see External Media Tests for information on setting up browsermob-proxy for network bandwidth limiting tests.
  • You must fill in values for $PROFILE and $BROWSERMOB-PROXY-SCRIPT

Default tests

./mach external-media-tests

Default tests with e10s

./mach external-media-tests --e10s

Basic YouTube API

./mach external-media-tests dom/media/test/external/external_media_tests/playback/youtube/manifest.ini

Basic YouTube API with e10s

./mach external-media-tests dom/media/test/external/external_media_tests/playback/youtube/manifest.ini --e10s

Basic YouTube API with medium length videos

./mach external-media-tests dom/media/test/external/external_media_tests/playback/youtube/manifest.ini --urls dom/media/test/external/external_media_tests/urls/youtube/medium1-60.ini

Basic YouTube API with long videos

./mach external-media-tests dom/media/test/external/external_media_tests/playback/youtube/manifest.ini --urls dom/media/test/external/external_media_tests/urls/youtube/long2-720.ini
  • Job symbol: y-long
  • Videos: Four very long videos, ranging from 58:50 to 6:00:00
  • What it tests: Full playback of long videos. Useful to check for memory leaks. Not run very often, if at all.
  • pf-jenkins:
    • None

Run videos under various network configuratons.

./mach external-media-tests --profile $PROFILE --browsermob-script $BROWSERMOB-PROXY-SCRIPT --browsermob-port 999 dom/media/test/external/external_media_tests/playback/test_playback_limiting_bandwidth.py

Test Netflix playback.

./mach external-media-tests dom/media/test/external/external_media_tests/playback/eme.ini --profile $PROFILE --urls=dom/media/test/external/external_media_tests/urls/netflix/default.ini

Test Netflix with bandwidth limits in place.

./mach external-media-tests dom/media/test/external/external_media_tests/playback/netflix_limiting_bandwidth.ini --browsermob-script=$BROWSERMOB-PROXY-SCRIPT --browsermob-port 999 --urls=dom/media/test/external/external_media_tests/urls/netflix/default.ini  --profile $PROFILE

Contacts

  • :SingingTree (primary contact)
  • :sydpolk

Where to file bugs

Test failures seen on Treeherder should be filed under Core::Audio/Video: Playback

Harness/infrastructure bustage should be filed under Testing: external-media-tests

Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Projects/Platform_Quality/External_Media_Tests&oldid=1132499"