Platform/Features/Camera API - Phase 2 (getUserMedia)/Test Plan

Strategy

Manual/Automated API Testing
- For anything testing the specification of the API itself, aim to do testing via automation as much as possible to what's feasible
- Additionally including anything that makes use of the MediaStream itself - audio tags, video tags for example
- Will start manually during the definition phase for different API and DOM test cases, but move to automation after clear definition
Direct Camera and Audio Involvement - Test Cases
- For testing the underlying hardware situations with real web cams
- Requires explicit step by step procedures in sample html pages
- Needs to be ran on a variety of hardware types
Hardware Crowd-Source Testing
- Crowd-source testing for basic sanity tests for various hardware configurations
- Use test days, blog posts, dev docs, etc to get users to try out basic pieces of the feature with their web cam and audio recording device and encourage them to file bugs

Test Cases

Direct Camera & Audio Involvement - Test Cases

Smoke-Level

[In MozTrap] Launch a basic view of a video stream from the camera with a video tag and accept permissions to use the camera, verify that you can see yourself in the camera as it runs for a few seconds, ensure no errors or weird behavior shows up in the error console or web console
[In MozTrap] Launch a basic view of a audio stream from the user's audio recording device with the audio tag and accept permissions to use the mic, verify that as you generate sound, you hear it back, ensure no errors or weird behavior shows up in the error console or web console
[In MozTrap] Launch a view of a video with video and audio being captured together in a video tag and accept permissions to use the camera. Verify that you can see a video stream of yourself, and ensure no errors or weird behavior shows up in the error console or web console. Then, speak a few words into the mic. Verify that the audio is sent back and heard, and ensure no errors or weird behavior shows up in the error console or web console.

BFT-Level - TC Possible

[In MozTrap] Launch a basic view of a video stream from the camera and don't accept the permission prompt that appears in firefox. Verify that the video stream from your camera does not start.
[In MozTrap] Launch a basic view of an audio stream from the camera and don't accept the permission prompt that appears in firefox. Verify that the video stream from your camera does not start.
[In MozTrap] Launch a basic view of an audio and video stream from the camera and don't accept the permission prompt that appears in firefox. Verify that the video stream from your camera does not start.
[Revised In MozTrap] Launch two basic views of the video stream for the camera in separate tabs and accept permissions to use the camera in the 1st tab. Verify that the video stream plays fine in the first tab, but the second tab does not play the video stream and reports an error that the hardware is not available. No errors or weird behavior in the web console or error console. No permission prompt should appear either on the 2nd tab.
[Revised in MozTrap] Launch two basic views of the audio stream for the audio recording device in separate tabs and accept permissions to use the camera in the 1st tab. Verify that the audio can be successfully recorded in the first tab, but fails on the second tab with the hardware being unavailable. No weird behavior or unexpected errors in the web console or error console. No permission prompt should appear either on the 2nd tab.
[Revised in MozTrap] Launch a request for the camera on a machine with no camera support. Verify that an error is thrown indicating that there is no camera on this machine. No permission prompt should appear.
[Revised in MozTrap] Launch a request for the audio on a machine with no audio recording support. Verify that an error is thrown indicating that there is no audio on this machine. No permission prompt should appear.
[Revised in MozTrap] Launch a request for camera and audio with no camera support. Verify that no permission prompt appears and error is fired indicating that one of the devices requested was not found.
[Revised in MozTrap] Launch a request for camera and audio with no mic support. Verify that no permission prompt appears and error is fired indicating that one of the devices requested was not found.
[Revised in MozTrap] Launch a request for camera and audio with no camera and mic support. Verify that no permission prompt appears and error is fired indicating that both devices requested was not found.
[Revised in MozTrap] Launch a request for the camera video stream and accept permissions. Quit Firefox. Restart it. Launch a request the camera video stream again and accept permissions. Verify the video stream works in a video tag, no weird errors or behavior seen.
[Revised in MozTrap] Launch a request for the audio video stream and accept permissions. Quit Firefox. Restart it. Launch a request the camera video stream again and accept permissions. Verify the audio stream works in a video tag, no weird errors or behavior seen.
[Revised in MozTrap] Test that if I allow permissions to the camera for one website, if I request it for another tab, I should be prompted again and not be given by default
[Revised in MozTrap] Test that if I allow permissions to the mic for one website, if I request it for another tab, I should be prompted again and not be given by default
[Revised in MozTrap] Test that if I allow permissions to the camera and mic for one website, if I request it for another tab, I should be prompted again and not be given by default
[In MozTrap] Launch a request for a camera with both built-in and a USB camera hooked into the machine at once. Allow permissions for the built-in camera. Verify that the video stream plays okay with no errors from the built-in camera. No activity should be seen from the USB camera.
[In MozTrap] Launch a request for a camera with both built-in and a USB camera hooked into the machine at once. Allow permissions for the USB camera. Verify that the video stream plays okay with no errors from the USB camera. No activity should be seen from the built-in camera.
[In MozTrap] Launch a request for a audio with both built-in and a USB mic hooked into the machine at once. Allow permissions for the built-in mic. Verify that the audio stream plays okay with no errors from the built-in mic. No activity should be seen from the USB mic.
[In MozTrap] Launch a request for a audio with both built-in and a USB mic hooked into the machine at once. Allow permissions for the USB mic. Verify that the audio stream plays okay with no errors from the USB mic. No activity should be seen from the built-in mic.
[In MozTrap] Launch a request for the camera and audio with a USB camera, built-in camera, and built-in mic. Allow permissions for the built-in camera and built-in mic. Verify that I can see myself via the built-in camera and that if I generate sounds to the built-in mic, that I hear them back. The USB camera also should not be active.
[In MozTrap] Launch a request for the camera and audio with a built-in camera, USB mic, and built-in mic. Allow permissions from the built-in camera and USB mic. Verify that I can see myself via the built-in camera and make and hear sounds from the USB mic. Generating sound via the built-in mic should do nothing.
[In MozTrap] Launch a request for the camera and audio with a USB camera, built-in camera, USB mic, and a built-in mic. Allow permissions for the USB camera and USB mic. Verify that I can see myself from the USB camera and that if I generate sounds from the USB mic, that I hear them back. The built-in camera and mic should not be active.
[In MozTrap] Launch a request for the camera and audio with a USB camera, built-in camera, USB mic, and a built-in mic. Allow permissions for the built-in camera and USB mic. Verify that I can see myself from the built-in camera and hear and generate sounds from the USB mic. The built-in mic and USB camera should show no activity.
[In MozTrap] Launch a request for the camera and audio with a USB camera, built-in camera, USB mic, and a built-in mic. Allow permissions for the built-in camera, but don't give mic access. Verify that an error is reported saying that one of the devices wasn't provided.
[In MozTrap] Launch a request for the camera and audio with a USB camera, built-in camera, USB mic, and a built-in mic. Allow permissions for the built-in mic, but don't give camera access. Verify that an error is reported saying that one of the devices wasn't provided.

BFT-Level TC Needed

[TC Wanted] Test that if I request the mic a second time after the first request in the same tab, then I should automatically get access to the mic without another prompt for permissions
[TC Wanted] Test that if I request the camera a second time after the first request in the same tab, then I should automatically get access to the camera without another prompt for permissions

Exploratory Ideas

Unplugging the device while in use
External program usage vs. Firefox usage
Broken hardware
Computer shutdown while in Firefox usage
IFrame usage
file:// protocol usage
Crash while hardware is in use
Sleep mode
Attaching dead local media streams that can't be played onto media elements
Concurrent use of streams with calling of stop() on one - make sure it only releases the device at one point
Integrated and USB devices used at the same time

Picture Test Case Archive

Launch a basic view of a picture generated from the camera with the image tag, verify that you can see a picture captured of you, ensure no errors or weird behavior shows up in the error console or web console
Launch two requests simulatenously for the picture. Verify that the picture is taken in one tab successfully, but fails on the second tab with the hardware being unavailble. No weird behavior or unexpected errors in the web console or error console.
Launch a request for a picture on a machine with no camera support. Verify that an error is thrown indicating that there is no camera support on this machine.
Launch a request for the video stream, accept permissions, and view on a video tag. Put the machine into sleep mode for 5 seconds. Resume. Verify the camera stream did not play when you were in sleep mode (i.e. camera not active), but on resume, it did play. No errors seen that seem unexpected.
Launch a request for a picture. Quit Firefox. Restart it. Launch a request for a picture again. Verify that a picture could still be taken with no weird errors or behavior seen.
Launch a request for a picture and immediately put the machine into sleep mode for 5 seconds. Resume. Verify that a picture was not taken during sleep mode on the machine.
Launch a request for a picture with both built-in and a USB camera hooked into the machine at once. Verify the picture is taken successfully.
Launch a request for a picture with both built-in and a USB camera hooked into the machine at once. Verify the picture is taken successfully.

API and DOM Test Cases

[NEW] Call mozGetUserMedia with {picture: true, audio: true} --> Ensure an error callback is fired indicating that invalid parameters were provided
[NEW] Call mozGetUserMedia with {picture: true, video: true} --> Ensure an error callback is fired indicating that invalid parameters were provided
[NEW] Call mozGetUserMedia with {video: true, audio: true, picture: true} --> Ensure an error callback is fired indicating that invalid parameters were provided
[NEW] Call mozGetUserMedia with {Video: true} (incorrect syntax) --> Ensure an error callback is fired indicating that invalid parameters were provided
[NEW] Call mozGetUserMedia with 1 (incorrect object) --> Ensure an error callback is fired indicating that invalid parameters were provided
[NEW] Call mozGetUserMedia with {} (no information) --> Valid callback, no media content enabled
[NEW] Call mozGetUserMedia with {video: false, picture: false, audio: true} --> Ensure audio recording stream is returned
[NEW] Call mozGetUserMedia with {video: true, picture: false, audio: false} --> Ensure video recording stream is returned
[NEW] Call mozGetUserMedia with null --> Ensure the function fails gracefully with no crashes
[NEW] Call mozGetUserMedia with a null callback function in success parameter, all other parameters are valid --> Ensure we fail gracefully with a checked exception thrown
[NEW] Call mozGetUserMedia with a null callback function in error parameter, all other parameters are valid --> Ensure we fail gracefully with a checked exception thrown
[NEW] Call mozGetUserMedia with incorrect types in each parameter --> Ensure that a TypeError is thrown for each
[NEW] On success callback of a MediaStream mozGetUserMedia call, the currentTime should be initialized to zero.
[NEW] After starting a MediaStream for three seconds and stopping it through the stop() call, the currentTime should roughly be equilvalent to three seconds
[NEW] After starting a MediaStream, checking currentTime every second three times should show currentTime changing by roughly a second
[NEW] After starting a video only MediaStream and stopping it, the camera should go from being actively used to being inactive
[NEW] After starting an audio only MediaStream and stopping it, the audio recording device should no longer capture audio
[NEW] Upon calling stop on an already stopped MediaStream, an error should be fired back to indicate that the MediaStream is already stopped
[NEW] Upon restarting a stream that was stopped previously, the MediaStream should start back up with no errors
[NEW] When onended is set, it should be called when the video stream stops running
[NEW] In a running MediaStream, pause it. Wait a few seconds. Then, restart the MediaStream. Verify that the stream has starts immediately with no buffering of content that happened while the video was paused.
[NEW] In a MediaStream, check that the duration of the stream is infinity
[NEW] In a MediaStream, try seeking to a different location. Verify an error is thrown and that the content of the video does not change.
[NEW] In a MediaStream, set the video to loop. Verify nothing changes to MediaStream content.
[NEW] Test a video tag with a MediaStream and a different video source (ogg file) with the camera attached - verify the webcam stream is shown
[NEW] Test a video tag with a MediaStream and a different video source (ogg file) without the camera attached - verify the webcam stream is shown
[NEW] Test an audio tag with a MediaStream and a different audio source (ogg file) with the camera attached - verify the webcam stream is shown
[NEW] Test an audio tag with a MediaStream and a different audio source (ogg file) without the camera attached - verify the webcam stream is shown
[NEW] In a video tag with a MediaStream with a running video, check the following values match up below:
- src = blob URL
- currentSrc = blob URL
- preload = 'None'
- buffered = length = 1, start and end are 0, empty time range
- currentTime = 0 at start, should change overtime
- duration = infinity
- seeking = false
- defaultPlaybackRate = 1
- playbackRate = 1
- played = length = 1, start(0) = 0, end(0) = currentTime
- seekable = length = 0, start = currentTime, end = currentTime, not seekable
- startOffestTime = NaN
- loop = false
[NEW] Hook up an onloadedmetadata callback to a video tag. Call gUM and hook it up to that video tag. Verify that event is fired with a valid event object.
[NEW] Call drawImage with a video generated from a MediaStream onto a canvas. Verify that the canvas renders the image generated from the MediaStream.

Edge Cases - Hardware Testing

We need to do crowd-source testing of basic sanity pieces of the feature across various hardware configurations. Hardware configurations were looking to get coverage on include:

Different USB versions
Different resolutions
Different frame rates
Different encodings (YUV, RGB, h264)

In my testing generally, I'll cover a typical case for USB vs. built-in cameras and audio recording devices.

Note - It was noted that hardware-specific issues will likely be 5% of the bugs, so this is lowest priority area to focus testing in.

Edge Cases - Direct Camera and Audio Involvement

Camera and/or Audio Recording Device Unplugged While Stream is Running
Multiple Cameras from different software or browser
No camera or audio recording hooked up
Camera Type - USB vs. Built-In
USB Type - USB1 vs. USB2 vs. USB3
Differences in max resolution and framerate
Raw YUV vs RGB (if any) vs encoded (in H.264)
Broken Camera and/or Audio Recording
Built in and USB hooked up at the same time
Multiple Audio Recording Requests from different software or browser
Multiple constraint types at the same time (e.g. video and audio)
Different styles of operating systems (OS X, Windows, Linux)
Browser Crashes while stream is in use
Shutdown of Firefox while video and/or audio stream is running
Enda, power management edge cases, what happens after system sleep\resume

DOM and API Analysis for Test Cases

mozGetUserMedia parameter dictionary - ideas to test
- Valid singular values - video, audio, picture true alone
- Valid combination - audio/video
- Invalid combinations - picture & audio, picture & video, picture & video & audio
- Incorrect value - Video capitalized
- Incorrect object type
- No parameters specified {}
- Ignore parameters that are false - video with other false params, audio with other false params
Callback functions on mozGetUserMedia - null, valid function, invalid object type
Error callback value - should be a string
Success callback value - MediaStream object testing
- currentTime at start of streaming
- currentTime after streaming has been running
- currentTime after the stream has stopped
- calling stop() on the stream already started
- calling stop() on a non-started stream
- Restarting the stream after it was stopped through stop()
- onended set, called when the stream stops running
- onended not set, stop stream
- Ensure that the stream cannot be buffered
- Ensure that the duration of the stream continues on forever
- Ensure that you cannot seek a DOM object utilizing a MediaStream
- Ensure that you cannot loop a DOM object utlizing a MediaStream
Different DOM use cases using a MediaStream (video, img, canvas, audio)
- Video 1 vs. multiple sources
- Video methods - changes of state with play, pause, load, and canPlayType
- Analyzing video properties - currentSrc, currentTime, videoWidth, videoHeight, duration, ended, error, paused, muted, seeking, volume, height, width
- Video events - play, pause, progress, error, timeupdate, ended, abort, empty, emptied, waiting, loadedmetadata
- Video timing of loading with preload or other attributes
- Audio 1 vs. multiple sources
- General HTML 5 Attributes and Events - ondurationchange, onended, onpause, onplay, onplaying, onratechange, onseeked, onseeking, onvolumechange, ontimeupdate, onloadeddata, onloadedmetadata, onloadstart, onprogress, onreadystatechange, onstalled, onsuspend, ontimeupdate, onwaiting
- img src vs. alt
- img width and height variations
Testing of the below attribute values for a Media Element
- src - local URI referencing media stream
- currentSrc - local URI referencing media stream
- preload - 'None' <-- cannot be preloaded
- buffered <-- length = 1, start and end are 0, empty time range for buffered
- currentTime <-- starts at 0, increments as time increments (value in current stream), ignore any attempts to modify
- duration <-- no pre-defined duration, so it should be infinity, on destruction of media stream is equals current time
- seeking <-- always false, as MediaStream is not seekable
- defaultPlaybackRate <-- always 1, cannot modify, as it is not seekable
- playbackRate <-- always 1, cannot modify, as it is not seekable
- played <-- length = 1, start(0) = 0, end(0) = currentTime <-- timeline
- seekable <-- length = 0, start = currentTime, end = currentTime, not seekable
- startOffSetTime <--- NaN, no timeline offset set
- loop <-- false, cannot be looped
Draw image on a canvas object from a video using a MediaStream
Analyze the different states of readyState on a video object

Out of Scope Edge Cases Seen in Spec

Testing of the spec MediaStream object
- label <-- uniqueness must always be maintained
- ended <-- true if stream is ended, false otherwise
- audioTracks <--- read only, MediaStreamTracks that can be enabled or disabled, must be same object on every call
- videoTracks <--- read only, MediaStreamTracks that can be enabled or disabled, must be same object on every call
- EventTarget <--- which DOM element triggered it
Testing of the spec MediaStreamTrack object
- kind - readonly type of MediaStreamTrack
- enabled - whether it is enabled or not
- label <-- object representing label of the underlying resource
- onended <--- Function callback for when the track ends
- onmute <-- Function callback on mute
- onunmute <-- Function callback on resuming after mute
- readyState <-- State of track (LIVE, MUTED, ENDED)
Testing of createObjectURL function for MediaStreams
Testing of the spec MediaStreamTrackList object
- length <-- length of tracks in the list
- onaddtrack - callback for when a track is added
- onremovetrack - callback for when a track is removed
- add <-- adds the track (error if it isn't finished)
- item <-- index against the track list
- remove <-- remove the track (error if it isn't finished)

Signoff Criteria

Note: As of right now, the UI has not been implemented, so merge signoff criteria shall be defined when those pieces land.

Infrastructure

Server to host test cases - mozqa.com?
Hardware for testing different hardware configurations for the camera

Operating Systems

Windows XP (P1)
Windows Vista (P2)
Windows 7 (P1)
OS X 10.6 (P1)
OS X 10.8 (P1)
Ubuntu 12 (P3)

Test Case Management

Automated API Tests - Managed in mozilla central
Standardized Manual Test Cases - Managed on a single server - maybe mozqa.com?
Hardware Manual Test Cases - Probably will reference the server test cases

Automation

Overview

Currently automation targets using fake media streams with the gum API. In the future, we'll move to using simulated hardware to exercise more of the full stack.

Currently Automated

Basic gum video test to run video from gum in a media element and ensuring that the right events fire and no surprising issues happen during cleanup
Basic gum audio test to run audio from gum in a media element and ensuring that the right events fire and no surprising issues happen during cleanup
Basic gum video & audio test to run video & audio from gum in a media element and ensuring that the right events fire and no surprising issues happen during cleanup
An exceptions test against the gum api to check that invalid inputs fire the right exceptions and do not cause a crash
A crashtest that checks that we do not crash with null values for the function callbacks

Automation Needed

Not Blocked - Bugs

Bug List

Not Blocked - Defined

A basic gum test that verifies that if two separate tabs request access to the camera, then one should get access, the other should get an error
A basic gum test that verifies that if two separate tabs request access to the audio, then one should get access, the other should get an error
A basic gum test that verifies that if two separate tabs request access to the camera+audio, then one should get access, the other should get an error
A basic gum test that verifies that if you running video through a media element, you quit and restart firefox, restart the video stream, then the video stream should start
A basic gum test that verifies that if you running audio through a media element, you quit and restart firefox, restart the video stream, then the video stream should start
A basic gum test that verifies that if you running video+audio through a media element, you quit and restart firefox, restart the video stream, then the video stream should start
A basic gum test that if a call to stop() on the stream while the video is running in a media element, then onended should be fired indicating the media element has stopped
A basic gum test that if a call to stop() on the stream while the audio is running in a media element, then onended should be fired indicating the media element has stopped
A basic gum test that if a call to stop() on the stream while the video+audio is running in a media element, then onended should be fired indicating the media element has stopped
A basic gum test that if I call top stop() on the stream while the video is running in a media element, then I grab the media stream off of the element and attach it onto another media element and play the element, then the element should fail to play
A basic gum test that if I call top stop() on the stream while the audio is running in a media element, then I grab the media stream off of the element and attach it onto another media element and play the element, then the element should fail to play
A basic gum test that if I call top stop() on the stream while the video+audio is running in a media element, then I grab the media stream off of the element and attach it onto another media element and play the element, then the element should fail to play
A basic gum test that calls stop() on a video media stream from gum while being played in a media element, then we call gum again for video, attach it to the media element and play the element, then the media should play
A basic gum test that calls stop() on an audio media stream from gum while being played in a media element, then we call gum again for audio, attach it to the media element and play the element, then the media should play
A basic gum test that calls stop() on a video+audio media stream from gum while being played in a media element, then we call gum again for video+audio, attach it to the media element and play the element, then the media should play
A basic gum test that calls stop() on a media stream from gum video playing in one tab, then we open another tab and try to call gum with video and play it through a media element - should successfully play
A basic gum test that calls stop() on a media stream from gum audio playing in one tab, then we open another tab and try to call gum with audio and play it through a media element - should successfully play
A basic gum test that calls stop() on a media stream from gum video+audio playing in one tab, then we open another tab and try to call gum with video+audio and play it through a media element - should successfully play

Automation Blocked

Permission Prompt Automation

We need support for testing the permission prompt for getUserMedia. However, the permission prompt is currently disabled for fake streams.

Tracking Bug: TBD

Tests to Automate:

Create a basic gum test that accepts the permission prompt for a video only request and plays the stream successfully in a media element
Create a basic gum test that accepts the permission prompt for a audio only request and plays the stream successfully in a media element
Create a basic gum test that accepts the permission prompt for a video+audio only request and plays the stream successfully in a media element
Create a basic gum test that denies the permission prompt for a video only request should report permission denied
Create a basic gum test that denies the permission prompt for a audio only request should report permission denied
Create a basic gum test that denies the permission prompt for a video+audio only request should report permission denied

Device Simulation Automation

We need to directly test with simulated devices, not fake streams, in order to run tests that deal with device-specific issues and interaction.