WebRTC/Test Plan

Purpose

This document intends to provide a high level overview of the testing that's needed for testing the entire stack of the WebRTC feature set.

Feature Ownership

Feature	QA Lead
getUserMedia	Jason Smith
Peer Connection	Jason Smith
Data Channel	Jason Smith

Other Resources

Resource	Lead
Security	Christoph Diehl
Automation Development	Henrik Skupin

Testing Scope

In Scope

Functionality

• getUserMedia and Local Media Streams DOM functionality
• End-to-end integration with real cameras/mics across operating systems with getUserMedia
• Peer Connection, Session Description, and Ice Candidate DOM API functionality
• Data Channel DOM API and it's integration with global Peer Connection DOM API
• End-to-end local and remote Peer Connection handshaking, ice candidates, data connections, etc.
• End-to-end local and remote Data Channel integration with Peer Connection and sending of data

Software Qualities

• Security - Fuzzing SDP, the DOM APIs, ordering of handshake, etc
• Performance - Time for round-trip calls, video/audio response time upon input, etc
• Scalability - Scaling to large network topologies - do we hold together without functionality/performance issues?
• Availability & Resilience - Downtime during & after handshake, lame network handling, etc
• Location - location of peers - close? Different country?

Out of Scope

• Actual third-party WebRTC apps using the API (outside of Mozilla-specific needs)
• Android and B2G for now, as we're targeting desktop primarily first
• Other unsupported getUserMedia, Local Media Streams, Peer Connections, and Data Channels DOM API functions we are not targeting for v1 release

Testing Lifecycle

Overview

This section provides an overview of the lifecycle of testing the WebRTC-based APIs for the first release of getUserMedia, PeerConnection, and DataChannel.

getUserMedia Device Integration Testing with No UI Integration

Summary

Testing the integration with real devices (camera, microphones) without the UI hooked up. Focuses on making sure integrated hardware works with real-time media streams received from getUserMedia for playback.

Signoff Criteria

Already signed off and finished.

getUserMedia DOM Testing and Automation

Summary

Testing and writing mochitest automation for LocalMediaStream, getUserMedia API, and integration with media element functionality in the DOM.

Signoff Criteria

Already signed off and finished.

getUserMedia Device Integration Testing with UI Integration

Summary

Testing the integration with real devices with the UI hooked up in the Firefox front-end. Focuses on making sure the full end to end pieces are hooked up with both integrated and USB hardware with real-time media streams received from the getUserMedia.

Dependencies

The testing for no UI integration flows needs to be completed.

Signoff Criteria

Already signed off and finished. For reference, see the Firefox 20 getUserMedia signoff checklist for the testing that was done.

Peer Connection - Localhost and Fake Stream Automation

Summary

Heavy focus on testing the Peer Connection DOM API through mochitests using fake streams. Such areas that will be focused on include - the peer connection handshake, addition of ice candidates, media flow, event handlers, and closing of connections.

Dependencies

The testing of no UI integration flows with getUserMedia needs to be completed.

Signoff Criteria

All issues classified as P1s (automation bustage) and P2s (automation highly needed) in this bug query needs to be completed.

Note: We should still work through P3s (automation want) even with this quality area signed off, but that won't block signoff for this quality area.

Data Channel - Localhost DOM API & Peer Connection Integration Automation

Summary

Focuses on testing the Data Channel DOM API and it's integration with general Peer Connections through mochitests. Such areas that will be focused on include establishing a handshake between two data channels local and remote, event handling, attribute state, closing of data channels, etc.

Dependencies

The basic smoke test automation for peer connections should be completed with a reusable automation framework.

Signoff Criteria

All issues classified as P1s (automation bustage) and P2s (automation highly needed) in this bug query needs to be completed.

Note: We should still work through P3s (automation want) even with this quality area signed off, but that won't block signoff for this quality area.

Peer Connection - Localhost and Real Stream End to End Testing

Summary

Focuses on testing of the Peer Connection DOM API through end to end tests on a local machine with real streams from cameras and microphones. The testing style is similar to how the automation will be developed with fake streams, except the big difference is the focus with real streams and testing areas that are non-trivial to automate. Such areas that will be focused on include - video/audio device integration, call rejection explicit vs. not explicit, call longevity, multiple calls, clean vs. non-clean shutdown, connection loss, flaky connection, etc.

Dependencies

getUserMedia UI integration testing should have basic flows passing and most complex flows passing.

Signoff Criteria

• A smoke test run executed with no major blockers found with localhost real stream peer connection scenarios
• A basic functional test run executed with no major blockers found with localhost real stream peer connection scenarios
• An exploratory test run executed with no major blockers found with localhost real stream peer connection scenarios
• Verify blocker bugs that are possible to verify with localhost real stream peer connection scenarios

Open Questions

• What's the purpose of the onopen callback on Peer Connection?
• What's the proper way of testing a simple end to end flow using addIceCandidate on the Peer Connection object?

Resources

• WebRTC W3C Spec
• Example WebRTC App
• WebRTC Test Pages
• WebRTC Tutorial
• WebRTC Architecture
• NAT Traversal
• STUN
• TURN
• SDP
• ICE