View source for WebRTC/Test Plan

== Purpose ==
This document intends to provide a high level overview of the testing that's needed for testing the entire stack of the WebRTC feature set.

== Feature Ownership ==
{| class="fullwidth-table"
|-
| '''Feature'''
| '''QA Lead''' 
|-
| [https://wiki.mozilla.org/Platform/Features/Camera_API_-_Phase_2_%28getUserMedia%29/Test_Plan getUserMedia]
| Jason Smith
|-
| [https://wiki.mozilla.org/WebRTC/Test_Plan/Peer_Connection Peer Connection]
| Jason Smith
|-
| [https://wiki.mozilla.org/WebRTC/Test_Plan/Data_Channel Data Channel]
| Jason Smith
|}

== Other Resources ==

{| class="fullwidth-table"
|-
| '''Resource'''
| '''Lead''' 
|-
| Security
| Christoph Diehl
|-
| [https://wiki.mozilla.org/Auto-tools/Automation_Development/Projects/WebRTC Automation Development]
| Henrik Skupin
|}

== Testing Scope ==

=== In Scope ===

==== Functionality ====

* getUserMedia and Local Media Streams DOM functionality
* End-to-end integration with real cameras/mics across operating systems with getUserMedia
* Peer Connection, Session Description, and Ice Candidate DOM API functionality
* Data Channel DOM API and it's integration with global Peer Connection DOM API
* End-to-end local and remote Peer Connection handshaking, ice candidates, data connections, etc.
* End-to-end local and remote Data Channel integration with Peer Connection and sending of data

==== Software Qualities ====

* Security - Fuzzing SDP, the DOM APIs, ordering of handshake, etc
* Performance - Time for round-trip calls, video/audio response time upon input, etc
* Scalability - Scaling to large network topologies - do we hold together without functionality/performance issues?
* Availability & Resilience - Downtime during & after handshake, lame network handling, etc
* Location - location of peers - close? Different country?

=== Out of Scope ===

* Actual third-party WebRTC apps using the API (outside of Mozilla-specific needs)
* Android and B2G for now, as we're targeting desktop primarily first
* Other unsupported getUserMedia, Local Media Streams, Peer Connections, and Data Channels DOM API functions we are not targeting for v1 release

== Testing Lifecycle ==

=== Overview ===

This section provides an overview of the lifecycle of testing the WebRTC-based APIs for the first release of getUserMedia, PeerConnection, and DataChannel.

=== getUserMedia Device Integration Testing with No UI Integration ===

==== Summary ====

Testing the integration with real devices (camera, microphones) without the UI hooked up. Focuses on making sure integrated hardware works with real-time media streams received from getUserMedia for playback.

==== Signoff Criteria ====

Already signed off and finished.

=== getUserMedia DOM Testing and Automation ===

==== Summary ====

Testing and writing mochitest automation for LocalMediaStream, getUserMedia API, and integration with media element functionality in the DOM.

==== Signoff Criteria ====

Already signed off and finished.

=== getUserMedia Device Integration Testing with UI Integration ===

==== Summary ====

Testing the integration with real devices with the UI hooked up in the Firefox front-end. Focuses on making sure the full end to end pieces are hooked up with both integrated and USB hardware with real-time media streams received from the getUserMedia.

==== Dependencies ====

The testing for no UI integration flows needs to be completed.

==== Signoff Criteria ====

Already signed off and finished. For reference, see the Firefox 20 getUserMedia [https://wiki.mozilla.org/Platform/Features/Camera_API_-_Phase_2_%28getUserMedia%29/Test_Plan/Signoff#Checklist signoff] checklist for the testing that was done.

=== Peer Connection - Localhost and Fake Stream Automation ===

==== Summary ====

Heavy focus on testing the Peer Connection DOM API through mochitests using fake streams. Such areas that will be focused on include - the peer connection handshake, addition of ice candidates, media flow, event handlers, and closing of connections.

==== Dependencies ====

The testing of no UI integration flows with getUserMedia needs to be completed.

==== Signoff Criteria ====

All issues classified as P1s (automation bustage) and P2s (automation highly needed) in this [https://bugzilla.mozilla.org/buglist.cgi?list_id=6005729&o1=equals&v1=795383&f1=blocked&order=Importance&resolution=---&query_format=advanced bug query] needs to be completed.

'''Note: ''' We should still work through P3s (automation want) even with this quality area signed off, but that won't block signoff for this quality area.

=== Data Channel - Localhost DOM API & Peer Connection Integration Automation ===

==== Summary ====

Focuses on testing the Data Channel DOM API and it's integration with general Peer Connections through mochitests. Such areas that will be focused on include establishing a handshake between two data channels local and remote, event handling, attribute state, closing of data channels, etc.

==== Dependencies ====

The basic smoke test automation for peer connections should be completed with a reusable automation framework.

==== Signoff Criteria ====

All issues classified as P1s (automation bustage) and P2s (automation highly needed) in this [https://bugzilla.mozilla.org/buglist.cgi?list_id=6005782&o1=equals&v1=767072&f1=blocked&order=Importance&resolution=---&query_format=advanced bug query] needs to be completed.

'''Note: ''' We should still work through P3s (automation want) even with this quality area signed off, but that won't block signoff for this quality area.

=== Peer Connection - Localhost and Real Stream End to End Testing ===

==== Summary ====

Focuses on testing of the Peer Connection DOM API through end to end tests on a local machine with real streams from cameras and microphones. The testing style is similar to how the automation will be developed with fake streams, except the big difference is the focus with real streams and testing areas that are non-trivial to automate. Such areas that will be focused on include - video/audio device integration, call rejection explicit vs. not explicit, call longevity, multiple calls, clean vs. non-clean shutdown, connection loss, flaky connection, etc.

==== Dependencies ====

getUserMedia UI integration testing should have basic flows passing and most complex flows passing.

==== Signoff Criteria ====

* A smoke test run executed with no major blockers found with localhost real stream peer connection scenarios
* A basic functional test run executed with no major blockers found with localhost real stream peer connection scenarios
* An exploratory test run executed with no major blockers found with localhost real stream peer connection scenarios
* Verify blocker bugs that are possible to verify with localhost real stream peer connection scenarios

=== Data Channel - Localhost and Peer Connection Integration Testing with Real Streams ===

==== Summary ====

Focuses on testing the Data Channel DOM API with end to end tests and integration with other Peer Connections. Such areas that will be focused on include reliability of messages sent, type of information sent, rate of information sent, and integration scenarios involving many of the areas talked about in the localhost Peer Connection end to end testing.

==== Dependencies ====

Basic end to end test scenarios for Peer Connections should be defined and passing for localhost cases.

==== Signoff Criteria ====

* A smoke test run executed with no major blockers found with localhost data channel scenarios
* A basic functional test run executed with no major blockers found with localhost data channel scenarios
* An exploratory test run executed with no major blockers found with localhost data channel scenarios
* Verify blocker bugs that are possible to verify with localhost data channel scenarios

=== Peer Connection - Remote Real Stream End to End Testing ===

==== Summary ====

Focuses on remote scenario testing with Peer Connections with real streams and microphones that both use Firefox. Such areas to focus on include single vs. multiple remote connections, connection loss, lame networks, peer locations, stream performance, type of server being used, explicit vs. not explicit call end, longevity of call.

==== Dependencies ====

Localhost peer connection real stream testing has basic flows passing and some complex flows passing.

==== Signoff Criteria ====

* A smoke test run executed with no major blockers found with remote peer connection scenarios
* A basic functional test run executed with no major blockers found with remote peer connection scenarios
* An exploratory test run executed with no major blockers found with remote peer connection scenarios
* Verify blocker bugs that are possible to verify with remote peer connection scenarios

=== Data Channel - Remote Peer Connection Integration Testing with Real Streams ===

==== Summary ====

Focuses on remote scenario testing with Data Channels with Peer Connection integration that use Firefox purely. Such areas to focus on include remote reliability, size of information sent, storage remaining on each system, type of data sent, rate of messages sent, single vs. multiple connections, and integration with remote Peer Connection scenarios.

==== Dependencies ====

Basic end to end scenarios for Data Channels should be defined and passing for localhost scenarios.

==== Signoff Criteria ====

* A smoke test run executed with no major blockers found with remote Data Channel scenarios
* A basic functional test run executed with no major blockers found with remote Data Channel scenarios
* An exploratory test run executed with no major blockers found with remote Data Channel scenarios
* Verify blocker bugs that are possible to verify with remote Data Channel scenarios

=== Peer Connection - Interoperability Testing with Chrome ===

==== Summary ====

Focuses on interoperability testing between Chrome and Firefox locally and remotely with Peer Connections. Areas to focus on are basically a summation of the focus areas of the localhost and remote peer connection testing except using differing browsers in the call.

==== Dependencies ====

Localhost and remote peer connection testing has basic flows passing and some complex flows passing.

==== Signoff Criteria ====

* A smoke test run executed with no major blockers found with interoperability peer connection scenarios
* A basic functional test run executed with no major blockers found with interoperability peer connection scenarios
* An exploratory test run executed with no major blockers found with interoperability peer connection scenarios
* Verify blocker bugs that are possible to verify with interoperability peer connection scenarios

=== Data Channel - Interoperability Testing with Chrome ===

==== Summary ====

Focuses on interoperability testing between Chrome and Firefox locally with Data Channels. Areas to focus are a summation of focus areas of the localhost and remote Data Channel testing except with differing browsers.

==== Dependencies ====

Localhost and remote Data Channel testing has basic flows and some complex flows passing.

==== Signoff Criteria ====

* A smoke test run executed with no major blockers found with interoperability data channel scenarios
* A basic functional test run executed with no major blockers found with interoperability data channel scenarios
* An exploratory test run executed with no major blockers found with interoperability data channel scenarios
* Verify blocker bugs that are possible to verify with interoperability data channel scenarios

== Infrastructure ==

* A variety of servers and networks used in connection with Peer Connections will be needed. Open question on the specifics of these servers and networks.
* Device driver emulation will be needed in automation to exercise most of the media stream stack with getUserMedia
* Multiple operating systems with multiple machines needs to be accessible in different geographic locations
* Webcam and microphones integrated and USB need to be available for getUserMedia integration testing
* Frame by frame measurement of streams in local and remote calls could be useful for performance testing
* Data channel testing might benefit from having a repository of test data to reuse for sending to other peers
* A repository of test cases hosted will be needed for formal testing

== Test Case Management ==

* Formal end to end test cases in MozTrap shall be defined with the webrtc tag and peer connection tag for peer connection test cases and data channel tag for data channel test cases
* Exploratory tests shall be tracked directly in the wiki as a list of ideas to try out along with results of past investigations
* Automation for the large part will be tracked and developed in the dom/media/tests directory in mozilla central

== Open Questions ==

* For infrastructure requirements, what networks and servers should be worth considering for remote Peer Connection testing? [EKR: Can you explain more?]

== Resources ==

* [http://dev.w3.org/2011/webrtc/editor/webrtc.html WebRTC W3C Spec]
* [http://apprtc.appspot.com/ Example WebRTC App]
* [http://mozilla.github.com/webrtc-landing/ WebRTC Test Pages]
* [http://www.html5rocks.com/en/tutorials/webrtc/basics/ WebRTC Tutorial]
* [https://wiki.mozilla.org/Media/WebRTC/Architecture WebRTC Architecture]
* [http://en.wikipedia.org/wiki/NAT_traversal NAT Traversal on Wikipedia]
* [http://en.wikipedia.org/wiki/STUN STUN on Wikipedia]
* [http://en.wikipedia.org/wiki/Traversal_Using_Relay_NAT TURN on Wikipedia]
* [http://en.wikipedia.org/wiki/Session_Description_Protocol SDP on Wikipedia]
* [http://en.wikipedia.org/wiki/Interactive_Connectivity_Establishment ICE on Wikipedia]
* [http://www.ietf.org/rfc/rfc4566.txt SDP Spec]
* [https://tools.ietf.org/html/rfc5245 ICE Spec]
* [https://wiki.mozilla.org/WebRTC/Test_Plan/Dogfooding WebRTC Dogfooding]
* [https://wiki.mozilla.org/WebRTC/Test_Plan/Top_Site_Testing WebRTC Top Site Testing]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/WebRTC WebRTC Daily Smoketesting]