Contents
Purpose
This document intends to provide a high level overview of the testing that's needed for testing the entire stack of the WebRTC feature set on Android.
Ownership and Status Overview - WebRTC in Firefox for Android
| Feature | Release Target | Dev Lead | QA Lead | Dev Status | QA Status | Health |
| WebRTC in Firefox for Android | Firefox 23/24 | Gian-Carlo Pascutto | Aaron Train | Landed (disabled) | In Progress | [ON TRACK] |
| Data Channel | Firefox 23/24 | Gian-Carlo Pascutto | Aaron Train | OK | In Progress | |
| Peer Connection | Firefox 23/24 | Gian-Carlo Pascutto | Aaron Train | OK | In Progress | |
| getUserMedia | Firefox 23/24 | Gian-Carlo Pascutto | Aaron Train | OK | In Progress |
Testing Scope
In Scope
Functionality
- getUserMedia and Local Media Streams DOM functionality
- End-to-end integration with Android device cameras/mics across different Android OS versions with getUserMedia
- Peer Connection, Session Description, and Ice Candidate DOM API functionality
- Data Channel DOM API and it's integration with global Peer Connection DOM API
- End-to-end local and remote Peer Connection handshaking, ice candidates, data connections, etc.
- End-to-end local and remote Data Channel integration with Peer Connection and sending of data
Testing Lifecycle
Overview
This section provides an overview of the lifecycle of testing the WebRTC-based APIs for the first release of getUserMedia, PeerConnection, and DataChannel.
getUserMedia Device Integration Testing with UI Integration
Summary
Testing the integration with real devices with the UI hooked up in the Firefox front-end. Focuses on making sure the full end to end pieces are hooked up with both integrated and USB hardware with real-time media streams received from the getUserMedia.
Dependencies
The testing for no UI integration flows needs to be completed.
Signoff Criteria
getUserMedia DOM Testing and Automation
Summary
Testing and writing mochitest automation for LocalMediaStream, getUserMedia API, and integration with media element functionality in the DOM.
Signoff Criteria
Covered by Jason; existing automation enabled on branch (mochitests/crash-tests/DOM tests).
getUserMedia Device Integration Testing with UI Integration
Summary
Testing the integration with real devices with the UI hooked up in the Firefox front-end. Focuses on making sure the full end to end pieces are hooked up with both integrated and USB hardware with real-time media streams received from the getUserMedia.
Dependencies
The testing for no UI integration flows needs to be completed.
Signoff Criteria
Peer Connection - Localhost and Real Stream End to End Testing
Summary
Focuses on testing of the Peer Connection DOM API through end to end tests on a local machine with real streams from cameras and microphones. The testing style is similar to how the automation will be developed with fake streams, except the big difference is the focus with real streams and testing areas that are non-trivial to automate. Such areas that will be focused on include - video/audio device integration, call rejection explicit vs. not explicit, call longevity, multiple calls, clean vs. non-clean shutdown, connection loss, flaky connection, etc.
Dependencies
getUserMedia UI integration testing should have basic flows passing and most complex flows passing.
Signoff Criteria
- A smoke test run executed with no major blockers found with localhost real stream peer connection scenarios
- A basic functional test run executed with no major blockers found with localhost real stream peer connection scenarios
- An exploratory test run executed with no major blockers found with localhost real stream peer connection scenarios
- Verify blocker bugs that are possible to verify with localhost real stream peer connection scenarios
Peer Connection - Remote Real Stream End to End Testing
Summary
Focuses on remote scenario testing with Peer Connections with real streams and microphones that both use Firefox. Such areas to focus on include single vs. multiple remote connections, connection loss, lame networks, peer locations, stream performance, type of server being used, explicit vs. not explicit call end, longevity of call.
Dependencies
Localhost peer connection real stream testing has basic flows passing and some complex flows passing.
Signoff Criteria
- A smoke test run executed with no major blockers found with remote peer connection scenarios
- A basic functional test run executed with no major blockers found with remote peer connection scenarios
- An exploratory test run executed with no major blockers found with remote peer connection scenarios
- Verify blocker bugs that are possible to verify with remote peer connection scenarios
Data Channel - Remote Peer Connection Integration Testing with Real Streams
Summary
Focuses on remote scenario testing with Data Channels with Peer Connection integration that use Firefox purely. Such areas to focus on include remote reliability, size of information sent, storage remaining on each system, type of data sent, rate of messages sent, single vs. multiple connections, and integration with remote Peer Connection scenarios.
Dependencies
Basic end to end scenarios for Data Channels should be defined and passing for localhost scenarios.
Signoff Criteria
- A smoke test run executed with no major blockers found with remote Data Channel scenarios
- A basic functional test run executed with no major blockers found with remote Data Channel scenarios
- An exploratory test run executed with no major blockers found with remote Data Channel scenarios
- Verify blocker bugs that are possible to verify with remote Data Channel scenarios
Peer Connection - Interoperability Testing with Chrome
Summary
Focuses on interoperability testing between Chrome and Firefox locally and remotely with Peer Connections. Areas to focus on are basically a summation of the focus areas of the localhost and remote peer connection testing except using differing browsers in the call.
Dependencies
Localhost and remote peer connection testing has basic flows passing and some complex flows passing.
Signoff Criteria
- A smoke test run executed with no major blockers found with interoperability peer connection scenarios
- A basic functional test run executed with no major blockers found with interoperability peer connection scenarios
- An exploratory test run executed with no major blockers found with interoperability peer connection scenarios
- Verify blocker bugs that are possible to verify with interoperability peer connection scenarios
Data Channel - Interoperability Testing with Chrome
Summary
Focuses on interoperability testing between Chrome and Firefox locally with Data Channels. Areas to focus are a summation of focus areas of the localhost and remote Data Channel testing except with differing browsers.
Dependencies
Localhost and remote Data Channel testing has basic flows and some complex flows passing.
Signoff Criteria
- A smoke test run executed with no major blockers found with interoperability data channel scenarios
- A basic functional test run executed with no major blockers found with interoperability data channel scenarios
- An exploratory test run executed with no major blockers found with interoperability data channel scenarios
- Verify blocker bugs that are possible to verify with interoperability data channel scenarios
Test Case Management
- Formal end to end test cases in MozTrap shall be defined with the webrtc tag and peer connection tag for peer connection test cases and data channel tag for data channel test cases
- Exploratory tests shall be tracked directly in the wiki as a list of ideas to try out along with results of past investigations
- Automation for the large part will be tracked and developed in the dom/media/tests directory in mozilla central
WebRTC/getUserMedia - Device Overview
The following matrix discerns device testing and quality based on results of the MozTrap getUserMedia BFT Test Suite: (21 Cases)
| Device | "Android" | QA Lead | Bugs | Quality | Signed Off |
| LG Nexus 4 | Android 4.2.2 (Jellybean) | Aaron | - | - | - |
| HTC One | Android 4.1 (Jellybean) | Aaron | - | - | - |
| Samsung Galaxy S IV | Android 4.2.2 (Jellybean) | Aaron | - | - | - |
| Samsung Galaxy Nexus | Android 4.2.2 (Jellybean) | Aaron | - | - | - |
| Samsung Galaxy Note II | Android 4.1 (Jellybean) | Aaron | - | - | - |
| Asus Nexus 7 | Android 4.2.2 (Jellybean) | Aaron | - | - | - |
| Samsung Galaxy S II | Android 4.0.4 (Ice Cream Sandwich) | Aaron | - | - | - |
| Asus Transformer Prime TF201 | Android 4.1 (Jellybean) | Aaron | - | - | - |
| Alcatel One Touch 8008X | Android 4.1.2 (Jellybean) | Aaron | - | - | - |
WebRTC Peer to Peer Connection Testing
The following matrix discerns testing of quality and duration of WebRTC AppRTC and Talky.io calls of > 5 minutes in length.
WebRTC
Firefox 24 (SoftVision)
Using the Moztrap test as a guideline, verify against Firefox 24 when making AppRTC calls of >5 minutes in length. When testing, here are the guidelines to follow:
- Have only one browser open on each machine and device at any given time
- Have only one call running at any given time on a particular machine to another device
- Make sure the caller and callee are always on *different* machines or devices.
- Please don't test any 3+-way calls for this sanity check -- We just want to see the results for 1:1 (basic) calling
- If you find regressions, report a bug and CC Randell and Maire; they can help track down if the regressions are real or not
- Tip: be sure to provide extremely detailed steps to reproduce and witnessed results, as well as detailed information about your test environment; more information is better than not enough information.
| Caller | Callee | Result | Notes |
| Firefox 24 (Android, Samsung Galaxy S4, Office WiFi) | Firefox 24 (Android, HTC One, Office WiFi) | [DONE] | Caller: PASS Callee: PASS |
| [AT RISK] | Caller: Callee: |
Firefox 26 (SoftVision)
Using the Moztrap test as a guideline, verify against Firefox 24 when making AppRTC calls of >5 minutes in length. When testing, here are the guidelines to follow:
- Have only one browser open on each machine and device at any given time
- Have only one call running at any given time on a particular machine to another device
- Make sure the caller and callee are always on *different* machines or devices.
- Please don't test any 3+-way calls for this sanity check -- We just want to see the results for 1:1 (basic) calling
- If you find regressions, report a bug and CC Randell and Maire; they can help track down if the regressions are real or not
- Tip: be sure to provide extremely detailed steps to reproduce and witnessed results, as well as detailed information about your test environment; more information is better than not enough information.
| Caller | Callee | Result | Notes |
| Nightly 26.0a1 (2013-09-03) (Android(4.0.3), Samsung Galaxy S2, Office WiFi) | Nightly 26.0a1 (2013-09-03) (Android(4.1.2), LG Otimus 4x, Office WiFi) | Paul Feher: Video seems to be working ok there where no artefacts present or noticeable drop in video call quality
but the audio is severely distorted (a metallic sound is present all the time) and you can't hear anything. I couldn’t establish a connection using Chrome for Android with the same devices. |
Caller: PASS Callee: PASS |
| Nightly 26.0a1 (2013-09-03) (Android(4.0.4), Asus Transformer TF 101, Office WiFi) | Nightly 26.0a1 (2013-09-03) (Android(4.1.1), Samsung Galaxy Nexus, Office WiFi) | TeoVermesan: Video works ok and the audio is distorted al the time. Managed to connect after a few tries.} | Caller: PASS Callee: PASS: |
| Nightly 26.0a1 (2013-09-03) (Android(4.0.4), Samsung Galaxy Tab, Office WiFi) | Nightly 26.0a1 (2013-09-03) (Android(4.2.2), LG Nexus 4, Office WiFi) | Mihai Pop: Tested on "http://mysecondwebrtc.appspot.com/", not able to connect on "https://apprtc.webrtc.org/". Video quality was very good with high fps rate, no artifacts, but with moments of video freeze(about 5 second).
Audio behavior was not consistent, speaking from phone to tablet, sometimes resulted in almost clear sound with no delay, but sometimes there we're background noises and the voice is not understandable and delays may appear. However speaking from tablet to phone resulted always in bad audio experience. |
Caller: PASS Callee: PASS |
Firefox 24 (Networking)
The goal here is to capture as many ~5 minute video/audio calls across as many network combinations we can to compare video/audio latency in FxAndroid vs. Chrome for Android Beta.
Test Case Steps
- Go to apprtc.appspot.com on the browser under test on device A under a target network configuration
- Accept permissions for camera/microphone for device A
- Go to the apprtc URL generated in step 1 on the browser under test on device B under a target network configuration
- Accept permissions for camera/microphone for device B
- Wait for the call to establish
- Begin communicating between each peer between device A and B for 5 minutes
- Record your observations in the below table
| Tester | Device A Network & Type | Device B Network & Type | FxAndroid 24 Call Observations | Chrome for Android 29 Call Observations |
| jsmith | Mountain View Mozilla Wifi with Samsung Galaxy Nexus on Android 4.2 | Mountain View Mozilla Wifi with Samsung Galaxy S3 on Android 4.3 | Call establishes. Video/audio latency is extremely bad to the point of being unusable from start to end of the call (e.g. chopped up syllables, choppy video). | |
| kbrosnan | SF Mozilla Wifi | SF Mozilla Wifi | ||
| Toronto Mozilla Wifi | Toronto Mozilla Wifi | |||
| AdrianT | Romania SV Wifi with Samsung Galaxy Tab 2 on Android 4.1 | Romania SV Wifi in Romania with Samsung Galaxy Note on Android 4.0 | Video freezes a lot and the audio is distorted at first and then starts to be more understandable but with ~2 min delay. | Eventually managed to connect after a few tries. Audio is distorted and hard to understand but of better quality then on Firefox |
Note: The browser under test should be the same between device A and B. For any bugs found, file the bug in the core webrtc component with signaling logs included.
Testing Documents
- WebRTC - Site Testing
- This document provides an overview of how to do top site testing around WebRTC web applications, see top site testing