WebRTC/Test Plan/Top Site Testing

Overview

This document provides an overview of how to do top site testing around WebRTC web applications.

Setup

• Install the Desktop Firefox build you intend to use for testing
  • Desktop Firefox Beta
  • Desktop Firefox Aurora
  • Desktop Firefox Nightly
• Install Chrome
• Install the Firefox for Android build you intend to use for testing
  • Firefox for Android Nightly
• Install Chrome Beta for Android
  • Enable WebRTC in chrome://flags
• Ensure your testing machine has a webcam and a microphone
• Multiple distinct desktop and android devices are needed to complete the breadth of this testing

Top Site Tests

• Try to do a 1 to 1 call across two different machines on the same wifi network using the demo on Firefox Beta on both machines. Then, try to talk to each other while you are in the call. How is the sound quality? How is the video quality?
• Try to do a 1 to 1 call across two different machines on different networks using the demo on Firefox Beta on both machines. Then, try to talk to each other while you are in the call. How is the sound quality? How is the video quality?
• Try to do a 1 to 1 call across two different machines on different networks using the demo on Firefox Beta on both machines. Then, kill the network connection on one of the machines. Does the call fail gracefully?
• Try to do a 1 to 1 call on the same machine on the same network using the demo on Firefox Beta on both machines. Then, try to talk to each other while you are in the call. How is the sound quality? How is the video quality?
• Try to do a 1 to 1 call on two different machines in locations far away from each other (e.g. different countries) using the demo on Firefox Beta on both machines. Then, try to talk to each other while you are in the call. How is the sound quality? How is the video quality?
• Try to do a 1 to 1 call on two different machines in locations far away from each other using the demo on Firefox Beta on both machines. Then, let the call run for about five minutes. next, try to talk to each other while you are in the call. How is the sound quality? How is the video quality?
• Try to do a 1 to 1 call on two different machines on the same network using the demo on Firefox Beta on both machines. Then, end the call on one of the machines by closing the tab running the tokbox demo. Does the call fail gracefully?
• Try to do a multi-person call with 3 people on three different machines on different networks using the demo on Firefox Beta on each machine. Then, try to talk to each other while you are in the call. How is the sound quality? How is the video quality?
• Try to do a multi-person call with 4 people on four different machines on different networks using the demo on Firefox Beta on each machine. Then, try to talk to each other while you are in the call. How is the sound quality? How is the video quality?
• Try to do a multi-person call with 4 people on four different machines on different networks using the demo on Firefox Beta on each machine. Then, try to kill the network on two of the machines. Does the call continue for the two users who still have connectivity and end the call gracefully for the two users who left?
• Try to do a multi-person call with 4 people on four different machines on different networks using the demo on Firefox Beta on each machine. Then, let the call run for five minutes. Next, try to talk to each other while you are in the call. How is the sound quality? How is the video quality?
• Try to do a 1 to 1 call on two different machines in locations far away from each other using the demo on Firefox Beta on one machine and Chrome on the second machine. Then, let the call run for about five minutes. next, try to talk to each other while you are in the call. How is the sound quality? How is the video quality?
• Try to do a 1 to 1 call across two different machines on different networks using the demo on Firefox Beta on one machine and Chrome on the other machine. Then, kill the network connection on one of the machines. Does the call fail gracefully?
• Try to do a 1 to 1 call on the same machine on the same network using the demo on Firefox Beta on one machine and Chrome on the other machine. Then, try to talk to each other while you are in the call. How is the sound quality? How is the video quality?
• Try to do a multi-person call with 4 people on four different machines on different networks using the demo on Firefox Beta on two machines and Chrome on two other machines. Then, let the call run for five minutes. Next, try to talk to each other while you are in the call. How is the sound quality? How is the video quality?

Note: WebRTC bugs found during running any of these tests should be filed here.

Top Sites Process

The goal for WebRTC top site testing is to find out if the site is a reliable WebRTC site to show off in Firefox or not using the target tests specified above. This means you need to analyze the following areas as part of running the above tests:

• What functionality works well in the above tests?
• What issues did you see while running the above tests?
• What bugs were filed when running the above tests?

The sites currently being tested as part of the top sites testing are the following:

• Tokbox
• freshtilledsoil
• Codassium
• Conversat.io