QA/Desktop Firefox/WebRTC

See: https://wiki.mozilla.org/Media/WebRTC/Testing for background on automated testing.

Routine Manual Testing Tasks

The following tasks are essentially mechanical, but at least for now the need a person.

The basic topology should include two separate machines on separate networks, each with their own NAT. We have some flexibility about which NAT, but for concreteness, I suggest a pair of Apple Airport Express NATs. I.e., the following topology:

The following test series should be run, with one machine on each network with Nightly.

Full test series on AppRTC

• A call on apprtc between each pair of desktop endpoints
  • (9 calls in all)
• A call on apprtc between Firefox and Chrome (Stable + Canary) on any platform
  • (2 calls in all).
• Each call should last no less than 5 minutes with no major failures (failure to connect, sustained freezes, etc.)

Connect series on WebRTCMe

• A call on webrtcme between two Firefox endpoints (any platform)
• A call on webrtcme between Firefox and Chrome Stable (and platform)
• Each call should connect and work for at least one minute

B2G Testing

• A call on webrtcme between a B2G phone and any desktop OS for the following phones;
  • Unagi, Peak
• This test needs to be run on a home-built build with Steven's patch
• Each call should connect and work for at least one minute with no major glitches (some video lag and freezing is acceptable).

aPlatforms to include all Windows, Mac OSX, Linux, Android, FirefoxOS

• Daily regression testing with Firefox Nightly across all platforms to include:
  • Calling with apprtc using various combinations of Firefox and Chrome
  • Calling with webrtcme using various combinations of Firefox and Chrome
  • Evaluation of audio/video delay
  • WebRTC-Landing tests pass
  • Daily might be too ambitious, can it be done weekly?

QA Tasks

• Daily verification of whatever fixes landed in that day's Nightly
• Weekly triage and attempted confirmation of unconfirmed/regression bugs
• Monthly re-evaluation of regression rate to inform adjustments to the above approach

Testing

• Manual
  • What must be done manually?
• Automated
  • What can be automated?