QA/Desktop Firefox/WebRTC

See: https://wiki.mozilla.org/Media/WebRTC/Testing for background on automated testing.

Routine Manual Testing Tasks

The following tasks are essentially mechanical, but at least for now the need a person.

The basic topology should include two separate machines on separate networks, each with their own NAT. We have some flexibility about which NAT, but for concreteness, I suggest a pair of Apple Airport Express NATs. I.e., the following topology:

The following test series should be run, with one machine on each network with Nightly.

Full test series on AppRTC

• A call on apprtc between each pair of desktop endpoints

 ** (9 calls in all)

• A call on apprtc between Firefox and Chrome (Stable + Canary) on any platform

  ** (2 calls in all).

• Each call should last no less than 5 minutes with no major failures (failure to connect, sustained freezes, etc.)

Connect series on WebRTCMe

• A call on webrtcme between two Firefox endpoints (any platform)
• A call on webrtcme between Firefox and Chrome Stable (and platform)
• Each call should connect and work for at least one minute

aPlatforms to include all Windows, Mac OSX, Linux, Android, FirefoxOS

• Daily regression testing with Firefox Nightly across all platforms to include:
  • Calling with apprtc using various combinations of Firefox and Chrome
  • Calling with webrtcme using various combinations of Firefox and Chrome
  • Evaluation of audio/video delay
  • WebRTC-Landing tests pass
  • Daily might be too ambitious, can it be done weekly?

QA Tasks

• Daily verification of whatever fixes landed in that day's Nightly
• Weekly triage and attempted confirmation of unconfirmed/regression bugs
• Monthly re-evaluation of regression rate to inform adjustments to the above approach

Testing

• Manual
  • What must be done manually?
• Automated
  • What can be automated?