|
|
| (42 intermediate revisions by 3 users not shown) |
| Line 1: |
Line 1: |
| See: https://wiki.mozilla.org/Media/WebRTC/Testing for background on automated testing. | | See: https://wiki.mozilla.org/Media/WebRTC/Testing for background on automated testing. |
|
| |
|
| == Routine Manual Testing Tasks ==
| | == Basic Topology == |
| The following tasks are essentially mechanical, but at least for now they
| |
| need a person. The goal should be to find ways to automate this work.
| |
| | |
| === Basic Topology ===
| |
| The basic topology should include two separate machines on separate networks, | | The basic topology should include two separate machines on separate networks, |
| each with their own NAT. We have some flexibility about which NAT, but | | each with their own NAT. We have some flexibility about which NAT, but |
| Line 15: |
Line 11: |
| The following test series should be run, with one machine on each network with Nightly. | | The following test series should be run, with one machine on each network with Nightly. |
|
| |
|
| ==== Endpoints ====
| | === Endpoints === |
| * Firefox Nightly <-> Firefox Nightly | | * Firefox Nightly <-> Firefox Nightly |
| * Firefox Nightly <-> Chrome Stable | | * Firefox Nightly <-> Chrome Stable |
| * Firefox Nightly <-> Chrome Canary | | * Firefox Nightly <-> Chrome Canary |
|
| |
|
| ==== Platforms ====
| | === Platforms === |
| * Desktop (Windows, Mac, Linux) | | * Desktop (Windows, Mac, Linux) |
| * B2G (Unagi, Peak) | | * B2G (Unagi, Peak) |
| * Android? | | * Android |
|
| |
|
| === AppRTC === | | === AppRTC === |
| Line 34: |
Line 30: |
| Run the [http://mozilla.github.com/webrtc-landing WebRTC-Landing tests] with no failures. | | Run the [http://mozilla.github.com/webrtc-landing WebRTC-Landing tests] with no failures. |
|
| |
|
| == Other Tasks == | | == Known Issues == |
| === FIXED Bugs ===
| | * {{bug|936034}} - Sound is distorted on one side on 1:1 call |
| The following tracks verification of WebRTC bug fixes. As you verify a bug fix please list it below.
| |
| * [http://mzl.la/196DkC0 549 more bugs] | |
| | |
| === UNCONFIRMED Bugs ===
| |
| The following tracks confirmation of WebRTC bugs. As you confirm a bug exists please list it below.
| |
| * [http://mzl.la/14a3qyW 17 more bugs]
| |
| | |
| === WebRTC Health Report ===
| |
| The following tries to estimate the relative "health" of WebRTC based on bugs in bugzilla. Update this table each week with the number of bugs reported, resolved, and reopened.
| |
| {| class="querytable"
| |
| | class="header" | Period
| |
| | class="header" | Bugs Reported
| |
| | class="header" | Bugs Resolved
| |
| | class="header" | Bugs Reopened
| |
| |-
| |
| | 2013.07.28 - 2013.08.03
| |
| | 9
| |
| | 32
| |
| | 4
| |
| |-
| |
| | 2013.07-21 - 2013-07-27
| |
| | 14
| |
| | 17
| |
| | 3
| |
| |-
| |
| | 2013.07.14 - 2013.07.20
| |
| | 12
| |
| | 11
| |
| | 2
| |
| |-
| |
| | 2013.07.07 - 2013.07.13
| |
| | 16
| |
| | 16
| |
| | 2
| |
| |-
| |
| | 2013.06.30 - 2013.07.06
| |
| | 11
| |
| | 17
| |
| | 3
| |
| |}
| |
| | |
| == Testing ==
| |
| === AppRTC ===
| |
| * Demo Site: [http://apprtc.webrtc.org/ apprtc.webrtc.org]
| |
| * Test Methodology:
| |
| ** Running the demo 1:1 with no other processes running
| |
| ** Running the demo 1:1 with another vanilla Firefox process running
| |
| ** Running the demo 1:1 with another heavy Firefox process running (Flash, videos, games, gmail, etc)
| |
| ** Restart the caller browser and repeat the above
| |
| ** Restart the callee browser and repeat the above
| |
| * Expected Result:
| |
| ** A call lasting 5 minutes or longer should be clear and accurate (no delays, no artifacts, no crashes or hangs, no degradation in audio or video quality from the previous revision)
| |
| * Platforms:
| |
| ** P1: Windows XP, Windows 7, Windows 8, Mac OSX 10.8, Ubuntu 12.04, B2G
| |
| ** P2: Windows Vista, Mac OSX 10.7, Ubuntu 13.04, Android
| |
| * Builds:
| |
| ** P1: Firefox Nightly, Chrome Stable, B2G Browser
| |
| ** P2: Firefox Aurora/Beta/Release, Chrome Canary, Firefox Android (all branches)
| |
| | |
| ==== Results: YYYY.MM.DD ====
| |
| {| class="querytable"
| |
| | class="header" style="width:30%" | Caller
| |
| | class="header" style="width:30%" | Callee
| |
| | class="header" style="width:40%" | Result
| |
| |-
| |
| | Windows XP w/Firefox Nightly YYYY.MM.DD
| |
| | Windows 7 w/Firefox Nightly YYYY.MM.DD
| |
| | '''RESULT''': detailed notes
| |
| |-
| |
| |}
| |
| | |
| == Activity Log ==
| |
| * 2013.08.20: Firefox 24.0b4 signed-off for parity with Firefox 22.0 [[https://wiki.mozilla.org/Releases/Firefox_24/Test_Plan#Regression_Tests_3 details]]
| |
| * 2013.08.15: Firefox 23.0.1 signed-off for parity with Firefox 22.0 [[https://wiki.mozilla.org/Releases/Firefox_23/Test_Plan#WebRTC details]]
| |
|
| |
|
| == Meetings == | | == Meetings == |
| * [https://etherpad.mozilla.org/qa-webrtc-meeting-20130814 August 14, 2013]: Kick-off meeting with WebRTC Devs | | * [https://etherpad.mozilla.org/qa-webrtc-meeting-20130814 August 14, 2013]: Kick-off meeting with WebRTC Devs |