QA/WebVR

Overview

Purpose

• The purpose of this wiki is to serve as a general test plan for the WebVR API.
• See https://wiki.mozilla.org/QA/MozVR for other test plans.

Ownership

Developer contacts: Kearwood Kip Gilbert
QA: Naoki Hirata (:nhirata_), Cornel Ionce (:cornel_ionce)

Testing Summary

Introduction

This is a hybrid test plan of ACC and traditional test plan in regards to the WebVR API

Scope of testing

In Scope

The current testing scope is to ensure that the WebVR API:

1. does not have regressions in performance or latency
2. is stable
3. is reliable ( visibility )

Requirements for testing

Environments

Testing will be performed on following OSes (x64 infrastructures):

• Windows 10

Hardware

The minimum system requirements for Windows OS are:

• Graphics
  • GTX 970 equivalent or greater
  • AMD 290 equivalent or greater
  • Quadro k600 or newer
• CPU
  • Intel i5-4590 equivalent or greater
  • AMD FX 8320 or greater
• 8GB+ RAM
• Compatible HDMI 1.3 video output
• 2x USB 3.0 ports
• Windows 7 SP1 or newer, plus the DirectX platform update

Devices

• Oculus Development Kit 2 with positional tracker
  • Note: soon to change to CV1
• Samsung Internet for Gear VR
  • Note: currently not daily smoked. need to talk to team.

Software

• Experimental Chrome
• Nightly Firefox
  • Archives of older builds are listed here : http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/

Setup

Oculus

• https://docs.google.com/document/d/16z5cAl21yeuaH_kYJBAs--g8HgXHj89snJI_8Py4VXQ/edit#
  • Note: needs updating

Software setup

• http://mozvr.com/#start

Test Strategy

Test Objectives

The test objective is to test the WebVR APIs for stability, visibility and performance.

Test Types

• Smoke test : this is to hunt for any major regressions that could prevent further testing.
• Unit Tests : ?
• Manual Tests: Integration / End to End Tests
  • Manual Test cases : Must hit target testing that can't be automated
  • Session Based testing : Use of Testing outline to perform test cases
• Automated Tests : Automating as much as possible for integration/end to end tests
• Performance : Need some way to automated / Dashboard
• Security : need security team review
• Usability/Accessibility : ?
• Metrics/Telemetry : ?

Testing Tools

• ACC Rough Draft : Google Doc
• Test Outline : etherpad
• Session Based Testing Notes: etherpad
• Smoke tests : Google Doc
• Unit Test cases : ?
• Manual Test cases : ?
• Performances test cases : ?
• Automated Test Cases : ?

References

Meta: Bug 1218482 - Enable WebVR By Default on all platforms

Test cases

Test suites

Full Test Suite

Bugs Work

Tracking bug: Meta bug 1229464

Archive

• Old Test Plan : https://docs.google.com/document/d/1miP0-eoHeoYwIZzj1fPQGuA9QVzXZknwSWoukXxpCSw/edit?pref=2&pli=1#
• Previous Test Results :
  • https://wiki.mozilla.org/QA/WebVR/TestResults_03.04.2016