QA/Aframe

Overview

Purpose

• The purpose of this wiki is to serve as a general test plan for A-Frame
• See https://wiki.mozilla.org/QA/MozVR for other test plans.

Ownership

Developer contacts:

Casey Yee UX Engineer @whoyee	Chris Van Wiemeersch UX Engineer @cvanw
Diego Marcos Developer @dmarcos	Kevin Ngo Developer @ngokevin_

Mozilla QA: Naoki Hirata (:nhirata_)

Testing Summary

Introduction

This is a hybrid test plan of ACC and traditional test plan in regards to the A-Frame

Scope of testing

In Scope

The current testing scope is to ensure that A-Frame

1. does not have regressions in changes
2. is stable
3. is reliable ( visibility )
4. performant ( no jerkiness )

Requirements for testing

Environments

Testing will be performed on following hardware

• iPhone:
  • iphone 6
  • iphone 6s ?
• Android:
  • Nexus s7 ?
  • Nexus s7 edge ?
  • nexus s6 ?
  • Galaxy Note 5
• Desktop Windows 10:
  • with headsets:
    • Oculus SV1 ?
    • HTC Vive
    • oculus DK2
  • without a headset.

Hardware

The minimum system requirements for Windows OS are:

• Graphics
  • GTX 970 equivalent or greater
  • AMD 290 R9 equivalent or greater
  • Quadro k600 or newer
• CPU
  • Intel i5-4590 equivalent or greater
  • AMD FX 8320 or greater
• 8GB+ RAM
• Compatible HDMI 1.3 video output
• 2x USB 3.0 ports
• Windows 7 SP1 or newer, plus the DirectX platform update

Devices

• Oculus Development Kit 2 with positional tracker
  • Note: soon to change to CV1
• Samsung Internet for Gear VR
  • Note: currently not daily smoked. need to talk to team.

Software

OS

• iOS ( latest ): 9.3.1
• Android : M 6.0 ?
• Windows : 10

Browsers

• Experimental Chrome
• Nightly Firefox
  • Archives of older builds are listed here : http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/

Setup

Oculus

• https://docs.google.com/document/d/16z5cAl21yeuaH_kYJBAs--g8HgXHj89snJI_8Py4VXQ/edit#
  • Note: needs updating

Software setup

• http://mozvr.com/#start

Test Strategy

Test Objectives

The test objective is to test the WebVR APIs for stability, visibility and performance.

Test Types

• Smoke test : this is to hunt for any major regressions that could prevent further testing; will need manual + automated on daily basis
• Unit Tests : this will be covered by developers and QA ; need to check status, @commit and @build time.
• Manual Tests: Integration / End to End Tests
  • Manual test cases : Must hit target testing that can't be automated
    • need to create small narrowed example web pages of each feature
  • Session Based testing :
    • width based testing : Use of Testing outline to perform test cases
    • depth based testing : Use of fixed bugs as a guideline for testing; ie bug verification means to check the bug fix + test around the bug fix.
• Automated Tests : Automating as much as possible for integration/end to end tests
  • weekly dashboard ? - need to find a way to do the automation and dashboard.
• Performance : Need some way to automated / Dashboard
• Security : need security team review
• Usability/Accessibility : ?
• Metrics/Telemetry : ?
• L10n : ?

Testing Tools

• ACC Rough Draft : [ Google Doc ]
• Test Outline : etherpad
• Session Based Testing Notes: etherpad
• Smoke tests : Google Doc
• Unit Test cases : github
• Manual Test cases : Google Doc
• Performances test cases : ? / Google Doc
• Automated Test Cases : ? / Selenium / Robotics? / Other frameworks?
  • Appium for mobile testing
• Socorro Nightly Reports : ie https://crash-stats.mozilla.com/topcrashers/?product=Firefox&version=48.0a1&days=7
• Community Feedback ( http://mozvr.com/#community )
• mozregression ( https://wiki.mozilla.org/Auto-tools/Projects/Mozregression )
• github
  • jshint
  • travis

Test cases

Test suites

Full Test Suite (TBD)

Bugs Work

Tracking bug:

Open A-Frame bugs

https://github.com/aframevr/aframe/issues?q=is%3Aopen+is%3Aissue

Closed A-Frame bugs

https://github.com/aframevr/aframe/issues?q=is%3Aissue+is%3Aclosed+sort%3Aupdated-desc