QA/e10s Multi

Overview

Purpose

This wiki details the testing that will be performed by QA and other teams for e10s-multi. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

• Areas of risk
• What will be tested
• How testing will be performed
  • criteria
  • deliverables
  • ownership
  • schedule
• What data will be monitored

To help ensure the best possible release and long term maintenance of this feature.

Risk Analysis

Risk area	Requirement	Status
Performance	Improved browser responsiveness	TBD
Memory Usage	No regression in comparable system memory usage	TBD
Security	No security holes found in fuzz testing	TBD
Stability	No regressions in crash rate	TBD
Major site compatibility	Improved perf at major websites	TBD

Testing Strategy

Scope

Test Areas	Covered
Private Window	Yes
Multi-Process Enabled	Yes
Single-Content Process Enabled	Yes (baseline)
e10s dis-abled	Yes (baseline)
Stability	Yes
UI
Interraction (scroll, zoom)	Yes
Multi-Tab/Window	Yes
Security
Fuzzing	Yes
Web Compatibility
Testing against target sites	Yes
Data Monitoring
Temporary or permanent telemetry monitoring	Yes

If it's not listed above, it is currently out of scope.

Test Strategy

Objectives

Criteria Description	Metric	Single-Content Process	Multi-Content Process	Criteria Met?	QA Owner
Manual testing	Tests passed	(test results)	(test results)	TBD (Date status updated)	SV?
Unit testing	Automated tests pass	(test results)	(test results)	TBD (Date status updated)	Blake
Talos testing	Startup time	ts_paint - perherder <link tbd>	ts_paint - perherder <link tbd>	TBD (Date status updated)	Talos team?
UI	Tab/Window responsiveness	Perherder <link(s) tbd>	Perherder <link(s) tbd>	TBD (Date status updated)	Talos team?
Memory monitoring	System memory use	<link tbd>	<link tbd>	TBD (Date status updated)	TBD
Fuzz testing	Automated tests pass	(test results)	(test results)	TBD (Date status updated)	Fuzzing team (tbd)
Stability	Crash rate stable	telemetry and crash stats	telemetry and crash stats	TBD (Date status updated)	Telemetry team

Testcases

TBD:

• General browser testing as in e10s
• e10s-multi specific tests
  • process assignment

Environments

Full Testing will be performed using Nightly for Desktop builds on:

• Windows
• Mac OS X
• Linux

Channel dependent settings (configs) and environment setups

• In Nightly dom.ipc.processCount should be 4 by default

Project Information

Ownership

Developer contacts: Blake Kaplan, Gabor Krizsanits
QA: Tracy Walker
QA Peer: TBD

Builds

This section should contain links for builds with the feature

• Links for Nightly builds

Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project	Q3/2016	Q3/2017
QA - Test plan creation	Q1/2017	Q1/2017
QA - Test cases/Env preparation	Q1/2017	Q2/2017
QA - Nightly Testing	Q2/2017	Q2/2017
QA - Aurora Testing	Q2/2017	Q3/2017
QA - Beta Testing	Q3/2017	Q3/2017
Release Date		FF55

References

• List and links for specs
  • [1] e10s multi wiki page
  • [2] e10s multi release strategy
  • [3] e10s release criteria

• Meta bug
  • [4] - Turn e10s-multi on in Nightly