Revision History

Date	Version	Author	Description
02/03/2017	1.0	Vasilica Mihasca	Created first draft
04/10/2017	1.1	Krupa Raj	Updated test strategy
05/19/2017	1.2	Vasilica Mihasca	Updates

Overview

With webext-oop, addon code runs in a separate process (bugzil.la/webext-oop). In the far future, addon code may also run in multiple processes.

Purpose

Entry Criteria

QA has access to all the PRDs, mocks and related documents
The feature has landed on Nightly

Exit Criteria

All the bugs against the feature have been triaged
All the P1/P2 bugs have been fixed
All the resolved bugs have been verified by QA
The find/fixed rate is going down over a predefined period of time

Acceptance Criteria

This section broadly outlines when the product is ready to ship

QA has signed off
All the required Telemetry is in place
All info is localized at least for a pre-defined set of locales
All the necessary PR/blogposts have been sent out

Scope

This section describes what parts of the feature will be tested and what parts won't be.

what's in scope?

Webextensions installation with e10s enabled
Webextensions running in their own process
Regression testing for the top 20 webextensions
Regression testing for webextensions examples
Regression testing for old XUL add-ons
Webextension crashing
Add-ons behavior after a Firefox crash

what's out of scope?

Security testing
Performance testing

Ownership

Dev Lead: > Kris Maglione; irc nick
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Vasilica Mihasca; irc nick :vasilica_mihasca
Add-ons QA: Valentina Peleski; irc nick :ValentinaV

Requirements for testing

Environments

OSed covered: Windows, Mac OS X, Linux

Channel dependent settings (configs) and environment setups

Nightly

Disable check for signing if testing on AMO dev or stage
Webextension process crashing:
- Set "extensions.webextensions.remote” to true
- Insert the following line in Browser Console input filed: Cu.import("resource://gre/modules/Extension.jsm").GlobalManager.extensionMap.values().next().value.parentMessageManager.loadProcessScript("data:,Components.utils.import('resource://gre/modules/ctypes.jsm').ctypes.cast(ctypes.uintptr_t(1), ctypes.int.ptr).contents", false)

Beta

[Unbranded build] Disable check for signing if testing on AMO dev or stage

Post Beta / Release

The feature is enabled by default.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref	Function	Test Objective	Test Type	Owners
TO-1	Installing with e10s enabled	To verify that installation process runs without any issue	Manual	Add-ons QA Team
TO-2	Webextensions running in their own process	To verify that webextensions run in separate process	Manual	Add-ons QA Team
TO-3	Regression testing for the top 20 webextensions	To verify that the most popular webextensions works without issues	Manual	Add-ons QA Team
TO-4	Regression testing for webextensions example master	To verify that the webextensions works without issues	Manual	Add-ons QA Team
TO-5	Webextensions testing in Private Window	To verify the webextension behavior in private winodow	Manual	Add-ons QA Team
TO-6	Webextension crashing	To verify that crashing the webextension process does not cause a Firefox crash	Manual	Add-ons QA Team
TO-7	Firefox crashing	To verify that Firefox will load all the add-ons properly without any loss of saved add-on data	Manual	Add-ons QA Team
TO-8	Regression testing on XUL add-ons	To verify that the old XUL add-ons works without any issue	Manual	Add-ons QA Team

Test Strategy

Here are the main ways in which we will ascertain that OOP functions as expected

Test all the webextensions listed at https://github.com/mdn/webextensions-examples and ensure there are no regressions
Check webextensions which are using APIs which are most affected by OOP. These APIs include -
- context Menus
- devtools
- omnibox
- tabs
- identity
- permissions
Check that a firefox doesn't crash if a webextension crashes. See Bug 1353959 for more details
Check that extensions work as expected after a firefox crash

   webRequest

Builds

This section should contain links for builds with the feature -

Link for Nightly builds
Link for Beta builds
Beta Unbranded builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project
Study documentation/specs received from developers
QA - Test plan creation	02-03-2017
QA - Test cases/Env preparation
QA - Nightly Testing	03-22-2017
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	[ TestRail] / [Google docs] / etherpad
Test case execution	[TestRail]
Bugs management	Bugzilla / Github

Status

Overview

Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Aurora
Track the dates and build number where feature was merged to Release/Beta

Risk analysis

The test strategy recommended by devs is based on spot-checking and regression testing.
- Mitigation stragety - QA will test all webextensions with the most number of users to ensure that the most impactful add-ons are covered.
Webextensions will have their own process but Content scripts will still run in the content process
- Mitigation stragety - QA will test with webextensions which run content scripts to make sure there is degradation of behavior
Product is yet to define what is the acceptable dip in performance for when webextensions run in their process

References

* List and links for specs
  PRD - 
* bug 1190679 - (webext-oop) Run WebExtensions out of process

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1

Bug Work

Tracking bug - bug 1190679

Bug fix verification

Logged bugs

Sign off

Criteria

Check list

All test cases should be executed
Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

Link for the tests run
- Daily Smoke, use template from [link]
- Full Test suite, use template from [ link]
- Regression Test suite, if needed/available

Merge to Aurora Sign-off List of OSes that will be covered by testing

Link for the tests run
- Full Test suite

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release		Email to be sent
QA Aurora - Full Testing
QA Signoff - Aurora Release		Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release		Email to be sent

Add-ons/QA/Testplan/OOP-WebExtensions

Contents

Overview

Purpose

Entry Criteria

Exit Criteria

Acceptance Criteria

Scope

what's in scope?

what's out of scope?

Ownership

Requirements for testing

Environments

Channel dependent settings (configs) and environment setups

Nightly

Beta

Post Beta / Release

Test Objectives

Test Strategy

Builds

Test Execution Schedule

Testing Tools

Status

Overview

Risk analysis

References

Bugzilla query error

Bug Work

Bug fix verification

Logged bugs

Sign off

Criteria

Results

Checklist