Add-ons/QA/Testplan/Telemetry for add-ons manager

Revision History

Date	Version	Author	Description
10/12/2018	1.0	Cosmin Badescu	Created first draft

Overview

TBD.

Purpose

This document purports to detail the test approach for the improvements of Context Menu Updates and includes Entry/Exit criteria, Scope for testing, links to test cases etc

Entry Criteria

• QA has access to all the PRDs, mocks and related documents
• The feature has landed on Nightly
• AMO parts has landed on dev

Exit Criteria

• All the bugs against the feature have been triaged
• All the P1/P2 bugs have been fixed
• All the resolved bugs have been verified by QA
• The find/fixed rate is going down over a predefined period of time

Acceptance Criteria

This section broadly outlines when the product is ready to ship

• QA has signed off
• All the required Telemetry is in place
• All info is localized at least for a pre-defined set of locales
• All the necessary PR/blogposts have been sent out

Scope

This section describes what parts of the feature will be tested and what parts won't be.

what's in scope?

• TBD.

what's out of scope?

• Security testing

Ownership

Dev Lead: Luca Greco; irc nick :rpl
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Cosmin Badescu; irc nick :CosminB

Requirements for testing

Environments

OSes covered: Windows, Mac OS X, Linux

Channel dependent settings (configs) and environment setups

Nightly

• TBD

Beta

• TBD

Release

The feature is enabled by default

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref	Function	Test Objective	Test Type	Owners
TO-1	Installing from AMO	To verify that the extension uses the API correctly	Manual	Add-ons QA Team
TO-2	Installing from local files	To verify that the extension uses the API correctly	Manual	Add-ons QA Team
TO-3	Add-on updates	To verify that the extension uses the API correctly	Manual	Add-ons QA Team
TO-4	Sideloading	To verify that the extension uses the API correctly	Manual	Add-ons QA Team

Builds

This section should contain links for builds with the feature -

• Link for Nightly builds
• Link for Beta builds
• Link for Release builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project
Study documentation/specs received from developers
QA - Test plan creation	10-12-2018
QA - Test cases/Env preparation		N/A
QA - Nightly Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	[ Docs] / TestRail
Test case execution	[ Docs] / TestRail
Bugs management	Bugzilla / Github

Status

Overview

Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Release/Beta

Risk analysis

Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix

References

* List and links for specs
  PRD - TBD
  Install flow - TBD
  

* bug 1465143 - [META] Telemetry completion

Full Query

ID	Priority	Component	Assigned to	Summary	Status	Resolution	Target milestone
1316074	P2	Add-ons Manager		get telemetry on started and completed extension/theme installs	RESOLVED	FIXED	---
1483002	P2	General	Luca Greco [:rpl] [:luca] [:lgreco]	Create WEBEXT probes keyed by addon_id	VERIFIED	FIXED	mozilla64

2 Total; 0 Open (0%); 1 Resolved (50%); 1 Verified (50%);

* bug 1316074 - get telemetry on started and completed extension/theme installs

Full Query

ID	Priority	Component	Assigned to	Summary	Status	Resolution	Target milestone
1338713	P1	Add-ons Manager	Andrew Swan [:aswan]	Basic telemetry for permissions notifications	VERIFIED	FIXED	mozilla55
1433334	P1	Add-ons Manager	Luca Greco [:rpl] [:luca] [:lgreco]	Store the installation source of the extension for telemetry	RESOLVED	FIXED	mozilla64
1433335	P1	Add-ons Manager	Luca Greco [:rpl] [:luca] [:lgreco]	Send telemetry events for each action on an extension	VERIFIED	FIXED	mozilla64
1486763	P2	Add-ons Manager	Luca Greco [:rpl] [:luca] [:lgreco]	Add installTelemetryInfo to addon installed from Firefox Sync	VERIFIED	FIXED	mozilla64
1496161	--	Add-ons Manager	Luca Greco [:rpl] [:luca] [:lgreco]	Add installTelemetryInfo to the WebExtensions converted themes installed from the LightweightThemeManager	VERIFIED	FIXED	mozilla64
1496163	--	Add-ons Manager	Luca Greco [:rpl] [:luca] [:lgreco]	Add installTelemetryInfo to the WebExtensions themes installed using the management.install API method	VERIFIED	FIXED	mozilla64
1496167	P1	New Tab Page	Luca Greco [:rpl] [:luca] [:lgreco]	Add installTelemetryInfo to addon installed from the ActivityStream	VERIFIED	FIXED	Firefox 64

7 Total; 0 Open (0%); 1 Resolved (14.29%); 6 Verified (85.71%);

Testcases

Overview

Summary of testing scenarios

Test Areas

Test Areas	Covered	Details
Installing from AMO
Installing from local files
Installing from thirdparty
Add-on updates
Sideloading
Other

Test suite

• Link for the [ Initial test planning]
• Link for the [ Google doc tests]
• Link for the [ TestRail tests]

Sign off

Criteria

Check list

• All test cases should be executed
• Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
• All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

• Link for the tests run
  • Full Test suite, use template from []

Merge to Beta Sign-off List of OSes that will be covered by testing

• Link for the tests run
  • Full Test suite

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation	10-12-2018
Test Cases Creation		N/A
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release		Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release		Email to be sent