Revision History

Date	Version	Author	Description
03-25-2019	2.0	Vlad Jiman	Updated all sections

Overview

To enable telemetry for about:addons internal in-page interactions and external events leading to about:addons, we require to establish what are the events which trigger data collection and are in need of logging while also providing an area where such events can be viewed based on criteria such as timestamp, category, type of performed action, target of action, etc. In addition to the previously mentioned, a preference for the user to enable/disable data collection has to be implemented, as well.

Purpose

This document proposes to detail a test approach to enabling telemetry for about:addons, which includes Entry/Exit/Acceptance criteria, Testing scope, references to test cases, etc.

Entry Criteria

• QA has access to all PRDs, mocks and related documentation
• The feature has landed on Nightly

Exit Criteria

• All feature related bugs have been triaged
• All P1/P2 bugs have been fixed
• All resolved bugs have been verified by QA
• The find/fixed bug ratio shows a descending trend over a defined time period

Acceptance Criteria

This section proposes to highlight the criteria concerning the shipment readiness status of the product.

• QA has signed off
• All the required Telemetry triggers are in place

Scope

This section outlines which parts of the new implemented feature will or will not be tested.

What is in scope

• Validation of the about:addons telemetry system, including all related UI features.

What is out of scope

• Security testing
• Device testing
• Performance testing

Ownership

Dev Lead: Mark Striemer ; irc nick:mstriemer
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Vlad Jiman; irc nick :VladJ

Requirements for testing

Environments

Covered OSes: Windows, Mac OS X, Linux

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered.

Ref	Function	Test Objective	Test Type	Owners
TO-01	Normal Browsing testing	Validate event triggering and logging during normal usage	Manual	Add-ons QA Team
TO-02	Private Browsing testing	Validate event triggering and logging during private usage	Manual	Add-ons QA Team
TO-03	Profile testing	Validate event triggering and logging while using new/used profiles	Manual	Add-ons QA Team

The objective of testing this feature is to assure that the triggers (Listed here: https://docs.google.com/spreadsheets/d/1ETub4hptzLTnEyLDg_q6MhtST4veID4-fg-rnG16ffU/edit#gid=618290056) are correctly logging user events.

Test Profiles

Ref	Profile Objective
TP-1	Validate if events are triggered and logged during normal browsing
TP-2	Validate if events are triggered and logged during private browsing
TP-3	Validate if events are triggered and logged during usage with a new profile
TP-4	Validate if events are triggered and logged during usage with a used profile

Builds

This section should contain links for builds with the feature -

• Link for Nightly builds
• Link for Beta builds
• Link for Release builds

Test Execution Schedule

The below table outlines the anticipated testing time frame available for test execution.

Project phase	Start Date	End Date
Start project	10/18/2018
Study documentation/specs received from developers	03/05/2019
QA - Test plan creation	03/05/2019
QA - Test cases/Env preparation	03/05/2019
QA - Nightly Testing	03/04/2019	03/08/2019
QA - Beta Testing
Release Date

Testing Tools

Exemplifies the tools used for test suite creation/execution.

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	TestRail
Test case execution	TestRail
Bugs management	Bugzilla

Status

Overview

Track the dates and build number where the feature was released to Nightly.
Track the dates and build number where the feature was merged to Release/Beta.

Risk analysis

Identify the high-risk assumptions.
Identify existing bugs on the feature with high risk.
Identify if other areas are affected by the fix.

References

* List and links for specs
  Trigger list - Google Docs
   
* bug 1500147 - [meta] Add telemetry for about:addons interface

Full Query

ID	Priority	Component	Assigned to	Summary	Status	Target milestone
1543640	--	General	Mark Striemer [:mstriemer]	‘Remove’ and ‘Undo’ events are not registered in telemetry when performed in the ‘Recent Updates’ tab	VERIFIED	---

1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);

Testcases

Overview

The test suite proposes a series of 4 test cases devised to cover 4 scenarios where event triggering and logging will occur in order to properly validate the feature.

• Event triggering and logging validation during Normal Browsing utilizing a New Profile
• Event triggering and logging validation during Normal Browsing utilizing a Used Profile
• Event triggering and logging validation during Private Browsing utilizing a New Profile
• Event triggering and logging validation during Private Browsing utilizing a Used Profile

Test Areas

Test Areas	Covered	Details
Normal browsing on a new profile
Normal browsing on a used profile
Private browsing on a new profile
Private browsing on a used profile

Sign off

Criteria

Check list

• All test cases should be executed
• All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

• Link for the tests run
  • Full Test suite, use template from []

Merge to Beta Sign-off List of OSes that will be covered by testing

• Link for the tests run
  • Full Test suite

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation	01-04-2019
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release	03-08-2019	Email sent
QA Beta - Full Testing
QA Signoff - Beta Release