QA/Web Compat System Add-on

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date	Version	Author	Description
12/15/2016	1.0	Ovidiu Boca	Created first draft

Overview

Purpose

Detail the purpose of this document. For example:

• The test scope, focus areas and objectives
• The test responsibilities
• The test strategy for the levels and types of test for this release
• The entry and exit criteria
• The basis of the test estimates
• Any risks, issues, assumptions and test dependencies
• The test schedule and major milestones
• The test deliverables

Scope

This wiki details the testing that will be performed by the project team for the webcompat system add-on project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

• By using the go faster system add-ons infrastructure and delivery mechanism, we can add and remove top web property site patches via out of band updates (i.e., not tied to the release train cadence). The intention is to provide a means to fix urgent issues until outreach efforts are successful, or until platform bugs or regressions are shipping in release builds as fixed.

Ownership

Development
Manager:

[Mike Taylor (irc: miketaylr)

Developers contacts:

Eric Tsai

Dennis Schubert (irc:denschub)

QA:
Rares Bologa - PM for QA team

Brindusa Tot QA team lead

Ovidiu Boca, QA

Feature short description

By using the go faster system add-ons infrastructure and delivery mechanism, we can add and remove top web property site patches via out of band updates (i.e., not tied to the release train cadence). The intention is to provide a means to fix urgent issues until outreach efforts are successful, or until platform bugs or regressions are shipping in release builds as fixed.

Testing summary

Scope of Testing

In Scope

The testing effort for Web Compat Add-on will be invested on the following areas:

• functionality testing: Basic and advanced functionality must be verified to see if they work according to the existing requirements.

• compatibility testing: We will verify if this add-on is compatible with all OS's and with x64 and x32 builds.

We will focus our work around the xpi that we will receive from the development team and we will test if this add-on is installed correctly, sites that are qualified for override are working, and also on the notification in the web console. After this will land and will be a system add-on a smoke testing will be performed.

Out of Scope

Following areas/features are considered out of scope and will not be considered as testing zones to be handled in this test plan.

1. Accessibility testing
2. Security testing

Requirements for testing

Environments

• Ubuntu
• OS X
• Windows

Channel dependent settings (configs) and environment setups

Nightly

text

Aurora

text

Beta

text

Post Beta / Release

text

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref	Function	Test Objective	Test Type	RAC	Owners
TO-1	Sites that need the override agent to work are properly displayed.	Firefox works on sites that have problems based on browser compatibility.	Manual	RAC-01, RAC-02	Eng Team
TO-2	The web compat system add-on is not affected if Firefox updates, crashes or restores.	Firefox works properly after update,crash or restore.	Manual	RAC-03	Eng Team
TO-3	The webcompat system add-on is working in private browsing.	The system add-on is not affected if Firefox is in private browsing mode.	Manual	RAC-04	Eng Team

Impact Score Breakdown:

• An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
• An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
• An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

RAC: Refers to the "Risk Assessment and Coverage" item that is covering.

Risk Assessment and Coverage

ID	Description / Threat Description	Covered by Test Objective	Magnitude	Probability	Priority	Impact Score
RAC-1	Installing the add-on may not work.	TO-1	3-High	1-Unlikely	3-High	9
RAC-2	The override of the user agent will not work on all sites.	TO-1	3-High	2-Possible	3-High	18
RAC-3	Crashes, updates or session restores can affect the functionality of the add-on.	TO-2	3-High	2-Possible	3-High	18
RAC-4	Private browsing interferes with the add-on.	TO-3	3-High	2-Possible	3-High	18

Values:

• Magnitude: 1- Low , 2-Moderate, 3-High

• Probability: 1-Unlikely, 2-Possible, 3-Almost Certain

• Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

• An impact value of 1 or 2 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
• An impact value of 4, 8 or 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
• An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Builds

Testing will be performed on latest Nightly - 53.0a1(link) using the add-on received from developers team:

• Link to the webcompat xpi: link

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project	12.15.2016
Study documentation/specs received from developers	12.15.2016	12.19.2016
QA - Test plan creation	12.19.2016	12.27.2016
QA - Test cases/Env preparation	12.21.2016	12.21.2016
QA - Add-on testing using Nightly builds	12.21.2016	01.06.2017
QA - Aurora Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	TestRail/ Google docs
Test case execution	TestRail
Bugs management	Bugzilla

Status

Overview

Tested the add-on on latest Nightly - 53.0a1 - from 01.05.2017 - link
Tests were performed on Windows 10 x32bit, Windows 7 x64bit, Ubuntu 16.04 x64, Mac 10.12 Due to some issues with implementation, the development team removed the developer console notification for this version(Nightly 54.0a1). Notification is added in Browser Console. The plan is to re-adding the original console in their second version. version,

References

• List and links for specs

 List and links for available specs - documents, user stories, specifications

• Meta bug 1287966 - (gofaster) Tracking Bug for WebCompat Go Faster Add-on

Testcases

Full testsuites can be found on TestRail at link

Test Areas

Test Areas	Covered	Details
Private Window	Yes
Multi-Process Enabled	Yes
Multi-process Disabled	Yes
Theme (high contrast)	No
UI
Mouse-only operation	No
Keyboard-only operation	No
Display (HiDPI)	No
Interaction (scroll, zoom)	No
Usable with a screen reader	No
Usability and/or discoverability testing	No
RTL build testing	No
Help/Support
Help/support interface required	No
Support documents planned(written)	No
Install/Upgrade
Feature upgrades/downgrades data as expected	Yes
Does sync work across upgrades	No
Requires install testing	Yes	We covered this for installing the add-on(xpi)
Affects first-run or onboarding	N/A
Does this affect partner builds? Partner build testing	N/A
Enterprise
Enterprise administration	N/A
Network proxies/autoconfig	N/A
ESR behavior changes	N/A
Locked preferences	N/A
Data Monitoring
Temporary or permanent telemetry monitoring	No
Telemetry correctness testing	No
Server integration testing	No
Offline and server failure testing	No
Load testing	No
Add-ons
Addon API required?	No
Comprehensive API testing	No
Permissions	No
Testing with existing/popular addons	No
Security		Mike Taylor said that they don't need a security review at this point .
3rd-party security review	N/A
Privilege escalation testing	N/A
Fuzzing	N/A
Web Compatibility
Testing against target sites	Yes
Survey of many sites for compatibility	No
Interoperability
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.	No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS	No
Interaction of this feature with other browser features	No

Bug Work

Tracking bug – meta Bug 1308271 - Ship v1 of webcompat go faster addon

Logged bugs ( blocking 1308271 )

Full Query

ID	Priority	Component	Assigned to	Summary	Status	Target milestone
1296011	--	Interventions	Dennis Schubert [:denschub][Vacation, back Sep. 22]	Create prototype go faster addon to do UA overrides	RESOLVED	---
1307788	--	Interventions	Eric Tsai (no more review request accepted)	Implement a Developer Tools notification when changes are applied	RESOLVED	---
1307792	--	Interventions	Dennis Schubert [:denschub][Vacation, back Sep. 22]	Implement a way to disable all site patches	RESOLVED	---
1316252	--	Interventions		Submit webcompat addon v1 patch from github repo to mozilla-central	RESOLVED	---
1317572	--	Interventions	Dennis Schubert [:denschub][Vacation, back Sep. 22]	Add UA overrides for Go Faster add-on V1	RESOLVED	---
1328853	--	Interventions	Dennis Schubert [:denschub][Vacation, back Sep. 22]	Console message gets logged every time, not only after overrides	VERIFIED	---
1330076	--	General	Luca Greco [:rpl] [:luca] [:lgreco]	Rewrite test_chrome_ext_shutdown_cleanup.html into an xpcshell test	RESOLVED	mozilla53

7 Total; 0 Open (0%); 6 Resolved (85.71%); 1 Verified (14.29%);

Bug fix verification

Full Query

ID	Priority	Component	Assigned to	Summary	Status	Target milestone
1296011	--	Interventions	Dennis Schubert [:denschub][Vacation, back Sep. 22]	Create prototype go faster addon to do UA overrides	RESOLVED	---
1307788	--	Interventions	Eric Tsai (no more review request accepted)	Implement a Developer Tools notification when changes are applied	RESOLVED	---
1307792	--	Interventions	Dennis Schubert [:denschub][Vacation, back Sep. 22]	Implement a way to disable all site patches	RESOLVED	---
1317572	--	Interventions	Dennis Schubert [:denschub][Vacation, back Sep. 22]	Add UA overrides for Go Faster add-on V1	RESOLVED	---
1328853	--	Interventions	Dennis Schubert [:denschub][Vacation, back Sep. 22]	Console message gets logged every time, not only after overrides	VERIFIED	---
1330076	--	General	Luca Greco [:rpl] [:luca] [:lgreco]	Rewrite test_chrome_ext_shutdown_cleanup.html into an xpcshell test	RESOLVED	mozilla53

6 Total; 0 Open (0%); 5 Resolved (83.33%); 1 Verified (16.67%);

Sign off

Criteria

Checklist

• All test cases should be executed
• Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
• All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

• Link for the tests run
  • Full Test suite, in TestRail at link
  • A Test Report can be found in Test Rail at link
  • Regression Test suite, if needed/available

Merge to Aurora Sign-off

• Testing covered the following OSes:
  • Test Results

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)	[DONE]
Testing Infrastructure setup	[DONE]
Test Plan Creation	[DONE]
Test Cases Creation	[DONE]
Full Functional Tests Execution	[DONE]
Automation Coverage	[DONE]	Automated tests to ensure that pages are loaded with the add-on installed.
Performance Testing	N/A
All Defects Logged	[DONE]
Critical/Blockers Fixed and Verified	[DONE]
Metrics/Telemetry
QA Signoff - Nightly Release
QA Signoff - Mid-Aurora
QA Aurora - Full Testing
QA Signoff - Aurora Release
QA Beta - Full Testing
QA Signoff - Beta Release