Add-ons/QA/Testplan/Add-ons ContentScript
Revision History
Date | Version | Author | Description |
---|---|---|---|
16/05/2017 | 1.0 | Vasilica Mihasca | Created first draft |
Overview
Currently, we load any script which doesn't run at document_start asynchronously. That means that we should (but apparently don't) be able to use off-main-thread compilation for at least those.
Purpose
A better approach would be to parse once and then clone the script each time we want to run it.
Entry Criteria
- QA has access to all the PRDs, mocks and related documents
- The feature has landed on Nightly
Exit Criteria
- All the bugs against the feature have been triaged
- All the P1/P2 bugs have been fixed
- All the resolved bugs have been verified by QA
- The find/fixed rate is going down over a predefined period of time
Acceptance Criteria
This section broadly outlines when the product is ready to ship
- QA has signed off
- All the required Telemetry is in place
- All info is localized at least for a pre-defined set of locales
- All the necessary PR/blogposts have been sent out
Scope
This section describes what parts of the feature will be tested and what parts won't be.
what's in scope?
- Webextensions/Add-ons installation is successfully
- Webextensions/Add-ons run without any issue
- Regression testing for the top 20 webextensions/add-ons
what's out of scope?
- Security testing
- Performance testing
Ownership
Dev Lead: > Kris Maglione; irc nick
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Vasilica Mihasca; irc nick :vasilica_mihasca
Add-ons QA: Valentina Peleski; irc nick :ValentinaV
Requirements for testing
Environments
OSed covered: Windows, Mac OS X, Linux
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
Ref | Function | Test Objective | Evaluation Criteria | Test Type | Owners |
---|
Test Strategy
Here are the main ways in which we will ascertain that OOP functions as expected
- Test the webextensions/add-ons which uses a large amount of content scripts that are not affected by https://bugzilla.mozilla.org/show_bug.cgi?id=1333990 changes.
- Check that there are no regressions related to content script while running webextension/add-ons.
Builds
This section should contain links for builds with the feature -
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | ||
Study documentation/specs received from developers | ||
QA - Test plan creation | 02-03-2017 | |
QA - Test cases/Env preparation | ||
QA - Nightly Testing | ||
QA - Aurora Testing | ||
QA - Beta Testing | ||
Release Date |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | [ TestRail] / [Google docs] / [etherpad] |
Test case execution | [TestRail] |
Bugs management | Bugzilla / Github |
Status
Overview
Track the dates and build number where feature was released to Nightly Track the dates and build number where feature was merged to Beta/Release
Risk analysis
- The test strategy recommended by devs is based on spot-checking and regression testing.
- Mitigation stragety - QA will test all webextensions/add-ons which run content scripts to make sure that no issues are encountered.
- QA will test all webextensions with the most number of users to ensure that the most popular webextensions/add-ons are covered.
References
* List and links for specs PRD - * bug 1333990 - We re-parse content scripts every time we run them
6 Total; 1 Open (16.67%); 3 Resolved (50%); 2 Verified (33.33%);
Bug Work
Tracking bug - bug 1333990
Bug fix verification
Logged bugs
Sign off
Criteria
Check list
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
List of OSes that will be covered by testing
- Link for the tests run
- Daily Smoke, use template from [link]
- Full Test suite, use template from [ link]
- Regression Test suite, if needed/available
Merge to Aurora Sign-off
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite
Checklist
Exit Criteria | Status | Notes/Details |
---|---|---|
Testing Prerequisites (specs, use cases) | ||
Testing Infrastructure setup | ||
Test Plan Creation | ||
Test Cases Creation | ||
Full Functional Tests Execution | ||
Automation Coverage | ||
Performance Testing | ||
All Defects Logged | ||
Critical/Blockers Fixed and Verified | ||
Metrics/Telemetry | ||
QA Signoff - Nightly Release | Email to be sent | |
QA Aurora - Full Testing | ||
QA Signoff - Aurora Release | Email to be sent | |
QA Beta - Full Testing | ||
QA Signoff - Beta Release | Email to be sent |