Add-ons/QA/Testplan/Recommendations on about:addons (themes & extensions): Difference between revisions
No edit summary |
No edit summary |
||
| Line 163: | Line 163: | ||
| Test plan creation || Mozilla wiki | | Test plan creation || Mozilla wiki | ||
|- | |- | ||
| Test case creation ||[ | | Test case creation ||[] | ||
|- | |- | ||
| Test case execution || [[ | | Test case execution || [[] | ||
|- | |- | ||
| Bugs management || [https://bugzilla.mozilla.org/show_bug.cgi?id= | | Bugs management || [https://bugzilla.mozilla.org/show_bug.cgi?id=1523406 Bugzilla] | ||
|} | |} | ||
| Line 182: | Line 182: | ||
= References = | = References = | ||
* List and links for specs | * List and links for specs | ||
OnePager - [https://docs.google.com/document/d/14BFLvv6njePn4ItZLyneRZmKNo4MKx_2IvjBtUWsm1c/edit Gdocs] | |||
Invision - [https://mozilla.invisionapp.com/share/SEPLAO9WKPY#/screens/336575346_Overview Presentation] | |||
* {{Bug| | * {{Bug|1523406}} - [meta] Recommendations on about:addons in Extensions & Themes | ||
<bugzilla> | <bugzilla> | ||
{ | { | ||
"blocks":[ | "blocks":[1523406], | ||
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone" | "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone" | ||
} | } | ||
Revision as of 11:40, 30 January 2019
Revision History
| Date | Version | Author | Description |
|---|---|---|---|
| 30-01-2019 | 1.0 | Vlad Jiman | Created first draft |
Please note: This document is still under works and sections might contain placeholders and not accurate data
Overview
The purpose of this feature is to add recommended extensions/themes on the appropriate tabs, as it is currently, suggestions are only shown on the 'Get Add-ons' tab. This feature is dependent on the modifications that will occur on the about:addons HTML page and any QA process will only begin once that is implemented.
Purpose
This document purports to detail the test approach for the 'Recommendations on about:addons page' feature, including Entry/Exit criteria, Scope for testing, links to test cases etc.
Entry Criteria
- QA has access to all the PRDs, mocks and related documents
- The feature has landed on Nightly
- AMO parts have landed on dev
Exit Criteria
- All the bugs against the feature have been triaged
- All the P1/P2 bugs have been fixed
- All the resolved bugs have been verified by QA
- The find/fixed rate is going down over a predefined period of time
Acceptance Criteria
This section broadly outlines when the product is ready to ship
- QA has signed off
- All the required Telemetry is in place
- All info is localized at least for a pre-defined set of locales
- All the necessary PR/blogposts have been sent out
Scope
This section describes what parts of the feature will be tested and what parts won't be.
what's in scope?
- Validation of the entire feature ( Recommendations on about:addons ) along with all of the attached UI features.
what's out of scope?
- Security testing
- Device testing
- Load testing
Ownership
Dev Lead: Mark Striemer ; irc nick:mstriemer
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Vlad Jiman; irc nick :CosminB
Requirements for testing
Environments
OSes covered: Windows, Mac OS X, Linux
Channel dependent settings (configs) and environment setups
Nightly
security.signed_app_signatures.policy with the default value 2
Beta
security.signed_app_signatures.policy with the default value 2
Release
Post Beta / Release
The feature is enabled by default.
Test Strategy
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
| Ref | Function | Test Objective | Test Type | Owners |
|---|---|---|---|---|
| TO-1 | Private Browsing testing | What is expected to happen? - Default suggestions | Manual | Add-ons QA Team |
| TO-2 | Profile testing | Including localization (Profiles made using different regions, having installed region-specific addons) | Manual | Add-ons QA Team |
| TO-3 | Toggle suggestions ON/OFF | Validate the on/off toggle of the suggestions from the settings menu for above scenarios | Manual | Add-ons QA Team |
| TO-4 | Performance testing | To verify that the system still performs under load | Automated | Add-ons QA Team |
Test profiles
| Ref | Profile Objective |
|---|---|
| TP-1 | Validate if default recommendations are shown for fresh profiles |
| TP-2 | Validate if personalized recommendations are shown for profile that has bookmarks and navigation history |
| TP-3 | Validate if personalized recommendations are shown for profile that has installed extensions |
| TP-4 | Validate if personalized recommendations are shown for profile that has both installed extensions and bookmarks |
| TP-5 | Validate if personalized recommendations are shown for localization profile |
| TP-6 | Validate if default recommendations are shown when the service is unavailable |
Builds
This section should contain links for builds with the feature -
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
| Project phase | Start Date | End Date |
|---|---|---|
| Start project | 10/18/2018 | |
| Study documentation/specs received from developers | 11/02/2018 | |
| QA - Test plan creation | 12/23/2018 | |
| QA - Test cases/Env preparation | 01/03/2019 | |
| QA - Nightly Testing | ||
| QA - Beta Testing | ||
| Release Date |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | [] |
| Test case execution | [[] |
| Bugs management | Bugzilla |
Status
Overview
Track the dates and build number where feature was released to Nightly Track the dates and build number where feature was merged to Release/Beta
Risk analysis
Identify the high-risk assumptions Identify existing bugs on the feature with high risk Identify if other areas are affected by the fix
References
* List and links for specs OnePager - Gdocs Invision - Presentation * bug 1523406 - [meta] Recommendations on about:addons in Extensions & Themes
8 Total; 1 Open (12.5%); 5 Resolved (62.5%); 2 Verified (25%);
Testcases
Overview
Summary of testing scenarios
Test Areas
| Test Areas | Covered | Details |
|---|---|---|
| Private browsing | ||
| Profile testing | ||
| System toggled ON/OFF | ||
| Performance testing | ||
| Cookie's round trip validation |
Bug Work
Tracking bug - []
Bug fix verification
[Verified] [ Bug xxxxxxx] - Display permissions prompt for webextensions installed using mozAddonManager
- ↳ 2017-01-10: verified fixed on 53.0a1 across platforms
[Verified] [ Bug xxxxxxx] - Prompt users with permissions for third-party webextensions installs
- ↳ 2015-04-21: verified fixed on 53.0a1 across platforms
Logged bugs
[ Bug xxxxxxx] - Misaligned icon and webextension name in permissions doorhanger
Sign off
Criteria
Check list
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite, use template from []
Merge to Beta Sign-off
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite
Checklist
| Exit Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | ||
| Testing Infrastructure setup | ||
| Test Plan Creation | 04-01-2019 | |
| Test Cases Creation | ||
| Full Functional Tests Execution | ||
| Automation Coverage | ||
| Performance Testing | ||
| All Defects Logged | ||
| Critical/Blockers Fixed and Verified | ||
| Metrics/Telemetry | ||
| QA Signoff - Nightly Release | Email to be sent | |
| QA Beta - Full Testing | ||
| QA Signoff - Beta Release | Email to be sent |