Add-ons/QA/Testplan/Recommendations on about:addons (themes & extensions): Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Created page with "'''Revision History''' {| class="wikitable" style="width:60%" |- ! Date !! Version !! Author !! Description |- | 30-01-2019 || 1.0 || Vlad Jiman || Created first draft |- |}...")
 
No edit summary
 
(13 intermediate revisions by one other user not shown)
Line 5: Line 5:
! Date !! Version !! Author !! Description  
! Date !! Version !! Author !! Description  
|-
|-
| 30-01-2019 || 1.0 || Vlad Jiman || Created first draft
| 02-04-2019 || 2.0 || Vlad Jiman || Updated all sections
|-
|-
|}
|}


'''Please note: This document is still under works and sections might contain placeholders and not accurate data'''
'''<big>Please note: This document is still under works and sections might contain placeholders and not accurate data</big>'''
= Overview =
= Overview =


To enable recommendations from AMO for TAAR, we need to supply a preference for the user to enable/disable recommendations and provide a mechanism to make the "client ID" accessible by AMO (to return TAAR recommendations). There are two separate parts, each with its own bug that are dependencies of the tracking bug.
The purpose of this feature is to add recommended extensions/themes on the appropriate tabs. As it is currently, suggestions are only shown on the 'Get Add-ons' tab. This feature is dependent on the modifications that will occur on the about:addons HTML page and any QA process will only begin once that is implemented.


== Purpose ==
== Purpose ==


This document purports to detail the test approach for the 'Recommendations on about:addons page' feature, including Entry/Exit criteria, Scope for testing, links to test cases etc.
This document proposes to detail a test approach for the “Recommendations on about:addons” feature, which includes Entry/Exit/Acceptance criteria, Testing scope, references to testcases, etc.


=== Entry Criteria ===
=== Entry Criteria ===
* QA has access to all the PRDs, mocks and related documents
* QA has access to all PRDs, mocks and related documentation
* The feature has landed on Nightly
* The feature has landed on Nightly
* AMO parts have landed on dev


=== Exit Criteria ===
=== Exit Criteria ===
* All the bugs against the feature have been triaged
* All feature related bugs have been triaged
* All the P1/P2 bugs have been fixed
* All P1/P2 bugs have been fixed
* All the resolved bugs have been verified by QA
* All resolved bugs have been verified by QA
* The find/fixed rate is going down over a predefined period of time
* The find/fixed bug ratio shows a descending trend over a defined time period


=== Acceptance Criteria ===
=== Acceptance Criteria ===
This section broadly outlines when the product is ready to ship
This section proposes to highlight the criteria concerning the shipment readiness status of the product.
* QA has signed off
* QA has signed off
* All the required Telemetry is in place
* All the required Telemetry is in place
* All info is localized at least for a pre-defined set of locales
* All the necessary PR/blogposts have been sent out


== Scope ==
== Scope ==
This section describes what parts of the feature will be tested and what parts won't be.
This section outlines which parts of the new implemented feature will or will not be tested.


===what's in scope?===
===What is in scope===
* Validation of the entire feature ( Recommendations on about:addons ) along with all of the attached UI features.
* Validation of the entire ‘Recommendations on about:addons’ feature, including all related UI features


===what's out of scope?===
===What is out of scope===
* Security testing
* Security testing
* Device testing
* Device testing
* Load testing
* Performance testing


== Ownership ==
== Ownership ==
Line 55: Line 52:
= Requirements for testing =
= Requirements for testing =
== Environments ==
== Environments ==
OSes covered: Windows, Mac OS X, Linux
Covered OSes: Windows, Mac OS X, Linux
 
== Channel dependent settings (configs) and environment setups ==
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
 
=== Nightly ===
<div class="mw-collapsible-content">
security.signed_app_signatures.policy with the default value 2
</div>
 
=== Beta ===
<div class="mw-collapsible-content">
security.signed_app_signatures.policy with the default value 2
</div>
 
=== Release ===
<div class="mw-collapsible-content">
 
</div>
 
=== Post Beta / Release ===
<div class="mw-collapsible-content">
The feature is enabled by default.
</div>
</div>


= Test Strategy =  
= Test Strategy =  


== Test Objectives ==
== Test Objectives ==
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
This section details the progression test objectives that will be covered.
This could be documented in bullet form or in a table similar to the one below.


{| class="wikitable"
{| class="wikitable" style="width:60%"
|-
|-
! Ref !! Function !! Test Objective !! Test Type !! Owners  
! Ref !! Function !! Test Objective !! Test Type !! Owners  
|-
|-
| TO-1 || Private Browsing testing || What is expected to happen? - Default suggestions || Manual || Add-ons QA Team
| TO1 || Private Browsing Testing || Validate if default recommendations are displayed  || Manual || Add-ons QA Team
|-
|-
| TO-2 || Profile testing || Including localization (Profiles made using different regions, having installed region-specific addons) || Manual || Add-ons QA Team
| TO2 || Profile Testing || Validate if default/personalized recommendations are displayed for new profiles/profiles with installed add-ons/browsing history/bookmarks  || Manual || Add-ons QA Team
|-
|-
| TO-3 || Toggle suggestions ON/OFF|| Validate the on/off toggle of the suggestions from the settings menu for above scenarios || Manual || Add-ons QA Team
| TO3 || Feature On/Off Toggling || Validate if activating/deactivating the feature yields default/personalized recommendations  || Manual || Add-ons QA Team
|-
|-
| TO-4 || Performance testing|| To verify that the system still performs under load|| Automated || Add-ons QA Team
|}
|}


== Test Profiles ==


 
{| class="wikitable" style="width:60%"
== Test profiles ==
 
{| class="wikitable"
|-
|-
! Ref !! Profile Objective
! Ref !! Profile Objective
|-
|-
| TP-1 || Validate if default recommendations are shown for fresh profiles
| TP1 || Validate if default recommendations are displayed for new profiles
|-
| TP2 || Validate if personalized recommendations are displayed for profiles that have both bookmarks and navigation history
|-
|-
| TP-2 || Validate if personalized recommendations are shown for profile that has bookmarks and navigation history
| TP3 || Validate if personalized recommendations are displayed for profiles that have installed add-ons (both extensions and themes)
|-
|-
| TP-3 || Validate if personalized recommendations are shown for profile that has installed extensions
| TP4 || Validate if personalized recommendations are displayed for profiles that have both installed add-ons and bookmarks
|-
|-
| TP-4 || Validate if personalized recommendations are shown for profile that has both installed extensions and bookmarks
| TP5 || Validate if default recommendations are displayed when the feature is deactivated
|-
|-
| TP-5 || Validate if personalized recommendations are shown for localization profile
| TP6 || Validate if ‘Create your own theme’ link is displayed and works as intended
|-
|-
| TP-6 || Validate if default recommendations are shown when the service is unavailable
|}
|}


== Builds ==
== Builds ==
This section should contain links for builds with the feature -  
This section should contain links for builds with the feature -  
* [http://archive.mozilla.org/pub/mobile/nightly/latest-date-android-api-16/ Link] for Nightly builds
* [https://archive.mozilla.org/pub/firefox/nightly/2019/ Link] for Nightly builds
* [http://archive.mozilla.org/pub/mobile/candidates/ Link] for Beta builds
* [https://archive.mozilla.org/pub/firefox/candidates/ Link] for Beta builds
* [http://archive.mozilla.org/pub/mobile/releases/ Link] for Release builds
* [https://archive.mozilla.org/pub/firefox/releases/ Link] for Release builds


== Test Execution Schedule ==
== Test Execution Schedule ==
The following table identifies the anticipated testing period available for test execution.
The below table outlines the anticipated testing time frame available for test execution.  
{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
|-
|-
Line 156: Line 126:


== Testing Tools ==
== Testing Tools ==
Detail the tools to be used for testing, for example see the following table:
Exemplifies the tools used for test suite creation/execution.
 
{| class="wikitable" style="width:50%"
{| class="wikitable" style="width:50%"
|-
|-
Line 163: Line 134:
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki
|-
|-
| Test case creation ||[https://testrail.stage.mozaws.net/index.php?/suites/view/3160 TestRail]
| Test case creation ||[https://testrail.stage.mozaws.net/index.php?/suites/view/4353&group_by=cases:section_id&group_order=asc TestRail]
|-
|-
| Test case execution || [[https://testrail.stage.mozaws.net/index.php?/suites/view/3160 TestRail]
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/suites/view/4353&group_by=cases:section_id&group_order=asc TestRail]
|-
|-
| Bugs management || [https://bugzilla.mozilla.org/show_bug.cgi?id=1499934 Bugzilla]
| Bugs management || [https://bugzilla.mozilla.org/show_bug.cgi?id=1523406 Bugzilla]
|}
|}
= Status =
== Overview ==
Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Release/Beta
== Risk analysis ==
Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix


= References =
= References =
  * List and links for specs
  * List and links for specs
   PRD - [https://docs.google.com/document/d/1U7XaMYdzPQmddt9KOTW6xdzgh9arcY0GWk70BwsuUQo/edit# Gdocs]
   OnePager - [https://docs.google.com/document/d/14BFLvv6njePn4ItZLyneRZmKNo4MKx_2IvjBtUWsm1c/edit Gdocs]
   Install flow - [https://docs.google.com/presentation/d/1KgrBsm62z5oIxaSlvKJEdmL8bp9XXuoad6SxYeCB6Sc/edit#slide=id.g1fbe570521_0_86 Presentation]
   Invision - [https://mozilla.invisionapp.com/share/SEPLAO9WKPY#/screens/336575346_Overview Presentation]
    
    
   
   
  * {{Bug|1499934}} - [META] Changes that need to ride the train for Disco-TAAR
  * {{Bug|1523406}} - [meta] Recommendations on about:addons in Extensions & Themes
 
<bugzilla>
<bugzilla>
     {
     {
         "blocks":[1499934],
         "blocks":[1523406],
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
     }
     }
Line 196: Line 158:
= Testcases =  
= Testcases =  
== Overview ==
== Overview ==
Summary of testing scenarios
The test suite proposes a series of test cases devised to cover scenarios where default or recommended extensions will be displayed, in order to validate the feature.
 
* Validation of default/personalized recommendations being displayed based on profile usage
* Validation of default recommendations being displayed during private browsing
 
== Test Areas ==
== Test Areas ==
{| class="wikitable" style="width:80%"
{| class="wikitable" style="width:80%"
|-
|-
! Test Areas !! Covered !! Details
! Test Areas !! Covered !! Details
|-
|-
| '''Private browsing'''
| '''Private Browsing'''
|style="text-align:center;" | ||  
|style="text-align:center;" | ||  
|-
|-
| '''Profile testing'''
| '''Profile Testing'''
|style="text-align:center;" |  ||
|-
| '''System toggled ON/OFF'''
|style="text-align:center;" |  ||  
|style="text-align:center;" |  ||  
|-
|-
| '''Performance testing'''
| '''Feature On/Off Toggling'''
|style="text-align:center;" |  ||  
|style="text-align:center;" |  ||  
|-
| '''Cookie's round trip validation'''
||  ||
|-
|-
|}
|}
= Bug Work =
Tracking bug - []
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Bug fix verification ======
<div class="mw-collapsible-content">
<font color="green">'''[Verified]'''</font> [ Bug xxxxxxx] - Display permissions prompt for webextensions installed using mozAddonManager
:↳ <small>'''2017-01-10'''</small>: verified fixed on 53.0a1 across platforms
<font color="green">'''[Verified]'''</font> [ Bug xxxxxxx] - Prompt users with permissions for third-party webextensions installs
:↳ <small>'''2015-04-21'''</small>: verified fixed on 53.0a1 across platforms
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Logged bugs ======
<div class="mw-collapsible-content">
[ Bug xxxxxxx] - Misaligned icon and webextension name in permissions doorhanger<br/>
</div>
</div>


= Sign off =
= Sign off =
Line 246: Line 184:
Check list
Check list
* All test cases should be executed
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
== Results ==
'''Nightly testing'''<br />
List of OSes that will be covered by testing<br />
*Link for the tests run
** Full Test suite, use template from []
'''Merge to Beta Sign-off'''
List of OSes that will be covered by testing<br />
*Link for the tests run
** Full Test suite


== Checklist ==
== Checklist ==

Latest revision as of 12:56, 30 April 2019

Revision History

Date Version Author Description
02-04-2019 2.0 Vlad Jiman Updated all sections

Please note: This document is still under works and sections might contain placeholders and not accurate data

Overview

The purpose of this feature is to add recommended extensions/themes on the appropriate tabs. As it is currently, suggestions are only shown on the 'Get Add-ons' tab. This feature is dependent on the modifications that will occur on the about:addons HTML page and any QA process will only begin once that is implemented.

Purpose

This document proposes to detail a test approach for the “Recommendations on about:addons” feature, which includes Entry/Exit/Acceptance criteria, Testing scope, references to testcases, etc.

Entry Criteria

  • QA has access to all PRDs, mocks and related documentation
  • The feature has landed on Nightly

Exit Criteria

  • All feature related bugs have been triaged
  • All P1/P2 bugs have been fixed
  • All resolved bugs have been verified by QA
  • The find/fixed bug ratio shows a descending trend over a defined time period

Acceptance Criteria

This section proposes to highlight the criteria concerning the shipment readiness status of the product.

  • QA has signed off
  • All the required Telemetry is in place

Scope

This section outlines which parts of the new implemented feature will or will not be tested.

What is in scope

  • Validation of the entire ‘Recommendations on about:addons’ feature, including all related UI features

What is out of scope

  • Security testing
  • Device testing
  • Performance testing

Ownership

Dev Lead: Mark Striemer ; irc nick:mstriemer
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Vlad Jiman; irc nick :CosminB

Requirements for testing

Environments

Covered OSes: Windows, Mac OS X, Linux

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered.

Ref Function Test Objective Test Type Owners
TO1 Private Browsing Testing Validate if default recommendations are displayed Manual Add-ons QA Team
TO2 Profile Testing Validate if default/personalized recommendations are displayed for new profiles/profiles with installed add-ons/browsing history/bookmarks Manual Add-ons QA Team
TO3 Feature On/Off Toggling Validate if activating/deactivating the feature yields default/personalized recommendations Manual Add-ons QA Team

Test Profiles

Ref Profile Objective
TP1 Validate if default recommendations are displayed for new profiles
TP2 Validate if personalized recommendations are displayed for profiles that have both bookmarks and navigation history
TP3 Validate if personalized recommendations are displayed for profiles that have installed add-ons (both extensions and themes)
TP4 Validate if personalized recommendations are displayed for profiles that have both installed add-ons and bookmarks
TP5 Validate if default recommendations are displayed when the feature is deactivated
TP6 Validate if ‘Create your own theme’ link is displayed and works as intended

Builds

This section should contain links for builds with the feature -

  • Link for Nightly builds
  • Link for Beta builds
  • Link for Release builds

Test Execution Schedule

The below table outlines the anticipated testing time frame available for test execution.

Project phase Start Date End Date
Start project 10/18/2018
Study documentation/specs received from developers 11/02/2018
QA - Test plan creation 12/23/2018
QA - Test cases/Env preparation 01/03/2019
QA - Nightly Testing
QA - Beta Testing
Release Date

Testing Tools

Exemplifies the tools used for test suite creation/execution.

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail
Test case execution TestRail
Bugs management Bugzilla

References

* List and links for specs
  OnePager - Gdocs
  Invision - Presentation
  

* bug 1523406 - [meta] Recommendations on about:addons in Extensions & Themes
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1540253 P2 Add-ons Manager Rob Wu [:robwu] Define a property for an extension's recommended status RESOLVED mozilla69
1550911 P1 Add-ons Manager Mark Striemer [:mstriemer] [about:addons] Show recommendations on the list view VERIFIED mozilla69
1557240 -- Add-ons Manager ‘Some of these recommendations are personalized...’ is displayed when using a new profile VERIFIED ---
1561538 P3 Add-ons Manager Samuel Grasse-Haroldsen Extensions manager empty state displays the ‘Manage Your Extensions’ title RESOLVED 90 Branch
1561541 -- Add-ons Manager Mark Striemer [:mstriemer] Warning message is displayed when attempting to install Firefox Color RESOLVED ---
1571277 -- Add-ons Manager "Recommended Extensions" and "Recommended Themes" are still visible in Extension and Themes tabs in about:addons/Add-on Manager while having disabled "Allow Firefox to make personalized extension recommendations" and "Recommend extensions as you browse" RESOLVED ---
1571279 -- Add-ons Manager Move out "Recommended Extensions" and "Recommended Themes" to its proper section in "Recommendations" tab in about:addons/Add-on Manager RESOLVED ---
1602275 P3 Add-ons Manager Allow users to rate add-ons directly from the about:addons page UNCONFIRMED ---

8 Total; 1 Open (12.5%); 5 Resolved (62.5%); 2 Verified (25%);


Testcases

Overview

The test suite proposes a series of test cases devised to cover scenarios where default or recommended extensions will be displayed, in order to validate the feature.

  • Validation of default/personalized recommendations being displayed based on profile usage
  • Validation of default recommendations being displayed during private browsing

Test Areas

Test Areas Covered Details
Private Browsing
Profile Testing
Feature On/Off Toggling

Sign off

Criteria

Check list

  • All test cases should be executed
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation 04-01-2019
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release Email to be sent
Retrieved from "https://wiki.mozilla.org/index.php?title=Add-ons/QA/Testplan/Recommendations_on_about:addons_(themes_%26_extensions)&oldid=1211471"