MozillaWiki

Add-ons/QA/Testplan/Recommendations on about:addons (themes & extensions): Difference between revisions

Page Discussion

Add-ons/QA/Testplan/Recommendations on about:addons (themes & extensions) (view source)

Revision as of 12:56, 30 April 2019

2,249 bytes removed ,  30 April 2019
no edit summary
No edit summary
No edit summary
 
(10 intermediate revisions by one other user not shown)
Line 5: Line 5:
! Date !! Version !! Author !! Description  
! Date !! Version !! Author !! Description  
|-
|-
| 30-01-2019 || 1.0 || Vlad Jiman || Created first draft
| 02-04-2019 || 2.0 || Vlad Jiman || Updated all sections
|-
|-
|}
|}
Line 12: Line 12:
= Overview =
= Overview =


The purpose of this feature is to add recommended extensions/themes on the appropriate tabs, as it is currently, suggestions are only shown on the 'Get Add-ons' tab. This feature is dependent on the modifications that will occur on the about:addons HTML page and any QA process will only begin once that is implemented.
The purpose of this feature is to add recommended extensions/themes on the appropriate tabs. As it is currently, suggestions are only shown on the 'Get Add-ons' tab. This feature is dependent on the modifications that will occur on the about:addons HTML page and any QA process will only begin once that is implemented.


== Purpose ==
== Purpose ==


This document purports to detail the test approach for the 'Recommendations on about:addons page' feature, including Entry/Exit criteria, Scope for testing, links to test cases etc.
This document proposes to detail a test approach for the “Recommendations on about:addons” feature, which includes Entry/Exit/Acceptance criteria, Testing scope, references to testcases, etc.


=== Entry Criteria ===
=== Entry Criteria ===
* QA has access to all the PRDs, mocks and related documents
* QA has access to all PRDs, mocks and related documentation
* The feature has landed on Nightly
* The feature has landed on Nightly
* AMO parts have landed on dev


=== Exit Criteria ===
=== Exit Criteria ===
* All the bugs against the feature have been triaged
* All feature related bugs have been triaged
* All the P1/P2 bugs have been fixed
* All P1/P2 bugs have been fixed
* All the resolved bugs have been verified by QA
* All resolved bugs have been verified by QA
* The find/fixed rate is going down over a predefined period of time
* The find/fixed bug ratio shows a descending trend over a defined time period


=== Acceptance Criteria ===
=== Acceptance Criteria ===
This section broadly outlines when the product is ready to ship
This section proposes to highlight the criteria concerning the shipment readiness status of the product.
* QA has signed off
* QA has signed off
* All the required Telemetry is in place
* All the required Telemetry is in place
* All info is localized at least for a pre-defined set of locales
* All the necessary PR/blogposts have been sent out


== Scope ==
== Scope ==
This section describes what parts of the feature will be tested and what parts won't be.
This section outlines which parts of the new implemented feature will or will not be tested.


===what's in scope?===
===What is in scope===
* Validation of the entire feature ( Recommendations on about:addons ) along with all of the attached UI features.
* Validation of the entire ‘Recommendations on about:addons’ feature, including all related UI features


===what's out of scope?===
===What is out of scope===
* Security testing
* Security testing
* Device testing
* Device testing
* Load testing
* Performance testing


== Ownership ==
== Ownership ==
Line 55: Line 52:
= Requirements for testing =
= Requirements for testing =
== Environments ==
== Environments ==
OSes covered: Windows, Mac OS X, Linux
Covered OSes: Windows, Mac OS X, Linux
 
== Channel dependent settings (configs) and environment setups ==
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
 
=== Nightly ===
<div class="mw-collapsible-content">
security.signed_app_signatures.policy with the default value 2
</div>
 
=== Beta ===
<div class="mw-collapsible-content">
security.signed_app_signatures.policy with the default value 2
</div>
 
=== Release ===
<div class="mw-collapsible-content">
 
</div>
 
=== Post Beta / Release ===
<div class="mw-collapsible-content">
The feature is enabled by default.
</div>
</div>


= Test Strategy =  
= Test Strategy =  


== Test Objectives ==
== Test Objectives ==
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
This section details the progression test objectives that will be covered.
This could be documented in bullet form or in a table similar to the one below.


{| class="wikitable"
{| class="wikitable" style="width:60%"
|-
|-
! Ref !! Function !! Test Objective !! Test Type !! Owners  
! Ref !! Function !! Test Objective !! Test Type !! Owners  
|-
|-
| TO-1 || Private Browsing testing || What is expected to happen? - Default suggestions || Manual || Add-ons QA Team
| TO1 || Private Browsing Testing || Validate if default recommendations are displayed  || Manual || Add-ons QA Team
|-
|-
| TO-2 || Profile testing || Including localization (Profiles made using different regions, having installed region-specific addons) || Manual || Add-ons QA Team
| TO2 || Profile Testing || Validate if default/personalized recommendations are displayed for new profiles/profiles with installed add-ons/browsing history/bookmarks  || Manual || Add-ons QA Team
|-
|-
| TO-3 || Toggle suggestions ON/OFF|| Validate the on/off toggle of the suggestions from the settings menu for above scenarios || Manual || Add-ons QA Team
| TO3 || Feature On/Off Toggling || Validate if activating/deactivating the feature yields default/personalized recommendations  || Manual || Add-ons QA Team
|-
|-
| TO-4 || Performance testing|| To verify that the system still performs under load|| Automated || Add-ons QA Team
|}
|}


== Test Profiles ==


 
{| class="wikitable" style="width:60%"
== Test profiles ==
 
{| class="wikitable"
|-
|-
! Ref !! Profile Objective
! Ref !! Profile Objective
|-
|-
| TP-1 || Validate if default recommendations are shown for fresh profiles
| TP1 || Validate if default recommendations are displayed for new profiles
|-
| TP2 || Validate if personalized recommendations are displayed for profiles that have both bookmarks and navigation history
|-
|-
| TP-2 || Validate if personalized recommendations are shown for profile that has bookmarks and navigation history
| TP3 || Validate if personalized recommendations are displayed for profiles that have installed add-ons (both extensions and themes)
|-
|-
| TP-3 || Validate if personalized recommendations are shown for profile that has installed extensions
| TP4 || Validate if personalized recommendations are displayed for profiles that have both installed add-ons and bookmarks
|-
|-
| TP-4 || Validate if personalized recommendations are shown for profile that has both installed extensions and bookmarks
| TP5 || Validate if default recommendations are displayed when the feature is deactivated
|-
|-
| TP-5 || Validate if personalized recommendations are shown for localization profile
| TP6 || Validate if ‘Create your own theme’ link is displayed and works as intended
|-
|-
| TP-6 || Validate if default recommendations are shown when the service is unavailable
|}
|}


== Builds ==
== Builds ==
This section should contain links for builds with the feature -  
This section should contain links for builds with the feature -  
* [http://archive.mozilla.org/pub/mobile/nightly/latest-date-android-api-16/ Link] for Nightly builds
* [https://archive.mozilla.org/pub/firefox/nightly/2019/ Link] for Nightly builds
* [http://archive.mozilla.org/pub/mobile/candidates/ Link] for Beta builds
* [https://archive.mozilla.org/pub/firefox/candidates/ Link] for Beta builds
* [http://archive.mozilla.org/pub/mobile/releases/ Link] for Release builds
* [https://archive.mozilla.org/pub/firefox/releases/ Link] for Release builds


== Test Execution Schedule ==
== Test Execution Schedule ==
The following table identifies the anticipated testing period available for test execution.
The below table outlines the anticipated testing time frame available for test execution.  
{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
|-
|-
Line 156: Line 126:


== Testing Tools ==
== Testing Tools ==
Detail the tools to be used for testing, for example see the following table:
Exemplifies the tools used for test suite creation/execution.
 
{| class="wikitable" style="width:50%"
{| class="wikitable" style="width:50%"
|-
|-
Line 163: Line 134:
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki
|-
|-
| Test case creation ||[https://testrail.stage.mozaws.net/index.php?/suites/view/3160 TestRail]
| Test case creation ||[https://testrail.stage.mozaws.net/index.php?/suites/view/4353&group_by=cases:section_id&group_order=asc TestRail]
|-
|-
| Test case execution || [[https://testrail.stage.mozaws.net/index.php?/suites/view/3160 TestRail]
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/suites/view/4353&group_by=cases:section_id&group_order=asc TestRail]
|-
|-
| Bugs management || [https://bugzilla.mozilla.org/show_bug.cgi?id=1499934 Bugzilla]
| Bugs management || [https://bugzilla.mozilla.org/show_bug.cgi?id=1523406 Bugzilla]
|}
|}
= Status =
== Overview ==
Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Release/Beta
== Risk analysis ==
Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix


= References =
= References =
  * List and links for specs
  * List and links for specs
   PRD - [https://docs.google.com/document/d/1U7XaMYdzPQmddt9KOTW6xdzgh9arcY0GWk70BwsuUQo/edit# Gdocs]
   OnePager - [https://docs.google.com/document/d/14BFLvv6njePn4ItZLyneRZmKNo4MKx_2IvjBtUWsm1c/edit Gdocs]
   Install flow - [https://docs.google.com/presentation/d/1KgrBsm62z5oIxaSlvKJEdmL8bp9XXuoad6SxYeCB6Sc/edit#slide=id.g1fbe570521_0_86 Presentation]
   Invision - [https://mozilla.invisionapp.com/share/SEPLAO9WKPY#/screens/336575346_Overview Presentation]
    
    
   
   
  * {{Bug|1499934}} - [META] Changes that need to ride the train for Disco-TAAR
  * {{Bug|1523406}} - [meta] Recommendations on about:addons in Extensions & Themes
 
<bugzilla>
<bugzilla>
     {
     {
         "blocks":[1499934],
         "blocks":[1523406],
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
     }
     }
Line 196: Line 158:
= Testcases =  
= Testcases =  
== Overview ==
== Overview ==
Summary of testing scenarios
The test suite proposes a series of test cases devised to cover scenarios where default or recommended extensions will be displayed, in order to validate the feature.
 
* Validation of default/personalized recommendations being displayed based on profile usage
* Validation of default recommendations being displayed during private browsing
 
== Test Areas ==
== Test Areas ==
{| class="wikitable" style="width:80%"
{| class="wikitable" style="width:80%"
|-
|-
! Test Areas !! Covered !! Details
! Test Areas !! Covered !! Details
|-
|-
| '''Private browsing'''
| '''Private Browsing'''
|style="text-align:center;" | ||  
|style="text-align:center;" | ||  
|-
|-
| '''Profile testing'''
| '''Profile Testing'''
|style="text-align:center;" |  ||
|-
| '''System toggled ON/OFF'''
|style="text-align:center;" |  ||  
|style="text-align:center;" |  ||  
|-
|-
| '''Performance testing'''
| '''Feature On/Off Toggling'''
|style="text-align:center;" |  ||  
|style="text-align:center;" |  ||  
|-
| '''Cookie's round trip validation'''
||  ||
|-
|-
|}
|}
= Bug Work =
Tracking bug - []
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Bug fix verification ======
<div class="mw-collapsible-content">
<font color="green">'''[Verified]'''</font> [ Bug xxxxxxx] - Display permissions prompt for webextensions installed using mozAddonManager
:↳ <small>'''2017-01-10'''</small>: verified fixed on 53.0a1 across platforms
<font color="green">'''[Verified]'''</font> [ Bug xxxxxxx] - Prompt users with permissions for third-party webextensions installs
:↳ <small>'''2015-04-21'''</small>: verified fixed on 53.0a1 across platforms
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Logged bugs ======
<div class="mw-collapsible-content">
[ Bug xxxxxxx] - Misaligned icon and webextension name in permissions doorhanger<br/>
</div>
</div>


= Sign off =
= Sign off =
Line 246: Line 184:
Check list
Check list
* All test cases should be executed
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
== Results ==
'''Nightly testing'''<br />
List of OSes that will be covered by testing<br />
*Link for the tests run
** Full Test suite, use template from []
'''Merge to Beta Sign-off'''
List of OSes that will be covered by testing<br />
*Link for the tests run
** Full Test suite


== Checklist ==
== Checklist ==
ACornestean
67

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1206930...1211471"