MozillaWiki

Add-ons/Permissions/Testplan: Difference between revisions

Page Discussion

Add-ons/Permissions/Testplan (view source)

Revision as of 07:08, 2 October 2017

4,185 bytes added ,  2 October 2017
→‎Ownership
 
(47 intermediate revisions by 3 users not shown)
Line 7: Line 7:
| 01/09/2017 || 1.0 || Krupa Raj || Created first draft
| 01/09/2017 || 1.0 || Krupa Raj || Created first draft
|-
|-
|  
| 01/20/2017 || 1.1 || Vasilica Mihasca || Updates
|-
| 01/23/2017 || 1.2 || Krupa Raj || Added Acceptance Criteria
|-
| 03/22/2017 || 1.3 || Vasilica Mihasca || Added TestRail links
|-
| 08/23/2017 || 1.4 || Cosmin Badescu || Updates: References, Overview, Purpose,Requirements for testing, Ownership
|-
|}
|}


= Overview =
= Overview =
Permissions is a feature which allows webextensions to have certain special powers. The permissions key specified in the manifest.json will grant special powers for your extension. This key is an array of strings, and each string is a request for a permission.
If you request permissions using this key, then the browser may inform the user at install time that the extension is requesting certain privileges, and ask them to confirm that they are happy to grant these privileges. The browser may also allow the user to inspect an extension's privileges after installation.
* As part of this feature, the optional permissions are used when they are needed for optional features in your extension.
== Purpose ==
== Purpose ==
This document purports to detail the test approach to Webextension Permissions.
This document purports to detail the test approach to Webextension Permissions and Optional permissions including Entry/Exit criteria, Scope for testing, links to testcases etc
Detail the purpose of this document. For example:
 
* The test scope, focus areas and objectives
=== Entry Criteria ===
* The test responsibilities
* QA has access to all the PRDs, mocks and related documents
* The test strategy for the levels and types of test for this release
* The feature has landed on Nightly
* The entry and exit criteria
* AMO parts has landed on dev
* The basis of the test estimates
 
* Any risks, issues, assumptions and test dependencies
=== Exit Criteria ===
* The test schedule and major milestones
* All the bugs against the feature have been triaged
* The test deliverables
* All the P1/P2 bugs have been fixed
* All the resolved bugs have been verified by QA
* The find/fixed rate is going down over a predefined period of time
 
=== Acceptance Criteria ===
This section broadly outlines when the product is ready to ship
* QA has signed off
* All the required Telemetry is in place
* All info is localized at least for a pre-defined set of locales
* All the necessary PR/blogposts have been sent out


== Scope ==
== Scope ==
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
This section describes what parts of the feature will be tested and what parts won't be.
* What will be tested
 
* How testing will be performed
===what's in scope?===
* Installation flow from AMO
* Installation flow from Add-ons Manager
* Installation flow from Discopane
* Installation flow from Third-party websites
* Add-on updates
* Add-on Validation related to permissions during submission
* Sideloading(?)
* Webextensions have access to the permissions requested
 
===what's out of scope?===
* Security testing
* Performance testing


== Ownership ==
== Ownership ==
What resources are needed, and when (developers, QA, PMs working on the feature)
Dev Lead: [mailto:aswan@mozilla.com Andrew Swan]; irc nick :aswan<br />
QA Manager: [mailto:kraj@mozilla.com Krupa Raj]; irc nick :krupa<br />
= Testing summary =
QA Lead: [mailto:vcarciu@softvision.ro Victor Carciu]; irc nick :victorc<br />
== Scope of Testing ==
Webextensions QA: [mailto:vasilica.mihasca@softvision.ro Vasilica Mihasca]; irc nick :vasilica_mihasca<br />
=== In Scope ===
Webextensions QA: [mailto:cosmin.badescu@softvision.ro Cosmin Badescu]; irc nick :CosminB<br />
Detail what is in scope from a testing perspective for the project team.
Add-ons QA: [mailto:valentina.peleskei@softvision.ro Valentina Virlics]; irc nick :ValentinaV<br />
 
=== Out of Scope ===
Detail what is out of scope from a testing perspective for the project team. Note: if usability testing is not in the scope of testing feature.


= Requirements for testing =
= Requirements for testing =
== Environments ==
== Environments ==
Specify OSes that need to be covered
OSes covered: Windows, MacOSX, Linux
Specify Devices that need to be covered
Specify other configuration/environmental setup needed
 


== Channel dependent settings (configs) and environment setups ==
== Channel dependent settings (configs) and environment setups ==
Line 51: Line 80:
=== Nightly ===
=== Nightly ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
text
* Enable extensions.webextPermissionPrompts from about:config
* Enable extensions.webextOptionalPermissionPrompts from about:config
* Disable check for signing if testing on AMO dev or stage
</div>
</div>


=== Aurora ===
=== Aurora ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
text
* Enable extensions.webextPermissionPrompts from about:config
* Enable extensions.webextOptionalPermissionPrompts from about:config
* Disable check for signing if testing on AMO dev or stage
</div>
</div>


=== Beta ===
=== Beta ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
text
* Enable extensions.webextPermissionPrompts from about:config
* Enable extensions.webextOptionalPermissionPrompts from about:config
* [Unbranded build] Disable check for signing if testing on AMO dev or stage
 
</div>
</div>


=== Post Beta / Release ===
=== Post Beta / Release ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
text
The feature is enabled by default.
</div>
</div>
</div>
</div>


= Test Strategy =  
= Test Strategy =  
== Test Objectives ==
== Test Objectives ==
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
Line 77: Line 114:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! Owners  
! Ref !! Function !! Test Objective !! Test Type !! Owners  
|-
| TO-1 || Validation during AMO || To verify that linter raise the proper error messages || Manual || Add-ons QA Team
|-
| TO-2 || AMO public pages || To verify that permissions are correctly displayed || Manual || Add-ons QA Team
|-
| TO-3 || Installing from AMO || To verify that the permissions flow is fully respected || Manual || Add-ons QA Team
|-
|-
| 1 || Name of the feature or sub-function being tested || The objective the test is trying to demonstrate || The criteria that will be evaluated to demonstrate the test is successful || Manual/ Automation/ Regression/ Performance/ Usability/ Security/ Telemetry || Eng Team
| TO-4 || Installing from Discopane || To verify that the permissions flow is fully respected || Manual || Add-ons QA Team
|-
|-
| 2 || Repeat for each feature/sub-function || || || ||
| TO-5 || Installing from Add-ons manager || To verify that the permissions flow is fully respected || Manual || Add-ons QA Team
|-
|-
| 3 || || || || ||
| TO-6 || Installing via sideloading || To verify that the permissions flow is fully respected || Manual || Add-ons QA Team
|-
| TO-7 || Installing from local files|| To verify that the permissions flow is fully respected || Manual || Add-ons QA Team
|-
| TO-8 || Installing from thirdparty || To verify that the permissions flow is fully respected || Manual || Add-ons QA Team
|-
| TO-9 || Add-on updates || To verify that the permissions flow is fully respected || Manual || Add-ons QA Team
|}
|}


== Builds ==
== Builds ==
This section should contain links for builds with the feature -  
This section should contain links for builds with the feature -  
* Links for Nightly builds
* [https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ Link] for Nightly builds
* Links for Aurora builds
* [https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-aurora/ Link] for Aurora builds
* Links for Beta builds
* [https://archive.mozilla.org/pub/firefox/candidates/ Link] for Beta builds


== Test Execution Schedule ==
== Test Execution Schedule ==
Line 106: Line 154:
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | ||  
|style="text-align:center;" | 01-09-2017 || 01-11-2017
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" |  ||  
|style="text-align:center;" |01-09-2017 ||  
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" |  ||  
|style="text-align:center;" |01-11-2017 ||  
|-
|-
| QA - Aurora Testing
| QA - Aurora Testing
Line 132: Line 180:
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki
|-
|-
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail] / [https://docs.google.com/spreadsheets/d/1b_W1iVVuo-MRtwilKeHt_Vjbemy3j1Pbiq4slL_BhE8/edit#gid=36647849 Google docs] / [https://public.etherpad-mozilla.org/p/webext-permissions etherpad]
|-
|-
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]
|-
|-
| Bugs management || Bugzilla
| Bugs management || Bugzilla / Github
|}
|}


Line 149: Line 197:
  Identify existing bugs on the feature with high risk
  Identify existing bugs on the feature with high risk
  Identify if other areas are affected by the fix
  Identify if other areas are affected by the fix
= References =
* List and links for specs
  PRD - https://docs.google.com/document/d/1jZQUeIfelisRXBMaTuPTo2hXiIcPT3IpdYwQXDpzEMI/edit#heading=h.jyc2dow656pq
  Install flow - https://www.figma.com/file/HrLiKUwoLQZsIUIVBKM8Wnnu/Install-Flow-showing-Permissions
  Permissions Copy - https://docs.google.com/document/d/1-aLncxcKpinCNpvksmknwWZhrM2bEChDmQ3C422C6Vc/edit#heading=h.cwg9zx7py4zk
   
   
= References =
* {{Bug|1308292}} - (webext-permissions) (tracking) Webextensions required permissions handling
* PRD - https://docs.google.com/document/d/1jZQUeIfelisRXBMaTuPTo2hXiIcPT3IpdYwQXDpzEMI/edit#heading=h.jyc2dow656pq
<bugzilla>
* Install flow - https://www.figma.com/file/HrLiKUwoLQZsIUIVBKM8Wnnu/Install-Flow-showing-Permissions
    {
* Permissions Copy - https://docs.google.com/document/d/1-aLncxcKpinCNpvksmknwWZhrM2bEChDmQ3C422C6Vc/edit#heading=h.cwg9zx7py4zk
        "blocks":[1308292],
* Tracker bug - [https://bugzilla.mozilla.org/show_bug.cgi?id=1308292 bug 1308292]
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
    }
</bugzilla>
 
* {{Bug|1197420}} - Implement permissions API and optional_permissions manifest property
<bugzilla>
    {
        "blocks":[1197420],
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
    }
</bugzilla>


= Testcases =  
= Testcases =  
Line 188: Line 252:


== Test suite ==
== Test suite ==
Full Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
* Link for the [https://public.etherpad-mozilla.org/p/webext-permissions Initial test planning]
Smoke Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
* Link for the [https://docs.google.com/spreadsheets/d/1b_W1iVVuo-MRtwilKeHt_Vjbemy3j1Pbiq4slL_BhE8/edit#gid=1522764364 Google doc tests]
Regression Test suite - Link with the gdoc - if available/needed.
* Link for the [https://testrail.stage.mozaws.net/index.php?/suites/view/747&group_by=cases:section_id&group_id=2394&group_order=asc TestRail tests]


= Bug Work =
= Bug Work =
Tracking bug – meta bug
Tracking bug - [https://bugzilla.mozilla.org/show_bug.cgi?id=1308292 bug 1308292]


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
Line 199: Line 263:
====== Bug fix verification ======
====== Bug fix verification ======
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
{| class="wikitable" style="width:50%"
<font color="green">'''[Verified]'''</font> [https://bugzilla.mozilla.org/show_bug.cgi?id=1308309 Bug 1308309] - Display permissions prompt for webextensions installed using mozAddonManager
|-
:↳ <small>'''2017-01-10'''</small>: verified fixed on 53.0a1 across platforms
! Bug No !! Summary !! Status !! Firefox Verion
<font color="green">'''[Verified]'''</font> [https://bugzilla.mozilla.org/show_bug.cgi?id=1308295 Bug 1308295] - Prompt users with permissions for third-party webextensions installs
|-
:↳ <small>'''2015-04-21'''</small>: verified fixed on 53.0a1 across platforms
|  123 || bug summary || NEW || Nighly 45
|-
| || || ||
|}
</div></div>
</div></div>


Line 213: Line 273:
====== Logged bugs ======
====== Logged bugs ======
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
Bug 111111<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1329942 Bug 1329942] - Misaligned icon and webextension name in permissions doorhanger<br/>  
Bug 211111<br/>
 
</div>
</div>
</div>
</div>


= Sign off =
= Sign off =
Line 231: Line 290:
List of OSes that will be covered by testing<br />
List of OSes that will be covered by testing<br />
*Link for the tests run
*Link for the tests run
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Full Test suite, use template from [https://testrail.stage.mozaws.net/index.php?/suites/view/747&group_by=cases:section_id&group_id=2394&group_order=asc link]
** Full Test suite, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
 
** Regression Test suite, if needed/available


'''Merge to Aurora Sign-off'''
'''Merge to Aurora Sign-off'''
ValentinaP
302

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1159324...1181530"