MozillaWiki

Add-ons/Webextensions Cutover/Testplan: Difference between revisions

Page Discussion

Add-ons/Webextensions Cutover/Testplan (view source)

Revision as of 17:09, 20 September 2017

3,302 bytes added ,  20 September 2017
Added links for TestRail, Google docs
(Added links for TestRail, Google docs)
 
(31 intermediate revisions by 2 users not shown)
Line 5: Line 5:
! Date !! Version !! Author !! Description  
! Date !! Version !! Author !! Description  
|-
|-
| 01/09/2017 || 1.0 || Krupa Raj || Created first draft
| 05/01/2017 || 1.0 || Krupa Raj || Created first draft
|-
|-
| 01/20/2017 || 1.1 || Vasilica Mihasca || Updates
| 08/17/2017 || 1.1 || Vasilica Mihasca || Updates
|-
| 01/23/2017 || 1.2 || Krupa Raj || Added Acceptance Criteria
|-
| 03/22/2017 || 1.3 || Vasilica Mihasca || Added TestRail links
|-
|-
|}
|}
Line 17: Line 13:


= Overview =
= Overview =
Permissions is a feature which allows webextensions to have certain special powers. The permissions key specified in the manifest.json will grant special powers for your extension. This key is an array of strings, and each string is a request for a permission.
In 57, Firefox Desktop Release will only support installation of Webextensions. All legacy extensions will fail to install on all channels except for Nightly. Complete themes support will also cease after 57.
 
This feature will be implemented in two stages :
== Stage 1:==


If you request permissions using this key, then the browser may inform the user at install time that the extension is requesting certain privileges, and ask them to confirm that they are happy to grant these privileges. The browser may also allow the user to inspect an extension's privileges after installation.
All non-mpc add-ons will be disabled on Nightly by default. This is being tracked at [https://bugzilla.mozilla.org/show_bug.cgi?id=1352204 Bug 1352204]. Users will be able to enable these disabled add-ons by going to about:config and toggling the pref extensions.allow-non-mpc-extensions to set it to "true"
 
==Stage 2: ==
This feature will land in 57 and will disable all legacy add-ons on firefox desktop release channel. Users will still be able to install legacy add-ons on Nightly after toggling "extensions.legacy.enabled" pref to true. Android will be unaffected and will continue to allow install of legacy extensions. Dictionaries and Language Packs should still work as always. Support for Complete Themes will fully cease in 57.


== Purpose ==
== Purpose ==
This document purports to detail the test approach to Webextension Permissions including Entry/Exit criteria, Scope for testing, links to testcases etc
This document purports to detail the test approach to the Webextensions Cutover including Entry/Exit criteria, Scope for testing, links to testcases etc


=== Entry Criteria ===
=== Entry Criteria ===
Line 45: Line 47:
This section describes what parts of the feature will be tested and what parts won't be.
This section describes what parts of the feature will be tested and what parts won't be.


===what's in scope?===
== what's in scope? ==
* Installation flow from AMO
The compat table listed below should provide all the permutations supported by Firefox and which will in turn get tested before release.
* Installation flow from Add-ons Manager
 
* Installation flow from Discopane
=== Compat. Table ===
* Installation flow from Third-party websites
 
* Add-on updates
The table below shows what is installable where when 57 hits the Firefox release channel. NOTE that for a while when 57 is in the Nightly channel, with the pref enabled - install of legacy extensions will fail on Nightly but will be supported on beta and release.
* Add-on Validation related to permissions during submission
 
* Sideloading(?)
{| class="wikitable"
* Webextensions have access to the permissions requested
|-
! Type !! Status !! Nightly !! Beta, Release
|-
| WebExtension || Signed by AMO || YES || YES
|-
| || Signed by Mozilla internally || YES || YES
|-
| || Unsigned || YES with pref || NO
|-
| || Unsigned loaded temporarily || YES || YES
|-
| Legacy Extension (bootstrapped, overlay, XUL etc) || Signed by AMO || YES with pref || NO
|-
| || Signed by Mozilla internally || YES || YES
|-
| || Unsigned || YES with pref || NO
|-
| || Unsigned loaded temporarily || YES || NO
|-
| WebExtension Experiment || Signed by AMO || YES  || NO
|-
| || Signed by Mozilla internally || YES || YES
|-
| || Unsigned || YES with pref || NO
|-
| || Unsigned loaded temporarily || YES || NO
|-
| New style static theme || Signed by AMO || YES  || YES
|-
| || Signed by Mozilla internally || YES || YES
|-
| || Unsigned || YES || YES
|-
| || Unsigned loaded temporarily || YES || YES
|-
| Telemetry Experiments || Signed by AMO || YES  || YES
|-
| || Signed by Mozilla internally || YES || YES
|-
| || Unsigned || YES with pref || YES
|-
| || Unsigned loaded temporarily || YES || YES
|-
| Legacy lightweight theme || Signed by AMO || YES  || YES
|-
| Complete Themes || Any combination || YES with pref || NO
|-
| Multiple item package || Any combination || NO  || NO
|-
| Language Pack || Any combination || YES  || YES
|-
| Dictionaries || Any combination || YES  || YES
|-
|}
 
=== AMO ===
Awaiting changelist from jorgev
 
=== Other ===
The other things will also get tested is
* Reverting to an older version of Firefox
* Switching channels across the same profile
* Full regression testing for Android
 


===what's out of scope?===
===what's out of scope?===
* Security testing
* Security testing
* Performance testing
* Performance testing
* Testing for ESR


== Ownership ==
== Ownership ==
Line 75: Line 141:
=== Nightly ===
=== Nightly ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
* Enable extensions.webextPermissionPrompts from about:config
* Stage 1: Set extensions.allow-non-mpc-extensions to false from about:config.
* Disable check for signing if testing on AMO dev or stage
* Stage 2: Set extensions.legacy.enabled to false from about:config.
</div>
* Starting with 2017-08-12 this feature will be enabled by default.
 
=== Aurora ===
<div class="mw-collapsible-content">
* Enable extensions.webextPermissionPrompts from about:config
* Disable check for signing if testing on AMO dev or stage
* Disable check for signing if testing on AMO dev or stage
</div>
</div>
Line 87: Line 149:
=== Beta ===
=== Beta ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
* Enable extensions.webextPermissionPrompts from about:config
* The feature is enabled by default.
* [Unbranded build] Disable check for signing if testing on AMO dev or stage
* [Unbranded build] Disable check for signing if testing on AMO dev or stage


Line 106: Line 168:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! Owners  
! Ref !! Function !! Test Objective !! Test Type !! Owners  
|-
|-
| TO-1 || Validation during AMO || To verify that linter raise the proper error messages || Manual || Add-ons QA Team
| TO-1 || Installing from AMO || To verify that legacy add-ons installation is successfully blocked || Manual || Add-ons QA Team
|-
|-
| TO-2 || AMO public pages || To verify that permissions are correctly displayed || Manual || Add-ons QA Team
| TO-2 || Installing from Discopane || To verify that legacy add-ons installation is successfully blocked || Manual || Add-ons QA Team
|-
|-
| TO-3 || Installing from AMO || To verify that the permissions flow is fully respected || Manual || Add-ons QA Team
| TO-3 || Installing from Add-ons manager || To verify that legacy add-ons installation is successfully blocked || Manual || Add-ons QA Team
|-
|-
| TO-4 || Installing from Discopane || To verify that the permissions flow is fully respected || Manual || Add-ons QA Team
| TO-4 || Installing via sideloading || To verify that legacy add-ons installation is successfully blocked || Manual || Add-ons QA Team
|-
|-
| TO-5 || Installing from Add-ons manager || To verify that the permissions flow is fully respected || Manual || Add-ons QA Team
| TO-5 || Installing from local files|| To verify that legacy add-ons installation is successfully blocked || Manual || Add-ons QA Team
|-
|-
| TO-6 || Installing via sideloading || To verify that the permissions flow is fully respected || Manual || Add-ons QA Team
| TO-6 || Installing from thirdparty || To verify that legacy add-ons installation is successfully blocked || Manual || Add-ons QA Team
|-
|-
| TO-7 || Installing from local files|| To verify that the permissions flow is fully respected || Manual || Add-ons QA Team
| TO-7 || Add-on updates || To verify that legacy add-on is successfully updated to a webextension version || Manual || Add-ons QA Team
|-
|-
| TO-8 || Installing from thirdparty || To verify that the permissions flow is fully respected || Manual || Add-ons QA Team
| TO-8 || Browser upgrade || To verify that legacy add-on are successfully disabled and displayed in Legacy Extensions tab || Manual || Add-ons QA Team
|-
|-
| TO-9 || Add-on updates || To verify that the permissions flow is fully respected || Manual || Add-ons QA Team
| TO-9 || Browser downgrade || To verify that legacy add-on are successfully enabled and functional || Manual || Add-ons QA Team
|}
|}


== Builds ==
== Builds ==
This section should contain links for builds with the feature -  
This section should contain links for builds with the feature -  
* [https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ Link] for Nightly builds
* [http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ Nightly builds]
* [https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-aurora/ Link] for Aurora builds
* [https://archive.mozilla.org/pub/firefox/candidates/ Beta builds]
* [https://archive.mozilla.org/pub/firefox/candidates/ Link] for Beta builds
* Beta Unbranded builds:
** [https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-win64-add-on-devel/ Windows Link]
** [https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-linux64-add-on-devel/ Linux Link]  
** [https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-macosx64-add-on-devel/ Mac Link]
* [https://archive.mozilla.org/pub/firefox/releases/ Release builds]
* Release Unbranded builds:
** [https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release-win64-add-on-devel/ Windows Link]
** [https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release-linux64-add-on-devel/ Linux Link]
** [https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release-macosx64-add-on-devel/ Mac Link]


== Test Execution Schedule ==
== Test Execution Schedule ==
Line 146: Line 216:
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | 01-09-2017 || 01-11-2017
|style="text-align:center;" | 01-05-2017 ||
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
Line 152: Line 222:
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" |01-11-2017  ||  
|style="text-align:center;" |14-06-2017  ||  
|-
| QA - Aurora Testing
|style="text-align:center;" |  || 
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
Line 172: Line 239:
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki
|-
|-
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail] / [https://docs.google.com/spreadsheets/d/1b_W1iVVuo-MRtwilKeHt_Vjbemy3j1Pbiq4slL_BhE8/edit#gid=36647849 Google docs] / [https://public.etherpad-mozilla.org/p/webext-permissions etherpad]
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/1489&group_by=cases:section_id&group_order=asc TestRail] / [https://docs.google.com/spreadsheets/d/1aqP3lBVXwwmuXBE44Vwdanwf1DKgDrQ2FWc-3zLtTHk/edit#gid=1 Google docs] / [https://public.etherpad-mozilla.org/p/webextensions-cutover etherpad]
|-
|-
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/runs/view/6276 TestRail]
|-
|-
| Bugs management || Bugzilla / Github
| Bugs management || Bugzilla / Github
Line 186: Line 253:


== Risk analysis ==
== Risk analysis ==
Identify the high-risk assumptions
This section identified high-risk areas related to testing-
Identify existing bugs on the feature with high risk
* Users reverting to an older version of Firefox: Users trying to work around the restrictions may see unexpected regressions related to <bug number>
Identify if other areas are affected by the fix
** Mitigation strategy: Reverting to an older version/ESR builds will be tested as part of the end-to-end testing of this feature.
* Switching channels using the same profile may result in unexpected behavior.


= References =
= References =
Line 198: Line 266:
     {
     {
         "blocks":[1352204],
         "blocks":[1352204],
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
    }
</bugzilla>
* {{Bug|1360777}} - about:addons indications for disabled legacy extensions
<bugzilla>
    {
        "blocks":[1360777],
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
     }
     }
Line 241: Line 316:


== Test suite ==
== Test suite ==
* Link for the [https://public.etherpad-mozilla.org/p/webext-permissions Initial test planning]
* Link for the [https://public.etherpad-mozilla.org/p/webextensions-cutover Initial test planning]
* Link for the [https://docs.google.com/spreadsheets/d/1b_W1iVVuo-MRtwilKeHt_Vjbemy3j1Pbiq4slL_BhE8/edit#gid=1522764364 Google doc tests]
* Link for the [https://docs.google.com/spreadsheets/d/1aqP3lBVXwwmuXBE44Vwdanwf1DKgDrQ2FWc-3zLtTHk/edit#gid=1 Google doc tests]
* Link for the [https://testrail.stage.mozaws.net/index.php?/suites/view/747&group_by=cases:section_id&group_id=2394&group_order=asc TestRail tests]
* Link for the [https://testrail.stage.mozaws.net/index.php?/suites/view/1489&group_by=cases:section_id&group_order=asc TestRail tests]


= Bug Work =
= Bug Work =
Tracking bug - [https://bugzilla.mozilla.org/show_bug.cgi?id=1308292 bug 1308292]
Tracking bug - [https://bugzilla.mozilla.org/show_bug.cgi?id=1360777 bug 1360777]


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Bug fix verification ======
====== Bug fix verification ======
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
Santa
31

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1169874...1180842"