Add-ons/QA/Testplan/Android Webextension API: Difference between revisions

Jump to navigation Jump to search

Add-ons/QA/Testplan/Android Webextension API (view source)

Revision as of 15:19, 17 July 2017

9,235 bytes added ,  17 July 2017
no edit summary
(Replaced content with "Test")
No edit summary
Line 1: Line 1:
Test
'''Revision History'''
 
{| class="wikitable" style="width:60%"
|-
! Date !! Version !! Author !! Description
|-
| 07/17/2017 || 1.0 || Cosmin Badescu || Created first draft
|-
|}
 
 
= Overview =
The implementation of the Android WebExtension API, is to ensure that the WebExtensions are functional on the Android platform.
 
== Purpose ==
 
=== Entry Criteria ===
* QA has access to all the PRDs, mocks and related documents
* The feature has landed on Nightly
 
=== Exit Criteria ===
* All the bugs against the feature have been triaged
* All the P1/P2 bugs have been fixed
* All the resolved bugs have been verified by QA
* The find/fixed rate is going down over a predefined period of time
 
=== Acceptance Criteria ===
This section broadly outlines when the product is ready to ship
* QA has signed off
* All the required Telemetry is in place
* All info is localized at least for a pre-defined set of locales
* All the necessary PR/blogposts have been sent out
 
== Scope ==
This section describes what parts of the feature will be tested and what parts won't be.
 
===what's in scope?===
* Webextensions installation with e10s enabled
* Webextensions running in their own process
* Regression testing for the top 20 WebExtensions
* Regression testing for WebExtensions examples
* Webextension crashing
* Add-ons behavior after a Firefox crash
 
===what's out of scope?===
* Security testing
* Performance testing
 
== Ownership ==
Dev Lead: [mailto:wehuang@mozilla.com Wesly Huang]; slack nick :weslyhuang <br />
QA Manager: [mailto:kraj@mozilla.com Krupa Raj]; irc nick :krupa<br />
QA Lead: [mailto:vcarciu@softvision.ro Victor Carciu]; irc nick :victorc<br />
Webextensions QA: [mailto:cosmin.badescu@softvision.ro Cosmin Badescu]; irc nick :CosminB<br />
 
 
= Requirements for testing =
== Environments ==
OSed covered: Android
 
== Channel dependent settings (configs) and environment setups ==
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
 
=== Nightly ===
<div class="mw-collapsible-content">
* Disable check for signing if testing on AMO dev or stage
* Webextension process crashing:
** Set "extensions.webextensions.remote” to true
** Insert the following line in Browser Console input filed: Cu.import("resource://gre/modules/Extension.jsm").GlobalManager.extensionMap.values().next().value.parentMessageManager.loadProcessScript("data:,Components.utils.import('resource://gre/modules/ctypes.jsm').ctypes.cast(ctypes.uintptr_t(1), ctypes.int.ptr).contents", false)
</div>
 
=== Beta ===
<div class="mw-collapsible-content">
* Disable check for signing if testing on AMO dev or stage
 
</div>
 
=== Post Beta / Release ===
<div class="mw-collapsible-content">
Disable check for signing if testing on AMO dev or stage.
</div>
</div>
 
== Test Objectives ==
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
This could be documented in bullet form or in a table similar to the one below.
 
{| class="wikitable"
|-
! Ref !! Function !! Test Objective !! Test Type !! Owners
|-
| TO-1 || Installing with e10s enabled || To verify that installation process runs without any issue || Manual || Add-ons QA Team
|-
| TO-2 || Webextensions running in their own process || To verify that webextensions run in separate process || Manual || Add-ons QA Team
|-
| TO-3 || Regression testing for the top 20 webextensions || To verify that the most popular webextensions works without issues || Manual || Add-ons QA Team
|-
| TO-4 || Regression testing for webextensions example master || To verify that the webextensions works without issues || Manual || Add-ons QA Team
|-
| TO-5 || Webextensions testing in Private Window|| To verify the webextension behavior in private winodow || Manual || Add-ons QA Team
|-
| TO-6 || Webextension crashing || To verify that crashing the webextension process does not cause a Firefox crash || Manual || Add-ons QA Team
|-
| TO-7 || Firefox crashing || To verify that Firefox will load all the add-ons properly without any loss of saved add-on data || Manual || Add-ons QA Team
|-
|}
 
== Test Strategy ==
Here are the main ways in which we will ascertain that Android Webextension API functions as expected
* Test the WebExtensions listed at https://addons-dev.allizom.org/en-US/android/ and ensure there are no regressions.
* Check that WebExtensions that are using Android Webextension API work as expected.
* Check that Firefox doesn't crash if a WebExtension crashes. See [https://bugzilla.mozilla.org/show_bug.cgi?id=1353959 Bug 1353959] for more details.
* Check that WebExtensions work as expected after a firefox crash.
 
== Builds ==
This section should contain links for builds with the feature:
* [https://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-api-15/ Link] for Nightly builds
* [https://archive.mozilla.org/pub/mobile/candidates/ Link] for Beta builds
 
== Test Execution Schedule ==
The following table identifies the anticipated testing period available for test execution.
{| class="wikitable" style="width:60%"
|-
! Project phase !! Start Date !! End Date
|-
| Start project
|style="text-align:center;" |  ||
|-
| Study documentation/specs received from developers
|style="text-align:center;" |  ||
|-
| QA - Test plan creation
|style="text-align:center;" | 07-17-2017 ||
|-
| QA - Test cases/Env preparation
|style="text-align:center;" | ||
|-
| QA - Nightly Testing
|style="text-align:center;" | 07-17-2017||
|-
| QA - Beta Testing
|style="text-align:center;" |  ||
|-
| Release Date
|style="text-align:center;" | ||
|}
 
== Testing Tools ==
Detail the tools to be used for testing, for example see the following table:
{| class="wikitable" style="width:50%"
|-
! Process !! Tool
|-
| Test plan creation || Mozilla wiki
|-
| Test case creation || [ TestRail] / [Google docs] /
|-
| Test case execution || [TestRail]
|-
| Bugs management || Bugzilla
|}
 
= Status =
== Overview ==
Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Release/Beta
 
== Risk analysis ==
* The test strategy recommended by devs is based on spot-checking and regression testing.
** Mitigation stragety - QA will test all webextensions with the most number of users to ensure that the most impactful WebExtensions are covered.
* Webextensions will have their own process but Content scripts will still run in the content process
** Mitigation stragety - QA will test with webextensions which run content scripts to make sure there is degradation of behavior
* Product is yet to define what is the acceptable dip in performance for when webextensions run in their process
 
= References =
* List and links for specs
  PRD -
* {{Bug|1185785}} - (webext-android) [tracking] Extension API for Android
<bugzilla>
    {
        "blocks":[1185785],
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
    }
</bugzilla>
 
 
= Bug Work =
Tracking bug - [https://bugzilla.mozilla.org/show_bug.cgi?id=1185785 bug 1185785]
 
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
 
====== Bug fix verification ======
<div class="mw-collapsible-content">
</div></div>
 
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
 
====== Logged bugs ======
<div class="mw-collapsible-content">
 
</div>
</div>
 
= Sign off =
== Criteria ==
Check list
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
 
== Results ==
'''Nightly testing'''<br />
 
List of OSes that will be covered by testing<br />
*Link for the tests run
** Daily Smoke, use template from [link]
** Full Test suite, use template from [ link]
** Regression Test suite, if needed/available
 
'''Merge to Aurora Sign-off'''
List of OSes that will be covered by testing<br />
*Link for the tests run
** Full Test suite
 
== Checklist ==
{| class="wikitable" style="width:60%"
|-
! Exit Criteria !! Status !! Notes/Details
|-
|  Testing Prerequisites (specs, use cases)
| style="text-align:center;" | 
| style="text-align:center;" |
|-
|  Testing Infrastructure setup
|style="text-align:center;" |  ||
|-
|  Test Plan Creation
| style="text-align:center;" |  ||
|-
|  Test Cases Creation
|style="text-align:center;" |  ||
|-
|  Full Functional Tests Execution
|style="text-align:center;" | 
|style="text-align:center;" | 
|-
|  Automation Coverage ||
|style="text-align:center;" |
|-
|  Performance Testing
|style="text-align:center;" |  ||
|-
|  All Defects Logged || ||
|-
|  Critical/Blockers Fixed and Verified || ||
|-
|  Metrics/Telemetry||
|style="text-align:center;" |
|-
|  QA Signoff  - Nightly Release||
|style="text-align:center;" | Email to be sent
|-
|  QA Beta - Full Testing
|style="text-align:center;" |  ||
|-
|  QA Signoff  - Beta Release ||
|style="text-align:center;" | Email to be sent
|}
CosminB
231

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1175854"

Navigation menu