Add-ons/QA/Testplan/OOP-WebExtensions

From MozillaWiki
Jump to: navigation, search

Revision History

Date Version Author Description
02/03/2017 1.0 Vasilica Mihasca Created first draft
04/10/2017 1.1 Krupa Raj Updated test strategy
05/19/2017 1.2 Vasilica Mihasca Updates


Overview

With webext-oop, addon code runs in a separate process (bugzil.la/webext-oop). In the far future, addon code may also run in multiple processes.

Purpose

Entry Criteria

  • QA has access to all the PRDs, mocks and related documents
  • The feature has landed on Nightly

Exit Criteria

  • All the bugs against the feature have been triaged
  • All the P1/P2 bugs have been fixed
  • All the resolved bugs have been verified by QA
  • The find/fixed rate is going down over a predefined period of time

Acceptance Criteria

This section broadly outlines when the product is ready to ship

  • QA has signed off
  • All the required Telemetry is in place
  • All info is localized at least for a pre-defined set of locales
  • All the necessary PR/blogposts have been sent out

Scope

This section describes what parts of the feature will be tested and what parts won't be.

what's in scope?

  • Webextensions installation with e10s enabled
  • Webextensions running in their own process
  • Regression testing for the top 20 webextensions
  • Regression testing for webextensions examples
  • Regression testing for old XUL add-ons
  • Webextension crashing
  • Add-ons behavior after a Firefox crash

what's out of scope?

  • Security testing
  • Performance testing

Ownership

Dev Lead: > Kris Maglione; irc nick
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Vasilica Mihasca; irc nick :vasilica_mihasca
Add-ons QA: Valentina Peleski; irc nick :ValentinaV

Requirements for testing

Environments

OSed covered: Windows, Mac OS X, Linux

Channel dependent settings (configs) and environment setups

Nightly

  • Disable check for signing if testing on AMO dev or stage
  • Webextension process crashing:
    • Set "extensions.webextensions.remote” to true
    • Insert the following line in Browser Console input filed: Cu.import("resource://gre/modules/Extension.jsm").GlobalManager.extensionMap.values().next().value.parentMessageManager.loadProcessScript("data:,Components.utils.import('resource://gre/modules/ctypes.jsm').ctypes.cast(ctypes.uintptr_t(1), ctypes.int.ptr).contents", false)

Beta

  • [Unbranded build] Disable check for signing if testing on AMO dev or stage

Post Beta / Release

The feature is enabled by default.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Test Type Owners
TO-1 Installing with e10s enabled To verify that installation process runs without any issue Manual Add-ons QA Team
TO-2 Webextensions running in their own process To verify that webextensions run in separate process Manual Add-ons QA Team
TO-3 Regression testing for the top 20 webextensions To verify that the most popular webextensions works without issues Manual Add-ons QA Team
TO-4 Regression testing for webextensions example master To verify that the webextensions works without issues Manual Add-ons QA Team
TO-5 Webextensions testing in Private Window To verify the webextension behavior in private winodow Manual Add-ons QA Team
TO-6 Webextension crashing To verify that crashing the webextension process does not cause a Firefox crash Manual Add-ons QA Team
TO-7 Firefox crashing To verify that Firefox will load all the add-ons properly without any loss of saved add-on data Manual Add-ons QA Team
TO-8 Regression testing on XUL add-ons To verify that the old XUL add-ons works without any issue Manual Add-ons QA Team

Test Strategy

Here are the main ways in which we will ascertain that OOP functions as expected

  • Test all the webextensions listed at https://github.com/mdn/webextensions-examples and ensure there are no regressions
  • Check webextensions which are using APIs which are most affected by OOP. These APIs include -
    • context Menus
    • devtools
    • omnibox
    • tabs
    • identity
    • permissions
  • Check that a firefox doesn't crash if a webextension crashes. See Bug 1353959 for more details
  • Check that extensions work as expected after a firefox crash
   webRequest

Builds

This section should contain links for builds with the feature -

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project
Study documentation/specs received from developers
QA - Test plan creation 02-03-2017
QA - Test cases/Env preparation
QA - Nightly Testing 03-22-2017
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation [ TestRail] / [Google docs] / etherpad
Test case execution [TestRail]
Bugs management Bugzilla / Github

Status

Overview

Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Aurora
Track the dates and build number where feature was merged to Release/Beta

Risk analysis

  • The test strategy recommended by devs is based on spot-checking and regression testing.
    • Mitigation stragety - QA will test all webextensions with the most number of users to ensure that the most impactful add-ons are covered.
  • Webextensions will have their own process but Content scripts will still run in the content process
    • Mitigation stragety - QA will test with webextensions which run content scripts to make sure there is degradation of behavior
  • Product is yet to define what is the acceptable dip in performance for when webextensions run in their process

References

* List and links for specs
  PRD - 
* bug 1190679 - (webext-oop) Run WebExtensions out of process
ID Priority Component Assigned to Summary Status Target milestone
1202478 -- WebExtensions: Untriaged Bill McCloskey (:billm) Run WebExtension callbacks asynchronously RESOLVED mozilla44
1224105 -- WebExtensions: Untriaged Bill McCloskey (:billm) Use a <browser> element for the background page RESOLVED mozilla46
1253748 P2 Graphics Compositor plugin window management should support multiple remote layer trees with plugin windows RESOLVED ---
1287004 P1 WebExtensions: General David Anderson [:dvander] Support remote <browser> elements in windowless browsers RESOLVED mozilla54
1287007 -- WebExtensions: Untriaged Rob Wu [:robwu] Start using ChildAPIManager for some APIs RESOLVED mozilla52
1287010 -- WebExtensions: Untriaged Rob Wu [:robwu] Some API methods need to run locally with OOP webextensions RESOLVED mozilla51
1287209 P3 WebExtensions: Frontend Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Popup and options UI resizing code is not compatible with remote browsers RESOLVED mozilla52
1287210 -- WebExtensions: Untriaged Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) setIcon calls with ImageData objects are not cross-process compatible RESOLVED ---
1287977 P2 WebExtensions: Untriaged Modify E10sUtils.jsm to run WebExtension tab pages in the WebExtensions process RESOLVED ---
1287978 -- WebExtensions: Untriaged Need to allow <browser> element to choose to run in a WebExtension process RESOLVED ---
1288276 -- WebExtensions: Untriaged Rob Wu [:robwu] closeProxyContext is never invoked RESOLVED mozilla51
1288279 -- WebExtensions: Untriaged Rob Wu [:robwu] WebExt contextId must be globally unique, even with multiple processes RESOLVED mozilla51
1302702 P1 WebExtensions: Developer Tools Luca Greco [:rpl] Make WebExtension Addon Debugging oop-compatible VERIFIED mozilla55
1305216 P3 WebExtensions: General Rob Wu [:robwu] Use ChildAPIManager for webNavigation API RESOLVED ---
1305217 P1 WebExtensions: General Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Make webRequest oop-compatible RESOLVED mozilla52
1306110 P2 WebExtensions: General Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Run WebExtension tests with both remote and non-remote APIs RESOLVED ---
1317101 -- WebExtensions: General Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Run WebExtension tests in both in-process and out-of-process modes RESOLVED mozilla53
1320395 -- WebExtensions: General Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Run remote extensions in their own process type RESOLVED mozilla53
1334557 P3 Security: Process Sandboxing Tighten sandboxing for extension content process NEW ---
1339144 P1 WebExtensions: General Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Use windowless remote <browser> element for background pages RESOLVED mozilla54
1353060 P1 Graphics: Layers Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Remote <browser>s are not visible as children of XUL <popup>s RESOLVED mozilla55
1353959 P1 WebExtensions: General Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Need a test webextension to crash the extension process RESOLVED ---
1355239 P2 WebExtensions: General We should attempt to recover from an extension process crash. NEW ---
1356317 P1 Graphics: Layers Stephen A Pohl [:spohl] Support remote layer trees in popups on Windows VERIFIED mozilla56
1357486 P1 WebExtensions: General Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Turn on OOP extensions by default on Windows RESOLVED mozilla56
1357487 P2 WebExtensions: General Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Turn on OOP extensions by default on Linux NEW ---
1357490 -- WebExtensions: General Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Run xpcshell tests in both in-process and OOP modes RESOLVED mozilla56
1357729 -- WebExtensions: Untriaged blank browserAction with extensions.webextensions.remote=true RESOLVED ---
1361661 P1 Telemetry Georg Fritzsche [:gfritzsche] Send extension process Telemetry in a separate payload section RESOLVED mozilla55
1362457 P1 WebExtensions: General Robert Helmer [:rhelmer] No crash report generated for extension process crashes RESOLVED ---
1365660 P3 WebExtensions: Frontend Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Add special handling for popups which will contain remote browsers NEW ---
1369559 P5 WebExtensions: General enable autocomplete popup test for remote popups NEW ---
1370131 P2 Widget: Cocoa Markus Stange [:mstange] Support OMTC on panels with shadows ASSIGNED ---
1375490 P1 WebExtensions: Untriaged Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Alerts do not work correctly in OOP popup browsers VERIFIED mozilla57
1379046 -- Graphics: Layers Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) RenderFrameParent inconsistently uses the wrong LayerManager in XUL popups RESOLVED mozilla56
1379508 P1 WebExtensions: Frontend Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) <select> dropdowns don't work in OOP popup browsers RESOLVED mozilla56
1380109 P1 WebExtensions: General Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) [oop webextensions] When moving a WebExtension to the panel, it closes the panel when clicked (doesn't open subview) RESOLVED mozilla56
1380156 P1 WebExtensions: Request Handling Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Loading temporary an unpacked extension breaks extension page's CSS in OOP Extensions VERIFIED mozilla56
1380290 P2 WebExtensions: Untriaged Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Options page can no longer receive messages from the background page VERIFIED mozilla56
1380294 P1 WebExtensions: General Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) [Firefox Notes] Formatting option tooltips are wrongly displayed on the latest Nightly VERIFIED mozilla57
1380622 P2 WebExtensions: Untriaged [Windows] Visual glitches while opening the webextension panel NEW ---
1381023 -- WebExtensions: General [Windows] "Screenshots went haywire" error message displayed for any screenshot taken and the Screenshots overlay wrongly remains displayed on root page VERIFIED mozilla56
1381086 -- WebExtensions: Untriaged extensions.webextensions.remote: true breaks Containers add-on RESOLVED ---
1381097 P3 Graphics: Layers Popups with APZ enabled to not repaint correctly after resize NEW ---
1381321 -- WebExtensions: Frontend Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) <select> popups don't work in OOP options browsers VERIFIED mozilla56
1381337 -- WebExtensions: Frontend Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Select dropdowns in popups should not be constrained to document viewport RESOLVED mozilla56
1381344 P1 WebExtensions: Untriaged Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) No context menu for OOP popups opened as standalone panel VERIFIED mozilla57
1381810 P3 WebExtensions: Untriaged [Ubuntu] Black borders displayed for webextension panel RESOLVED ---
1382228 -- WebExtensions: Untriaged Unable to Disable or Remove Awesome Screenshot Plus - Capture add-on RESOLVED ---
1382271 P1 WebExtensions: Request Handling Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) <script>s referencing non-existent moz-extension: scripts never resume parsing with protocol remoting enabled RESOLVED ---
1384078 P1 WebExtensions: Untriaged Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Enter closes WebExtensions popup VERIFIED mozilla57
1385403 P3 Graphics: Layers Turn on OOP extensions by default on Mac OS NEW ---
1385548 P2 WebExtensions: Frontend Luca Greco [:rpl] Confirm/alert/prompt dialogs do not work from options_ui with OOP enabled VERIFIED mozilla58
1385736 P2 IPC How to distinguish between content processes and Extension OOP processes? NEW ---
1385880 P5 WebExtensions: Untriaged Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) Disable/Enable hides extension options VERIFIED mozilla57
1387386 P5 WebExtensions: Untriaged Long URL breaks the webextension panel while alert pop-up is displayed UNCONFIRMED ---
1388336 -- Add-ons Layout issue in AdBlock for Firefox details tab NEW ---
1389496 P3 WebExtensions: Compatibility Slow getHost method in Page Zoom WE RESOLVED ---
1390276 P3 Add-ons Manager Mark Striemer [:mstriemer] Update options_ui to match photon VERIFIED mozilla58
1390445 P2 WebExtensions: Frontend Luca Greco [:rpl] HTML <select> drop-downs are incorrectly positioned in WebExtensions add-on Options page VERIFIED mozilla58
1392210 P2 WebExtensions: Frontend Shane Caraveo (:mixedpuppy) [OOP] onBeforeLinkTraversal is not called for extension frames VERIFIED mozilla57
1393150 P1 WebExtensions: General Shane Caraveo (:mixedpuppy) require e10s if webextension.remote=true VERIFIED mozilla57
1394208 -- WebExtensions: Frontend Popup with scrollable content in overflow menu has painting artifacts with OOP enabled RESOLVED ---
1394807 -- WebExtensions: General Tab pages for OOP extensions don't work when e10s is disbled RESOLVED ---
1395898 -- WebExtensions: Request Handling Haik Aftandilian [:haik] [OOP] jar cache incorrectly handled non-existent file paths VERIFIED mozilla57
1397210 P3 WebExtensions: Frontend [OOP] Opening sidebar link inconsistencies with new tab active state and container inheritance NEW ---
1401288 P2 WebExtensions: Developer Tools Luca Greco [:rpl] Ensure that the WebExtensions "devtools panel" browsers run in the same TabGroup of the other extension pages RESOLVED mozilla57
1403965 P2 WebExtensions: General The context menu is displayed in a wrong position after moving the "Notes" sidebar to right / left NEW ---
1405001 P3 Panning and Zooming Error processing PAPZCTreeManager::Msg_SetTargetAPZC message when opening dropdown in remote webextension's options panel NEW ---

69 Total; 15 Open (21.74%); 37 Resolved (53.62%); 17 Verified (24.64%);


Bug Work

Tracking bug - bug 1190679

Bug fix verification
Logged bugs

Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

  • Link for the tests run
    • Daily Smoke, use template from [link]
    • Full Test suite, use template from [ link]
    • Regression Test suite, if needed/available

Merge to Aurora Sign-off List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release Email to be sent
QA Aurora - Full Testing
QA Signoff - Aurora Release Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release Email to be sent