QA/WebExtensions

From MozillaWiki
< QA
Jump to: navigation, search

WebExtensions: Manual Testing

Ownership

Developer contacts:

QA:

Krupa Raj, :krupa on IRC - leading all QA efforts across different products

Victor Carciu, :VictorC on IRC - leading QA Softvision team

Vasilica Mihasca, :vasilica_mihasca on IRC; - platform owner on Firefox Client side testing

Valentina Peleskei, :ValentinaP on IRC; - peer for Firefox Client side testing

Madalin Cotetiu, :madalinc on IRC; - peer for Firefox Client side testing

Cosmin Badescu, :CosminB on IRC; - peer for Firefox Client side testing

Status

Overview

[LANDED] Nightly 48: first landed, Firefox 48 is the first release of Firefox where we are recommending people start to use WebExtension

[ON TRACK] Developer Edition 48: tbd

[ON TRACK] Beta 48: tbd

[ON TRACK] Release 48: tbd

Risk analysis

Event Probability Impact Current situation Mitigation steps
WebExtensions fails to be installed on Firefox probable high Not able to test yet. Periodical regression testing will cover this area.
WebExtensions are successfully installed using an incompatible Firefox version probable high Not able to test yet. Periodical regression testing will cover this area.
WebExtensions are successfully installed but breaks the browser improbable high Not able to test yet. Periodical regression testing will cover this area.
WebExtensions are successfully installed but the expected functionality is not meet probable high Not able to test yet. Periodical regression testing will cover this area.
WebExtensions are successfully installed but there are encountered issues in terms of UI or performance probable high Not able to test yet. Periodical regression testing will cover this area.

Summary

Context

About WebExtensions:

  • are small software programs that can modify and enhance the functionality of the browser
  • are a zipped bundle of files—HTML, CSS, JavaScript, images, and anything else you need
  • are essentially web pages, and they can use all the APIs that the browser provides to web pages, from XMLHttpRequest to JSON to HTML5.

Enabling the feature

WebExtensions installation will be available by default starting with Firefox 48 and there is no pref to control this.

AMO

WebExtensions will be able to be uploaded to addons.mozilla.org then reviewed, searched, installed and updated just like any other Addon Bug 1210037

References

Test cases

Desktop tests will focus on:

  • Install a Webextension on Firefox
  • Use available functionality
  • Compare Firefox's behaviour with Chrome's behaviour
  • Uninstall is clean and permanent

Coverage

WebExtensions test cases will be executed according to each OS's percentage of our ADi numbers.

  • All the webextensions will be tested across the Main OSes:

[1] Windows: Windows 7, Windows 10, Windows XP, Windows 8.1
[2] Mac: Mac OS X 10.11, Mac OS X 10.10, Mac OS X 10.9.5, Mac OS X 10.6
[3] Ubuntu: 14.04 LTS, 12.04 LTS

  • The popular webextensions will be also tested across the Secondary OSes:

[1] Windows: Windows 8, Windows Vista
[2] Mac: Mac OS X 10.7, Mac OS X 10.6

Overview

  • Draft test suites:
    • [DONE] Full WebExtensions Testing: [1].
    • [DONE] Sanity WebExtensions Testing: [2]

Targeted scenarios

Testing focused on basic functionality

Testing focused on interoperability

Testing based on recent bugs and regressions

Testing with e10s

Bug work

Tracking bug

meta: Bug 1161828

Bug triage +/- for verification
Bug fix verification
Logged bugs


Sign off

Criteria

  • All the test cases were executed.
  • All the blocker, critical, major bugs have been fixed.

Results

WebExtensions: Automation

Tracking Bug

bug 1252966

Contacts

Dave Hunt, :davehunt on IRC

John Dorlus, :silne30 on IRC

Krupa Raj, :krupa on IRC


Planned coverage for Firefox 48

must-have coverage

  • Install a webextension from AMO - being tracked at bug 1252288
  • Install a webextension from an external source - being tracked at bug 1252295
  • Uninstall a webextension - being tracked at bug 1252312
  • Test API calls for tabs - being tracked at bug 1252701