Add-ons/QA/Testplan/Notify user when add-on update requires new permissions
|07/10/2019||1.0||Alex Cornestean||Created first draft|
Please note: This document is still a work in progress and sections might contain placeholders and/or inaccurate data
- 1 Overview
- 2 Requirements for testing
- 3 Test Strategy
- 4 References
- 5 Testcases
- 6 Sign off
The purpose of this feature is to add a more noticeable prompt for users when new add-on permissions interrupt automatic updates for an add-on. The intent is to also reuse this UX for when an add-on is no longer recommended, and potentially for add-on blocking, due to the current UX, which silently displays a warning icon on top the hamburger menu, proving to be insufficient in making users aware of something needing their attention as users on older versions of their add-ons are a security risk and need to be minimized.
This document proposes to detail a test approach for the new UX designed to increase user awareness regarding the state of their add-ons, which includes Entry/Exit/Acceptance criteria, Testing scope, references to testcases, etc.
- QA has access to all PRDs, mocks and related documentation
- The feature has landed on Nightly and Beta
- All feature related bugs have been triaged
- All P1/P2 bugs have been fixed
- All resolved bugs have been verified by QA
- The find/fixed bug ratio shows a descending trend over a defined time period
This section proposes to highlight the criteria concerning the shipment readiness status of the product.
- QA has signed off
- All the required Telemetry is in place
This section outlines which parts of the new implemented feature will or will not be tested.
What is in scope
- Validation of the new UX when an add-on changes permissions and can’t be updated because of it, when an installed add-on is no longer recommended and can’t be updated because of it and when add-ons are prohibited from being installed (blocked).
What is out of scope
- Security testing
- Device testing
- Performance testing
Dev Lead????????????: Franziskus Kiefer ; irc nick: fkiefer or :franziskus
QA Manager: Krupa Raj; irc nick: krupa
QA Lead: Victor Carciu; irc nick: victorc
Webextensions QA: Alex Cornestean; irc nick: AlexC_
Requirements for testing
Covered OSes: Windows, macOS, Linux
Channel dependent settings (configs) and environment setups
Post Beta / Release
The feature is enabled by default.
This section details the progression test objectives that will be covered.
|Ref||Function||Test Objective||Test Type||Owners|
|TO-1||Checking if the prompt is shown/not shown when auto-updating an add-on with changed/unchanged permissions||To verify that the prompt is displayed/not displayed accordingly||Manual||Add-ons QA Team|
|TO-2||Checking if the prompt can be accepted/cancelled||To verify that the auto-update proceeds or is cancelled accordingly||Manual||Add-ons QA Team|
|TO-3||Checking if the prompt is shown/not shown when auto-updating to a non-recommended/to a recommended of an add-on||To verify that the prompt is displayed/not displayed accordingly||Manual||Add-ons QA Team|
|TO-4||Checking if the prompt is shown/not shown when attempting to install a blacklisted/non-blacklisted add-on||To verify that the prompt is displayed/not displayed accordingly||Manual||Add-ons QA Team|
This section contains links to builds with the feature
Test Execution Schedule
The below table outlines the anticipated testing time frame available for test creation and execution.
|Project phase||Start Date||End Date|
|Study documentation/specs received from developers||05/31/2019|
|QA - Test plan creation||07/10/2019|
|QA - Test cases/Env preparation||07/10/2019|
|QA - Nightly Testing|
|QA - Beta Testing|
Exemplifies the tools used for test suite creation/execution.
|Test plan creation||Mozilla wiki|
|Test case creation||TestRail|
|Test case execution||TestRail|
* List and links for specs PRD - Gdocs Install flow - Presentation * bug 1403838 - [Meta] Multiple-signed add-ons
|ID||Priority||Component||Assigned to||Summary||Status||Target milestone|
|1169532||--||Security||extension XPI signing still uses SHA1 for digests; should use SHA2||VERIFIED||---|
|1357815||P1||Security: PSM||Dana Keeler (she/her) (use needinfo) (:keeler for reviews)||support SHA-256 when verifying PKCS7 signatures on addons||VERIFIED||mozilla58|
|1403840||P1||Security: PSM||Franziskus Kiefer [:franziskus] (Away until October 2019)||Implement COSE for the new add-on signatures||RESOLVED||mozilla59|
|1403844||P1||Security: PSM||Franziskus Kiefer [:franziskus] (Away until October 2019)||Integrate COSE rust library in PSM||VERIFIED||mozilla59|
|1415991||P1||Security: PSM||Dana Keeler (she/her) (use needinfo) (:keeler for reviews)||remove support for verifying signed unpacked add-ons||RESOLVED||mozilla59|
|1421413||P1||Security: PSM||Dana Keeler (she/her) (use needinfo) (:keeler for reviews)||add a preference to control the accepted signature algorithms for add-ons||VERIFIED||mozilla59|
|1421816||P1||Security: PSM||Dana Keeler (she/her) (use needinfo) (:keeler for reviews)||add an option to sign_app.py to include a COSE signature||RESOLVED||mozilla59|
|1422904||--||Add-ons Manager||Dana Keeler (she/her) (use needinfo) (:keeler for reviews)||add an integration test for an add-on signed with sha256||RESOLVED||mozilla59|
|1436948||--||Security: PSM||Franziskus Kiefer [:franziskus] (Away until October 2019)||Update cbor lib||RESOLVED||mozilla60|
|1471185||--||Security||Greg Guthe [:g-k] [:gguthe]||Implement COSE XPI signing in Autograph||RESOLVED||---|
|1472104||P1||Security: PSM||Franziskus Kiefer [:franziskus] (Away until October 2019)||Test autograph-signed extension||VERIFIED||mozilla63|
|1475084||P1||Security: PSM||Dana Keeler (she/her) (use needinfo) (:keeler for reviews)||add tampered signature testcases for COSE-signed add-ons (like we have for PKCS7)||RESOLVED||mozilla63|
|1545836||P2||Security: PSM||Require COSE signatures for extensions||NEW||---|
13 Total; 1 Open (7.69%); 7 Resolved (53.85%); 5 Verified (38.46%);
The test suite proposes a series of test cases devised to cover different add-on installation scenarios under the effects of the SHA256 signing mechanism.
- Validation of the installation process of SHA256 and SHA1 signed add-ons using different methods.
|Installing from local storage|
|Installing as a temporary extension|
|Installing from thirdparty|
|Installing via sideloading|
|Installing extensions with missing/corrupted signatures|
|Installing from AMO|
- All test cases should be executed
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
|Testing Prerequisites (specs, use cases)|
|Testing Infrastructure setup|
|Test Plan Creation||07-10-2019|
|Test Cases Creation||07-10-2019|
|Full Functional Tests Execution|
|All Defects Logged|
|Critical/Blockers Fixed and Verified|
|QA Signoff - Nightly Release||Email to be sent|
|QA Beta - Full Testing|
|QA Signoff - Beta Release||Email to be sent|