Add-ons/QA/Testplan/Notify user when add-on update requires new permissions
Revision History
Date | Version | Author | Description |
---|---|---|---|
07/10/2019 | 1.0 | Alex Cornestean | Created first draft |
Please note: This document is still a work in progress and sections might contain placeholders and/or inaccurate data
Overview
The purpose of this feature is to add a more noticeable prompt for users when new add-on permissions interrupt automatic updates for an add-on. The intent is to also reuse this UX for when an add-on is no longer recommended, and potentially for add-on blocking, due to the current UX, which silently displays a warning icon on top the hamburger menu, proving to be insufficient in making users aware of something needing their attention as users on older versions of their add-ons are a security risk and need to be minimized.
Purpose
This document proposes to detail a test approach for the new UX designed to increase user awareness regarding the state of their add-ons, which includes Entry/Exit/Acceptance criteria, Testing scope, references to testcases, etc.
Entry Criteria
- QA has access to all PRDs, mocks and related documentation
- The feature has landed on Nightly and Beta
Exit Criteria
- All feature related bugs have been triaged
- All P1/P2 bugs have been fixed
- All resolved bugs have been verified by QA
- The find/fixed bug ratio shows a descending trend over a defined time period
Acceptance Criteria
This section proposes to highlight the criteria concerning the shipment readiness status of the product.
- QA has signed off
- All the required Telemetry is in place
Scope
This section outlines which parts of the new implemented feature will or will not be tested.
What is in scope
- Validation of the new UX when an add-on changes permissions and can’t be updated because of it, when an installed add-on is no longer recommended and can’t be updated because of it and when add-ons are prohibited from being installed (blocked).
What is out of scope
- Security testing
- Device testing
- Performance testing
Ownership
Dev Lead????????????: Franziskus Kiefer ; irc nick: fkiefer or :franziskus
QA Manager: Krupa Raj; irc nick: krupa
QA Lead: Victor Carciu; irc nick: victorc
Webextensions QA: Alex Cornestean; irc nick: AlexC_
Requirements for testing
Environments
Covered OSes: Windows, macOS, Linux
Channel dependent settings (configs) and environment setups
Nightly
none
Beta
none
Release
none
Post Beta / Release
The feature is enabled by default.
Test Strategy
Test Objectives
This section details the progression test objectives that will be covered.
Ref | Function | Test Objective | Test Type | Owners |
---|---|---|---|---|
TO-1 | Checking if the prompt is shown/not shown when auto-updating an add-on with changed/unchanged permissions | To verify that the prompt is displayed/not displayed accordingly | Manual | Add-ons QA Team |
TO-2 | Checking if the prompt can be accepted/cancelled | To verify that the auto-update proceeds or is cancelled accordingly | Manual | Add-ons QA Team |
TO-3 | Checking if the prompt is shown/not shown when auto-updating to a non-recommended/to a recommended of an add-on | To verify that the prompt is displayed/not displayed accordingly | Manual | Add-ons QA Team |
TO-4 | Checking if the prompt is shown/not shown when attempting to install a blacklisted/non-blacklisted add-on | To verify that the prompt is displayed/not displayed accordingly | Manual | Add-ons QA Team |
Builds
This section contains links to builds with the feature
Test Execution Schedule
The below table outlines the anticipated testing time frame available for test creation and execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | 05/31/2019 | |
Study documentation/specs received from developers | 05/31/2019 | |
QA - Test plan creation | 07/10/2019 | |
QA - Test cases/Env preparation | 07/10/2019 | |
QA - Nightly Testing | ||
QA - Beta Testing | ||
Release Date |
Testing Tools
Exemplifies the tools used for test suite creation/execution.
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | TestRail |
Test case execution | TestRail |
Bugs management | Bugzilla |
References
* List and links for specs PRD - Gdocs Install flow - Presentation * bug 1403838 - [Meta] Multiple-signed add-ons
ID | Priority | Component | Assigned to | Summary | Status | Target milestone |
---|---|---|---|---|---|---|
1169532 | -- | Security | extension XPI signing still uses SHA1 for digests; should use SHA2 | VERIFIED | --- | |
1357815 | P1 | Security: PSM | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | support SHA-256 when verifying PKCS7 signatures on addons | VERIFIED | mozilla58 |
1403840 | P1 | Security: PSM | Franziskus Kiefer [:franziskus] | Implement COSE for the new add-on signatures | RESOLVED | mozilla59 |
1403844 | P1 | Security: PSM | Franziskus Kiefer [:franziskus] | Integrate COSE rust library in PSM | VERIFIED | mozilla59 |
1415991 | P1 | Security: PSM | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | remove support for verifying signed unpacked add-ons | RESOLVED | mozilla59 |
1421413 | P1 | Security: PSM | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | add a preference to control the accepted signature algorithms for add-ons | VERIFIED | mozilla59 |
1421816 | P1 | Security: PSM | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | add an option to sign_app.py to include a COSE signature | RESOLVED | mozilla59 |
1422904 | -- | Add-ons Manager | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | add an integration test for an add-on signed with sha256 | RESOLVED | mozilla59 |
1436948 | -- | Security: PSM | Franziskus Kiefer [:franziskus] | Update cbor lib | RESOLVED | mozilla60 |
1471185 | -- | Security | Greg G | Implement COSE XPI signing in Autograph | RESOLVED | --- |
1472104 | P1 | Security: PSM | Franziskus Kiefer [:franziskus] | Test autograph-signed extension | VERIFIED | mozilla63 |
1475084 | P1 | Security: PSM | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | add tampered signature testcases for COSE-signed add-ons (like we have for PKCS7) | RESOLVED | mozilla63 |
12 Total; 0 Open (0%); 7 Resolved (58.33%); 5 Verified (41.67%);
Testcases
Overview
The test suite proposes a series of test cases devised to cover different add-on installation scenarios under the effects of the SHA256 signing mechanism.
- Validation of the installation process of SHA256 and SHA1 signed add-ons using different methods.
Test Areas
Test Areas | Covered | Details |
---|---|---|
Installing from local storage | ||
Installing as a temporary extension | ||
Installing from thirdparty | ||
Installing via sideloading | ||
Installing extensions with missing/corrupted signatures | ||
Installing from AMO |
Sign off
Criteria
Check list
- All test cases should be executed
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Checklist
Exit Criteria | Status | Notes/Details |
---|---|---|
Testing Prerequisites (specs, use cases) | ||
Testing Infrastructure setup | ||
Test Plan Creation | 07-10-2019 | |
Test Cases Creation | 07-10-2019 | |
Full Functional Tests Execution | ||
Automation Coverage | ||
Performance Testing | ||
All Defects Logged | ||
Critical/Blockers Fixed and Verified | ||
Metrics/Telemetry | ||
QA Signoff - Nightly Release | Email to be sent | |
QA Beta - Full Testing | ||
QA Signoff - Beta Release | Email to be sent |