Add-ons/QA/Testplan/Signing add-ons with SHA+ (with backwards compatibility)
Revision History
Date | Version | Author | Description |
---|---|---|---|
04/30/2019 | 1.0 | Alex Cornestean | Created first draft |
10/09/2019 | 1.1 | Alex Cornestean | Updated several sections |
Please note: This document is still a work in progress and sections might contain placeholders and not accurate data
Overview
The purpose of this feature is to migrate from the current combination of extension signing mechanisms supported by Firefox, PKCS#7 (with SHA1 or SHA256) and COSE (SHA256), to the exclusive usage of the strong COSE (SHA256) mechanism. As such, with Firefox 70, only COSE (SHA256) signature/ (SHA256) manifest add-ons will be allowed to run. Consequently, the "security.signed_app_signatures.policy" pref will be changed from the current default value of “2” to “4” as to only allow re-signed add-ons to work.
With these future changes, add-ons having signatures based solely on the PKCS#7 standard will fail to install or run in Firefox. On the other hand, add-ons with COSE signatures [ES256] (which supports only SHA256 and does not support SHA1) will be install-able and run after the mandatory verification of the strong COSE signature. If there are any other PKCS#7 signatures (either SHA1 or SHA256) alongside the COSE [ES256] signature, these must verify as well, after the mandatory verification of the COSE signature.
Purpose
This document proposes to detail a test approach for the SHA feature, which includes Entry/Exit/Acceptance criteria, Testing scope, references to testcases, etc.
Entry Criteria
- QA has access to all PRDs, mocks and related documentation
- The feature has landed on Nightly
Exit Criteria
- All feature related bugs have been triaged
- All P1/P2 bugs have been fixed
- All resolved bugs have been verified by QA
- The find/fixed bug ratio shows a descending trend over a defined time period
Acceptance Criteria
This section proposes to highlight the criteria concerning the shipment readiness status of the product.
- QA has signed off
- All the required Telemetry is in place
Scope
This section outlines which parts of the new implemented feature will or will not be tested.
What is in scope
- Validation of the transition from the current extension signing mechanisms PKCS#7 (SHA1 and SHA256) and COSE to the exclusive usage of COSE which allows only SHA256 signature/SHA256 manifest add-ons or re-signed COSE add-ons.
What is out of scope
- Security testing
- Device testing
- Performance testing
Ownership
Dev Lead: Franziskus Kiefer ; irc nick: fkiefer or :franziskus
QA Manager: Krupa Raj; irc nick: krupa
QA Lead: Victor Carciu; irc nick: victorc
Webextensions QA: Alex Cornestean; irc nick: AlexC_
Requirements for testing
Environments
Covered OSes: Windows, Mac OS X, Linux
Channel dependent settings (configs) and environment setups
Nightly
security.signed_app_signatures.policy with the default value 4
Beta
security.signed_app_signatures.policy with the default value 4
Release
security.signed_app_signatures.policy with the default value 4
Post Beta / Release
The feature is enabled by default.
Test Strategy
Test Objectives
This section details the progression test objectives that will be covered.
Ref | Function | Test Objective | Test Type | Owners |
---|---|---|---|---|
TO-1 | Installing PKCS#7 and COSE extensions from local storage | To verify that the extension uses the API correctly | Manual | Add-ons QA Team |
TO-2 | Installing PKCS#7 and COSE extensions as temporary addons | To verify that the extension uses the API correctly | Manual | Add-ons QA Team |
TO-3 | Installing PKCS#7 and COSE extensions from thirdparty | To verify that the extension uses the API correctly | Manual | Add-ons QA Team |
TO-4 | Installing PKCS#7 and COSE extensions via sideloading | To verify that the extension uses the API correctly | Manual | Add-ons QA Team |
TO-5 | Installing PKCS#7 and COSE extensions with missing/corrupted signatures | To verify that the extension uses the API correctly | Manual | Add-ons QA Team |
TO-6 | Installing PKCS#7 and COSE extensions from AMO | To verify that the extension uses the API correctly | Manual | Add-ons QA Team |
Builds
This section should contain links for builds with the feature -
Test Execution Schedule
The below table outlines the anticipated testing time frame available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | 04/19/2019 | 08/23/2019 |
Study documentation/specs received from developers | 04/19/2019 | 05/15/2019 |
QA - Test plan creation | 04/30/2019 | 05/15/2019 |
QA - Test cases/Env preparation | 05/03/2019 | 05/15/2019 |
QA - Nightly Testing | 05/06/2019 | 08/23/2019 |
QA - Beta Testing | 05/06/2019 | 08/23/2019 |
Release Date | - | 08/23/2019 |
Testing Tools
Exemplifies the tools used for test suite creation/execution.
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | TestRail |
Test case execution | TestRail |
Bugs management | Bugzilla |
References
* List and links for specs PRD - Gdocs Install flow - Presentation * bug 1403838 - [Meta] Multiple-signed add-ons
ID | Priority | Component | Assigned to | Summary | Status | Target milestone |
---|---|---|---|---|---|---|
1169532 | -- | Security | extension XPI signing still uses SHA1 for digests; should use SHA2 | VERIFIED | --- | |
1357815 | P1 | Security: PSM | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | support SHA-256 when verifying PKCS7 signatures on addons | VERIFIED | mozilla58 |
1403840 | P1 | Security: PSM | Franziskus Kiefer [:franziskus] | Implement COSE for the new add-on signatures | RESOLVED | mozilla59 |
1403844 | P1 | Security: PSM | Franziskus Kiefer [:franziskus] | Integrate COSE rust library in PSM | VERIFIED | mozilla59 |
1415991 | P1 | Security: PSM | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | remove support for verifying signed unpacked add-ons | RESOLVED | mozilla59 |
1421413 | P1 | Security: PSM | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | add a preference to control the accepted signature algorithms for add-ons | VERIFIED | mozilla59 |
1421816 | P1 | Security: PSM | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | add an option to sign_app.py to include a COSE signature | RESOLVED | mozilla59 |
1422904 | -- | Add-ons Manager | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | add an integration test for an add-on signed with sha256 | RESOLVED | mozilla59 |
1436948 | -- | Security: PSM | Franziskus Kiefer [:franziskus] | Update cbor lib | RESOLVED | mozilla60 |
1471185 | -- | Security | Greg G | Implement COSE XPI signing in Autograph | RESOLVED | --- |
1472104 | P1 | Security: PSM | Franziskus Kiefer [:franziskus] | Test autograph-signed extension | VERIFIED | mozilla63 |
1475084 | P1 | Security: PSM | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | add tampered signature testcases for COSE-signed add-ons (like we have for PKCS7) | RESOLVED | mozilla63 |
12 Total; 0 Open (0%); 7 Resolved (58.33%); 5 Verified (41.67%);
Testcases
Overview
The test suite proposes a series of test cases devised to cover different add-on installation scenarios under the effects of the SHA256 signing mechanism.
- Validation of the installation process of SHA256 and SHA1 signed add-ons using different methods.
Test Areas
Test Areas | Covered | Details |
---|---|---|
Installing from local storage | Yes | Passed |
Installing as a temporary extension | Yes | Passed |
Installing from thirdparty | Yes | Passed |
Installing via sideloading | Yes | Passed |
Installing extensions with missing/corrupted signatures | Yes | Passed |
Installing from AMO | Yes | Passed |
Sign off
Criteria
Check list
- All test cases should be executed
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Checklist
Exit Criteria | Status | Notes/Details |
---|---|---|
Testing Prerequisites (specs, use cases) | - | - |
Test Plan Creation | 04-30-2019 | Shared for review |
Test Cases Creation | 05-03-2019 | Shared for review |
Full Functional Tests Execution | 07-08-2019 | - |
QA Signoff - Nightly Release | - | Nightly testing not required |
QA Beta - Full Testing | Complete | Passed |
QA Signoff - Beta Release | 08-23-2019 | Green Sign-Off sent |