Add-ons/QA/Testplan/SHA
Revision History
Date | Version | Author | Description |
---|---|---|---|
11/09/2017 | 1.0 | Cosmin Badescu | Created first draft |
Contents
Overview
The purpose of this feature is to switch from the current extension-signing mechanism (SHA1) to a more newer one (SHA+).
Purpose
This document purports to detail the test approach to SHA and including Entry/Exit criteria, Scope for testing, links to testcases etc
Entry Criteria
- QA has access to all the PRDs, mocks and related documents
- The feature has landed on Nightly
- AMO parts has landed on dev
Exit Criteria
- All the bugs against the feature have been triaged
- All the P1/P2 bugs have been fixed
- All the resolved bugs have been verified by QA
- The find/fixed rate is going down over a predefined period of time
Acceptance Criteria
This section broadly outlines when the product is ready to ship
- QA has signed off
- All the required Telemetry is in place
- All info is localized at least for a pre-defined set of locales
- All the necessary PR/blogposts have been sent out
Scope
This section describes what parts of the feature will be tested and what parts won't be.
what's in scope?
- The transition that involves, in first place, the introduction of the new SHA+, along with the SHA1 and in the final phase, the cut-over of the old SHA1.
what's out of scope?
- Performance testing
Ownership
Dev Lead: Franziskus Kiefer ; irc nick:fkiefer or :franziskus
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Cosmin Badescu; irc nick :CosminB
Add-ons QA: Valentina Peleski; irc nick :ValentinaV
Requirements for testing
Environments
OSes covered: Windows, Mac OS X, Linux
Channel dependent settings (configs) and environment setups
Nightly
security.signed_app_signatures.policy with the default value 2
Beta
security.signed_app_signatures.policy with the default value 2
Release
Post Beta / Release
The feature is enabled by default.
Test Strategy
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
Ref | Function | Test Objective | Test Type | Owners |
---|---|---|---|---|
TO-1 | Installing from AMO | To verify that the extension uses the API correctly | Manual | Add-ons QA Team |
TO-2 | Installing from local files | To verify that the extension uses the API correctly | Manual | Add-ons QA Team |
TO-3 | Installing from thirdparty | To verify that the extension uses the API correctly | Manual | Add-ons QA Team |
TO-4 | Add-on updates | To verify that the extension uses the API correctly | Manual | Add-ons QA Team |
TO-5 | Sideloading | To verify that the extension uses the API correctly | Manual | Add-ons QA Team |
Builds
This section should contain links for builds with the feature -
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | ||
Study documentation/specs received from developers | ||
QA - Test plan creation | 11-09-2017 | |
QA - Test cases/Env preparation | ||
QA - Nightly Testing | ||
QA - Beta Testing | ||
Release Date |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | [ Docs] / [ TestRail] |
Test case execution | [ Docs] / [ TestRail] |
Bugs management | Bugzilla / Github |
Status
Overview
Track the dates and build number where feature was released to Nightly Track the dates and build number where feature was merged to Release/Beta
Risk analysis
Identify the high-risk assumptions Identify existing bugs on the feature with high risk Identify if other areas are affected by the fix
References
* List and links for specs PRD - Gdocs Install flow - Presentation * bug 1403838 - [Meta] Multiple-signed add-ons
ID | Priority | Component | Assigned to | Summary | Status | Target milestone |
---|---|---|---|---|---|---|
1169532 | -- | Security | extension XPI signing still uses SHA1 for digests; should use SHA2 | VERIFIED | --- | |
1357815 | P1 | Security: PSM | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | support SHA-256 when verifying PKCS7 signatures on addons | VERIFIED | mozilla58 |
1403840 | P1 | Security: PSM | Franziskus Kiefer [:franziskus] | Implement COSE for the new add-on signatures | RESOLVED | mozilla59 |
1403844 | P1 | Security: PSM | Franziskus Kiefer [:franziskus] | Integrate COSE rust library in PSM | VERIFIED | mozilla59 |
1415991 | P1 | Security: PSM | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | remove support for verifying signed unpacked add-ons | RESOLVED | mozilla59 |
1421413 | P1 | Security: PSM | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | add a preference to control the accepted signature algorithms for add-ons | VERIFIED | mozilla59 |
1421816 | P1 | Security: PSM | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | add an option to sign_app.py to include a COSE signature | RESOLVED | mozilla59 |
1422904 | -- | Add-ons Manager | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | add an integration test for an add-on signed with sha256 | RESOLVED | mozilla59 |
1436948 | -- | Security: PSM | Franziskus Kiefer [:franziskus] | Update cbor lib | RESOLVED | mozilla60 |
1471185 | -- | Security | Greg G | Implement COSE XPI signing in Autograph | RESOLVED | --- |
1472104 | P1 | Security: PSM | Franziskus Kiefer [:franziskus] | Test autograph-signed extension | VERIFIED | mozilla63 |
1475084 | P1 | Security: PSM | Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) | add tampered signature testcases for COSE-signed add-ons (like we have for PKCS7) | RESOLVED | mozilla63 |
12 Total; 0 Open (0%); 7 Resolved (58.33%); 5 Verified (41.67%);
Testcases
Overview
Summary of testing scenarios
Test Areas
Test Areas | Covered | Details |
---|---|---|
Installing from AMO | ||
Installing from local files | ||
Installing from thirdparty | ||
Add-on updates | ||
Sideloading | ||
Other |
Test suite
- Link for the [ Initial test planning]
- Link for the [ Google doc tests]
- Link for the [ TestRail tests]
Bug Work
Tracking bug - []
Bug fix verification
[Verified] [ Bug xxxxxxx] - Display permissions prompt for webextensions installed using mozAddonManager
- ↳ 2017-01-10: verified fixed on 53.0a1 across platforms
[Verified] [ Bug xxxxxxx] - Prompt users with permissions for third-party webextensions installs
- ↳ 2015-04-21: verified fixed on 53.0a1 across platforms
Logged bugs
[ Bug xxxxxxx] - Misaligned icon and webextension name in permissions doorhanger
Sign off
Criteria
Check list
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite, use template from []
Merge to Beta Sign-off
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite
Checklist
Exit Criteria | Status | Notes/Details |
---|---|---|
Testing Prerequisites (specs, use cases) | ||
Testing Infrastructure setup | ||
Test Plan Creation | 11-09-2017 | |
Test Cases Creation | ||
Full Functional Tests Execution | ||
Automation Coverage | ||
Performance Testing | ||
All Defects Logged | ||
Critical/Blockers Fixed and Verified | ||
Metrics/Telemetry | ||
QA Signoff - Nightly Release | Email to be sent | |
QA Beta - Full Testing | ||
QA Signoff - Beta Release | Email to be sent |