Add-ons/QA/Testplan/SHA

From MozillaWiki
Jump to: navigation, search

Revision History

Date Version Author Description
11/09/2017 1.0 Cosmin Badescu Created first draft


Overview

The purpose of this feature is to switch from the current extension-signing mechanism (SHA1) to a more newer one (SHA+).

Purpose

This document purports to detail the test approach to SHA and including Entry/Exit criteria, Scope for testing, links to testcases etc

Entry Criteria

  • QA has access to all the PRDs, mocks and related documents
  • The feature has landed on Nightly
  • AMO parts has landed on dev

Exit Criteria

  • All the bugs against the feature have been triaged
  • All the P1/P2 bugs have been fixed
  • All the resolved bugs have been verified by QA
  • The find/fixed rate is going down over a predefined period of time

Acceptance Criteria

This section broadly outlines when the product is ready to ship

  • QA has signed off
  • All the required Telemetry is in place
  • All info is localized at least for a pre-defined set of locales
  • All the necessary PR/blogposts have been sent out

Scope

This section describes what parts of the feature will be tested and what parts won't be.

what's in scope?

  • The transition that involves, in first place, the introduction of the new SHA+, along with the SHA1 and in the final phase, the cut-over of the old SHA1.

what's out of scope?

  • Performance testing

Ownership

Dev Lead: Franziskus Kiefer ; irc nick:fkiefer or :franziskus
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Cosmin Badescu; irc nick :CosminB
Add-ons QA: Valentina Peleski; irc nick :ValentinaV

Requirements for testing

Environments

OSes covered: Windows, Mac OS X, Linux

Channel dependent settings (configs) and environment setups

Nightly

security.signed_app_signatures.policy with the default value 2

Beta

security.signed_app_signatures.policy with the default value 2

Release

Post Beta / Release

The feature is enabled by default.

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Test Type Owners
TO-1 Installing from AMO To verify that the extension uses the API correctly Manual Add-ons QA Team
TO-2 Installing from local files To verify that the extension uses the API correctly Manual Add-ons QA Team
TO-3 Installing from thirdparty To verify that the extension uses the API correctly Manual Add-ons QA Team
TO-4 Add-on updates To verify that the extension uses the API correctly Manual Add-ons QA Team
TO-5 Sideloading To verify that the extension uses the API correctly Manual Add-ons QA Team

Builds

This section should contain links for builds with the feature -

  • Link for Nightly builds
  • Link for Beta builds
  • Link for Release builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project
Study documentation/specs received from developers
QA - Test plan creation 11-09-2017
QA - Test cases/Env preparation
QA - Nightly Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation [ Docs] / [ TestRail]
Test case execution [ Docs] / [ TestRail]
Bugs management Bugzilla / Github

Status

Overview

Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Release/Beta

Risk analysis

Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix

References

* List and links for specs
  PRD - Gdocs
  Install flow - Presentation
  

* bug 1403838 - [Meta] Multiple-signed add-ons
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1169532 -- Security extension XPI signing still uses SHA1 for digests; should use SHA2 VERIFIED ---
1357815 P1 Security: PSM Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) support SHA-256 when verifying PKCS7 signatures on addons VERIFIED mozilla58
1403840 P1 Security: PSM Franziskus Kiefer [:franziskus] Implement COSE for the new add-on signatures RESOLVED mozilla59
1403844 P1 Security: PSM Franziskus Kiefer [:franziskus] Integrate COSE rust library in PSM VERIFIED mozilla59
1415991 P1 Security: PSM Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) remove support for verifying signed unpacked add-ons RESOLVED mozilla59
1421413 P1 Security: PSM Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) add a preference to control the accepted signature algorithms for add-ons VERIFIED mozilla59
1421816 P1 Security: PSM Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) add an option to sign_app.py to include a COSE signature RESOLVED mozilla59
1422904 -- Add-ons Manager Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) add an integration test for an add-on signed with sha256 RESOLVED mozilla59
1436948 -- Security: PSM Franziskus Kiefer [:franziskus] Update cbor lib RESOLVED mozilla60
1471185 -- Security Greg G Implement COSE XPI signing in Autograph RESOLVED ---
1472104 P1 Security: PSM Franziskus Kiefer [:franziskus] Test autograph-signed extension VERIFIED mozilla63
1475084 P1 Security: PSM Dana Keeler (she/her) (use needinfo) [:keeler] (on leave) add tampered signature testcases for COSE-signed add-ons (like we have for PKCS7) RESOLVED mozilla63

12 Total; 0 Open (0%); 7 Resolved (58.33%); 5 Verified (41.67%);


Testcases

Overview

Summary of testing scenarios

Test Areas

Test Areas Covered Details
Installing from AMO
Installing from local files
Installing from thirdparty
Add-on updates
Sideloading
Other

Test suite

  • Link for the [ Initial test planning]
  • Link for the [ Google doc tests]
  • Link for the [ TestRail tests]

Bug Work

Tracking bug - []

Bug fix verification

[Verified] [ Bug xxxxxxx] - Display permissions prompt for webextensions installed using mozAddonManager

2017-01-10: verified fixed on 53.0a1 across platforms

[Verified] [ Bug xxxxxxx] - Prompt users with permissions for third-party webextensions installs

2015-04-21: verified fixed on 53.0a1 across platforms
Logged bugs

[ Bug xxxxxxx] - Misaligned icon and webextension name in permissions doorhanger

Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite, use template from []


Merge to Beta Sign-off List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation 11-09-2017
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release Email to be sent