Add-ons/QA/Testplan/Content handler API
Revision History
Date | Version | Author | Description |
---|---|---|---|
09/17/2018 | 1.0 | Vlad Jiman | Created first draft |
Contents
Overview
A content handler API would provide the means for extensions to handle responses and files for specific file (MIME) types.
Purpose
This document intends to detail the test approach to the Webextensions content handler API, including Entry/Exit criteria. Scope for testing, links to testcases etc
Entry Criteria
- QA has access to all the PRD's, mocks and related documents
- Kick-off meeting has been established
- The feature has landed on Nightly
- Testing extensions have been provided
Exit Criteria
- All the bugs against the feature have been triaged
- All the bugs have been fixed
- All the resolved bugs have been verified by QA
- The find/fixed rate is going down over a predefined period of time
Acceptance Criteria
This section broadly outlines when the product is ready to ship
- QA has signed off
Scope
This section describes what parts of the feature will be tested and what parts won't be.
what's in scope?
- Validating the provided API testing extension.
- Regression testing for the webextension after bugs have been fixed
- API handler's behaviour after certain interruptions, on start-up, session restore, closing an active window, closing an active tab and restoring it
what's out of scope?
- Performance testing
- Security testing
Ownership
Dev Lead: David Durst ; irc nick:ddurst
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Vlad Jiman; irc nick :VladJ
Requirements for testing
Environments
OSes covered: Windows, Mac OS X, Linux
Channel dependent settings (configs) and environment setups
Nightly
security.signed_app_signatures.policy with the default value 2
Beta
security.signed_app_signatures.policy with the default value 2
Release
Post Beta / Release
The feature is enabled by default.
Test Strategy
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
Ref | Function | Test Objective | Test Type | Owners |
---|---|---|---|---|
TO-1 | Installing API test extension | To verify that the installation process runs without any issue | Manual | Add-ons QA Team |
TO-2 | Extension first run | To check that the extension will be able to show intercepted API responses | Manual | Add-ons QA Team |
TO-3 | Extension run using different MIME types | Open various URL's in Firefox using different MIME types and verify that the extension is able to intercept the API request and handle it | Manual | Add-ons QA Team |
TO-4 | Run tests while having the pref ON | Pref: devtools.jsonview.enabled should be set to True | Manual | Add-ons QA Team |
TO-5 | Extension run using local files | To validate if Firefox can handle closing/starting up using local test files for this extension | Manual | Add-ons QA Team |
TO-6 | Extension run using various file location | URL's, ftp, HTTP redirrect, responses that are aborted | Manual | Add-ons QA Team |
Builds
This section should contain links for builds with the feature -
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | 09/12/2018 | |
Study documentation/specs received from developers | 09/12/2018 | |
QA - Test plan creation | 09-14-2018 | |
QA - Test cases/Env preparation | 09-14-2018 | |
QA - Nightly Testing | ||
QA - Beta Testing | ||
Release Date |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | [ Docs] / [ TestRail] |
Test case execution | [ Docs] / [ TestRail] |
Bugs management | Bugzilla / Github |
Status
Overview
Track the dates and build number where feature was released to Nightly Track the dates and build number where feature was merged to Release/Beta
Risk analysis
Identify the high-risk assumptions Identify existing bugs on the feature with high risk Identify if other areas are affected by the fix
- The test strategy recommended by devs is based on a provided extension that we will have to assume correctly handles the and intercepts responses
- Mitigation strategy - QA will test only the provided webextension with the most common file types to ensure that the most impactful scenarios are covered
References
* List and links for specs PRD - Gdocs * bug 1457500 - Support content handlers in WebExtensions
Test suite
- Link for the [ Initial test planning]
- Link for the [ Google doc tests]
- Link for the [ TestRail tests]
Bug Work
Tracking bug - [1457500]
Sign off
Criteria
Check list
- All test cases should be executed
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite, use template from []
Merge to Beta Sign-off
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite
Checklist
Exit Criteria | Status | Notes/Details |
---|---|---|
Testing Prerequisites (specs, use cases) | ||
Testing Infrastructure setup | ||
Test Plan Creation | 09-14-2018 | |
Test Cases Creation | ||
Full Functional Tests Execution | ||
Automation Coverage | ||
Performance Testing | ||
All Defects Logged | ||
Critical/Blockers Fixed and Verified | ||
Metrics/Telemetry | ||
QA Signoff - Nightly Release | QA Sign-off missed | |
QA Beta - Full Testing | ||
QA Signoff - Beta Release |