Add-ons/QA/Testplan/Find Addons Replacement
From MozillaWiki
< Add-ons
Revision History
Date | Version | Author | Description |
---|---|---|---|
05/10/2017 | 1.0 | Carciu Victor | Created first draft |
Contents
Overview
A new feature is being added to Firefox with a link to AMO to find alternative add-ons for Legacy extensions which do not meet the Firefox standards anymore.
Purpose
Reducing delays in reviews (delays present due to decreased reviewer activity and increase submissions) by automatically reviewing some add-ons (that meet a list of criteria).
Entry Criteria
Current Status
- The feature was released.
Exit Criteria
- All related bugs triaged
- All blockers fixed
- All resolved bugs verified by QA
- Found-fixed bugs rate going down in time
Scope
what's in scope?
- Add a counter for approvals since last automatic review
- Get extensions auto-approved if they meet the following criteria:
- Have been manually approved once
- Are not flagged for admin review (this can happen automatically on submission in some cases, but is generally done manually by reviewers)
- Are WebExtensions
- Active Daily Users are under 10K (initially set to this value, but configurable).
- Are not triggering any of the following linter warnings:
- CSP
- Native Messaging
Not yet implemented:
- Auto-approve extensions that are not triggering content scripts warnings for all URLs at validation
what's out of scope?
- Human unreviewed extensions
- Embedded Webextensions
Ownership
Product Manager: Jorge Villalobos; irc nick :jorgev
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Add-ons QA: Valentina Virlics; irc nick :ValentinaV
Requirements for testing
Environments
- Windows
- Mac OS
Servers
- Stage: https://addons.allizom.org/en-US/
- Dev: https://addons-dev.allizom.org/en-US/
- Production: https://addons.mozilla.org/en-US/
Channels
- Release
Test Strategy
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | 31.01.2017 | |
Study PRD/mocks received | 01.02.2017 | |
QA - Test plan creation | 21.04.2017 | |
QA - Test cases preparation | 21.04.2017 | |
QA - Test cases execution | 24.04.2017 | |
Release Date | 27.04.2017 (Q1 - 2017) |
Created an etherpad with more detailed scenarios - Auto-Approvals
Testing Tools
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | TestRail |
Test case execution | TestRail |
Bugs management | Github |
References
* PRD - https://docs.google.com/document/d/1VxvvzQQvfDOVCylSVlpfeXv0tDYRLZfIMvWOoraANGU/edit# * Tracker: 4533
Testcases
Test Areas
- Submission Tools/Reviewer Tools
Test suite
Full Test Suite
Bug Work
- Feature implementation bug - 4533
Open bugs
- 325 - Automatically flag add-ons using native messaging for admin review
- 5273 - Add ability to store reviewer approval for auto-approved add-ons
- 5274 - Stop sending email to developers when requesting super-review of an auto-approved add-on
- 5275 - Implement rejection of auto-approved add-ons by human reviewers
- 5292 - Add locked/has_info_request/has_admin_review flags to AutoApprovalSummary
Closed bugs
- 5155 - Automate auto-approval task
- 5154 - Add a list of auto-approved add-ons in reviewer tools
- 4752 - Maintain a counter of the number of times an add-on has been approved
- 4855 - Set up auto_approve command logging to be shown in syslog and console
- 4730 - Remove old auto validation code
- 5164 - Reset AddonApprovalsCounter for add-ons when they are auto approved
Sign off
Criteria
Check list
- All test cases should be executed
- All blockers must be fixed and verified or have an agreed-upon timeline for being fixed