Add-ons/QA/Testplan/Add-ons Post Reviews Process

Revision History

Date	Version	Author	Description
20/06/2017	1.0	Valentina Virlics	Created first draft

Overview

• Add-ons listed on AMO are all pre-reviewed by a team of employees and volunteers. This process creates problems with waiting times, especially for add-ons escalated for special review, leading to developers to choose self-hosting or not develop for Firefox at all.
• We aim to a post-review model (for WebExtensions), where they are approved and published - if they pass a series of automatic tests - and manually post-reviewed.

Goals

• Automatically approve WebExtensions without human intervention
  • Decrease the waiting time for publishing a WebExtension or a WebExtension update to less than one day. Currently, waiting times vary between a few hours to several weeks.
  • Improve the value of user feedback channels (abuse reports, ratings on AMO) so they can be used to prioritize post-reviews.
  • Better expose developers to documentation on add-on policies and rules during the submission process.

Entry Criteria

• QA has access to PRD and some mocks (found in bugs)
• The feature has landed in -dev

Current Status

• The feature is under continuous development

Exit Criteria

• All related bugs triaged
• All blockers fixed
• All resolved bugs verified by QA
• Found-fixed bugs rate going down in time

Scope

what's in scope?

• Deploy Auto-approval
  • • WebExtensions will continue to be submitted through the regular flow, and a command will be run regularly (~every hour) to evaluate and auto-approve some of them, based on criteria defined in the linked PRD.

• Implement post-review list for auto-approved add-ons
  • • Add-ons that are auto-approved will appear in the post-review queue from Reviewer Tools.
      • List will contain: add-on name and version number (linking to the corresponding review page), last review - time since last manual review (in days), Flags and Weight (sorted after weight)

In discussions:

what's out of scope?

• Add-ons/Webextensions functionality

Ownership

Product Manager: Jorge Villalobos; irc nick :jorgev
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Add-ons QA: Valentina Virlics; irc nick :ValentinaV

Requirements for testing

Environments

• Windows
• Mac OS

Servers

• Stage: https://addons.allizom.org/en-US/
• Dev: https://addons-dev.allizom.org/en-US/
• Production: https://addons.mozilla.org/en-US/

Channels

• Release

Test Strategy

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project	14.03.2017
Study PRD/mocks received	20.04.2017
QA - Test plan creation	20.06.2017
QA - Test cases preparation
QA - Test cases execution
Release Date	Firefox Release 57

Testing Tools

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	[TestRail] / [Google docs]
Test case execution	[TestRail]
Bugs management	Github

References

• PRD - https://docs.google.com/document/d/1BdiJSYD4aOMClasGxx5wVShHEk8VbxefE3Gf_6jpgkQ/edit#
• Tracker: https://github.com/mozilla/addons-server/issues/5211
• Add-ons Auto Approvals test plan: https://wiki.mozilla.org/Add-ons/QA/Testplan/Auto-Approvals
• Implementation plan for Q2: https://docs.google.com/document/d/1rZvM2QGk8WtkNUDRvLEvrbqZ190hvHfOle4zVRllQmw/edit
• Post-review weights for prioritization: https://docs.google.com/spreadsheets/d/1CZEECHHqEmK87fNiWNslAsFQBL7zxYF-c63-zZVZuqw/edit#gid=0

Testcases

Test Areas

• Review of add-ons

Test suite

Full Test Suite: [here]

Bug Work

• Feature implementation bug - 5210

Bug fix verification

• Open
• Closed

Sign off

Criteria

Check list

• All test cases should be executed
• All blockers must be fixed and verified or have an agreed-upon timeline for being fixed