Add-ons/QA/Testplan/Add-ons Post Reviews Process: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(40 intermediate revisions by the same user not shown)
Line 6: Line 6:
|-
|-
| 20/06/2017 || 1.0 || Valentina Virlics || Created first draft
| 20/06/2017 || 1.0 || Valentina Virlics || Created first draft
|-
| 31/08/2017 || 2.0 || Valentina Virlics || Updated
|-
|-
|}
|}
Line 16: Line 18:
== Goals ==
== Goals ==
* '''Automatically approve WebExtensions without human intervention'''
* '''Automatically approve WebExtensions without human intervention'''
** Decrease the waiting time for publishing a WebExtension or a WebExtension update to less than one day. Currently, waiting times vary between a few hours to several weeks.
** Decrease the waiting time for publishing a WebExtension or a WebExtension update to approximately 15 minutes. Currently, waiting times vary between a few hours to several weeks.
** Improve the value of user feedback channels (abuse reports, ratings on AMO) so they can be used to prioritize post-reviews.
** Improve the value of user feedback channels (abuse reports, ratings on AMO) so they can be used to prioritize post-reviews.
** Better expose developers to documentation on add-on policies and rules during the submission process.
** Better expose developers to documentation on add-on policies and rules during the submission process.
Line 36: Line 38:
== Scope ==
== Scope ==
===what's in scope?===
===what's in scope?===
'''1. Deploy Auto-approval'''  
'''1. Deploy Auto-approval'''  
*WebExtensions will continue to be submitted through the regular flow, and a command will be run regularly (~every hour) to evaluate and auto-approve some of them, based on criteria defined in the linked [https://docs.google.com/document/d/1VxvvzQQvfDOVCylSVlpfeXv0tDYRLZfIMvWOoraANGU/edit#heading=h.diha6b7e1on6 PRD].
* WebExtensions will continue to be submitted through the regular flow, and a command will be run regularly (~every hour) to evaluate and auto-approve some of them, based on criteria defined in the linked [https://docs.google.com/document/d/1VxvvzQQvfDOVCylSVlpfeXv0tDYRLZfIMvWOoraANGU/edit#heading=h.diha6b7e1on6 PRD].
    
    
'''2. Implement post-review list for auto-approved add-ons'''
'''2. Implement post-review queue for auto-approved add-ons'''
*Add-ons that are auto-approved will appear in the post-review queue from [https://addons-dev.allizom.org/en-US/editors/queue/auto_approved Reviewer Tools]. List will contain: add-on name and version number (linking to the corresponding review page), last review - time since last manual review (in days), Flags and Weight (sorted after weight)  
* Add-ons that are auto-approved will appear in the post-review queue from [https://addons-dev.allizom.org/en-US/editors/queue/auto_approved Reviewer Tools].  
* List will contain: add-on name and version number (linking to the corresponding review page), last review (time since last manual review - in days), Flags and Weight (sorted after weight)  
    
    
'''3. Changes to Reviewer Tools'''
'''3. Changes to Reviewer Tools'''
*If the last review for an add-on was done manually or the user looking at the page doesn’t have the Addons:PostReview permission, show the current reviewer page.  
* If the last review for an add-on was done manually or the user looking at the page doesn’t have the Addons:PostReview permission, show the current reviewer page.  
*If the last review for an add-on was automatic and the user looking at the page has the Addons:PostReview permission, show the reviewer page with the following changes:
* If the last review for an add-on was automatic and the user looking at the page has the Addons:PostReview permission, show the reviewer page with the following changes:
- Display recent user ratings (3 stars of fewer) and abuse reports (for the add-on or the developers, if there are any reports), with links to the full lists, below the add-on metadata and right above “More about this add-on”.   
# Display recent user ratings (3 stars of fewer) and abuse reports (for the add-on or the developers, if there are any reports), with links to the full lists, below the add-on metadata and right above “More about this add-on”.   
- The "Confirm Approval" resolution should be available, and shouldn’t display the form for comments and canned responses. Instead, it should only show the Save button. Confirming doesn’t send any information to the developer or change its status. It only records it so the last manually-approved version is used to calculate the code changes compared to the latest version.   
# The "Confirm Approval" resolution should be available, and shouldn’t display the form for comments and canned responses. Instead, it should only show the Save button. Confirming doesn’t send any information to the developer or change its status. It only records it so the last manually-approved version is used to calculate the code changes compared to the latest version.   
- The Request super-review action shouldn’t send an email to the developer.
# Reject Multiple Versions - should allow the reviewer to select a range of versions to reject (disable) with a single review message.
- The Reject action should include a way to select which versions of the add-on are affected by the rejection. All of those versions should be disabled once the review is saved, and should be listed in the email sent to the developer.
# Reviewer reply - should work the same as with regular reviews. (combined with the possibility of a more info request checkbox option).
 
# Requesting super-review - should increase weight if the add-on wasn’t flagged for super-review before.
'''4. Post-review prioritization'''
# Adding a comment - should work the same as with regular reviews.
*The post-review list will be sorted according to a weighted sum of the following risk factors:
 
- The add-on has the admin review flag.   
'''4. Post-review prioritization'''  
- Flags raised during static analysis.
*The post-review list will be sorted according to a weighted sum of the following risk factors:  
- Size of code changed since last manual approval.   
# The add-on has the admin review flag.   
- User feedback obtained from abuse reports (for the add-on and the developers).   
# Flags raised by static validation after webextension submission: eval(), document.write(), setInterval/setTimeout (with a string, not a function), document.write, innerHTML, or a custom CSP;
- User feedback obtained from ratings left on add-on listings.   
# Size of code changed since last manual approval.   
- Add-on [https://github.com/mozilla/addons-server/issues/5520#event-1129062485 reputation], set by admin reviewers.
# User feedback obtained from abuse reports (for the add-on and the developers).   
- Number of active users.   
# User feedback obtained from ratings left on add-on listings.   
- Past rejection history.   
# Add-on [https://github.com/mozilla/addons-server/issues/5520#event-1129062485 reputation], set by admin reviewers.
# Number of active users.   
# Past rejection history.   
    
    
'''Note''': Add-on Reputation - is an admin-set override that helps rank down popular add-ons that are known to be high-quality and would generally rank higher due to code complexity and high volume of user feedback. This also includes add-ons developed by Mozilla.   
'''Note''': Add-on Reputation - is an admin-set override that helps rank down popular add-ons that are known to be high-quality and would generally rank higher due to code complexity and high volume of user feedback. This also includes add-ons developed by Mozilla.   
The reputation is an integer ranging between 0 and 3, that is set per-add-on, defaulting to 0.   
The reputation is an integer ranging between 0 and 3, that is set per-add-on, defaulting to 0.   
    
    
'''5. Submission process updates'''  
'''5. Submission process updates'''    
*The submission flow will have the following changes:
* The submission flow will have the following changes:
- Replace the Developer Agreement step with a step linking to the 3 main documents: Developer Agreement, Review Policy, and Review Rules. Each document will have a checkbox next to it indicating the developer has read them and agreed to them.
# New submissions should show the new Distribution Agreement/Review Policy with links to MDN .
- Ensure all developers see these documents on their next submission, even for an update.
# The last submission step should indicate the add-on will be available soon and not refer to waiting for review.
- For WebExtensions, a subset of flags raised by static validation will be shown after it completes. They are specified on this spreadsheet, in the column “Flag with developer during submission”.
# After submission, the uploaded version should be publicly available on AMO within 15 minutes (probably less time than that).
- For WebExtensions, the text in the last step should be adjusted to reflect the new process.
# Check that the add-on status is appropriately updated in the Developer Hub.
# Check that the add-on appears in the auto-approval list (requires the tester to have the Addons:PostReview permission).
# Check the add-ons and [https://github.com/mozilla/addons-server/issues/6033 weights] to verify they are being calculated correctly based on the spec.
    
    
'''6. Remove auto-approval restrictions'''
'''6. Remove auto-approval restrictions'''  
*All WebExtension submissions will be post-reviewed after this point
* All WebExtension submissions will be post-reviewed after this point


===what's out of scope?===
===what's out of scope?===
* Add-ons/Webextensions functionality
* Add-ons/Webextensions functionality


== Risks ==
*This is major change in the way we review add-ons. The security implications are significant, so getting security review and approval as early as possible will ensure this project won’t be delayed.
*The current reviewer team is trained to use the pre-review system, and are more familiar with legacy APIs than WebExtensions.
*A group of contractors is in process of being hired to help with WebExtension reviews, and they would be the first to handle the post-review process. Any delays in their on-boarding will lead to insufficient post-review staffing and higher security risk.
 
== Ownership ==
== Ownership ==
Product Manager: [mailto:jorge@mozilla.com Jorge Villalobos]; irc nick :jorgev<br />
Product Manager: [mailto:jorge@mozilla.com Jorge Villalobos]; irc nick :jorgev<br />
Line 83: Line 95:
Add-ons QA: [mailto:valentina.peleskei@softvision.ro Valentina Virlics]; irc nick :ValentinaV<br />
Add-ons QA: [mailto:valentina.peleskei@softvision.ro Valentina Virlics]; irc nick :ValentinaV<br />


= Requirements for testing =
== Requirements for testing ==
== Environments ==
=== Environments ===
* Windows
* Windows
* Mac OS
* Mac OS


== Servers ==
=== Servers ===
* Stage: https://addons.allizom.org/en-US/  
* Stage: https://addons.allizom.org/en-US/  
* Dev: https://addons-dev.allizom.org/en-US/  
* Dev: https://addons-dev.allizom.org/en-US/  
* Production: https://addons.mozilla.org/en-US/  
* Production: https://addons.mozilla.org/en-US/


== Channels ==
=== Channels ===
* Release
* Release


= Test Strategy =  
== Test Strategy ==
== Test Execution Schedule ==
=== Test Execution Schedule ===
The following table identifies the anticipated testing period available for test execution.
The following table identifies the anticipated testing period available for test execution.
{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
Line 113: Line 125:
|-
|-
| QA - Test cases preparation  
| QA - Test cases preparation  
|style="text-align:center;" | ||  
|style="text-align:center;" | 01.08.2017 ||  
|-
|-
| QA - Test cases execution   
| QA - Test cases execution   
|style="text-align:center;" | ||  
|style="text-align:center;" | 01.09.2017 ||  
|-
|-
| Release Date  
| Release Date  
|style="text-align:center;" | Firefox Release 57
|style="text-align:center;" | 15.09.2017
|}
|}


== Testing Tools ==
=== Testing Tools ===
{| class="wikitable" style="width:50%"
{| class="wikitable" style="width:50%"
|-
|-
Line 136: Line 148:
|}
|}


= References =
== References ==
* PRD - https://docs.google.com/document/d/1BdiJSYD4aOMClasGxx5wVShHEk8VbxefE3Gf_6jpgkQ/edit#
* PRD - https://docs.google.com/document/d/1hVx-NbuVRc0wKXSdt8eHceObexY93YFzMXtqYC95nUs/edit
* Tracker: https://github.com/mozilla/addons-server/issues/5211  
* Tracker bugs:  
* Add-ons Auto Approvals test plan: https://wiki.mozilla.org/Add-ons/QA/Testplan/Auto-Approvals
**https://github.com/mozilla/addons-server/issues/5211  
* Implementation plan for Q2: https://docs.google.com/document/d/1rZvM2QGk8WtkNUDRvLEvrbqZ190hvHfOle4zVRllQmw/edit
**https://github.com/mozilla/addons-server/issues/5579
* Post-review weights for prioritization: https://docs.google.com/spreadsheets/d/1CZEECHHqEmK87fNiWNslAsFQBL7zxYF-c63-zZVZuqw/edit#gid=0
* Add-ons [[Add-ons/QA/Testplan/Auto-Approvals|Auto Approvals]] test plan
* [https://docs.google.com/document/d/1rZvM2QGk8WtkNUDRvLEvrbqZ190hvHfOle4zVRllQmw/edit Implementation plan for Q2] 
* [https://docs.google.com/document/d/1dV_4mzAq-u7KQTqq6HdVuN955YvJTYMsCz1hvKwrLJY/edit# Post-review testing plan]
* [https://docs.google.com/spreadsheets/d/1CZEECHHqEmK87fNiWNslAsFQBL7zxYF-c63-zZVZuqw/edit#gid=0 Post-review weights for prioritization]


= Testcases =  
== Testcases ==  
== Test Areas ==
=== Test Areas ===
* Review of add-ons 
* Review of extensions


== Test suite ==
=== Test suite ===
  Full Test Suite: [here]
  Full [https://testrail.stage.mozaws.net/index.php?/suites/view/1429 Test Suite].


= Bug Work =
== Bug Work ==
* Feature implementation bug - [https://github.com/mozilla/addons-server/issues/5211 5210]   


== Bug fix verification ==
=== Bug fix verification ===
* [https://github.com/mozilla/addons-server/labels/project%3A%20auto%20approval Open]
* [https://github.com/mozilla/addons-server/labels/project%3A%20auto%20approval Open]
* [https://github.com/mozilla/addons-server/issues?q=label%3A%22project%3A+auto+approval%22+is%3Aclosed Closed]
* [https://github.com/mozilla/addons-server/issues?q=label%3A%22project%3A+auto+approval%22+is%3Aclosed Closed]


= Sign off =
== Sign off ==
== Criteria ==
=== Criteria ===
Check list
Check list
* All test cases should be executed
* All test cases should be executed
* All blockers must be fixed and verified or have an agreed-upon timeline for being fixed
* All blockers must be fixed and verified or have an agreed-upon timeline for being fixed

Latest revision as of 11:27, 4 September 2017

Revision History

Date Version Author Description
20/06/2017 1.0 Valentina Virlics Created first draft
31/08/2017 2.0 Valentina Virlics Updated

Overview

  • Add-ons listed on AMO are all pre-reviewed by a team of employees and volunteers. This process creates problems with waiting times, especially for add-ons escalated for special review, leading to developers to choose self-hosting or not develop for Firefox at all.
  • We aim to a post-review model (for WebExtensions), where they are approved and published - if they pass a series of automatic tests - and manually post-reviewed.


Goals

  • Automatically approve WebExtensions without human intervention
    • Decrease the waiting time for publishing a WebExtension or a WebExtension update to approximately 15 minutes. Currently, waiting times vary between a few hours to several weeks.
    • Improve the value of user feedback channels (abuse reports, ratings on AMO) so they can be used to prioritize post-reviews.
    • Better expose developers to documentation on add-on policies and rules during the submission process.


Entry Criteria

  • QA has access to PRD and some mocks (found in bugs)
  • The feature has landed in -dev

Current Status

  • The feature is under development

Exit Criteria

  • All related bugs triaged
  • All blockers fixed
  • All resolved bugs verified by QA
  • Found-fixed bugs rate going down in time

Scope

what's in scope?

1. Deploy Auto-approval

  • WebExtensions will continue to be submitted through the regular flow, and a command will be run regularly (~every hour) to evaluate and auto-approve some of them, based on criteria defined in the linked PRD.

2. Implement post-review queue for auto-approved add-ons

  • Add-ons that are auto-approved will appear in the post-review queue from Reviewer Tools.
  • List will contain: add-on name and version number (linking to the corresponding review page), last review (time since last manual review - in days), Flags and Weight (sorted after weight)

3. Changes to Reviewer Tools

  • If the last review for an add-on was done manually or the user looking at the page doesn’t have the Addons:PostReview permission, show the current reviewer page.
  • If the last review for an add-on was automatic and the user looking at the page has the Addons:PostReview permission, show the reviewer page with the following changes:
  1. Display recent user ratings (3 stars of fewer) and abuse reports (for the add-on or the developers, if there are any reports), with links to the full lists, below the add-on metadata and right above “More about this add-on”.
  2. The "Confirm Approval" resolution should be available, and shouldn’t display the form for comments and canned responses. Instead, it should only show the Save button. Confirming doesn’t send any information to the developer or change its status. It only records it so the last manually-approved version is used to calculate the code changes compared to the latest version.
  3. Reject Multiple Versions - should allow the reviewer to select a range of versions to reject (disable) with a single review message.
  4. Reviewer reply - should work the same as with regular reviews. (combined with the possibility of a more info request checkbox option).
  5. Requesting super-review - should increase weight if the add-on wasn’t flagged for super-review before.
  6. Adding a comment - should work the same as with regular reviews.

4. Post-review prioritization

  • The post-review list will be sorted according to a weighted sum of the following risk factors:
  1. The add-on has the admin review flag.
  2. Flags raised by static validation after webextension submission: eval(), document.write(), setInterval/setTimeout (with a string, not a function), document.write, innerHTML, or a custom CSP;
  3. Size of code changed since last manual approval.
  4. User feedback obtained from abuse reports (for the add-on and the developers).
  5. User feedback obtained from ratings left on add-on listings.
  6. Add-on reputation, set by admin reviewers.
  7. Number of active users.
  8. Past rejection history.

Note: Add-on Reputation - is an admin-set override that helps rank down popular add-ons that are known to be high-quality and would generally rank higher due to code complexity and high volume of user feedback. This also includes add-ons developed by Mozilla. The reputation is an integer ranging between 0 and 3, that is set per-add-on, defaulting to 0.

5. Submission process updates

  • The submission flow will have the following changes:
  1. New submissions should show the new Distribution Agreement/Review Policy with links to MDN .
  2. The last submission step should indicate the add-on will be available soon and not refer to waiting for review.
  3. After submission, the uploaded version should be publicly available on AMO within 15 minutes (probably less time than that).
  4. Check that the add-on status is appropriately updated in the Developer Hub.
  5. Check that the add-on appears in the auto-approval list (requires the tester to have the Addons:PostReview permission).
  6. Check the add-ons and weights to verify they are being calculated correctly based on the spec.

6. Remove auto-approval restrictions

  • All WebExtension submissions will be post-reviewed after this point

what's out of scope?

  • Add-ons/Webextensions functionality

Risks

  • This is major change in the way we review add-ons. The security implications are significant, so getting security review and approval as early as possible will ensure this project won’t be delayed.
  • The current reviewer team is trained to use the pre-review system, and are more familiar with legacy APIs than WebExtensions.
  • A group of contractors is in process of being hired to help with WebExtension reviews, and they would be the first to handle the post-review process. Any delays in their on-boarding will lead to insufficient post-review staffing and higher security risk.

Ownership

Product Manager: Jorge Villalobos; irc nick :jorgev
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Add-ons QA: Valentina Virlics; irc nick :ValentinaV

Requirements for testing

Environments

  • Windows
  • Mac OS

Servers

Channels

  • Release

Test Strategy

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Date
Start project 14.03.2017
Study PRD/mocks received 20.04.2017
QA - Test plan creation 20.06.2017
QA - Test cases preparation 01.08.2017
QA - Test cases execution 01.09.2017
Release Date 15.09.2017

Testing Tools

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail / Google doc
Test case execution TestRail
Bugs management Github

References

Testcases

Test Areas

  • Review of extensions

Test suite

Full Test Suite.

Bug Work

Bug fix verification

Sign off

Criteria

Check list

  • All test cases should be executed
  • All blockers must be fixed and verified or have an agreed-upon timeline for being fixed
Retrieved from "https://wiki.mozilla.org/index.php?title=Add-ons/QA/Testplan/Add-ons_Post_Reviews_Process&oldid=1179830"