Add-ons/Reviewers/Guide/Reviewing: Difference between revisions

Jump to navigation Jump to search

Add-ons/Reviewers/Guide/Reviewing (view source)

Revision as of 19:45, 21 January 2015

3,070 bytes removed ,  21 January 2015
Cleaned up Feature Review section
m (Formatting)
(Cleaned up Feature Review section)
Line 218: Line 218:
= Step 4: Feature Review  =
= Step 4: Feature Review  =


The last step in a review is to install and test the add-on.  
The last step in a review is to install and test the add-on. This step is only necessary for an add-ons that are requesting Full Review.


== Testing setup  ==
Add-ons are normally cross-platform, in which case there will only be a single file to review. If the add-on is offered for a limited number of platforms or has different files for different platforms, there will be individual links for each one in the Add-on History entry. In this case all supported platforms should be tested.


These are a few settings and tools you should use or consider using when setting up your add-on testing environment:
Regarding application support, you don't need to test the add-on for all of them. If the add-on supports Firefox and others, it's OK to only test on Firefox.


*'''Always use a separate profile for testing''', never your main profile. See [https://developer.mozilla.org/en/Setting_up_extension_development_environment Setting up an extension development environment].
== Testing setup  ==
*Ideally you should perform your tests in a virtual machine. It is always useful in case you need to test in multiple platforms. [http://www.virtualbox.org/ VirtualBox] is free and supports most platforms.
*Enable full Error Console reporting, as described in [https://developer.mozilla.org/en/Setting_up_extension_development_environment#Development_preferences the Development Preferences section].
*These add-ons help you inspect add-on behavior and find possible solutions for problems:
**[https://addons.mozilla.org/en-US/firefox/addon/6622 DOM Inspector]. Analyze XUL and HTML layout, CSS and even JS objects.
**[https://addons.mozilla.org/en-US/firefox/addon/1815 Console<sup>2</sup>] for detailed Console logging.
**[https://addons.mozilla.org/en-US/firefox/addon/966 Tamper Data] or [https://addons.mozilla.org/en-US/firefox/addon/3829 Live HTTP Headers] for HTTP traffic analysis.
**[https://addons.mozilla.org/en-US/firefox/addon/2490 Leak Monitor] to detect some types of memory leak. See [https://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_3.0_TestPlan:Leaks:LeakTesting-How-To#Leak_Gauge Leak Gauge] for a more general solution.
**[https://addons.mozilla.org/en-US/firefox/addon/extension-test Extension Test] to detect loose variables and DOM IDs, prototype extension, dangerous category registration, and other difficult to spot problems. Also automatically sets the required dom.report_all_js_exceptions and javascript.options.showInConsole preferences.
*To test Fennec add-ons, you'll need to [https://wiki.mozilla.org/Fennec#Test_Builds install Fennec]. It is preferred that you test in a supported mobile device. If a Fennec nomination has been waiting for long, it's OK to test with the desktop XULRunner application.
*For online malware scanning, you can use [http://www.virustotal.com/ Virus Total], [http://www.kaspersky.com/scanforvirus Kaspersky online scan] and [http://virusscan.jotti.org Jotti online scan]. AMO performs virus checks, and binary add-ons should be admin-reviewed anyway, but if you suspect anything, those are good tools to use.


== Installing and testing ==
* '''Always use a separate profile for testing''', never your main profile. See [https://developer.mozilla.org/en/Setting_up_extension_development_environment Setting up an extension development environment].
 
* Ideally you should perform your tests in a virtual machine. It is always useful in case you need to test in multiple platforms. [http://www.virtualbox.org/ VirtualBox] is free and works well.  
Add-ons are normally cross-platform, so there will only be a single file to review, in the same section where the validator and source links are located. If the add-on is offered for a limited number of platforms, there will be individual links for each one of them. In this case all supported platforms should be tested.
* The [https://addons.mozilla.org/addon/extension-test Extension Test add-on] helps detect loose variables and DOM IDs, prototype extension, dangerous category registration, and other difficult to spot problems.
 
* Test with the Browser Console always open, and look for errors or log messages generated by the add-on.
Regarding applications, you don't need to test the add-on for all applications it supports. If the add-on supports Firefox and others, it's OK to only test on Firefox. If, however, an add-on update introduces Fennec or other application support, the add-on should be tested on it. The applications we support for reviews are listed on [https://addons.mozilla.org/en-US/firefox/pages/appversions this page].


== Policies and actions  ==
== Policies and actions  ==


{| width="700" cellspacing="0" cellpadding="1" border="0"
{| width="80%" cellspacing="0" cellpadding="1" border="0"
|-
|-
! style="border-bottom: 2px solid black" scope="col" | Issue  
! style="border-bottom: 2px solid black" scope="col" | Issue  
Line 250: Line 239:
! style="border-bottom: 2px solid black" scope="col" | Notes
! style="border-bottom: 2px solid black" scope="col" | Notes
|- style="vertical-align: top;" id="security-violations"
|- style="vertical-align: top;" id="security-violations"
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Security violations  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Security violations.
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Reject  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Reject  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Adding HTTP content to secure pages. Visit HTTPS sites like addons.mozilla.org and make sure the identity button is unchanged. This is specially important for add-ons that insert scripts into sites.
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Adding HTTP content to HTTPS pages. If the add-on injects content like iframes or images, make sure to visit HTTPS sites the add-on supports and look for any security warnings in the URL bar.
[[Image:Identity-button.png|center|Identity button]]
 
|- style="vertical-align: top;" id="no-surprises-violations"
|- style="vertical-align: top;" id="no-surprises-violations"
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | [https://addons.mozilla.org/en-US/developers/docs/policies/reviews#section-defaults No Surprises] violation  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | [https://addons.mozilla.org/en-US/developers/docs/policies/reviews#section-defaults No Surprises] violation  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Preliminary Review
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Preliminary review
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Changing homepage, default search provider, including unexpected ads or content changes without explicit user opt-in.
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Changing homepage, default search provider, including unexpected ads or content changes without explicit user opt-in.
|- style="vertical-align: top;" id="privacy-violations"
|- style="vertical-align: top;" id="privacy-violations"
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Privacy violations  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Privacy violations.
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Preliminary Review
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Preliminary review
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Incorrect or insufficient privacy policies, not respecting Private Mode.
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Incorrect or insufficient privacy policies, not respecting Private Browsing Mode.
|- style="vertical-align: top;" id="modal-startup"
|- style="vertical-align: top;" id="modal-startup"
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Showing a modal dialog at startup  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Showing a modal dialog at startup.
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Preliminary Review
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Preliminary review
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Many add-ons open dialogs or new tabs at startup, mostly offering information on getting started. This is useful, but it shouldn't block the user from using the browser. Opening modal (blocking) dialogs at startup is not allowed. Non-modal dialogs, separate windows or new tabs are allowed.
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Opening modal (blocking) dialogs at startup is not allowed. Non-modal dialogs, separate windows or new tabs are allowed.
|- style="vertical-align: top;" id="errors"
|- style="vertical-align: top;" id="errors"
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Errors in the Error Console  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Errors in the Browser Console.
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Preliminary Review
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Preliminary review
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" |  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Make sure the errors only occur with the add-on installed and are generated from add-on code and not Firefox code. In the latter case, it should only be noted.
|- style="vertical-align: top;" id="confusing"
|- style="vertical-align: top;" id="confusing"
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Add-on is very hard to use without instructions  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Add-on is very hard to use even with instructions.
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Preliminary Review
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Preliminary review
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | If the add-on is difficult to use, there should be instructions included in the add-on descriptions, or in a startup page or window.
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | If the testing instructions are missing, use Request more information. If even then it is too hard to use, give Preliminary review.
|- style="vertical-align: top;" id="sticky-toolbar-buttons"
|- style="vertical-align: top;" id="sticky-toolbar-buttons"
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Toolbar buttons are not customizable  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Toolbar buttons are not customizable.
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Preliminary Review
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Preliminary review
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" |
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | It must be possible to remove add-on buttons from the toolbar and move them to the menu panel.
|- style="vertical-align: top;" id="leaks"
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Memory leaks from content or chrome
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Preliminary Review
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | If an add-on touches the content in any way, we need to test the add-on on a page where it works, close the page, and then look in about:memory?verbose. Remaining compartments related to the page indicate a memory leak. After disabling the add-on (and minimizing memory usage) the add-on should no longer use any memory. If it does, it leaks to chrome. More details [https://developer.mozilla.org/en/Extensions/Performance_best_practices_in_extensions#Avoid_Creating_Memory_Leaks here] and
[https://developer.mozilla.org/en/Zombie_Compartments#Proactive_checking_of_add-ons here].
|- style="vertical-align: top;" id="affiliate"
|- style="vertical-align: top;" id="affiliate"
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Affiliate linking  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Affiliate linking.
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Preliminary Review
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Preliminary review
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | See details below.
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Some add-ons add affiliate codes to Amazon links (or similar) in order to make money. At the moment we allow this as long as (1) the add-on follows the No Surprises policy, (2) the feature doesn't replace or remove any existing affiliate codes, (3) the affiliate codes aren't inserted in the merchant website's links (inserting Amazon affiliate codes in Amazon.com pages).
|- style="vertical-align: top;" id="third-party-software"
|- style="vertical-align: top;" id="third-party-software"
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Requires third party software or paid registration  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Requires third party software or paid registration.
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Admin Review  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Admin Review  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | This excludes add-ons that require other add-ons to function, like Firebug extensions.
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" |
|- style="vertical-align: top;" id="content-ads"
|- style="vertical-align: top;" id="content-ads"
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Inserts ads into content  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Inserts ads into content.
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Admin Review  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | Admin Review  
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | The rules in these cases are complex. They need to be clearly labeled as coming from the add-on (otherwise Prelim). They can't remove or replace existing ads (otherwise Reject). They need to follow No Surprises (otherwise Prelim). And there are also security concerns (Reject) and privacy concerns (Prelim).
| style="padding: .5ex 1ex 1ex 0; border-bottom: 1px solid black;" | The rules in these cases are complex. They need to be clearly labeled as coming from the add-on (otherwise Prelim). They can't remove or replace existing ads (otherwise Reject). They need to follow No Surprises (otherwise Prelim). And there are also security concerns (Reject) and privacy concerns (Prelim).
|}
|}


Other tests to perform:  
Other tests to perform:


*Visit a very simple website like example.org and inspect its DOM, looking for any changes. Again, this is particularly important for extensions that insert scripts or make DOM changes.
*Open the add-on's preferences window, from the Add-ons Manager and elsewhere, and verify that preference changes apply properly. Make sure the window fits all of its contents (a common problem in Mac OS).
*Test all add-on features, within reason. If there too many, focus on the main features.
*Test all add-on features, within reason. If there too many, focus on the main features.
*Remove all added toolbar buttons, disable all added toolbars, and restart the browser. Make sure that buttons and toolbars are all removable and do not reappear on restart. Make sure that missing toolbar buttons to not cause errors to appear in the Error Console.
*Remove all added toolbar buttons, disable all added toolbars, and restart the browser. Make sure that buttons and toolbars are all removable and do not reappear on restart. Make sure that missing toolbar buttons to not cause errors to appear in the Error Console.
*Open the Customize Toolbar dialog and make sure that all buttons have appropriate icons and label text.
*Open the Customize Toolbar dialog and make sure that all buttons have appropriate icons and label text.
*Affiliate linking. Some add-ons add affiliate codes to Amazon links (or similar) in order to make money. At the moment we allow this as long as (1) the add-on follows the No Surprises policy, (2) the feature doesn't replace or remove any existing affiliate codes, (3) the affiliate codes aren't inserted in the merchant website's links (inserting Amazon affiliate codes in Amazon.com pages).
*Open the add-on's preferences window, from the Add-ons Manager and elsewhere, and verify that preference changes apply properly. Make sure the window fits all of its contents (a common problem in Mac OS).


= Step 5: Resolution  =
= Step 5: Resolution  =
Jorge.villalobos
canmove, Confirmed users
1,448

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1049557"

Navigation menu