View source for Marketplace/Reviewers/Apps/Testing

== System Setup ==
WebApp (aka App) installation is currently supported on Nightly and Aurora channels and is officially released on Aurora channel.  It can be useful to test on Nightly in case of bugs but Aurora is normally the best channel to routinely use.

Get both Desktop and Mobile (for Tablets too - we need the native, rather than xul version) Nightly from [http://nightly.mozilla.org/ here].  Similarly Aurora from [http://www.mozilla.org/firefox/aurora/ here]. Once installed you should be able to update from within the App.

For FirefoxOS the optimal test environment is an actual device.  For trivial testing (e.g. a minor UX change in the app or a check to make sure the manifest installs correctly) the [https://addons.mozilla.org/en-US/firefox/addon/firefox-os-simulator/ Firefox OS Simulator] may be used.  The simulator is ''not'' sufficient for initial testing of device compatibility.

Packaged App installation requires [[Marketplace/Reviewers/Apps/InstallingReviewerCerts|adding additional certificates]] to the phone.

== Known Issues ==
=== General Runtime issues ===
* Desktop gecko is (obviously) a lot more mature than on Android/FxOS so rendering glitches are rare.  There are nonetheless still some occasional issues with the webapp runtime implementation, especially as the development focus has been launching on Android first.  If something odd happens it can sometimes help to try and run the app content in the browser directly.
* Web apps on Android and FirefoxOS regularly throw up bugs that effect either all web content, or the webapp implementation specifically.  All issues should be logged as bugs on [https://bugzilla.mozilla.org/ Bugzilla].

=== Specific Bugs ===
<bugzilla>
    {
        "bug_id": "818871,818924,816891,810360,803044,793900,791281,774727,749792"
    }
</bugzilla>
(edit page to add more to bug_id list when they occur)

== Apps Queue ==

There are four App queues (plus moderated user reviews) but this guide relates to the primary 'Apps' Queue (aka pending; nominated; submissions).  The Apps queue is sorted by waiting time from the original submission, with the oldest App at the top - this means older apps will appear at the top of the queue even if they've only just been resubmitted. 

The columns: 
*App name. This is a link to the review page. A lock icon next to it means that another reviewer is looking at this review page.
*Flags.
** Pending info request: a blue icon with an 'i' in the middle. A reviewer has requested additional information from the developer(s). Developers will reply to the mailing list.
** Reviewer comments: a user icon with a speech bubble. The comment should be read carefully since it may be important. 
** Packaged App: a yellow cube.  This is a packaged, rather than hosted, app which may be priviledged so have more abilities than a hosted app.  Currently only Staff should review these add-ons.
*Waiting time: this is from the first submission date, not the most recent update.
*Devices: The devices this App supports. 
*Payments: Free, or not Free.
*Abuse Reports.

Where possible try to review Apps in queue order, prioritising the Apps at the top of the queue. (Taking into account outstanding issues and info requests)

== Testing Procedure - Hosted webapps ==
Generally we go as many steps as possible to test the App, rather stopping at the first issue and rejecting.  I.e. in the steps below 'reject' doesn't mean reject immediately, just that the reason should be noted down and it shouldn't be approved after you've exhausted the steps below.

Apps that are multi-platform (more than one of Desktop, Android, FirefoxOS) should be tested on all supported platforms.  (The slight exception is if the app appears to work on Desktop and Android Mobile its not always necessary to test on an Android Tablet).

See the [https://developer.mozilla.org/docs/Apps/Marketplace_review_criteria Marketplace Review Criteria] for details of what we allow and don't allow in Apps for listing on Marketplace.  The steps below outline the brief procedure, not the policy.
  
* Check the app has a sensible name, summary, description and icon.  The description should be extensive enough for a user to understand what the app does (you may need to revisit this after launching the app). If not, reject.
* Have a quick look at the app manifest (the 'View' link next to the manifest url).  If the manifest obviously isn't valid JSON/isn't found it won't install anyway.  The [https://developer.mozilla.org/docs/Apps/Manifest manifest spec] should be consulted if you aren't sure about syntax.  Any issues, reject.
* Take note of any requested permissions in the manifest.  There is a [[Marketplace/Reviewers/Apps/Permissions|Security Checklist]] of available APIs and what they might be used/abused for.  There are only a few APIs are available to hosted apps (alarms, desktop-notification, geolocation, fmradio)
* Press the install button. It will install natively so on Desktop you have to go find it in your Applications folder, start menu or desktop. On mobile platforms a shortcut will appear on your homescreen.
* Check the app's shortcut has an icon.  The default rocketship icon is not allowed any more.  If not, reject (there is a canned response).  There is an occasional issue on Windows where sometimes the icon shown is cached from previous installs or appears broken at first so if it seems to be missing open the properties dialog and see if an icon is shown in the dialog.
* Launch the app.  If the app doesn't launch as you would expect (just a random website, 404, directory listing, etc) then its likely that they need to specify a launch_path.  Reject - there is a canned response.
* Give the app a quick try and see what experience a new user would have.  
* Some apps require a login.  If its straightforward you should register as a new user (to see what experience an actual user would have).  If the app requires paid credentials; specific details; or isn't in a language you can understand sufficiently you can request a username & password - there is a canned response - with Request Information.
* All links load within the App unless explicitly specified otherwise so we need to check the app doesn't have any links (often in a navigation bar) that take the user out of the App 'experience' and strand them in a normal website (remember there is no back or home button in an app!).  If there are such links then reject and explain they need to add the target="_blank" parameter to launch the link externally (in Firefox proper)
* A common issue on FirefoxOS is 'download halted' during installation - this is most often due to a broken appcache, either because it doesn't exist.  You can check the manifest at [http://manifest-validator.com/ manifest-validator.com]
* Its important to note that we don't make any relevance or quality judgements about how the app ''looks'' in an App Review, only that it functions correctly.  The [https://developer.mozilla.org/docs/Apps/Marketplace_review_criteria review criteria] document should be consulted. You can make suggestions about how to improve the app though if you notice anything that would make it better. 

If there is anything that requires Admin/Staff attention you can request a super review.  This will remove the app from the current queue and place it in the Escalation queue.

== Testing Procedure - Packaged Apps ==

''These should only be reviewed by Marketplace Staff currently.''

The procedure is similar to [[#Testing_Procedure_-_Hosted_webapps|hosted apps]].  Currently packaged apps are only supported on FirefoxOS.  Packaged App installation requires [[Marketplace/Reviewers/Apps/InstallingReviewerCerts|adding additional certificates]] to the phone.

See the [https://developer.mozilla.org/docs/Apps/Marketplace_review_criteria Marketplace Review Criteria] for details of what we allow and don't allow in Apps for listing on Marketplace.  The steps below outline the brief procedure, not the policy.

* Check the app has a sensible name, summary, description and icon.  The description should be extensive enough for a user to understand what the app does (you may need to revisit this after launching the app). If not, reject.
* The manifest url (view) link only contains some details from the actual manifest, which is inside the (zip) package.  To download the package for offline inspection, etc, click the 'package_path' link - this shouldn't be routinely necessary.
* In the version table at the bottom of the view load the validation report and inspect any warnings/errors.
* Then inspect the app contents via the 'contents' link.
* The first file should be the manifest.  
* Check the type entry.  If there is no type entry in the manifest, or its 'web', the app should be treated the same as a hosted one so it is not necessary to check the js code.
* If the type is 'privileged' then the app has access to extra APIs and all code needs to be inspected before approval.  (See subsequent steps)
* Take note of any requested permissions in the manifest.  There is a [[Marketplace/Reviewers/Apps/Permissions|Security Checklist]] of available APIs and what they might be used/abused for.  
* Check all the files, in particular the .js files (thankfully inline js and external files aren't allowed by the CSP), paying attention to how any permissions requested are used. **Need to expand here a little**
* Launch the app on the device and give the app a quick try and see what experience a new user would have.  
* Some apps require a login.  If its straightforward you should register as a new user (to see what experience an actual user would have).  If the app requires paid credentials; specific details; or isn't in a language you can understand sufficiently you can request a username & password - there is a canned response - with Request Information.
* Its important to note that we don't make any relevance or quality judgements about how the app ''looks'' in an App Review, only that it functions correctly.  The [https://developer.mozilla.org/docs/Apps/Marketplace_review_criteria review criteria] document should be consulted. You can make suggestions about how to improve the app though if you notice anything that would make it better.

== Communication with other App Reviewers / Admins / Platform Developers / Marketplace Developers ==
=== IRC ===
* #amo-editors - for review policy questions (make it clear its about an App rather than Addon review - there isn't a dedicated channel yet)
* #marketplace - for Marketplace.mozilla.org technical/bug questions.  
* #openwebapps - for technical questions about the App Runtime.  Normally the first place to ask before logging a bug.

=== Email ===
* App-reviews@mozilla.org - for any concerns or questions related to the review process or changing manifest URLs, and anything related to administration of the Marketplace. This list includes the Marketplace reviewers.
* marketplace-developer-support@mozilla.org - for questions related to potential Marketplace bugs, like "My app manifest won't validate". This list includes Marketplace dev, QA, and support folks.