Reviewing add-ons in the sandbox

What is the sandbox?

What add-ons are in the sandbox?

The sandbox is where all add-ons that are hosted on AMO go, to start. It contains new versions of public add-ons, as well as all versions for add-ons that are not made public. When a new add-on, or an update to an existing add-on, is submitted to AMO, it is placed in the sandbox.

Some add-ons, and their specific versions, are made public after the review process indicates that they are ready and appropriate for public display. Other add-ons will remain in the sandbox indefinitely, where they are available to users who choose to browse the sandbox list and experiment with the software there.

How do add-ons become public?

Add-ons are reviewed by AMO users who opt into viewing the sandbox and testing the packages found there. The reviews that AMO users write will indicate whether an add-on is sufficiently useful, well-written and polished to be put in front of all of Firefox's users. These reviews, possibly in addition to other reviews and inspections by the AMO team, are used to determine whether a given add-on should be made public, whether it needs more work to be polished for wider visibility, or whether it's not suitable for promotion on the AMO site outside of the sandbox.

How do I get my add-on promoted to public status?

If you believe that your add-on (and your behaviour!) meets the criteria for a public add-on, you can nominate it from the developer control panel.

What are the criteria for public add-ons?

An add-on that's made public on AMO should be of high quality, and give users an improved web experience. We look for the following things when deciding whether an add-on is appropriate for the public side of AMO:

Are you responsive?

We expect that an author who is promoting their add-on to Firefox's many users is responsive to problem reports, maintains their contact information, and updates their add-on promptly to keep current with Firefox releases and changes in AMO policies. This doesn't mean that you have to reply to every question that someone posts in the discussions, or that you even need to fix every bug, but we do expect that you will respond to issues in a manner that's appropriate to the severity of the issue in question.

Is the add-on clearly and accurately described?

It's of the utmost importance to us that users get what they expect when they try a new add-on. Your description should provide details about what the add-on does, how a user should take advantage of it, and what the user should expect when they install it. Links to external docs for detailed instructions are fine, but the description itself should cover the basics and leave users confident that they know what they'll get.

Also, it is important that you maintain version notes appropriately as you improve and change your add-on. Users should be able to see what's new in an add-on they may have tried previously, and should be made aware of changes that might affect their current use of the add-on when they update. (Right now, users don't see the version notes when they're prompted for an update within the browser, but we'll work to fix that. If you maintain the version notes well, your users will benefit greatly before and after that work is complete.)

Are all privacy and security concerns clearly spelled out?

This is an aspect of a clear and accurate description, but such an important one that we feel it deserves specific mention. Many very useful and well-written add-ons manipulate some form of user data, or can present security hazards if misused; they are welcome on the public portion of AMO, but they must make it very clear to users what risks they might encounter, and what they can do to protect themselves.

Has the add-on been well-tested, and is it free of obvious or serious defects?

One important thing that we look for when considering an add-on for public access is whether its sandbox reviews indicate that it has received thorough testing, and that it doesn't have serious problems or negative impacts on the browser. If reviewers report problems such as major performance issues, crashes, frequent problems using the functions of the add-on, or spamming of messages to the error console, you should take those reports to heart, and re-nominate your add-on after you've addressed them as best you can. We don't expect you to perfectly optimize or have zero bugs -- Firefox itself undergoes constant improvement in these areas -- but we do want you to take reasonable efforts to minimize downsides, and to clearly call out cases where users may be surprised by those that remain.

If your add-on has been tested outside of the AMO Sandbox process, such as by a group of users of your service or an in-house QA team, you should indicate that in your nomination message. It certainly helps us establish what the level of testing has been, and can help get your add-on up on the site.

Do the add-on and add-on author both treat the user respectfully?

Your software should not intrude on the user unnecessarily, try to trick the user, or conceal any of its activities from the user. Users (or even non-users) are sometimes rude in their comments, and while we will do our best to filter those out as they're reported to us, we do expect that authors will avoid retaliating with rudeness of their own.

Is the add-on useful to an appropriately wide portion of Firefox's users?

Your add-on doesn't need to be the next Greasemonkey or FireBug, but if it is only useful to people at your company or who are part of a small web community, we may feel that it's not yet appropriate to put it in front of all of Firefox's users.

We are constantly looking at ways to improve the organization of the site to better accommodate add-ons that are exemplary in other ways, but are aimed at only a small community of potential users. Correctly categorizing and maintaining the metadata of your add-on will help us figure out how we can surface more of those sorts of add-ons to people who are most likely to benefit from them.

If your add-on just provides bookmarks or other simple access points to your site, it's probably not appropriate for the public part of the site. Like the rest of the Mozilla project, we love web applications and new web services, but Firefox add-ons should provide an improved browsing experience for the user and not just be a way to promote a new site or service through an AMO listing. If the description for your add-on is mostly about the service rather than the improvements it makes to the user's browser experience, you're probably not on the right track.

Is the add-on free of unlicensed trademarks or copyrights?

Though you may mean no harm to the holder of a trademark, or the owner of a copyrighted work, we can't host add-ons that infringe on trademarks or copyrights. If you don't have permission to use a trademarked name or image, please do not submit your add-on to AMO. If your add-on includes code that is copyrighted by someone else, and is not licensed to you to use in your add-on, please do not submit your add-on to AMO. (If the holder of a trademark or copyright objects to the use of their trademark, we will very likely have to have the request for removal reviewed by counsel, and we will remove the add-on if it's deemed legally necessary. This is an expensive process in terms of the project's resources, including time and money, so we ask you to be respectful and not cause us undue difficulty.)

If you're not sure if the name of your add-on, or use of something within it, will prevent it from being listed on the site, you can ask amo-editors@mozilla.org for guidance. IMPORTANT: Please note that this group is not able to provide legal advice, and that even if we feel that your usage is acceptable, we may revisit that decision in light of complaints from rights-holders and advice from legal counsel.

In terms of reuse of source code from other add-ons, if the author has not clearly stated that you are permitted to use her code in your own work -- such as by placing it under an open source license -- then you should assume that you do not have the right to do so. You can contact the author to seek such permission, but we can't provide you with any special rights to it just because it's been on AMO, or because the author isn't responding to your request. (And, again, we can't provide legal advice, just advice about how your add-on is likely to interact with the policies of the site.)

This applies to the Mozilla Foundation's trademarks as well, including "Mozilla", "Firefox", and "Thunderbird". The Mozilla policy on trademark use is designed to protect against confusion, and prevent the trademarks from being overturned due to lack of protection; please respect the need for such protection, and help us preserve some of the most valuable assets of the Mozilla Foundation.

What happens after I nominate something?

Once your add-on has been nominated, it is evaluated by a team of AMO Editors according to the criteria described above. If it is deemed ready for public display, it will be pushed to the public side once it has been evaluated, and you will receive an email notification.

If we feel that the add-on isn't appropriate for the public side of AMO at this time, you'll receive an email notification indicating why, and your nomination will be removed from the queue. If and when you feel that you've addressed the concerns expressed in that notification, and you want to be evaluated again, you can do so at your discretion. Repeated nominations without meaningful improvements in the add-on are not looked upon with favour, so please do exercise discretion; you are more likely to anger us than to wear us down.

Can I nominate someone else's add-on?

Currently, we ask that an add-on's author nominate their own work for publication. We want to make sure that the author is comfortable with the increased exposure and that feel the add-on in its current state appropriately reflects the quality of their work. If you believe that an add-on is polished, that the author is abiding by the letter and spirit of the AMO policies, and that it would benefit Firefox, our users, and the web in general to have it made available to nearly a hundred million users around the world, you should feel free to encourage the author of the add-on to nominate their creation.

My add-on has been in the nomination queue for a long time, do you hate me?

We don't hate you. We love add-on developers, and we work hard to make them happy and productive, so that users all over the world can benefit from their work. But being on the public side of AMO has value precisely because we take care in what ends up there, so we can't rush just to make it go faster. We appreciate that it can be frustrating to wait for your add-on to be evaluated, and we want to keep the turnaround time as short as possible. The more people provide careful and clear reviews of add-ons in the sandbox, the easier it is to perform these evaluations, so you might also consider helping out on that side if you're so inclined.

I found a serious bug in my add-on, and I really want to get the fix up there quickly. What should I do?

If there is a serious bug (security, stability, major functionality problem) in an add-on for which you need to get an update out promptly, you should indicate that in the "reviewer notes" when submitting the update -- as well as in the version notes, obviously! You may also want to enlist some existing users of your add-on to test the update and report their results in detail in the sandbox. Popping into #addons on irc.mozilla.org can help make people aware of the situation, but please be patient and polite if you do so.

Please don't cry wolf. We try to jump quickly on high-priority updates, but it costs us time evaluating other nominated add-ons or versions, and often it costs us sleep or time with our families and friends, so we take a dim view of people who try to take advantage of this mechanism to "jump the queue". If you're not sure whether you should go this route, asking on #addons on irc.mozilla.org may well help you decide.

I think I was treated unfairly in the evaluation of my add-on. What should I do?

If you believe that your add-on was incorrectly evaluated, and that it was denied public status in error, you should send an email to amo-editors@mozilla.org with the details of your reasoning. Please be polite and clear in your email, and make sure that you have specifics about how the add-on was misjudged.

(If you have fixed *all* the things that were listed as problems in your notification mail, you shouldn't appeal the evaluation, but should instead re-nominate for consideration through the Developer Control Panel.)

My add-on used to be public, and now it's only in the sandbox. What happened?

If an add-on no longer meets the criteria for being on the public side of the site, we may move it back to the sandbox. Unless we are legally prevented from doing so, we will notify you by email when that happens, and indicate the reasons for doing so.

It's also possible that you've found a bug in the site, in which case you should report it via Bugzilla; use the "addons.mozilla.org" product and the "Public Pages" component for your report, and include as much detail as you can.

My add-on is public, and people seem to love it. How can I get in the list of recommended add-ons?

If you believe that your add-on is a shining example of the power of add-ons, that it demonstrates and furthers Mozilla's values for the extensible and user-controlled web, and that it provides a great user experience, you can ask to have it considered for addition to the list of recommended add-ons. To do this, you should send an email to amo-editors@mozilla.org explaining why your add-on is great.

Your mail should include at least information about these things:

• how the web experience is improved for users
• how your add-on is appropriate for a large portion of Firefox's users
• how your add-on demonstrates and/or serves the values of the Mozilla project, especially with respect to putting the user in charge, protection of privacy and security, universal access to the web, and open standards and data
• how your add-on differs from other similar add-ons (in ways that are both better and worse)
• what reaction you've seen from users, reviewers, bloggers, astronauts, or your household pets, both positive and negative

The more complete the information you provide in your request the more amenable we are likely to be to granting it, though even a wonderfully-written and exhaustive application is no guarantee of placement in the recommended list. Ultimately, that list must be -- and is -- maintained at the discretion of Mozilla, and user experience and protection must trump all else.