User:Clouserw/AMO/Perf

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

The current state of things

Originally the plan for this was someone uploads an add-on, AMO sends the add-on to Talos (build machines), the build machines install the add-on on Firefox with a clean profile on several OSs, runs it a few times recording interesting numbers and then sends those numbers back to AMO. AMO compares those numbers with an average (also reported from Talos) and if the numbers are too far apart it shows performance warnings on AMO.

The system was constructed and launched briefly but there was backlash from add-on developers since we never really communicated with them and were suddenly flagging their add-ons as slowing down Firefox. Many bugs were filed asking for clarification of process and pointing out holes in the system (like add-ons that only slow Firefox down after the browser is used or settings are set). The Talos processing proved to be unreliable and sometimes took 20 minutes for a single add-on. The code was turned off in production and has been ignored since.

Next steps for performance

Firefox sluggishness due to add-ons is still an important metric to track and report since it's one of the larger complaints of end users. However, automating the process is dangerous (as we've witnessed) so the new plan is to remove the previous code and implement a new system which has 100% manual reporting. Performance warnings will only be shown on add-ons where someone (at Mozilla) has run tests and confirmed the add-on is causing performance problems.

Technical Details and Steps

The basic steps to accomplish the next phase:

• Remove the current system from the code and database.
  • Save the performance warning div's on the details pages. We can reuse them.
• Create a CRUD form to enter this data
  • Group permissions are AddonPerf:Edit
  • Form is accessible from a link associated with a file. That is to say, Performance data is associated with individual .xpi files
  • The form consists of:
    • Operating System: No version. Options: Windows, OS X, Linux, Android
    • Application: No version. Options: Firefox
    • Performance metric: Options: startup, memory, general
    • Severity: Options: no impact, small impact, major impact
    • Notes: An optional field for any notes from the reviewer
• Hook up the performance warning div's on the details pages, adjusting the wording for the new severity levels
• Send the new performance data through the API:

API

A <warnings> element would be added to the <addon> element, which contains <performance> elements to describe performance warnings. It would be present in:

• Keyword search API (to display warnings when searching for addons to install)
• GUID search API (to display warnings for installed addons, and in install confirmation dialogs)

This also apples to non-hosted addons, although they would only be returned in a GUID search. All <addon> elements would indicate if the addon was hosted or not (same as the <addon_compatibility> elements do). If an non-hosted addon has no warnings, then the returned data would not include a <addon> element for that addon. If a hosted addon has no warnings, then the <warnings> element would not be included (as opposed to being included, but empty).

Warnings for all versions of the addon need to be included, as the installed version may not be the latest version. Although, this is only a hard requirement for GUID search - keyword search can potentially only include data for the latest version.

Based on the assumption that many warnings will apply to more than one OS, the os attribute of <performance> elements is a comma-separated list of OS names. If warnings are different for the different OSes, then multiple <performance> elements can be included, with different values for the os attribuite. The values need to match the strings used by nsIXULRuntime.OS - "Darwin" for OSX, "WINNT" for Windows, "Linux" for Linux, "Android" for Android.

Each performance metric has a severity. Rather than use numeric values like usual (which has caused issues in the past when retroactively adding levels), the value would be a string. If the severity is "no impact", then that metric isn't included.

<?xml version="1.0" encoding="utf-8" ?>
<searchresults total_results="1">
    <addon hosted="false">
        <name>Addon Xtreme</name>
        <guid>xtreme@mozzarella.com</guid>
        <warnings>
            <performance application="{firefox-guid}" os="WINNT" version="1.2.3">
                <metric type="startup" severity="minor"/>
                <metric type="general" severity="major"/>
             </performance>
            <performance application="{firefox-guid}" os="Darwin,Linux,WINNT" version="1.2.0">
                <metric type="memory" severity="major"/>
             </performance>
        </warnings>
    </addon>
</searchresults>

Open Questions

• (An add-on can have many versions which can have many files) AMO for the most part advertises versions to the end user and figures out what file to serve (often only differing in OS) automatically. Would it be more appropriate to associate performance data with add-on versions rather than files?
• A similar question for add-ons not hosted at AMO. Associating with GUID+Version sounds like the right thing to do there anyway.
• How this works for add-ons not hosted at AMO is ignored in the steps above still.