User:Clouserw/AMO/Perf

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

The current state of things

Originally the plan for this was someone uploads an add-on, AMO sends the add-on to Talos (build machines), the build machines install the add-on on Firefox with a clean profile on several OSs, runs it a few times recording interesting numbers and then sends those numbers back to AMO. AMO compares those numbers with an average (also reported from Talos) and if the numbers are too far apart it shows performance warnings on AMO.

The system was constructed and launched briefly but there was concern from add-on developers since we never really communicated with them and were suddenly flagging their add-ons as slowing down Firefox. Many bugs were filed asking for clarification of process and pointing out holes in the system (like add-ons that only slow Firefox down after the browser is used or settings are set). The Talos processing proved to be unreliable and sometimes took 20 minutes for a single add-on. The code was turned off in production and has been ignored since.

Next steps for performance

Firefox sluggishness due to add-ons is still an important metric to track and report since it's one of the larger complaints of end users. However, automating the process is dangerous (as we've witnessed) so the new plan is to remove the previous code and implement a new system which has 100% manual reporting. Performance warnings will only be shown on add-ons where someone (at Mozilla) has run tests and confirmed the add-on is causing performance problems.

Technical Details and Steps

The basic steps to accomplish the next phase:

• Remove the current system from the code and database.
  • Save the performance warning div's on the details pages. We can reuse them.
• Create a CRUD form to enter this data
  • Group permissions are AddonPerf:Edit
  • Form is associated with Add-on versions (not specific files)
  • The form consists of:
    • (checkbox) Operating System: No version. Options: WINNT, Darwin, Linux, Android
    • (checkbox) Application: No version. Options: Firefox
    • (radio) Performance metric: Options: startup, memory, general
    • (radio) Severity: Options: no impact, small impact, major impact
    • (textarea) Notes: An optional field for any notes from the reviewer. Standard "some HTML allowed" textarea.
• backend of the form is a new db table which stores GUID, version, and associated data (similar to blocklisted guids table)
  • There isn't an enforced FK to the add-ons table because we need to store info for non-hosted add-ons also
  • Simple admin tool to search the table and allow results to be edited
  • bonus points: for hosted add-ons add a link on their "Manage Status & Versions" pages
• Hook up the performance warning div's on the details pages, adjusting the wording for the new severity levels
• Send the new performance data through the API (details below)

API

A <warnings> element would be added to the <addon> element, which contains <performance> elements to describe performance warnings. It would be present in in all <addon> output sections including:

• Keyword search API (to display warnings when searching for addons to install)
• GUID search API (to display warnings for installed addons, and in install confirmation dialogs)

This also apples to non-hosted addons, although they would only be returned in a GUID search. All <addon> elements would indicate if the addon was hosted or not (same as the <addon_compatibility> elements do). If an non-hosted addon has no warnings, then the returned data would not include a <addon> element for that addon. If a hosted addon has no warnings, then the <warnings> element would not be included (as opposed to being included, but empty).

Warnings for all versions of the addon need to be included, as the installed version may not be the latest version. Although, this is only a hard requirement for GUID search - keyword search can potentially only include data for the latest version. (let's be consistent between the two for now and include all. --clouserw)

Based on the assumption that many warnings will apply to more than one OS, the os attribute of <performance> elements is a comma-separated list of OS names. If warnings are different for the different OSes, then multiple <performance> elements can be included, with different values for the os attribuite. The values need to match the strings used by nsIXULRuntime.OS - "Darwin" for OSX, "WINNT" for Windows, "Linux" for Linux, "Android" for Android.

Each performance metric has a severity. Rather than use numeric values like usual (which has caused issues in the past when retroactively adding levels), the value would be a string. If the severity is "no impact", then that metric isn't included.

<?xml version="1.0" encoding="utf-8" ?>
<searchresults total_results="1">
    <addon hosted="false">
        <name>Addon Xtreme</name>
        <guid>xtreme@mozzarella.com</guid>
        <warnings>
            <performance application="{firefox-guid}" os="WINNT" version="1.2.3">
                <metric type="startup" severity="minor"/>
                <metric type="general" severity="major"/>
             </performance>
            <performance application="{firefox-guid}" os="Darwin,Linux,WINNT" version="1.2.0">
                <metric type="memory" severity="major"/>
             </performance>
        </warnings>
    </addon>
</searchresults>

Open Questions

• How this works (like, the actual storing in the db) for add-ons not hosted at AMO is ignored in the steps above still.