Motivation

We are changing our addon story from a "disabled by default" to a "compatible by default" architecture. This means that we will now have all addons be compatible until we learn otherwise. We need a testing system for this both in the short term and the long term.

This project covers the short term testing so that we can determine:

• If making addons compatible by default break the browser, and if so which one/combination of them?
• If the browser is broken will it still be able to update to a newer version of Firefox so that the addon blacklist can be updated to force-disable problem addons.

All work for this project will be tracked on bug 718403.

Deliverable

• Use Mozmill Endurance and Update tests to test functionality and updatability with the addons installed.
• We will use a list of top 100 addons, as provided by AMO
• The results will be sent to a server for tracking
• The tests will be:
  • one endurance/update test run with just the top 10 installed
  • Then, a sequence of tests. For groups of 5 in the remaining addons in the top 100:
    • install this group of 5 addons in the browser PLUS the top 10, then run the endurance and update tests again

Team

• Henrik (whimboo) - working on mozmill wrapper script to install addons and run tests
• Clint (ctalbert) - working with IT to provision machines and ensure the reporting machine is set up
• Will (wlach) - to get us the list of addons for addon base

Deadline

• The first round of this needs to be done by end of January

Dependencies

• The list of 100 addons
• Firefox top add-ons: https://addons.mozilla.org/en-US/firefox/compatibility/4.0?type=all
• We need some VMs to run this on - provisioned in bug 718081
• The list of addons we need to install for our addon base (wlach to provide)

Execution Guideline

To get the DTC tests executed for a new beta/release of Firefox the following steps have to be done:

• Create the configuration file - Therefore the example of the mozmill-automation repository can be used. The testrun settings will have to be updated each time, especially for the source builds to use. For the list of top 100 extensions see bug 718660 or ask the AMO team in regular intervals for updates.
• Upload the config file to all the boxes which have to run the tests. Clint has the list of IPs and user credentials to do so.
• Start the testrun by executing the testrun_compat_addons.py script with the config file as argument.
• Wait until all builds and add-ons have been staged
• Once done check the Mozmill Dashboard for test-results

Notes

• 2012-01-13 Notes