Auto-tools/Projects/Compatibile By Default Automation

Motivation

We are changing our addon story from a "disabled by default" to a "compatible by default" architecture. This means that we will now have all addons be compatible until we learn otherwise. We need a testing system for this both in the short term and the long term.

This project covers the short term testing so that we can determine:

• If making addons compatible by default break the browser, and if so which one/combination of them?
• If the browser is broken will it still be able to update to a newer version of Firefox so that the addon blacklist can be updated to force-disable problem addons.

All work for this project will be tracked on bug 718403.

Deliverable

• Use Mozmill Endurance and Update tests to test functionality and updatability with the addons installed.
• We will use a list of top 100 addons, as provided by AMO
• The results will be sent to a server for tracking
• The tests will be:
  • one endurance/update test run with just the top 10 installed
  • Then, a sequence of tests. For groups of 5 in the remaining addons in the top 100:
    • install this group of 5 addons in the browser PLUS the top 10, then run the endurance and update tests again

Team

• Henrik (whimboo) - working on mozmill wrapper script to install addons and run tests
• Clint (ctalbert) - working with IT to provision machines and ensure the reporting machine is set up
• Will (wlach) - to get us the list of addons for addon base

Deadline

• The first round of this needs to be done by end of January

Dependencies

• The list of 100 addons
• Firefox top add-ons: https://addons.mozilla.org/en-US/firefox/compatibility/4.0?type=all
• We need some VMs to run this on - provisioned in bug 718081
• The list of addons we need to install for our addon base (wlach to provide)

Notes

• 2012-01-13 Notes