Auto-tools/Projects/Compatibile By Default Automation

Motivation

We are changing our addon story from a "disabled by default" to a "compatible by default" architecture. This means that we will now have all addons be compatible until we learn otherwise. We need a testing system for this both in the short term and the long term.

This project covers the short term testing so that we can determine:

• If making addons compatible by default break the browser, and if so which one/combination of them?
• If the browser is broken will it still be able to update to a newer version of Firefox so that the addon blacklist can be updated to force-disable problem addons.

All work for this project will be tracked on bug 718403.

Deliverable

• Use Mozmill Endurance and Update tests to test functionality and updatability with the addons installed.
• The results will be sent to a server for tracking
• The test will run:
  • for addons in top 100 addons:
    • install the top 10 addons in each and every test-run
    • install a group of five additional add-on from the list
    • Run endurance and update tests, upload results
    • Consider to run an additional endurance test or even functional test on the updated build.

Team

• Henrik (whimboo) - working on mozmill wrapper script to install addons and run tests
• Clint (ctalbert) - working with IT to provision machines and ensure the reporting machine is set up
• Will (wlach) - to get us the list of addons for addon base

Deadline

• The first round of this needs to be done by end of January

Dependencies

• The list of 100 addons
• Firefox top add-ons: https://addons.mozilla.org/en-US/firefox/compatibility/4.0?type=all
• We need some VMs to run this on - provisioned in bug 718081
• The list of addons we need to install for our addon base (wlach to provide)

Notes

• 2012-01-13 Notes