Auto-tools/Projects/Compatibile By Default Automation: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
 
(7 intermediate revisions by 3 users not shown)
Line 5: Line 5:
* If making addons compatible by default break the browser, and if so which one/combination of them?
* If making addons compatible by default break the browser, and if so which one/combination of them?
* If the browser is broken will it still be able to update to a newer version of Firefox so that the addon blacklist can be updated to force-disable problem addons.
* If the browser is broken will it still be able to update to a newer version of Firefox so that the addon blacklist can be updated to force-disable problem addons.
All work for this project will be tracked on {{bug|718403}}.


= Deliverable =
= Deliverable =
* Use Mozmill Endurance and Update tests to test functionality and updatability with the addons installed.
* Use Mozmill Endurance and Update tests to test functionality and updatability with the addons installed.
* We will use a list of top 100 addons, as provided by AMO
* The results will be sent to a server for tracking
* The results will be sent to a server for tracking
* The test will run:
* The tests will be:
** for addons in top 100 addons:
** one endurance/update test run with just the top 10 installed
*** install the top 5 addons
** Then, a sequence of tests. For groups of 5 in the remaining addons in the top 100:
*** Run endurance and update tests, upload results
*** install this group of 5 addons in the browser PLUS the top 10, then run the endurance and update tests again


= Team =
= Team =
Line 24: Line 27:
= Dependencies =
= Dependencies =
* The list of [https://docs.google.com/document/d/1prS9gvDUF15M7ASlRcZcK2zNBROTPuJgakhInr2Vnk4/edit?hl=en_US 100 addons]
* The list of [https://docs.google.com/document/d/1prS9gvDUF15M7ASlRcZcK2zNBROTPuJgakhInr2Vnk4/edit?hl=en_US 100 addons]
* Firefox top add-ons: https://addons.mozilla.org/en-US/firefox/compatibility/4.0?type=all
* We need some VMs to run this on - provisioned in {{bug|718081}}
* We need some VMs to run this on - provisioned in {{bug|718081}}
* The list of addons we need to install for our addon base (wlach to provide)
* The list of addons we need to install for our addon base (wlach to provide)
= Execution Guideline =
To get the DTC tests executed for a new beta/release of Firefox the following steps have to be done:
* Create the configuration file - Therefore the [http://hg.mozilla.org/qa/mozmill-automation/file/default/configs/testrun_compat_addons.json.example example] of the mozmill-automation repository can be used. The testrun settings will have to be updated each time, especially for the source builds to use. For the list of top 100 extensions see {{bug|718660}} or ask the AMO team in regular intervals for updates.
* Upload the config file to all the boxes which have to run the tests. Clint has the list of IPs and user credentials to do so.
* Start the testrun by executing the ''testrun_compat_addons.py'' script with the config file as argument.
* Wait until all builds and add-ons have been staged
* Once done check the [http://mozmill-addons.blargon7.com/ Mozmill Dashboard] for test-results


= Notes =
= Notes =
* Dburns notes
* [https://wiki.mozilla.org/images/8/80/Dburnsnotes_addoncompattesting.pdf 2012-01-13 Notes]

Latest revision as of 19:59, 22 March 2012

Motivation

We are changing our addon story from a "disabled by default" to a "compatible by default" architecture. This means that we will now have all addons be compatible until we learn otherwise. We need a testing system for this both in the short term and the long term.

This project covers the short term testing so that we can determine:

  • If making addons compatible by default break the browser, and if so which one/combination of them?
  • If the browser is broken will it still be able to update to a newer version of Firefox so that the addon blacklist can be updated to force-disable problem addons.

All work for this project will be tracked on bug 718403.

Deliverable

  • Use Mozmill Endurance and Update tests to test functionality and updatability with the addons installed.
  • We will use a list of top 100 addons, as provided by AMO
  • The results will be sent to a server for tracking
  • The tests will be:
    • one endurance/update test run with just the top 10 installed
    • Then, a sequence of tests. For groups of 5 in the remaining addons in the top 100:
      • install this group of 5 addons in the browser PLUS the top 10, then run the endurance and update tests again

Team

  • Henrik (whimboo) - working on mozmill wrapper script to install addons and run tests
  • Clint (ctalbert) - working with IT to provision machines and ensure the reporting machine is set up
  • Will (wlach) - to get us the list of addons for addon base

Deadline

  • The first round of this needs to be done by end of January

Dependencies

Execution Guideline

To get the DTC tests executed for a new beta/release of Firefox the following steps have to be done:

  • Create the configuration file - Therefore the example of the mozmill-automation repository can be used. The testrun settings will have to be updated each time, especially for the source builds to use. For the list of top 100 extensions see bug 718660 or ask the AMO team in regular intervals for updates.
  • Upload the config file to all the boxes which have to run the tests. Clint has the list of IPs and user credentials to do so.
  • Start the testrun by executing the testrun_compat_addons.py script with the config file as argument.
  • Wait until all builds and add-ons have been staged
  • Once done check the Mozmill Dashboard for test-results

Notes

Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Projects/Compatibile_By_Default_Automation&oldid=410741"