QA/Mozmill Test Automation
OUT OF DATE WARNING
This page is outdated since beginning of July 2011 when the Automation Services team has been formed. Check its team page for further details.
Mozilla QA is using Mozmill to automate existing manual tests which were added to Litmus and folded into different testgroups, like Smoketests, Basic Functional Tests (BFT), and Full Funcional Tests (FFT). The primary goal is to lower the active testing time for any of our QA engineers and contributors. As a result we will be able to cover more than only one platform and locale for BFT testing. To get a better impression of the enhancements just compare those raw numbers: To run all the BFT tests manually it takes about one man day while running all the tests on every platform in parallel will eventually take about 40 minutes with Mozmill!
Given the number of new features and regression fixes for any new Firefox major or security release, we will continuously enhance our functional testing framework by adding more tests of any type, increasing the number of platforms and locale builds to run the tests on, and by making the analysis of test reports more user friendly.
Team Members and Assignments
|Henrik Skupinfirstname.lastname@example.org||Employee|| Team Lead|
(Details see Areas of Work)
|Anthony Hughesemail@example.com||Employee||Fixing failures and writing new tests, Endurance Tests|
|Geo Mealerfirstname.lastname@example.org||Employee||MozMill testing, updating Shared APIs, Update Tests|
|Dave Huntemail@example.com||Employee||Endurance Tests|
|Aaron Trainfirstname.lastname@example.org||Employee||Fixing test failures, writing new tests, testing frameworks|
|Owen Couttsemail@example.com||Employee (Intern)||Revising Mozmill Dashboard|
|Vlad Maniacfirstname.lastname@example.org||Contractor||Mozmill tests|
|Alex Lakatosemail@example.com||Contractor||Fixing test failures, writing new tests, testing frameworks|
|Remus Popfirstname.lastname@example.org||Contractor||Fixing test failures, writing new tests, testing frameworks|
Meetings, Work Weeks, and Dates
We have a bi-weekly public team project meeting. If you want to join the meetings, please check the meeting page for details.
The QA Test Automation team has a couple of ongoing projects. Priorities for each of those will be re-defined at the beginning of each quarter.
- Our high-level goal is to complete the work on the API refactoring project. It's necessary for a more streamlined and consistent way of creating Mozmill tests for Firefox. In combination with the basic API changes, new shared modules and ui maps have to be created for each single component our Mozmill tests cover. For this quarter we want to focus on the browser interface and the new Add-ons Manager.
- With the new Add-ons Manager in-place for Firefox 4 there are needs for automating the existing manual tests on Litmus. We want to start with the tests for the Discovery Pane (Get Add-ons).
- To follow-up on the great feedback we got for Endurance Tests last quarter, we will work on ways to retrieve even more performance and system related data. That's necessary to have a broader range of Endurance Tests (not only memory usage) and data for our own analysis.
- With more and more parallel releases coming up in the future our internal testing strategy has to be improved by relying on the Pulse system for distributed messages. Those will be used to broadcast testing tasks across machines. It's useful for update testing but also for our functional tests.
Areas of Work
As you can imagine a project like Mozmill automation has a couple of areas where work is happening or at least planned for the future. The following chapter will give you a good overview about all of our projects.
|New Mozmill Tests||Create automated tests to minimize our manual testing efforts.|
|New Shared Modules|| |
|Endurance Tests|| |
|Handling of Broken Tests||Making sure to always have a green test-run for Mozmill tests.|
|Shared Modules Refactor|| |
|Test Modules Refactor||Work in parallel with the Shared Modules Refactor project to simplify the test modules.|
|Software Update Tests||Create Mozmill tests to ensure lesser problems during an upgrade of Firefox.|
|Add-ons Tests||Create Mozmill tests to run functional tests against add-ons.|
|L10n Tests||Create Mozmill tests to ensure that localized builds are not broken.|
|Crowd-sourced Testing|| |
|Automated Test-runs|| |
|Triggered Update Testing|| |
|On-Demand Update Testing|| |
|Web Dashboard|| Henrik Skupin
|Shared Modules||Documentation on how to use shared modules and their feature set.|
|Accessibility Tests||Create Mozmill tests to ensure Firefox is accessible by everyone.|
|Panorama Testing||Create a high-quality suite of performance and functional tests to support the Panorama feature of Firefox 4|
|Result Data||Collecting necessary result data of test-runs for each test type.|
Each of the above mentioned sub-projects needs love from someone with passion to work on such a challenging task. There will be a project lead who will offer help in various ways.
If you are interested in helping us to push Mozmill a big step forward, please get in contact with Henrik Skupin. We will figure out what's the best way for your contribution.
Joining an existing project can be hard especially when you see all the stuff in the mind map. To give you a handy guide for the first steps a rich documentation has been created. It covers the following areas:
- Installation of Mozmill
- Getting in touch with the mozmill-test repository
- Executing existent Mozmill tests
All those documents should bring you into a state to be able to play around with Mozmill. At least by running those tests against nightly or release builds of Firefox and reporting failed tests would be a great help for us.
There are a couple of resources which are related to Mozmill. A nearly complete list you will find below: