QA/Desktop Firefox/Automation
UNDER CONSTRUCTION
This is being refactored from the old documentation here
If you are looking for the Automation Services team, please go to their team page.
Overview
Mozilla QA uses Mozmill, a tool developed for test automation of applications based on the Gecko Platform (XUL Runner).
We use this tool to automate our manual tests in Litmus. The primary goal is to lessen the time we spend actively testing Smoketest, Basic Functional, and Full Functional level regression tests; enabling us to focus our time more on deep testing of bleeding-edge features and bugs.
An added benefit of using automation is that we can run tests across multiple platforms and locales in parallel, thereby increasing the coverage of regression tests in a smaller amount of time. To put that all into perspective, a single person running the BFTs on one platform and one locale takes about 8 hours. Mozmill can run the same tests across all platforms and all locales in a couple of hours.
The Mozmill tool and APIs are maintained and developed by the Automation Services team. The relatively small Desktop Automation team is responsible for developing the tests. This is where we need you, the community.
The Team
| Person | Location | Role |
| Anthony Hughes IRC: ashughes |
Vancouver, Canada | Team Lead |
| Vlad Maniac IRC: vladmaniac |
Cluj, Romania | Developer Lead |
| Alex Lakatos IRC: AlexLakatos |
Cluj, Romania | Developer |
| Remus Pop IRC: remuspop |
Cluj, Romania | Developer |
You can get in contact with us in a few different ways:
- Take part in a discussion on the mozmill-dev mailing list
- Join the #mozmill IRC channel to ask about Mozmill
- Join the #automation IRC channel to ask about test automation
Weekly Status
Weekly Project Status
Every week we maintain an archive of the work accomplished and discussion had. Please select a link below to see that week's status page.
Areas of Work
Projects
| Project | Summary | Bugs | Available |
| Good First Bug | Hand-picked bugs ideal for a launching point into test development | good first bug | 3 Bugs Available |
| Failures | Repair broken tests | [mozmill-test-failure] | UNASSIGNED |
| Endurance | Tests measuring resource usage in various scenarios | [mozmill-endurance] | UNASSIGNED |
| Smoketests | Tests covering the most basic functionality | [mozmill-smoketest] | worksheet |
| Restart | Tests requiring Firefox to restart | [mozmill-restart] | worksheet |
| Functional | Tests covering more advanced functionality | [mozmill-functional] | worksheet |
| l10n | Tests for the numerous localizations/languages | [mozmill-l10n] | worksheet |
| Add-ons | Tests for Firefox add-ons | worksheet | |
| Local Data | Creating local data for all of our remote test cases | ||
| Test Refactoring | Refactoring existing tests to use new APIs | mozmill-test-refactor | 5 Bugs Available |
Prioritized by increasing complexity and decreasing need
How to Help
There are numerous ways you can help us with Mozmill test automation, listed below in order of simplest to most complex.
- Run the tests using the Mozmill Crowd extension, results can be seen here
- Fixing one of our good-first-bug bugs
- Fixing one of our test failure bugs
- Create a new Endurance test
- Create a new Functional test
- Getting Started
- Familiarize yourself with some of our existing functional tests
- Familiarize yourself with the Mozmill Tests documentation and try running some tests
- Review the list of open and unassigned bugs from one of the projects above, try to find something you want to work on
- Sign onto the #automation channel on IRC to get help or send an email to Anthony Hughes
Everyone on the automation team is willing and able to both help you get started and teach you the skills necessary to be successful. If you aren't sure where to get started or fear you don't have the skills to help, just ask!
Test Failures
The Test Failures project focuses on repairing tests which have started failing. Most of the time this results in a test having to be refactored due to a change in Firefox. Once in a while this ends up being a Firefox regression. It is of utmost importance that we tackles these issues as swiftly as possible.
- Policy for handling Failures
- Any failures which only affect the mozilla-1.9.2 branch will be disabled, the bug resolved WONTFIX
- Otherwise, follow this process:
- Disable the test
- Investigate if the bug is a failure of the test or Firefox
- If it's a problem with the test, the test will remain disabled until it can be fixed
- If it's a problem with Firefox, file a Firefox bug, re-enable the test and close the test failure bug
Endurance Tests
The Endurance Tests project focuses on development of automated tests which try to expose performance regressions in Firefox. Currently, they only measure memory usage but there are future plans to expand this out to CPU and I/O usage.
Smoketests
The Smokestests project focuses on development of automated tests for the manual smoketests run with every release. The aim here is to lessen the time spent running manual smoketests, devoting it to more important areas of release testing.
Restart Tests
The Restart Tests project focuses on development of automated tests requiring a restart of the browser (Add-on installation for example). They are very similar to Smoketests except that each test is broken into a separate file (book-ended by restarts). One of the unique benefits of Mozmill is that it can persist across the restart of a browser and therefor test in ways other frameworks cannot.
Functional Tests
more to come soon...
l10n Tests
The l10n Tests project focuses on development of automated tests for the numerous language translations (locales). The majority of these tests are UI focused, checking to make sure the localized strings do not break the look and feel of Firefox.
Add-ons Tests
More to come soon...
Good First Bug
The good-first-bug project is all about getting your feet wet with Mozmill Tests development. These bugs are hand-picked from all other projects. They are ideal candidates for your first attempt at writing a patch and fixing a bug for Mozilla.
Test Refactoring
The Test Refactoring project is about refactoring (changing) the code in existing tests to use current APIs. The Mozmill APIs undergo changes frequently to make tests easier to write, implement new features, or to stabilize the test suite. Some of the tests are well over a year old and need to be updated to these new improvements. This is a great way to learn the structure of tests, the APIs and how they are used.
Contributing
Before you Begin
- Learn about installing Mozmill
- Learn about the test repository
- Learn about running tests
- Learn about writing tests
Getting Started
- Set up your Mozmill development environment (if you have not done so already)
- Pick an project which fits best with your area of interest
- Familiarize yourself with some existing code
- Pick a project which interests you
- Assign a bug to yourself or file a new one
- Start coding
- Submit your first patch into the review process
- Remember to ask for help via email or on irc.mozilla.org #automation
What to do when RESOLVED FIXED?
- Wait until the next day's testrun to appear on the dashboard, and verify that your test passes
- Assuming it passes, update the worksheet
- Add the following boilerplate to the Litmus test
<hr /> Covered by Mozmill test:<br> <a href="http://hg.mozilla.org/qa/mozmill-tests/file/default/tests/testFolder/testModule.js"> testFolder/testModule.js </a>
- Add the Litmus test to Aurora Mozmill Tests subgroup
- should you be unable to perform any of the above duties, please comment in your bug
Where to get Help
- Joining the #automation channel on IRC and ask for help
- Sending an email directly to Anthony Hughes
Commit Policy
Check out our commit policy on MDN