If you are looking for the Automation Services team, please go to their team page.
- 1 Overview
- 2 The Team
- 3 Weekly Project Status
- 4 How You Can Help
Mozilla QA uses Mozmill, a tool developed for test automation of applications based on the Gecko Platform (XUL Runner).
We use this tool to automate our manual tests in Litmus. The primary goal is to lessen the time we spend actively testing Smoketest, Basic Functional, and Full Functional level regression tests; enabling us to focus our time more on deep testing of bleeding-edge features and bugs.
An added benefit of using automation is that we can run tests across multiple platforms and locales in parallel, thereby increasing the coverage of regression tests in a smaller amount of time. To put that all into perspective, a single person running the BFTs on one platform and one locale takes about 8 hours. Mozmill can run the same tests across all platforms and all locales in a couple of hours.
The Mozmill tool and APIs are maintained and developed by the Automation Services team. The relatively small Desktop Automation team is responsible for developing the tests. This is where we need you, the community.
| Anthony Hughes
|Vancouver, Canada||Team Lead|
| Vlad Maniac
|Cluj, Romania||Developer Lead|
| Alex Lakatos
| Andreea Matei
You can get in contact with us in a few different ways:
- Take part in a discussion on the mozmill-dev mailing list
- Join the #mozmill IRC channel to ask about Mozmill
- Join the #automation IRC channel to ask about test automation
Weekly Project Status
The following is an archive of our weekly accomplishments and discussions.
How You Can Help
There are numerous ways you can help us with Mozmill test automation.
Running the Tests
By far, the easiest way you can help us is by running the tests using the Mozmill Crowd extension. Results can be seen here. If you notice any failures, report them by filing a bug.
Getting Started with Development
- Familiarize yourself with some of our existing functional tests
- Familiarize yourself with the Mozmill Tests documentation and try running some tests
- Review the list of open and unassigned bugs from one of the projects below, try to find something you want to work on
- Sign onto the #automation channel on IRC to get help or send an email to Anthony Hughes
- Be sure to review our development process and commit policy so you know what to expect, and what we expect of you
Everyone on the automation team is willing and able to both help you get started and teach you the skills necessary to be successful. If you aren't sure where to get started or fear you don't have the skills to help, just ask!
The following is the life-cycle for a chemspill regression test.
- MozTrap smoketest is created, in-moztrap+ added to the bug in bugzilla
- Automate the MozTrap smoketest
- Remove the MozTrap smoketest from active testruns, in-qa-automation+ added to the bug in bugzilla
- [DROPPED] bug 756850 Some combinations of Hebrew vowels and consonants not displayed in certain fonts with harfbuzz
- Out of scope for Mozmill, will be covered through in-testsuite automation
- [DROPPED] bug 736731 Hotmail no longer auto-updates
- Out of scope for Mozmill due to live-site requirements, will be included in manual spotchecks
- Not Doable
The following tests are not possible to automate due to technical dependencies and scope.
- Download and view a PDF (we could automate once the internal PDF viewer is enabled by default)
- Breakpad end-to-end test (probably only needs Mozmill 2.0 with crash detection)
- Printing common file formats
- Downloading and opening common file formats
- Uninstalling Firefox
- Mozilla maintenance service
- Mac OSX code-sign verification
- Default installation and start Firefox
The following tests are possible to automate and are either assigned or available for anyone to assign to themselves.
- [ASSIGNED] Video playback of popular formats (bug 489474)
- [FIXED] Switch-to-Tab in the Awesomebar (bug 779127)
- [ASSIGNED] Share your geo location (bug 758187)
- [FIXED] Quit Firefox (bug 757916
|Test Failures||Fixing tests which have caused a failure and investigating for possible Firefox bugs||mozmill-test-failure|
|Test Refactoring||Updating tests so that they conform to new coding guidelines and modules||mozmill-test-refactor|
|Good First Bug||Bugs which have been hand-picked for new contributors||good-first-bug|
|Local Data||Writing minimized test cases to be used by the tests (web pages, add-ons, etc)||mozmill-test-data|
|Endurance Tests||Writing tests which repeat the same task to stress memory usage||mozmill-endurance|
|Smoketests||Writing tests for the most basic functionality||mozmill-smoketest|
|Functional Tests||Writing tests for more detailed functionality||mozmill-functional|
|l10n Tests||Writing tests to verify compatibility with the numerous languages supported by Firefox||mozmill-l10n|
|Restart Tests||Writing tests which verify functionality requiring restart (updates, add-ons, etc)||mozmill-restart|
|Add-ons Tests||Writing tests for specific add-ons (Selenium IDE, Google Toolbar, etc)||mozmill-addons|
Before you Begin
- Learn about installing Mozmill
- Learn about the test repository
- Learn about running tests
- Learn about writing tests
- Learn about our commit policy
Where to get Help
- Joining the #automation channel on IRC and ask for help
- Sending an email directly to Anthony Hughes
- Set up your Mozmill development environment (if you have not done so already)
- Pick an project which fits best with your area of interest
- Familiarize yourself with some existing code
- Pick a project which interests you
- Assign a bug to yourself or file a new one
- Start coding
- Submit your first patch into the review process
- Remember to ask for help via email or on irc.mozilla.org #automation
What to do when RESOLVED FIXED?
- Wait until the next day's testrun to appear on the dashboard, and verify that your test passes
- Assuming it passes, update the worksheet
- Add the following boilerplate to the Litmus test
<hr /> Covered by Mozmill test:<br> <a href="http://hg.mozilla.org/qa/mozmill-tests/file/default/tests/testFolder/testModule.js"> testFolder/testModule.js </a>
- Add the Litmus test to Aurora Mozmill Tests subgroup
- should you be unable to perform any of the above duties, please comment in your bug