QA/Desktop Firefox/Automation: Difference between revisions
| Line 85: | Line 85: | ||
| [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=0 worksheet] | | [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=0 worksheet] | ||
|- | |- | ||
| Restart | | [[#Restart_Tests | Restart]] | ||
| Tests requiring Firefox to restart | | Tests requiring Firefox to restart | ||
| 19 / 29 | | 19 / 29 | ||
| Line 133: | Line 133: | ||
* Familiarize yourself with the [https://developer.mozilla.org/en/Mozmill_Tests Mozmill Tests documentation] and try running some tests | * Familiarize yourself with the [https://developer.mozilla.org/en/Mozmill_Tests Mozmill Tests documentation] and try running some tests | ||
* Review our [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=0 worksheet] and find an UNASSIGNED test you would like to try automating | * Review our [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=0 worksheet] and find an UNASSIGNED test you would like to try automating | ||
* File a bug and sign onto the #automation channel on IRC to get help | |||
=== Restart Tests === | |||
The Restart Tests project focuses on development of automated tests requiring a restart of the browser (Add-on installation for example). They are very similar to Smoketests except that each test is broken into a separate file (book-ended by restarts). One of the unique benefits of Mozmill is that it can persist across the restart of a browser and therefor test in ways other frameworks cannot. | |||
; How to Help | |||
* Familiarize yourself with some of our existing [http://hg.mozilla.org/qa/mozmill-tests/file/default/tests/functional/restartTests functional tests] | |||
* Familiarize yourself with the [https://developer.mozilla.org/en/Mozmill_Tests Mozmill Tests documentation] and try running some tests | |||
* Review our [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=8 worksheet] and find an UNASSIGNED test you would like to try automating | |||
* File a bug and sign onto the #automation channel on IRC to get help | * File a bug and sign onto the #automation channel on IRC to get help | ||
Revision as of 00:30, 2 December 2011
UNDER CONSTRUCTION
This is being refactored from the old documentation here
If you are looking for the Automation Services team, please go to their team page.
Overview
Mozilla QA uses Mozmill, a tool developed for test automation of applications based on the Gecko Platform (XUL Runner).
We use this tool to automate our manual tests in Litmus. The primary goal is to lessen the time we spend actively testing Smoketest, Basic Functional, and Full Functional level regression tests; enabling us to focus our time more on deep testing of bleeding-edge features and bugs.
An added benefit of using automation is that we can run tests across multiple platforms and locales in parallel, thereby increasing the coverage of regression tests in a smaller amount of time. To put that all into perspective, a single person running the BFTs on one platform and one locale takes about 8 hours. Mozmill can run the same tests across all platforms and all locales in a couple of hours.
The Mozmill tool and APIs are maintained and developed by the Automation Services team. The relatively small Desktop Automation team is responsible for developing the tests. This is where we need you, the community.
The Team
| Person | Photo | Location | Role |
| Anthony Hughes IRC: ashughes |
Vancouver, Canada | Team Lead | |
| Vlad Maniac IRC: vladmaniac |
Cluj, Romania | Contractor Lead | |
| Alex Lakatos IRC: AlexLakatos |
Cluj, Romania | Developer | |
| Remus Pop IRC: remuspop |
Cluj, Romania | Developer |
You can get in contact with us in a few different ways:
- Take part in a discussion on the mozmill-dev mailing list
- Join the #mozmill IRC channel to ask about Mozmill
- Join the #automation IRC channel to ask about test automation
Meetings
Every two weeks we meet to discuss the progress we have made, where we want to go, and how we are going to get there. You can learn a lot about what we do just by listening in to one of our meetings. These meetings are open to everyone.
Meeting Information:
When: Mondays, 8am PT / 4pm GMT Dial-in: +1 (800) 707-2533 Password 369 Conference 654
Areas of Work
Projects
| Project | Summary | Status | % COMPLETE | Documents |
| Failures | Repair broken tests | 6 / 31 | 19 % | worksheet |
| Endurance | Tests measuring resource usage in various scenarios | 6 / 27 | 23 % | worksheet |
| Smoketests | Tests covering the most basic functionality | 15 / 32 | 46 % | worksheet |
| Restart | Tests requiring Firefox to restart | 19 / 29 | 66 % | worksheet |
| Functional | Tests covering more advanced functionality | 75 / 122 | 62 % | worksheet |
| l10n | Tests for the numerous localizations/languages | 2 / 25 | 8 % | worksheet |
Prioritized by increasing complexity and decreasing need
Test Failures
The Test Failures project focuses on repairing tests which have started failing. Most of the time this results in a test having to be refactored due to a change in Firefox. Once in a while this ends up being a Firefox regression. It is of utmost importance that we tackles these issues as swiftly as possible.
- How to Help
- Familiarize yourself with some of our existing functional tests
- Familiarize yourself with the Mozmill Tests documentation and try running some tests
- Review our worksheet and find an UNASSIGNED test you would like to try fixing
- If you can't find something, try checking our dashboard and see if a new failure has happened which you would like to solve
- Once you've found a test you want to fix, see if you can reproduce it locally (on your machine)
- File a bug and sign onto the #automation channel on IRC to get help
Endurance Tests
The Endurance Tests project focuses on development of automated tests which try to expose performance regressions in Firefox. Currently, they only measure memory usage but there are future plans to expand this out to CPU and I/O usage.
- How to Help
- Familiarize yourself with the existing endurance tests
- Familiarize yourself with the detailed project documentation and try running some tests
- Review our worksheet and find an UNASSIGNED test you would like to try automating
- File a bug and sign onto the #automation channel on IRC to get help
Smoketests
The Smokestests project focuses on development of automated tests for the manual smoketests run with every release. The aim here is to lessen the time spent running manual smoketests, devoting it to more important areas of release testing.
- How to Help
- Familiarize yourself with some of our existing functional tests
- Familiarize yourself with the Mozmill Tests documentation and try running some tests
- Review our worksheet and find an UNASSIGNED test you would like to try automating
- File a bug and sign onto the #automation channel on IRC to get help
Restart Tests
The Restart Tests project focuses on development of automated tests requiring a restart of the browser (Add-on installation for example). They are very similar to Smoketests except that each test is broken into a separate file (book-ended by restarts). One of the unique benefits of Mozmill is that it can persist across the restart of a browser and therefor test in ways other frameworks cannot.
- How to Help
- Familiarize yourself with some of our existing functional tests
- Familiarize yourself with the Mozmill Tests documentation and try running some tests
- Review our worksheet and find an UNASSIGNED test you would like to try automating
- File a bug and sign onto the #automation channel on IRC to get help
l10n Tests
l10n Tests are used to verify the numerous language translations (locales) do not break functionality and UI in Firefox.
- Manual testcases which need automation
- Worksheet for tracking UNASSIGNED and FIXED work
- Tracking bug for l10n test development
- Source code of the l10n Tests
- Source code of the shared modules
- Dashboard of l10n test results
Contributing
- Before you Begin
- Learn about installing Mozmill
- Learn about the test repository
- Learn about running tests
- Learn about writing tests
- Getting Started
- Set up your Mozmill development environment (if you have not done so already)
- Pick an project which fits best with your area of interest
- Familiarize yourself with some existing code
- Find a test you want to automate using the worksheet
- File a bug:
- instructions for bug...
- Start coding
- Submit your first patch into the review process
- What to do when RESOLVED FIXED?
- Wait until the next day's testrun to appear on the dashboard, and verify that your test passes
- Assuming it passes, update the worksheet
- Add the following boilerplate to the Litmus test
<hr /> Covered by Mozmill test:<br> <a href="http://hg.mozilla.org/qa/mozmill-tests/file/default/tests/testFolder/testModule.js"> testFolder/testModule.js </a>
- Add the Litmus test to Aurora Mozmill Tests subgroup
- should you be unable to perform any of the above duties, please comment in your bug
- Where to get Help
- Joining the #automation channel on IRC and ask for help
- Sending an email directly to Anthony Hughes