QA/Desktop Firefox/Automation: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(44 intermediate revisions by 7 users not shown)
Line 1: Line 1:
= UNDER CONSTRUCTION =
''This is being refactored from the old documentation [https://wiki.mozilla.org/QA/Mozmill_Test_Automation here]''
''If you are looking for the Automation Services team, please go to their [[QA/Automation_Services|team page]].''
''If you are looking for the Automation Services team, please go to their [[QA/Automation_Services|team page]].''


Line 15: Line 11:


= The Team =
= The Team =
{|class="querytable"
{|class="querytable" style="width:60%"
  |-
  |-
  | class="header" | Person
  | class="header" | Person
| class="header" | Photo
  | class="header" | Location
  | class="header" | Location
  | class="header" | Role
  | class="header" | Role
  |-
  |-
  | [mailto:ahughes@mozilla.com Anthony Hughes] <br> IRC: ashughes
  | [mailto:ahughes@mozilla.com Anthony Hughes] <br> IRC: ashughes
|
  | Vancouver, Canada
  | Vancouver, Canada
  | Team Lead
  | Team Lead
  |-
  |-
  | Vlad Maniac <br> IRC: vladmaniac
  | Vlad Maniac <br> IRC: vladmaniac
|
  | Cluj, Romania
  | Cluj, Romania
  | Contractor Lead
  | Developer Lead
  |-
  |-
  | Alex Lakatos <br> IRC: AlexLakatos
  | Alex Lakatos <br> IRC: AlexLakatos
|
  | Cluj, Romania
  | Cluj, Romania
  | Developer
  | Developer
  |-
  |-
  | Remus Pop <br> IRC: remuspop
  | Andreea Matei <br> IRC: AndreeaMatei
|
  | Cluj, Romania
  | Cluj, Romania
  | Developer
  | Developer
Line 48: Line 39:
* Join the [http://www.mibbit.com/chat/?server=irc.mozilla.org&channel=%23automation #automation] IRC channel to ask about test automation
* Join the [http://www.mibbit.com/chat/?server=irc.mozilla.org&channel=%23automation #automation] IRC channel to ask about test automation


= Meetings =
= Weekly Project Status =
; Meetings are canceled as of 2012-02-03 - see Weekly Project Status section below
The following is an archive of our weekly accomplishments and discussions.
 
* '''March 2012''': [[QA/Desktop_Firefox/Automation/Status:2012-03-02 | 2nd]], [[QA/Desktop_Firefox/Automation/Status:2012-03-09 | 9th]]
* '''February 2012''': [[QA/Desktop_Firefox/Automation/Status:2012-02-03 | 3rd]], [[QA/Desktop_Firefox/Automation/Status:2012-02-10 | 10th]], [[QA/Desktop_Firefox/Automation/Status:2012-02-17 | 17th]], [[QA/Desktop_Firefox/Automation/Status:2012-02-24 | 24th]]
 
= How You Can Help =
There are numerous ways you can help us with Mozmill test automation.
 
=== Running the Tests ===
By far, the easiest way you can help us is by running the tests using the [https://addons.mozilla.org/en-US/firefox/addon/mozmill-crowd/ Mozmill Crowd] extension. Results can be seen [http://mozmill-crowd.blargon7.com here]. If you notice any failures, report them by [http://bit.ly/wHfgTC filing a bug].
 
=== Getting Started with Development ===
* Familiarize yourself with some of our existing [http://hg.mozilla.org/qa/mozmill-tests/file/default/tests/functional functional tests]
* Familiarize yourself with the [https://developer.mozilla.org/en/Mozmill_Tests Mozmill Tests documentation] and try running some tests
* Review the list of open and unassigned bugs from one of the projects below, try to find something you want to work on
* Sign onto the #automation channel on IRC to get help or [mailto:ahughes@mozilla.com send an email] to Anthony Hughes
* Be sure to review our [https://developer.mozilla.org/en/Mozmill_Tests#Writing_Mozmill_tests development process] and [https://developer.mozilla.org/en/Mozmill_Tests/Commit_Policy commit policy] so you know what to expect, and what we expect of you
''Everyone on the automation team is willing and able to both help you get started and teach you the skills necessary to be successful. If you aren't sure where to get started or fear you don't have the skills to help, '''just ask!'''''
 
=== P1 Projects ===
==== Chemspill Regressions ====
The following is the life-cycle for a chemspill regression test.
# MozTrap smoketest is created, in-moztrap+ added to the bug in bugzilla
# Automate the MozTrap smoketest
# Remove the MozTrap smoketest from active testruns, in-qa-automation+ added to the bug in bugzilla


Every two weeks we meet to discuss the progress we have made, where we want to go, and how we are going to get there. You can learn a lot about what we do just by listening in to one of our meetings. '''These meetings are open to everyone.'''
Firefox 13.0.1:
* {{drop|}} {{bug|756850}} Some combinations of Hebrew vowels and consonants not displayed in certain fonts with harfbuzz
** Out of scope for Mozmill, will be covered through in-testsuite automation
* {{drop|}} {{bug|736731}} Hotmail no longer auto-updates
** Out of scope for Mozmill due to live-site requirements, will be included in manual spotchecks


Meeting Information:
==== MozTrap Smoketests ====
<pre>
; Not Doable
  When: Mondays, 8am PT / 4pm GMT
The following tests are not possible to automate due to technical dependencies and scope.
Dial-in: +1 (800) 707-2533 Password 369 Conference 654
* [https://moztrap.mozilla.org/manage/case/328/ Download and view a PDF] (we could automate once the internal PDF viewer is enabled by default)
</pre>
* [https://moztrap.mozilla.org/manage/case/325/ Breakpad end-to-end test] (probably only needs Mozmill 2.0 with crash detection)
* [https://moztrap.mozilla.org/manage/case/334/ Printing common file formats]
* Downloading and opening common file formats
** [https://moztrap.mozilla.org/manage/case/330/ DMG]
** [https://moztrap.mozilla.org/manage/case/331/ TAR]
** [https://moztrap.mozilla.org/manage/case/332/ EXE]
* Uninstalling Firefox
** [https://moztrap.mozilla.org/manage/case/324/ Linux]
** [https://moztrap.mozilla.org/manage/case/323/ Mac]
** [https://moztrap.mozilla.org/manage/case/321/ Win XP]
** [https://moztrap.mozilla.org/manage/case/322/ Win Vista, 7, 8]
* [https://moztrap.mozilla.org/manage/case/278/ Mozilla maintenance service]
* [https://moztrap.mozilla.org/manage/case/318/ Mac OSX code-sign verification]
* [https://moztrap.mozilla.org/manage/case/260/ Default installation and start Firefox]


Past Meetings:
; Doable
* [[QA/Desktop_Firefox/Automation/Meetings:2012-01-09 | 2012-01-09]]
The following tests are possible to automate and are either assigned or available for anyone to assign to themselves.
* [[QA/Desktop_Firefox/Automation/Meetings:2011-12-12 | 2011-12-12]]


= Weekly Project Status =
* '''[ASSIGNED]''' [https://moztrap.mozilla.org/manage/case/266/ Video playback of popular formats] ({{bug|489474}})
Every week we maintain an archive of the work accomplished and discussion had. Please select a link below to see that week's status page.
* '''[FIXED]''' [https://moztrap.mozilla.org/manage/case/327/ Switch-to-Tab in the Awesomebar] ({{bug|779127}})
* '''[ASSIGNED]''' [https://moztrap.mozilla.org/manage/case/329/ Share your geo location] ({{bug|758187}})
* '''[FIXED]''' [https://moztrap.mozilla.org/manage/case/333/ Quit Firefox] ({{bug|757916}}


; February 2012
==== Firefox Features ====
* [[QA/Desktop_Firefox/Automation/Status:2012-02-03 | 2012-02-03]]
* [https://moztrap.mozilla.org/manage/cases/?filter-suite=31 Silent Updates]
* [https://moztrap.mozilla.org/manage/cases/?filter-suite=33 Show in-line PDF]
* [https://moztrap.mozilla.org/manage/cases/?filter-suite=46 In-content Preferences]


= Areas of Work =
=== P2 Projects ===
== Projects ==
{|class="querytable"
{| class="querytable"
|class="header" style="width:20%" | Project
|class="header" style="width:60%" | Description
|class="header" style="width:20%" | Open Bugs
  |-
  |-
  | class="header" | Project
  | '''Test Failures'''
  | class="header" | Summary
  | Fixing tests which have caused a failure and investigating for possible Firefox bugs
  | class="header" | Bugs
  | [http://bit.ly/wByTFy mozmill-test-failure]
| class="header" | Availability
  |-
  |-
  | [[#Good_First_Bug | Good First Bug]]
  | '''Test Refactoring'''
  | Hand-picked bugs ideal for a launching point into test development
  | Updating tests so that they conform to new coding guidelines and modules
  | [http://bit.ly/szrMtZ good first bug]
  | [http://bit.ly/t5HWZM mozmill-test-refactor]
| 3 Bugs Available
|-
| [[#Test_Failures | Failures]]
| Repair broken tests
| [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=mozmill-test-failure [mozmill-test-failure]]
| [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=2 worksheet]
  |-
  |-
  | [[#Endurance_Tests | Endurance]]
  | '''Good First Bug'''
  | Tests measuring resource usage in various scenarios
  | Bugs which have been hand-picked for new contributors
  | [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=mozmill-endurance [mozmill-endurance]]
  | [http://bit.ly/szrMtZ good-first-bug]
| [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=6 worksheet]
  |-
  |-
  | [[#Smoketests | Smoketests]]
  | '''Local Data'''
  | Tests covering the most basic functionality
  | Writing minimized test cases to be used by the tests (web pages, add-ons, etc)
  | [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=mozmill-smoketest [mozmill-smoketest]]
  | [http://bit.ly/x8zbLG mozmill-test-data]
| [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=0 worksheet]
  |-
  |-
  | [[#Restart_Tests | Restart]]
  | '''Endurance Tests'''
  | Tests requiring Firefox to restart
  | Writing tests which repeat the same task to stress memory usage
  | [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=mozmill-restart [mozmill-restart]]
  | [http://bit.ly/yJq6VH mozmill-endurance]
| [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=8 worksheet]
  |-
  |-
  | Functional
  | '''Smoketests'''
  | Tests covering more advanced functionality
  | Writing tests for the most basic functionality
  | [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=mozmill-functional [mozmill-functional]]
  | [http://bit.ly/zGnG31 mozmill-smoketest]
| [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=7 worksheet]
  |-
  |-
  | [[#l10n_Tests | l10n]]
  | '''Functional Tests'''
  | Tests for the numerous localizations/languages
  | Writing tests for more detailed functionality
  | [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=mozmill-l10n [mozmill-l10n]]
  | [http://bit.ly/yTP5ri mozmill-functional]
| [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=9 worksheet]
  |-
  |-
  | [[#Add-ons_Tests | Add-ons]]
  | '''l10n Tests'''
  | Tests for Firefox add-ons
  | Writing tests to verify compatibility with the numerous languages supported by Firefox
|
  | [http://bit.ly/xogyOF mozmill-l10n]
  | [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=10 worksheet]
  |-
  |-
  | Local Data
  | '''Restart Tests'''
| Creating local data for all of our remote test cases
  | Writing tests which verify functionality requiring restart (updates, add-ons, etc)
  |  
  | [http://bit.ly/ytXscT mozmill-restart]
  |
  |-
  |-
  | [[#Test_Refactoring | Test Refactoring]]
  | '''Add-ons Tests'''
  | Refactoring existing tests to use new APIs
  | Writing tests for specific add-ons (Selenium IDE, Google Toolbar, etc)
  | [http://bit.ly/t5HWZM mozmill-test-refactor]
  | [http://bit.ly/zJxpLS mozmill-addons]
| 5 Bugs Available
  |}
  |}
''Prioritized by increasing complexity and decreasing need''
=== Test Failures ===
The Test Failures project focuses on repairing tests which have started failing. Most of the time this results in a test having to be refactored due to a change in Firefox. Once in a while this ends up being a Firefox regression. It is of utmost importance that we tackles these issues as swiftly as possible.
; How to Help
* Familiarize yourself with some of our existing [http://hg.mozilla.org/qa/mozmill-tests/file/default/tests/functional functional tests]
* Familiarize yourself with the [https://developer.mozilla.org/en/Mozmill_Tests Mozmill Tests documentation] and try running some tests
* Review our [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=2 worksheet] and find an UNASSIGNED test you would like to try fixing
* If you can't find something, [http://mozmill-release.brasstacks.mozilla.com/#/functional try checking our dashboard] and see if a new failure has happened which you would like to solve
* Once you've found a test you want to fix, see if you can reproduce it locally (on your machine)
* File a bug and sign onto the #automation channel on IRC to get help
=== Endurance Tests ===
The Endurance Tests project focuses on development of automated tests which try to expose performance regressions in Firefox. Currently, they only measure memory usage but there are future plans to expand this out to CPU and I/O usage.
; How to Help
* Familiarize yourself with the existing [http://hg.mozilla.org/qa/mozmill-tests/file/default/tests/endurance endurance tests]
* Familiarize yourself with the [[QA/Mozmill_Test_Automation/Endurance_Tests | detailed project documentation]] and try running some of the tests
* Check out our [http://mozmill-release.brasstacks.mozilla.com/#/endurance Endurance Dashboard] to see some recent results
* Review our [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=6 worksheet] and find an UNASSIGNED test you would like to try automating
* File a bug and sign onto the #automation channel on IRC to get help
=== Smoketests ===
The Smokestests project focuses on development of automated tests for the manual smoketests run with every release. The aim here is to lessen the time spent running manual smoketests, devoting it to more important areas of release testing.
; How to Help
* Familiarize yourself with some of our existing [http://hg.mozilla.org/qa/mozmill-tests/file/default/tests/functional functional tests]
* Familiarize yourself with the [https://developer.mozilla.org/en/Mozmill_Tests Mozmill Tests documentation] and try running some of the tests
* Check out our [http://mozmill-release.brasstacks.mozilla.com/#/functional Functional Dashboard] to see some recent results
* Review our [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=0 worksheet] and find an UNASSIGNED test you would like to try automating
* File a bug and sign onto the #automation channel on IRC to get help
=== Restart Tests ===
The Restart Tests project focuses on development of automated tests requiring a restart of the browser (Add-on installation for example). They are very similar to Smoketests except that each test is broken into a separate file (book-ended by restarts). One of the unique benefits of Mozmill is that it can persist across the restart of a browser and therefor test in ways other frameworks cannot.
; How to Help
* Familiarize yourself with some of our existing [http://hg.mozilla.org/qa/mozmill-tests/file/default/tests/functional/restartTests functional tests]
* Familiarize yourself with the [https://developer.mozilla.org/en/Mozmill_Tests Mozmill Tests documentation] and try running some of the tests
* Check out our [http://mozmill-release.brasstacks.mozilla.com/#/functional Functional] and [http://mozmill-release.brasstacks.mozilla.com/#/remote Remote Dashboards] to see some recent results
* Review our [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=8 worksheet] and find an UNASSIGNED test you would like to try automating
* File a bug and sign onto the #automation channel on IRC to get help
=== Functional Tests ===
more to come soon...
=== l10n Tests ===
The l10n Tests project focuses on development of automated tests for the numerous language translations (locales). The majority of these tests are UI focused, checking to make sure the localized strings do not break the look and feel of Firefox.
; How to Help
* Familiarize yourself with some of our existing [http://hg.mozilla.org/qa/mozmill-tests/file/default/tests/l10n l10n tests] and [https://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&branch_id=44&testgroup_id=271&subgroup_id= Litmus tests]
* Familiarize yourself with the [https://developer.mozilla.org/en/Mozmill_Tests Mozmill Tests documentation] and try running some of the tests
* Check out our [http://mozmill-release.brasstacks.mozilla.com/#/l10n/reports L10n Dashboard] to see some recent results
* Review our [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=9 worksheet] and find an UNASSIGNED test you would like to try automating
* File a bug and sign onto the #automation channel on IRC to get help
=== Add-ons Tests ===
More to come soon...
[http://mozmill-release.brasstacks.mozilla.com/#/addons dashboard]
=== Good First Bug ===
The ''good-first-bug'' project is all about getting your feet wet with Mozmill Tests development. These bugs are hand-picked from all other projects. They are ideal candidates for your first attempt at writing a patch and fixing a bug for Mozilla.
* Be sure to familiarize yourself with our documentation about [https://developer.mozilla.org/en/Mozmill_Tests#Writing_Mozmill_tests writing Mozmill tests]
* Pick [http://bit.ly/szrMtZ one of these bugs] to work on and assign it to yourself
* Comment in the bug introducing yourself and ask for help getting started
* Feel free to [mailto:ahughes@mozilla.com email me directly] if you have any questions
=== Test Refactoring ===
The ''Test Refactoring'' project is about refactoring (changing) the code in existing tests to use current APIs. The Mozmill APIs undergo changes frequently to make tests easier to write, implement new features, or to stabilize the test suite. Some of the tests are well over a year old and need to be updated to these new improvements. This is a great way to learn the structure of tests, the APIs and how they are used.
* Make sure you have read and understand [[QA/Desktop_Firefox/Automation#Contributing | contributing]]
* Pick one of the [http://bit.ly/t5HWZM test refactoring bugs] and assign it to yourself
* Introduce yourself to the team and ask for help getting started by commenting in the bug
* If you have any questions [mailto:ahughes@mozilla.com email me directly]


== Contributing ==
== Contributing ==
Line 211: Line 157:
* Learn about [https://developer.mozilla.org/en/Mozmill_Tests#Running_Mozmill_tests running tests]
* Learn about [https://developer.mozilla.org/en/Mozmill_Tests#Running_Mozmill_tests running tests]
* Learn about [https://developer.mozilla.org/en/Mozmill_Tests#Writing_Mozmill_tests writing tests]
* Learn about [https://developer.mozilla.org/en/Mozmill_Tests#Writing_Mozmill_tests writing tests]
* Learn about our [https://developer.mozilla.org/en/Mozmill_Tests/Commit_Policy commit policy]
=== Where to get Help ===
* Joining the [http://www.mibbit.com/chat/?server=irc.mozilla.org&channel=%23automation #automation] channel on IRC and ask for help
* Sending an email directly to [mailto:ahughes@mozilla.com Anthony Hughes]


=== Getting Started ===
=== Getting Started ===
Line 235: Line 186:
* Add the Litmus test to ''Aurora Mozmill Tests'' subgroup
* Add the Litmus test to ''Aurora Mozmill Tests'' subgroup
* ''should you be unable to perform any of the above duties, please comment in your bug''
* ''should you be unable to perform any of the above duties, please comment in your bug''
=== Where to get Help ===
* Joining the [http://www.mibbit.com/chat/?server=irc.mozilla.org&channel=%23automation #automation] channel on IRC and ask for help
* Sending an email directly to [mailto:ahughes@mozilla.com Anthony Hughes]
== Commit Policy ==
The following details our policy for being a Commiter for mozmill-tests.
; What is a Committer?
A committer is someone who physically checks code into the repository. This comes with a lot of responsibility and is typically only granted to those who have proven themselves over an extended period of contribution.
; How do I become a Committer?
When the team feels you are ready to become a committer, you will need to go through the application process. Mozilla has a standard Committer Agreement with everyone who wishes to check code into any repository. '''([http://www.mozilla.org/hacking/committer/ Mozilla Committer Agreement])'''
; Privileges
A new committer will be granted the following privileges:
# Can land patches to skip tests after one peer-review
# Can land patches to fix tests after one peer-review and one super-review
# Can land patches to new tests after one peer-review and one super-review
# Can land patches to litmus-data after one peer-review and one super review
# Can land patches to endurance tests after one peer-review, one super-review, and one feedback-review from the Endurance team
# Can land patches to shared modules after one peer-review, one super-review and one feedback-review from the Automation Services team
; Responsibilities
A committer will be responsible for the following:
* prompt check-in of patches they have reviewed
* checking the live results the following day to see if regressions are caused
* prompt back-out of patches when they regress
* landing patches across other branches once it has passed verification on a live run
* '''NOTE''' fixing a regression patch is the responsibility of the author but check-in/back-out is responsibility of the committer
; Life-cycle of a Patch
If the patch only affects a particular branch
# a patch passes all reviews, lands on that branch, bug is marked RESOLVED FIXED
# committer reviews the live results the following day
# if the test passes, bug is marked VERIFIED FIXED
# if the test fails, committer backs out patch, bug is marked REOPENED
# author must submit a new patch and the process repeats
If the patch affects multiple branches
# a patch passes all reviews, lands on default, bug is marked RESOLVED FIXED
# committer reviews the live results the following day
# if the test passes, patch is landed on aurora, beta, and release
# committer reviews the live results the following day
# if the test passes, bug is marked VERIFIED FIXED
# if the test fails at any point, committer backs out the patch from all affected branches, bug is marked REOPENED
# author must submit a new patch and the process repeats
; Non-responsibilities
The following is what the committer is NOT responsible for and should never do:
* landing patches on the ESR branch is prohibited (this will be managed by Anthony Hughes)
* merging branches is prohibited (this will be managed by Anthony Hughes)

Latest revision as of 14:08, 18 October 2012

If you are looking for the Automation Services team, please go to their team page.

Overview

Mozilla QA uses Mozmill, a tool developed for test automation of applications based on the Gecko Platform (XUL Runner).

We use this tool to automate our manual tests in Litmus. The primary goal is to lessen the time we spend actively testing Smoketest, Basic Functional, and Full Functional level regression tests; enabling us to focus our time more on deep testing of bleeding-edge features and bugs.

An added benefit of using automation is that we can run tests across multiple platforms and locales in parallel, thereby increasing the coverage of regression tests in a smaller amount of time. To put that all into perspective, a single person running the BFTs on one platform and one locale takes about 8 hours. Mozmill can run the same tests across all platforms and all locales in a couple of hours.

The Mozmill tool and APIs are maintained and developed by the Automation Services team. The relatively small Desktop Automation team is responsible for developing the tests. This is where we need you, the community.

The Team

Person Location Role
Anthony Hughes
IRC: ashughes 		Vancouver, Canada Team Lead
Vlad Maniac
IRC: vladmaniac 		Cluj, Romania Developer Lead
Alex Lakatos
IRC: AlexLakatos 		Cluj, Romania Developer
Andreea Matei
IRC: AndreeaMatei 		Cluj, Romania Developer

You can get in contact with us in a few different ways:

  • Take part in a discussion on the mozmill-dev mailing list
  • Join the #mozmill IRC channel to ask about Mozmill
  • Join the #automation IRC channel to ask about test automation

Weekly Project Status

The following is an archive of our weekly accomplishments and discussions.

How You Can Help

There are numerous ways you can help us with Mozmill test automation.

Running the Tests

By far, the easiest way you can help us is by running the tests using the Mozmill Crowd extension. Results can be seen here. If you notice any failures, report them by filing a bug.

Getting Started with Development

  • Familiarize yourself with some of our existing functional tests
  • Familiarize yourself with the Mozmill Tests documentation and try running some tests
  • Review the list of open and unassigned bugs from one of the projects below, try to find something you want to work on
  • Sign onto the #automation channel on IRC to get help or send an email to Anthony Hughes
  • Be sure to review our development process and commit policy so you know what to expect, and what we expect of you

Everyone on the automation team is willing and able to both help you get started and teach you the skills necessary to be successful. If you aren't sure where to get started or fear you don't have the skills to help, just ask!

P1 Projects

Chemspill Regressions

The following is the life-cycle for a chemspill regression test.

  1. MozTrap smoketest is created, in-moztrap+ added to the bug in bugzilla
  2. Automate the MozTrap smoketest
  3. Remove the MozTrap smoketest from active testruns, in-qa-automation+ added to the bug in bugzilla

Firefox 13.0.1:

  • [DROPPED] bug 756850 Some combinations of Hebrew vowels and consonants not displayed in certain fonts with harfbuzz
    • Out of scope for Mozmill, will be covered through in-testsuite automation
  • [DROPPED] bug 736731 Hotmail no longer auto-updates
    • Out of scope for Mozmill due to live-site requirements, will be included in manual spotchecks

MozTrap Smoketests

Not Doable

The following tests are not possible to automate due to technical dependencies and scope.

Doable

The following tests are possible to automate and are either assigned or available for anyone to assign to themselves.

Firefox Features

P2 Projects

Project Description Open Bugs
Test Failures Fixing tests which have caused a failure and investigating for possible Firefox bugs mozmill-test-failure
Test Refactoring Updating tests so that they conform to new coding guidelines and modules mozmill-test-refactor
Good First Bug Bugs which have been hand-picked for new contributors good-first-bug
Local Data Writing minimized test cases to be used by the tests (web pages, add-ons, etc) mozmill-test-data
Endurance Tests Writing tests which repeat the same task to stress memory usage mozmill-endurance
Smoketests Writing tests for the most basic functionality mozmill-smoketest
Functional Tests Writing tests for more detailed functionality mozmill-functional
l10n Tests Writing tests to verify compatibility with the numerous languages supported by Firefox mozmill-l10n
Restart Tests Writing tests which verify functionality requiring restart (updates, add-ons, etc) mozmill-restart
Add-ons Tests Writing tests for specific add-ons (Selenium IDE, Google Toolbar, etc) mozmill-addons

Contributing

Before you Begin

Where to get Help

Getting Started

  • Set up your Mozmill development environment (if you have not done so already)
  • Pick an project which fits best with your area of interest
  • Familiarize yourself with some existing code
  • Pick a project which interests you
  • Assign a bug to yourself or file a new one
  • Start coding
  • Submit your first patch into the review process
  • Remember to ask for help via email or on irc.mozilla.org #automation

What to do when RESOLVED FIXED?

  • Wait until the next day's testrun to appear on the dashboard, and verify that your test passes
  • Assuming it passes, update the worksheet
  • Add the following boilerplate to the Litmus test
<hr />
Covered by Mozmill test:<br>
<a href="http://hg.mozilla.org/qa/mozmill-tests/file/default/tests/testFolder/testModule.js">
  testFolder/testModule.js
</a>
  • Add the Litmus test to Aurora Mozmill Tests subgroup
  • should you be unable to perform any of the above duties, please comment in your bug
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Desktop_Firefox/Automation&oldid=480971"