QA/Desktop Firefox/Automation: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 59: Line 59:
= Areas of Work =
= Areas of Work =
== Projects ==
== Projects ==
{| class="querytable sortable"
{| class="querytable"
  |-
  |-
  | class="header" | Test Run
  | class="header" | Project
  | class="header" | Automated
  | class="header" | Summary
| class="header" | Priority
| class="header" | Status
  | class="header" | % COMPLETE
  | class="header" | % COMPLETE
  | class="header" | Documents
  | class="header" | Documents
  |-
  |-
  | Smoketests: simple functional tests, run with every Firefox release.
  | Failures
| Repair broken tests
| P1
| 5 / 9
| 31.3 %
| [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=2 worksheet]
|-
| Endurance
| Tests measuring resource usage in various scenarios
| P2
| 0 / 21
| 0 %
| [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=6 worksheet]
|-
| Smoketests
| Tests covering the most basic functionality
| P3
  | 15 / 33
  | 15 / 33
  | 45.5 %
  | 45.5 %
  | [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=0 worksheet]
  | [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=0 worksheet]
  |-
  |-
  | Functional Tests: detailed functional tests, run once per Firefox milestone
| Restart
| Tests requiring Firefox to restart
| P4
| 2 / 9
| 22.2 %
| [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=8 worksheet]
|-
  | Functional
| Tests covering more advanced functionality
| P5
  | 3 / 21
  | 3 / 21
  | 14.3 %
  | 14.3 %
  | [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=7 worksheet]
  | [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=7 worksheet]
  |-
  |-
  | Restart Tests : similar to Functional tests, these require Firefox to restart
  | Remote
  | 2 / 9
  | Tests requiring remote content
| 22.2 %
  | P6
| [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=8 worksheet]
  |-
| Remote Tests: tests which require remote content
  | 6 / 7
  | 6 / 7
  | 85.7 %
  | 85.7 %
  | [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=5 worksheet]
  | [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=5 worksheet]
  |-
  |-
  | l10n Tests: tests for Firefox's numerous localizations
  | l10n
| Tests for the numerous localizations/languages
| P7
  | 0 / 0
  | 0 / 0
  | n/a
  | n/a
  | [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=9 worksheet]
  | [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=9 worksheet]
|-
| Endurance Tests: tests performance (ie. memory usage) in Firefox
| 0 / 21
| 0 %
| [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=6 worksheet]
|-
| Test Failures: fixing broken tests
| 5 / 9
| 31.3 %
| [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdEhjOXRUak1MY1ViaS1zT2hBRWd4QlE&hl=en_US#gid=2 worksheet]
  |}
  |}
''Prioritized by increasing complexity and decreasing need''


== Functional Tests ==
== How To Help ==
This project entails development of tests for Firefox features, smoketests, and BFTs based on the manual tests we already have in [http://litmus.mozilla.org/ Litmus]. The following details the current areas of development we are focusing on.
 
{| class="querytable sortable"
|-
| class="header" | Component
| class="header" | CHECKED-IN
| class="header" | BLOCKED
| class="header" | ASSIGNED
| class="header" | UNASSIGNED
| class="header" | % COMPLETE
|-
| Add-ons Manager
| 14
| 3
| 9
| 31
| 24%
|-
| App Tabs
| 2
| 0
| 0
| 1
| 66%
|-
| Audio & Video
| 0
| 0
| 1
| 5
| 0%
|-
| Awesomebar
| 10
| 0
| 1
| 5
| 62%
|-
| Bookmarks
| 2
| 0
| 4
| 4
| 20%
|-
| Content Handling
| 0
| 0
| 0
| 8
| 0%
|-
| Cookies
| 4
| 0
| 0
| 0
| 100%
|-
| Downloading
| 6
| 4
| 4
| 3
| 35%
|-
| Feedback
| 0
| 0
| 0
| 10
| 0%
|-
| Find in Page
| 1
| 0
| 0
| 4
| 20%
|-
| Form Manager
| 7
| 0
| 0
| 0
| 100%
|-
| General
| 6
| 1
| 0
| 9
| 37%
|-
| Geolocation
| 0
| 0
| 0
| 5
| 0%
|-
| History
| 0
| 0
| 2
| 8
| 0%
|-
| Home Tab
| 0
| 0
| 0
| 1
| 0%
|-
| Import
| 0
| 0
| 0
| 1
| 0%
|-
| Installation
| 1
| 3
| 0
| 5
| 11%
|-
| Layout
| 1
| 3
| 0
| 9
| 7%
|-
| Library
| 0
| 0
| 0
| 12
| 0%
|-
| Lightweight Themes
| 0
| 0
| 0
| 3
| 0%
|-
| Menus
| 0
| 0
| 0
| 13
| 0%
|-
| Microsummaries
| 0
| 0
| 3
| 0
| 0%
|-
| Preferences
| 8
| 0
| 0
| 0
| 100%
|-
| Panorama
| 2
| 13
| 6
| 2
| 8%
|-
| Password Manager
| 10
| 0
| 0
| 0
| 100%
|-
| Plug-ins
| 1
| 3
| 0
| 18
| 4%
|-
| Pop-up Blocking
| 2
| 0
| 1
| 7
| 20%
|-
| Printing
| 0
| 1
| 0
| 6
| 0%
|-
| Private Browsing
| 14
| 0
| 0
| 1
| 93%
|-
| RSS
| 0
| 1
| 0
| 5
| 0%
|-
| Search
| 10
| 0
| 0
| 2
| 83%
|-
| Security
| 16
| 2
| 0
| 1
| 84%
|-
| Session Store
| 1
| 0
| 0
| 15
| 6%
|-
| Software Update
| 3
| 0
| 0
| 2
| 60%
|-
| Sync
| 0
| 0
| 0
| 12
| 0%
|-
| Tabbed Browsing
| 5
| 7
| 0
| 9
| 23%
|-
| Technical Tools
| 1
| 0
| 0
| 3
| 25%
|-
| Toolbar Customization
| 0
| 0
| 0
| 5
| 0%
|-
| Uninstall
| 0
| 1
| 0
| 2
| 0%
|-
| Web Console
| 0
| 0
| 0
| 9
| 0%
|-
| Discovery Pane
| 6
| 6
| 1
| 0
| 46%
|}
 
''Check out this [https://docs.google.com/spreadsheet/ccc?key=0AlroormSQgiRcEFQNVk1QUgzLVRsLXdSb05nQnVqVVE&hl=en_US spreadsheet] for more details regarding active work.''
 
== Failing Tests ==
This project entails fixing tests which are currently failing. Tests fail for several reasons: a change in content, a change in the APIs, a change in Mozmill, a change in Firefox, or even a change in the host environment. It is of primary concern that we fix these failures in as little time as possible. Here are some useful links to get you started:
 
* Dashboard of results from recent automated testruns: [http://mozmill-release.brasstacks.mozilla.com/#/functional/reports functional], [http://mozmill-release.brasstacks.mozilla.com/#/update/reports update], [http://mozmill-release.brasstacks.mozilla.com/#/l10n/reports localization], [http://mozmill-release.brasstacks.mozilla.com/#/endurance/reports endurance], [http://mozmill-release.brasstacks.mozilla.com/#/remote/reports remote], and [http://mozmill-release.brasstacks.mozilla.com/#/addons/reports addons] tests
* Mercurial repository containing the tests: http://hg.mozilla.org/qa/mozmill-tests/
* Documentation about the tests, APIs, and Mozmill framework: https://developer.mozilla.org/en/Mozmill_Tests
 
; Life-cycle of a Failure
* Check the dashboard for recent failures
* Check if the failure is already captured in Bugzilla
* If not, file a new bug
* Once a bug is filed, assign it to a developer (you)
* Debug the failure (see our tips below) -- '''failures older than 24 hours need to be disabled first'''
* Code your fix and attach a patch to the bug
* Once your patch has passed [https://developer.mozilla.org/en/Mozmill_Tests#The_review_process the review process] it will be checked in
 
 


= Getting Started =
= Getting Started =

Revision as of 23:09, 26 November 2011

UNDER CONSTRUCTION

This is being refactored from the old documentation here


If you are looking for the Automation Services team, please go to their team page.

Overview

Mozilla QA uses Mozmill, a tool developed for test automation of applications based on the Gecko Platform (XUL Runner).

We use this tool to automate our manual tests in Litmus. The primary goal is to lessen the time we spend actively testing Smoketest, Basic Functional, and Full Functional level regression tests; enabling us to focus our time more on deep testing of bleeding-edge features and bugs.

An added benefit of using automation is that we can run tests across multiple platforms and locales in parallel, thereby increasing the coverage of regression tests in a smaller amount of time. To put that all into perspective, a single person running the BFTs on one platform and one locale takes about 8 hours. Mozmill can run the same tests across all platforms and all locales in a couple of hours.

The Mozmill tool and APIs are maintained and developed by the Automation Services team. The relatively small Desktop Automation team is responsible for developing the tests. This is where we need you, the community.

The Team

Person Photo Location Role
Anthony Hughes
IRC: ashughes 		Vancouver, Canada Team Lead
Vlad Maniac
IRC: vladmaniac 		Cluj, Romania Contractor Lead
Alex Lakatos
IRC: AlexLakatos 		Cluj, Romania Developer
Remus Pop
IRC: remuspop 		Cluj, Romania Developer

You can get in contact with us in a few different ways:

  • Take part in a discussion on the mozmill-dev mailing list
  • Join the #mozmill IRC channel to ask about Mozmill
  • Join the #automation IRC channel to ask about test automation

Meetings

Every two weeks we meet to discuss the progress we have made, where we want to go, and how we are going to get there. You can learn a lot about what we do just by listening in to one of our meetings. These meetings are open to everyone.

Meeting Information: 

   When: Mondays, 8am PT / 4pm GMT
Dial-in: +1 (800) 707-2533 Password 369 Conference 654

Areas of Work

Projects

Project Summary Priority Status % COMPLETE Documents
Failures Repair broken tests P1 5 / 9 31.3 % worksheet
Endurance Tests measuring resource usage in various scenarios P2 0 / 21 0 % worksheet
Smoketests Tests covering the most basic functionality P3 15 / 33 45.5 % worksheet
Restart Tests requiring Firefox to restart P4 2 / 9 22.2 % worksheet
Functional Tests covering more advanced functionality P5 3 / 21 14.3 % worksheet
Remote Tests requiring remote content P6 6 / 7 85.7 % worksheet
l10n Tests for the numerous localizations/languages P7 0 / 0 n/a worksheet

Prioritized by increasing complexity and decreasing need

How To Help

Getting Started

To help you get started, we have put together a couple of guides:

Of course, if you have any questions, you can always reach us by:

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Desktop_Firefox/Automation&oldid=373288"