QA/Fennec

Overview

The team focuses on testing current and upcoming releases of Firefox on Android. Some of our responsibilities include qualifying builds prior to a maintenance or milestone release. Some of the responsibilities include, but are not limited to:

• Verifying bugs using a triage strategy
• Running basic functional tests and smoke tests - mostly manual; but automated in the near future
• Running our manual test suites on localized builds
• Making sure web content is viewable, acceptable and ready for consumption in Firefox for Android

In addition to qualifying builds prior to a release, we perform ongoing tasks on the development branches to ensure no major problems get uplifted as we merge changes to the next branch. For example, on a daily basis we:

• Confirm new unconfirmed bugs
• Verify bugs on development branches like Aurora and Beta
• Engage community and new volunteers in helping improve Firefox through test day events
• Interact with developers to help them test the features they implement
• Perform exploratory testing on new features while on the Aurora branch
• Write test cases in Litmus for new features
• Monitor crash-stats to assess stability
• File and track new crasher bugs as they appear in crash-stats
• Manage web compatibility testing through community involvement or crowdsourcing companies that help in testing different Android device configurations and top-site testing

All of which help assess the quality of individual new features as well as the Firefox version as a whole.

Team Details

The team is lead by both Kevin Brosnan and Aaron Train who coordinate testing of their respective areas among several MoCo employees and community volunteers. Traditionally new features are tested by several MoCo QA employees. Anyone is welcome and encouraged to participate.

• Lead - Kevin Brosnan (:kbrosnan)
• Lead - Aaron Train (:aaronmt)

Team Members and Assignments

Name	Contact	Availability	Project Assignments
Aaron Train	atrain@mozilla.com	MoCo Employee (full time)	Lead - New features testing coordination.
Kevin Brosnan	kbrosnan@mozilla.com	MoCo Employee (full time)	Lead - New features testing coordination.
Naoki Hirata	nhirata@mozilla.org	Moco Employee (full time)	Crash analysis, user feedback monitoring,

Current Projects

Project Test Plan	Description	Scope	Timeframe
Firefox for Android 16	QA activities revolving around Firefox Beta in the time frame for Firefox 16 final release.	Features in Firefox for Android 16	August 28th, 2012 (beta).
Feature Ownership	Testing, Planning, and Bug Triage of new and stable Firefox features

Release Testing

• Mobile Release Checklist

Fennec Crash Reporting

Crash Reports can be seen from Socorro and they are manually being done until there doesn't need to be a person looking at the socorro crash signatures for separation (crashes that have the same signatures but are completely different crashes) or combination (for signatures that are separate signatures but are really the same crashes).

Crash reports are done mostly for the Nightly as the developers can be the most effective there. Here is an example of a crash report: Dec 26, 2011 Crash Report

What We Use

Tools

Litmus

• Fennec Test Runs - A Manual test repository of execution tests for Mobile Firefox. Consisted of smoketests, BFTs, and latest feature coverage.

Automation

• Robocop - With advent of NativeFennec, a new framework was needed to test the UI of the Mobile Browser. Robotium was identified as a useful framework to take advantage of, but is not complete. Robocop incorporates all of Activity, Instrumentation, and Robotium Classes to inject events into the new Java front-end of Fennec.

Device Anywhere

• DeviceAnywhere - A virtual set of Android devices for assorted device coverage when you physically dont have on. Great for testday coverage, regression hunting, and device specific bug investigation.

Performance Graphs

• Performance Graphs

Resources

Device Anywhere Tutorial

• Device Anywhere Tutorial

Mobile (HG) Repository

• mozilla-central

Mobile Source Code

• Fennec MXR Source Code Web-Based Viewer

Builds

• Mobile Nightly Builds
• Hourly Tinderbox Builds
• Fennec L10n Dashboard

Where Do We Need Help

Test Tool Development

• Write automated testcases
• Port Automation Frameworks to WinCE
• Find, Fix, Verify UnitTest Failures

Execution

• Triage Unconfirmed Fennec Bugs
• Create Litmus testcases for bugs flagged as in-litmus?
  • Fennec Component Ownership in Litmus
• TestDay Promotions within Mobile Community Channels
• Write technical documentation to perform quality execution

Performance Benchmarking

• Current Instructions on Performance Testing

Previous Performance testing

• Scratch pad of notes
• Performance Testplan
• Performance Benchmarks

Website Compatibility Testing

• Scratch pad of notes
• Web Compatibility Testplan

Sync Performance Instrumentation

Fennec sync continues to cause performance issues on first time syncing with very large amounts of data. How you can help, is to provide sync logging through a custom extension, and submit your data to the Mozilla Services team. Detailed instructions can be found here.

More historical information on the project:

• Philikon's Sync Instrumentation Research
• Blog on profiling Javascript Objects

How You Can Contact Us

If you're interested in helping out with anything below (or want to help with something else), please contact jmaher or aakashd. If you want to talk to the Mozilla Mobile Community, you can find us in our IRC Channel.

• Mailing Lists
  • mobile@mozilla.com
• Newsgroups
  • mozilla.dev.platforms.mobile
  • mozilla.dev.quality
• 3rd Party Testing (Waverley)
  • Execution testplan for Smoketests & BFTs