Auto-tools/Projects/TestHarnesses

Team

• The A-Team
  • different projects have different people responsible, ping ahal, jgriffin, jmaher, or ted

Problem

Mozilla has thousands of tests which run 10+ harnesses. As Firefox evolves in features and supported platforms, the test harnesses need to accommodate the needs and challenges faced to keep our new and existing tests running reliably.

Goals & Considerations

Our primary goal is to ensure our harnesses share as much code as possible and remove old code which depended on older features (of python, firefox, etc.) or was not written in a scalable way. Other goals are to simplify our manifests used while allowing the manifest solution to have more flexibility in scheduling tests.

Non-Goals

We are not interested in rewriting our harnesses from scratch or making the perfect harness. We are not looking to switch from Python and Javascript.

Dependencies / Who will use this

Harnesses are primarily used by automation and developers. We need to support harnesses from up to 9 revisions ago (almost a year) when we look at ESR and older B2G branches.

Design and Approach

Overview of test jobs and their harnesses]

Milestones and Dates

As of March 2015, our current priorities are:

• fine grain test selection
• json manifests android, webgl, imptests
• mochitest chrome on b2g
• ssltunnel to android
• error reporting for asan xpcshell
• windows process ctrl+c
• run from tests.zip

Implementation

• reftest
• xpcshell
• mochitest
• talos
• mozbase - libraries reused between harnesses

Getting Involved

Getting Started

The best way is to find a bug and ask the mentor. In general, it is best to have a local build of Firefox

Expectations

For the majority of our work, it lives in mozilla-central (hg, not git) and we follow these guidelines.

Bugs

Bugzilla query error

Query options must be valid JSON.1