The OrangeFactor web app is a tool for tracking and analysing intermittent test failures encountered that occur during Firefox/gecko continuous integration automation.
NB: OrangeFactor is considered near end of life. A replacement will likely use Treeherder's API as a backend instead.
For help with OrangeFactor, please contact #ateam or :emorley, :gbrown, :jmaher.
The OrangeFactor web app can be run locally. See the instructions at: https://hg.mozilla.org/automation/orangefactor/file/default/README.txt
- SSL termination occurs on the box.
- Only listens on port 443, since HTTP->HTTPS redirection performed by Zeus.
- OrangeFactor REST API:
- Python FastCGI app reverse proxied by OpenResty.
- Read-only API apart from job classification submissions from Treeherder, which use Hawk authentication.
- API responses are a combination of results from OrangeFactor's ES instance and the public hg.mozilla.org pushlog.
- ES queries are made using pyes, plus a helper library we've written on top of it, mozautoeslib.
- OrangeFactor Elasticsearch instance:
- Index `bugs`: The intermittent test failure records submitted by Treeherder via OrangeFactor's REST API.
- Index `bzcache`: A cache of public `keyword:intermittent-failure` bugs populated via a brasstacks cron job.
- OrangeFactor bzcache refresh task:
- brasstacks cron job run every four hours, that populates the `bzcache` index on the OrangeFactor Elasticsearch instance.
- Fetches bug data using unauthenticated requests to Bugzilla's REST API.
- OrangeFactor mailer task:
- OrangeFactor bug commenter task:
- brasstacks cron job run in both a daily and weekly variant, that adds failure summary comments to bugs associated with the intermittent test failures (example).
- Fetches data from the OrangeFactor REST API.
- Posts bug comments using the email@example.com account, which has no additional permissions beyond a standard user account.
- ActiveData mirror of OrangeFactor Elasticsearch data:
- Currently synced manually by :eykle.
- Plans to automate this in the future (bug 1344253).
Making Oranges Interesting
Currently, our intermittent oranges are not very interesting. After they've been identified, they are usually more-or-less ignored. This has caused us to accumulate oranges to the point where we have to deal with several of them for every commit (and by 'deal with', I mean 'log it and forget it'), which is time consuming for the sheriffs and for anyone who pushes a commit. At the same time, it demotivates any effort to actually fix them.
We'd like to help change that. We think we can help by creating a dashboard to analyze oranges in the following ways:
- identify the oranges that occur most frequently; these are the oranges that would produce the greatest improvement in our orange factor if fixed
- identify significant changes in the frequency of a given orange; if a known intermittent orange suddenly begins to occur more frequently, it may be related to a recent code change, and this might give developers more information about when/why it occurred, which would hopefully help in fixing it
- identify interesting patterns in failures; some failures may occur more frequently on certain OS's, build types, architectures, or other factors; by providing views which can track oranges across a range of factors, we might be able to provide developers with data that would help them reproduce failures or give them insight into their cause
- identify overall trends in orange occurrences, already part of the legacy Orange Factor app; this can help track the 'orangeness' of a product over time, and can help measure the helpfulness of orange-fixing activities
These projects are deprecated and replaced by the new War on Orange/OrangeFactor application.
Topfails was the first database-driven orange tracker developed in our team. It shows failures in terms of overall occurrences. It suffers from a buggy log parser, and a UI with relatively few views.
Old Orange Factor
Orange Factor is a newer dashboard by jmaher. It calculates the average number of oranges per push (the 'orange factor'), and tracks that number over time. We're currently using it as a base to explore the usefulness of other statistics.
The OrangeFactor ElasticSearch metadata is replicated to ActiveData, and can be queried there using the "orange_factor" index:
The War on Orange site pulls its data from a REST API. Other applications can hook into this to get the raw orange data.
The API root is at http://brasstacks.mozilla.com/orangefactor/api/. Parameters are passed via the query string, eg. ?key1=value1&key2=value2. Example: http://brasstacks.mozilla.com/orangefactor/api/count?startday=2011-05-21&endday=2011-05-27&tree=mozilla-central
All returned data is in JSON format.
Provides a date-indexed list of oranges, with bug numbers, along with minimal details of each bug.
- startday: Mandatory. In ISO format, e.g. 2011-05-27.
- endday: Mandatory. Also in ISO format.
- bugid: Optional. Return orange data for this bug only.
- tree: Optional. Return information about this tree only. Defaults to mozilla-central. Pass "all" for orange data on all trees.
- type: Optional. Return information for this build type only. Must be "opt" or "debug". Defaults to none (both build types).
Returns an object with two properties:
- oranges: An object with dates as properties, e.g. data['oranges']['2011-05-27']. Each property is another object with orange data for the day, with the following properties:
- orangecount: total number of oranges for that day, e.g. 54.
- testruns: number of test runs that day, e.g. 24. The "Orange Factor" is orangecount/testruns.
- oranges: details of the oranges that occurred that day. It is an array of objects, each one having these simple properties:
- bugs: An object with bug ids as properties. Each bug in the above list of oranges is represented here. The information is gathered via pulse and thus is quicker to access than querying Bugzilla. Only a few basic properties are available; for more detailed info, you will have to consult Bugzilla:
Returns a date-indexed summary of orange data.
The parameters are the same as for bybug. The returned data is also the same, except that the 'bugs' property is not returned, and the array of orange details (data['oranges'][<date>]['oranges']) is empty. This is a faster way to get just the summarized numerical data for, e.g., Orange Factor calculations.
Returns minimal details for one or more bugs.
The only parameter is "bugid", which takes a bug id or a comma-separated list of bug ids.
Returns the same data as the "bugs" property of the bybug returned data.