Auto-tools/Projects/OrangeFactor
Status
We have weekly meetings!
Wednesdays, 2PM PDT, 1-800-707-2533, PIN: 369, Conf# 304 (or) 650-903-0800 ext: 92, Conf# 304
Goals
Primary Goal
To develop a web dashboard that is useful for identifying and tracking the state of intermittent oranges in our tinderbox unit tests. This should help developers identify which oranges are most 'interesting', and should give people a notion of the overall state of oranges over time.
Secondary Goals
Since the implementation of the dashboard will require tinderbox failures be put into a database, we could potentially use this database in the tinderbox+pushlog UI, which would allow it to query data from a (fast) database, rather than parsing buildbot logs as it sometimes currently does.
History
Topfails
Topfails was the first database-driven orange tracker developed in our team. It shows failures in terms of overall occurrences. It suffers from a buggy log parser, and a UI with relatively few views.
source: http://hg.mozilla.org/automation/topfails/
Orange Factor
Orange Factor is a newer dashboard by jmaher. It calculates the average number of oranges per push (the 'orange factor'), and tracks that number over time. We're currently using it as a base to explore the usefulness of other statistics.
source: http://github.com/jmaher/Orange-Factor
Architecture
The system will have several moving parts:
- an HBASE database (bug 601028), hosted by the metrics team, which will store data parsed out from the buildbot logs, and which will provide some type of queryable interface that will provide test failure information in JSON format
- a flume agent, which will move logs from the build/test system to a storage area hosted by metrics
- a unittest logparser (bug 601216), that will parse buildbot test logs and produce output that gets fed into the HBASE db
- a web dashboard that pulls data from the database and displays various interesting statistics about it
Making Oranges Interesting
Currently, our intermittent oranges are not very interesting. After they've been identified, they are usually more-or-less ignored. This has caused us to accumulate oranges to the point where we have to deal with several of them for every commit (and by 'deal with', I mean 'log it and forget it'), which is time consuming for the sheriffs and for anyone who pushes a commit. At the same time, it demotivates any effort to actually fix them.
We'd like to help change that.