Auto-tools/Projects/Automation Papercuts

From MozillaWiki
< Auto-tools‎ | Projects
Revision as of 13:59, 1 November 2011 by Wlach (talk | contribs) (Created page with "'''NOTE: This is only a proposal at this time''' Some parts of this document will refer to the Art of Unix Programming, which I (wlach) recommend highly. == Problem == Many of...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

NOTE: This is only a proposal at this time

Some parts of this document will refer to the Art of Unix Programming, which I (wlach) recommend highly.

Problem

Many of our automation harnesses (talos, mochitest, etc.) are overly involved to set up and don't give clear and understandable error messages when they fail. This in turn leads to several problems:

  • Difficult to bring new members of automation & release engineering up to speed on projects
  • Difficult to attract new developers to work on our stuff (when things fail on first try for unknown reasons, it's discouraging!)
  • Difficult for developers to test their software (hard to disambiguate problems in their own code with failures in the automation)

In most cases, this isn't actually due to systemic problems within the infrastructure but rather just small "papercuts" where we're not doing some combination of the following:

  • Automating setup steps
  • Checking preconditions before executing code
  • Handling errors when they occur (see the of repair: "when you must fail, fail noisily and as soon as possible")
  • Giving clear and understandable output and debugging messages (but only when the user asks for it, see the [rule of silence|http://catb.org/~esr/writings/taoup/html/ch01s06.html#id2878450])

Goals

The goal of this project is to quite simply correct this problems bug by bug. There is no systemic problem in our code that's causing the problems outlined above: usually individual problems are fixable by small patches. In some cases, we may need to refactor a subsystem in our automation to fix a problem, but this should be rare.

As a bonus, often these issues make "good first bugs", which will hopefully get the community excited about hacking on our infrastructure.

FIXME: By what criteria could we consider this project 'done'? FIXME: Would changing the way we fail in automation interfere with the way we're flagging oranges currently? If so, what can we do to mitigate this?

List of Papercuts

  • run_tests.py should err out early if firefox is already running ({bug|698561}) (potential mentors: wlach, jmaher, jhammel)
  • pageloader extension/talos should die more gracefully when pageset doesn't exist ({bug|695937}) (potential mentors: wlach, jmaher, jhammel)
  • FIXME: Need more of these!

Questions/Comments/Concerns

Feel free to drop us a line over at the #ateam channel!

Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Projects/Automation_Papercuts&oldid=363070"