Auto-tools/Projects/Proposed/RegressionHunter

Summary and Goal

We need to update (or create) our regression hunting tools so that they are branch aware. They should be able to bisect along a repo's changes, and upon encountering a merge commit from a project branch, they should follow the thread of changes into that project branch and continue bisecting.

They can use nightly builds for binaries to test. Once the script has narrowed the window to two nightly builds, it then continues bisecting by building each changeset between these nightlies. It will build the binaries in the cloud, download them, test them, then figure out the next changeset to build in the cloud. Once it finds the offending changeset, it reports that.

Proposed By

Ctalbert, Bmoss, Harth

Ideal ETA

End of Q2?

Requirements/Features

If you want to leave some ideas on requirements/features/thoughts, do so in a section under your name. That makes it easy to come back to you if we have questions later.

We'll file bugs once we have nailed down what we're implementing.

Ctalbert

• simple interface, requires minimal installation of other tools on users' machines
• must NOT require local build infrastructure on local machine
• must have mechanism for communicating with cloud builder across networks/vpns etc
• should be accessible without LDAP, ideally.
• should be have multiple means to determine whether a changeset is exhibiting the issue or not.
  • phase 1: checks return code of a specified test program - nonzero return code indicates failure
  • phase 1: (alternatively) starts the browser, asks the user if the problem remains in the build
  • phase 2: allows submitting a simplified version of our file-based tests (mochi, ref etc) and runs it, analyzes test output and uses the test result as the means for continuing to bisect or not.

Use Cases

• You find a regression, you start bisecting. The tool launches each version of the browser for you and you manually determine if the problem is still present by answering interactive prompts from the tool.
• You find a regression, you start bisecting, giving the tool a test to run. The tool whirs away, automatically determining whether the test is passing or failing based on the return code from the test (useful for crashes, for instance).

Related Bugs, Dependencies, & Starting Points

Links to any related bugs, ongoing work, proof of concept, etc.

Non-Goals

Other Stuff

• If the tool will need metadata in the commit message in order to follow the trail of commits back to project repos, then a script should be created for project repo maintainers to ensure this metadata will always be present. (Hopefully this is unnecessary)

Legend (remove if you like)