TestcaseManagementIdeas

Introduction

• A web-based tool (with an additional XML-RPC interface) to manage testcases for mozilla.org.

What is being tracked

• Testcases. Three main types-
  • Fully manual testcases (BFTs, smoketests, etc...)
    • These consist of a series of steps to perform and a description of the intended behavior of the test.
  • Semi-automated testcases
    • These consist of some content that must be displayed, along with steps for a user to follow to determine if the test has passed or failed.
  • Fully automated testcases
    • Test consist of content that determines if the test has passed or failed and gives an appropriate (js) result.
• Testcases have state. All states are set on a per-platform and per-branch basis. States are:
  • Pass
  • Fail
• Associated with a particular product are platforms and branches. A test therefore has multiple states, such that a test which is broken on the mac trunk is not assumed to be broken on the windows branch.
• Testcases can also have flags associated with them. We would want to be able to add/remove flags on the fly, but initially flags would include such bits as:
  • unconfirmed - a user-submitted test that must be confirmed before it should get widespread use
  • broken - the test does not work (e.g. it is hard to understand or not applicable)

Unanswered questions

• Where do testcases live? In the system or in a directory structure that the system points to?