TestcaseManagementIdeas

From MozillaWiki
Jump to: navigation, search

Introduction

  • A web-based tool to manage testcases for mozilla.org.
  • Keep track of the tests we have, when they are run, and their status on various platforms.

Some (very rough) mockups

What is being tracked

  • Testcases. Three main types-
    • Fully manual testcases (BFTs, smoketests, etc...)
      • These consist of a series of steps to perform and a description of the intended behavior of the test.
    • Semi-automated testcases
      • These consist of some content that must be displayed, along with steps for a user to follow to determine if the test has passed or failed.
    • Fully automated testcases
      • Test consist of content that determines if the test has passed or failed and gives an appropriate (js) result.
  • Testcases have state. All states are set on a per-platform and per-branch basis. States are:
    • Pass
    • Fail
  • Associated with a particular product are platforms and branches. A test therefore has multiple states, such that a test which is broken on the mac trunk is not assumed to be broken on the windows branch.
  • Testcases can also have flags associated with them. We would want to be able to add/remove flags on the fly, but initially flags would include such bits as:
    • unconfirmed - a user-submitted test that must be confirmed before it should get widespread use
    • broken - the test does not work (e.g. it is hard to understand or not applicable)
  • Organization of testcases --
    • Products
    • Groups -- each product consists of one or more test groups
    • Subgroups -- smaller divisions such as bookmarks, etc...
    • In addition, any number of tests can be put together to make a test suite.
      • Suites can be defined via a query (e.g. a set of all Firefox UI tests related to bookmarks => the Firefox BFT bookmarks suite)
      • Or suites can be defined by arbitrary test numbers (e.g. test numbers 64, 98, and 101 => the Firefox Browser smoketests)
      • Suites can also consist of other suites (e.g. all the Firefox smoketest suites => the Firefox smoketests).
      • At any point, tests can be reordered within a suite, probably by using some kind of fancy dhtml UI.

Formats

  • Testcases can have a format associated with them. For example, a standard BFT or smoketest might have a manual steps format, consisting of a series of actions to perform and a description of expected results. Other tests may have formats more suited to automation, such as a xul test format consisting of a chunk of XUL and a snippit of javascript.
  • New formats can be created as testing evolves.
  • Not everything has to fit into a "do these steps, look for this result" model.

Unanswered questions

  • Where do testcases live? In the system or in a directory structure that the system points to?
  • The "format" for testcases, particularly automated testcases. What form should tests be in so they can be managed by the system?
  • Editing testcases in ways other than html forms.