TestcaseManagementIdeas: Difference between revisions
Jump to navigation
Jump to search
Zachlipton (talk | contribs) No edit summary |
m (Reverted edit of Jepomsfjw, changed back to last version by Zachlipton) |
||
| (6 intermediate revisions by 2 users not shown) | |||
| Line 2: | Line 2: | ||
* A web-based tool to manage testcases for mozilla.org. | * A web-based tool to manage testcases for mozilla.org. | ||
* Keep track of the tests we have, when they are run, and their status on various platforms. | * Keep track of the tests we have, when they are run, and their status on various platforms. | ||
=== Some (very rough) mockups === | |||
* [http://www.zachlipton.com/litmusmockups/ Home] | |||
* [http://www.zachlipton.com/litmusmockups/confirmedfailures.html Query] | |||
* [http://www.zachlipton.com/litmusmockups/product.html Product page] | |||
* [http://www.zachlipton.com/litmusmockups/test-manual.html testcase] - UNFINISHED | |||
=== What is being tracked === | === What is being tracked === | ||
| Line 17: | Line 23: | ||
* Testcases can also have flags associated with them. We would want to be able to add/remove flags on the fly, but initially flags would include such bits as: | * Testcases can also have flags associated with them. We would want to be able to add/remove flags on the fly, but initially flags would include such bits as: | ||
** unconfirmed - a user-submitted test that must be confirmed before it should get widespread use | ** unconfirmed - a user-submitted test that must be confirmed before it should get widespread use | ||
** broken - the test does not work (e.g. it is hard to understand or not applicable) | ** broken - the test does not work (e.g. it is hard to understand or not applicable) | ||
* Organization of testcases -- | |||
** Products | |||
** Groups -- each product consists of one or more test groups | |||
*** For instance, the Firefox product might have a menubar group, a toolbar group, etc... See [http://test.bclary.com/docs/test-organization.xhtml Bob's document on test organization]. | |||
** Subgroups -- smaller divisions such as bookmarks, etc... | |||
** In addition, any number of tests can be put together to make a test suite. | |||
*** Suites can be defined via a query (e.g. a set of all Firefox UI tests related to bookmarks => the Firefox BFT bookmarks suite) | |||
*** Or suites can be defined by arbitrary test numbers (e.g. test numbers 64, 98, and 101 => the Firefox Browser smoketests) | |||
*** Suites can also consist of other suites (e.g. all the Firefox smoketest suites => the Firefox smoketests). | |||
*** At any point, tests can be reordered within a suite, probably by using some kind of fancy dhtml UI. | |||
=== Formats === | |||
* Testcases can have a format associated with them. For example, a standard BFT or smoketest might have a manual steps format, consisting of a series of actions to perform and a description of expected results. Other tests may have formats more suited to automation, such as a xul test format consisting of a chunk of XUL and a snippit of javascript. | |||
* New formats can be created as testing evolves. | |||
* Not everything has to fit into a "do these steps, look for this result" model. | |||
=== Unanswered questions === | === Unanswered questions === | ||
* Where do testcases live? In the system or in a directory structure that the system points to? | * Where do testcases live? In the system or in a directory structure that the system points to? | ||
* The "format" for testcases. What form should tests be in so they can be managed by the system? | * The "format" for testcases, particularly automated testcases. What form should tests be in so they can be managed by the system? | ||
* Editing testcases in ways other than html forms. | |||
Latest revision as of 10:30, 25 November 2006
Introduction
- A web-based tool to manage testcases for mozilla.org.
- Keep track of the tests we have, when they are run, and their status on various platforms.
Some (very rough) mockups
- Home
- Query
- Product page
- testcase - UNFINISHED
What is being tracked
- Testcases. Three main types-
- Fully manual testcases (BFTs, smoketests, etc...)
- These consist of a series of steps to perform and a description of the intended behavior of the test.
- Semi-automated testcases
- These consist of some content that must be displayed, along with steps for a user to follow to determine if the test has passed or failed.
- Fully automated testcases
- Test consist of content that determines if the test has passed or failed and gives an appropriate (js) result.
- Fully manual testcases (BFTs, smoketests, etc...)
- Testcases have state. All states are set on a per-platform and per-branch basis. States are:
- Pass
- Fail
- Associated with a particular product are platforms and branches. A test therefore has multiple states, such that a test which is broken on the mac trunk is not assumed to be broken on the windows branch.
- Testcases can also have flags associated with them. We would want to be able to add/remove flags on the fly, but initially flags would include such bits as:
- unconfirmed - a user-submitted test that must be confirmed before it should get widespread use
- broken - the test does not work (e.g. it is hard to understand or not applicable)
- Organization of testcases --
- Products
- Groups -- each product consists of one or more test groups
- For instance, the Firefox product might have a menubar group, a toolbar group, etc... See Bob's document on test organization.
- Subgroups -- smaller divisions such as bookmarks, etc...
- In addition, any number of tests can be put together to make a test suite.
- Suites can be defined via a query (e.g. a set of all Firefox UI tests related to bookmarks => the Firefox BFT bookmarks suite)
- Or suites can be defined by arbitrary test numbers (e.g. test numbers 64, 98, and 101 => the Firefox Browser smoketests)
- Suites can also consist of other suites (e.g. all the Firefox smoketest suites => the Firefox smoketests).
- At any point, tests can be reordered within a suite, probably by using some kind of fancy dhtml UI.
Formats
- Testcases can have a format associated with them. For example, a standard BFT or smoketest might have a manual steps format, consisting of a series of actions to perform and a description of expected results. Other tests may have formats more suited to automation, such as a xul test format consisting of a chunk of XUL and a snippit of javascript.
- New formats can be created as testing evolves.
- Not everything has to fit into a "do these steps, look for this result" model.
Unanswered questions
- Where do testcases live? In the system or in a directory structure that the system points to?
- The "format" for testcases, particularly automated testcases. What form should tests be in so they can be managed by the system?
- Editing testcases in ways other than html forms.