Testrunner

Introduction

Testrunner is a test development tool built on top of Bugzilla, our installation of which is located at http://testrunner.mozilla.org (t.m.o.) --where QA (quality assurance) can create, modify and monitor test plans, test cases, test runs and test results.

Examples of other Testrunner installations:

• Eduardo Fuentetaja's server running v0.6.1-beta: http://testrunner.citat.se/

• Opi's server running a recent CVS pull (0.6.1+ 14.04.05) (Warning: Under development! Might go poof at any time, but ideally this is what Seamonkey QA might use): http://www2.le-bit.de/bugzilla/index.cgi

At present the tests in Testrunner are manual, with a black-box persepective.

To-do list

1. Upgrade from Testrunner 0.2 to 0.6 (or whatever the most recent version).
2. Migrate existing test plans, functional groups and test cases into the more recent Testrunner version. Also, ensure that they remain accessible and useable in the new version.
   • Note that the old, pre-Bugzilla Component Makeover components had been used. We'll need to use the latest component organization.
3. More seamless integration with the https://bugzilla.mozilla.org (b.m.o.) database, such as
   • Allowing bug links in test cases and results (regardless of result status).
   • Cross-referencing products and components from b.m.o. into Testrunner, instead of recreating them.
4. Define (and publically document) usage guidelines for Testrunner, such as describing priviledges, outlining typical workflow, etc. This might also include a tutorial (as time permits).
   • Allow flags as an option, such a review request (for a test plan or case) or certification request (for a test run). Would be good to have some flexibility so that requests aren't required all the time, which might make the process slow or awkward. However, this would be useful for localization teams needing certification, or MoFo QA needing to certify release candidates at key points in the release cycle, etc.
5. Improve access control:
   • Define who can create testplans, which could differ from those who can modify test plans.
   • Same goes for test cases, functional groups and components.
   • For test runs, control who can modify which runs, but still have flexibility to allow >1 user per active run —particularly useful for long test runs like BFT's.
6. Improve Testrunner UI:
   • Clarify navigation amongst test plans, test cases, test runs and test results. (Might be resolved in a more recent Testrunner build.)
   • Improve maintenance, creation/editing and monitoring of test plans, test cases, test runs and test results. It'd be nice to be able to change the order of test cases in a functional group, too.
   • Allow for easier entry of comments for test runs and test cases.
   • Display last-modified date and user who modified a test plan or test case.
7. Integrate with Mister, the proposed tool to integrate https://update.mozilla.org (u.m.o.), Bouncer, the build systems (including Tinderbox/Bonsai), QA (noteably Testrunner and b.m.o.) and the release process.
8. Integrate with other QA tools, notably more automated and/or white/grey-box oriented testing (Eggplant? Smoketest extension?)
9. Integrate with Talkback (tracking build ids, test results, crash bugs, etc.)

Known issues and bugs

Some of these are observed in t.m.o., in which case they might've been resolved in a more recent Testrunner build.

1. (t.m.o.) We found that the old style products and components are being used, and currently hinder organization of test cases such as those for Seamonkey. (More or less point (2) under "To-do list" above.)
2. (t.m.o.) In spite of a case being in the "confirmed" state, a few such cases still don't appear in a test run. Why?
3. (t.m.o.) Unable to delete cases (and perhaps functional groups, components, etc.) when there's a test run including that cases. Might need a more obvious-looking "disabled" state?
4. (t.m.o.) Currently unable to change the order of test cases within a given functional group; frustrating when the order of tests matters.
5. (t.m.o.) Currently doesn't (easily) display the last-modified date or user who modified a given test plan, group or case.