- 1 Overview
- 2 Detailed description of workflow
- For all fixed bugs, the state of test coverage should be clear and unambiguous
- Developers have the best insight into which bugs carry the most risk and require attention from QA
- QA is the best group to effectively test fixes, and has the best insight into which bugs require additional testing and verification
The thrust of the proposal is simple: we should replace the current assortment of keywords, flags, and whiteboard annotations with a single multi-state flag, and make setting this flag part of the required workflow when resolving a bug in a product component as FIXED. The value of this flag will indicate the next step in testing a bug, or denote what the final state of testing is for that bug. This proposed flag would be known as "testing-status" for the purpose of this proposal.
Detailed description of workflow
On resolving a bug (Developer)
For a developer resolving a bug, one of three options should be chosen:
The fix is not effectively testable through manual or automated means. No further action will be required.
The fix has sufficient coverage through automated testing. No further action will be required.
If neither case above applies, QA should assess this bug for further action. (This replaces the verifyme keyword.) Developers are asked to provide as much relevant insight and information as possible for QA when setting this value.
When testing-needed is set (QA)
For all bugs where the flag is set to testing-needed, QA will review that bug and set one of the following four options:
The fix should be tested through automated testing, but current testing frameworks are insufficient.
The automation tools team (A-Team) is responsible for assessing these bugs and providing framework support on a priority basis. Once a framework exists, the flag should be set to automated-test-needed
The fix can be tested through existing frameworks.
Once a test is created and checked in, the flag should be set to in-automated-testsuite
QA has determined that this bug cannot be tested through automated means, but should be periodically tested as part of manual test runs.
Once a test has been documented and added to moztrap, the flag should be set to in-manual-testsuite
QA has determined one-off testing is sufficient, and no followup action is required. At QA's discretion, time spent on one-off testing may be as little as zero. No further action will be required.