Calendar:QA Procedures For Litmus

The Litmus Testing System is a great tool. This page will detail how we will handle the results from our test runs on the Calendar QA project. Note that all of this information is subject to change as Litmus develops. It is a great tool, but since it is being developed at the same time that we are using it, then many of these procedures will probably change.

Litmus Feature Wish List

These are items that will make working with Litmus (as described below) much easier.

• Ability to query on the "Vetted" status of failures and unclear results. This enables us to filter out results that have been "Vetted" out of our queries.
• Ability to query for all failures that do (or do not) have a bugzilla ID.
• Ability to query for all failures that are associated with a bugzilla ID AND where that Bug has been resolved. (In other words, ability to query for all test cases that should be retested due to fixed or resolved bugs.)

What to do with Test Cases that have Failed

If a test case has failed in Litmus, then follow these steps to vet the test case.

• Ensure that a Bug ID was entered for the failure.
  • If there is no Bug ID specified, search bugzilla for an existing bug and enter its number into the test case.
  • If no existing Bug can be found, please ensure that the issue is an actual bug (i.e. not a tester error), and please enter a new Bug and add its number to the test case.
• Once the test case has an actual Bug ID associated with it, please ensure the Valid checkbox is checked, and click the "Vet Result" button.

What to do with Test Cases that are Unclear/Broken

If a test case is marked as unclear or broken, we need to do follow these steps:

• Usually, testers will leave a comment if the test case is unclear.
• Understand the reason why the test case was unclear and correct the test case.
• Once the test case has been corrected and made more clear, ensure the Valid Checkbox is checked on the result, and click the "Vet Result" button.

If a test case previously passed and now fails

• This is very serious, and usually indicates a regression
• Ensure a bug ID is specified, and when the bug is fixed, be sure to retest this test case

If a test case previously failed and now passes

• This indicates that a long standing problem has now been fixed. In this case make sure (if possible) that you retested issue using the same architecture, operating system and of course product (sunbird or lighining).
• Ensure that the associated bug ID was resolved or fixed.
  • If the bug ID was NOT resolved or fixed, then this bug may have been fixed by a side-effect en change from another bug. In this case, make a comment about this in Bugzilla, inviting the developers to understand how the bug was suddenly fixed.

If a test case has both previously passed and failed

• This indicates that either the test case was unclear or the code in this area continues to break and to be fixed.
• Try to understand which issue this is, contact previous testers, review tester comments.
• If the test case seems to be unclear, try to re-write its steps and expected behaviors.
• If the test case does seem to test code that regresses and breaks often, alert the developers to this. Also, be sure to test that area of code any time you download a nightly or begin a test day.