Test Case Writing Guidelines

Background

After a few years of Firefox OS development and testing, we (the Firefox OS QA team) found that the style of writing test cases in MozTrap is hard to maintain. Therefore we would like to come up with a test case writing guideline that promotes readability, maintainability and boosts QA productivity.

The test cases we refer to are functional/regression cases written in MozTrap. They may potentially be written into automation scripts.

Guidelines

1. Keep multiple conditions in one test case

   Example 1:

   Don't write two cases, one for testing turning on WiFi, one for testing turning off WiFi

   Do write one test case only like so: 1. Turn on WiFi, verify the Internet connection is working 2. Turn off WiFi, verify the Internet connection is not working.

   Example 2

   Put conditions, like connection type (WiFi, mobile data, etc.) for an app that requires Internet access, in one single case.

2. Don't write multiple cases that only verifies the UI matches the UX spec, instead write a single case that says "Verify App X's UI matches the UX spec (https://link.to.ux.spec.pdf)"
3. Avoid hard-coding UI texts in steps, otherwise you might need to update the case frequently when UI changes.

   Example 1:

   Don't write "A popup box saying 'No connection found' appears, click the "OK" button to dismiss it" (The "No connection found" and "OK" button may change, and it's not l10n-friendly.)

   Do provide the UX spec and l10n documentation, and write something like "A warning message appears, dismiss it".

   Example 2:

   Don't write "Open https://www.mozilla.org to verify the Internet is OK"

   Do write "Open some website to verify the Internet is OK"

4. Don't write multiple cases that only verifies aborting/cancelling works

   Example 1:

   Say, we have a setting sequence that goes through steps A -> B -> C, each step has a continue and cancel option.

   Don't write 4 cases like

   1. A -> cancel
   2. A -> B -> cancel
   3. A -> B -> C -> cancel
   4. A -> B -> C -> done

   Do write one case for A -> B -> C -> done, but if you think the cancelling is important for the target under test, write "Repeat the case, but try cancel in every step" in the end of the test.

5. If multiple cases shares the same complex setup steps, write a wiki for that. Attach the wiki link in the test description. Also, maintain versioning in the wiki link, so old test will point to old steps.
6. If the UX spec has a standard name for UI elements (e.g. the menu button), use that name instead of something like "The button on the top right corner with a hamburger icon".

Contributing

If you wish to contribute, create a section below and write your thoughts, then send and email to qa-b2g-internal for dicussion If you wish to report bugs in this doc, email to slyu@mozilla.com