Mozilla QA Community:Test Case Writing Day Guidelines

From MozillaWiki
Jump to: navigation, search

A LITMUS TEST CASE WRITING PRIMER

  • Version 1.0
  • Author: Marcia Knous
  • Date: 1/5/07
  • I. UNDERSTANDING LITMUS TEST CASES
  • II. UNDERSTANDING TERMINOLOGY USED IN TEST CASES
  • III. OVERALL CAVEATS FOR WRITING TEST CASES FOR LITMUS
  • IV. WRITING TEST CASES FOR LITMUS
    • A. TEST CASE TITLE
    • B. TEST CASE STEPS
    • C. EXPECTED RESULTS
    • D. SAMPLE TEST CASES

Writing a clear, concise manual test case is an important part of quality assurance. There are a number of things you can do to make sure a manual tester has easy steps to follow when he/she runs a test case in Litmus (litmus.mozilla.org), Mozilla's manual testing database. Also, be aware that a good chunk of tests that we write cover "functional testing", which means the emphasis is on how the feature works. Part of manual testing is confirming that an existing feature works as expected. As we continue development on our product, we may occasionally break certain functionality, and the purpose of the manual tests in Litmus is a "gut check" that everything works as expected. We want our browser and email users to be happy!

I. UNDERSTANDING LITMUS TEST CASES

In Litmus, we have three sets of testgroups, each is a superset of the other.

  • Full Functional Tests
  • Basic Functional Tests
  • Smoketests

Full Functional Tests are usually run before major releases or when significant changes have been made to an area of the browser or email client. The Full Functional Tests cover *every* aspect of a feature, and try to include interaction between features (or more commonly known as end-to-end testing). An example would be: A Full Functional Test of the toolbar would include tests that cover each area of functionality, including adding a new toolbar, resetting the toolbar to default, resizing the icons, dragging and dropping the icons, clicking incessantly in the toolbar area, etc.

Basic Functional Tests are a subset of the Full Functionals, and are run before every release. Basic Functionals are aptly named since the cover the minimum basic functionality of a feature. A basic functional test of the Firefox toolbar might cover more of the common operations, and wouldn't drill down and test every level of functionality.

Smoketests are commonly run on nightly builds, and are meant to catch regressions. They are a much smaller subset of the Basic Functionals, but cover many of the common operations that browser users may do while browsing. A smoketest of the Firefox toolbar would probably cover customizing the toolbar at one level and then making sure when you restarted that Firefox honored those settings (again, this has to do with the fact that smoketests are more focused on catching regressions).

Q: Where should my test case go once it has been created?

A: Don’t worry too much about which group a testcase should belong to, the Mozilla QA team can help you with putting your test case in the right bucket.

II. UNDERSTANDING TERMINOLOGY USED IN TEST CASES There is certain terminology associated with many areas of the browser and the email client. An example is the browser chrome, which is another name for the graphical framework and elements of the web browser window. The browser chrome includes things such as the title bar, toolbar buttons, scrollbars, status bar and window frame. An example in the Thunderbird mail client is the various panes that are used in the mail window, such as the thread pane, etc.

It is probably best not to include terminology such as "chrome" in Litmus test cases, since a wide variety of testers come to Litmus and may not know what the "browser chrome" is. A better approach is to call out specifically the elements of the chrome, such as the window frame, status bar or scrollbars. Even if you don't know the technical name for things, be as descriptive as possible. A Mozilla QA person can always go back in and assist you in cleaning up any unclear areas.

There are some standard items in the Browser User Interface that we like to refer to, and these include:

  • 1. Search Bar (area on the right hand side of the browser that has a little logo in it, used for searches)
  • 2. Location Bar (also known as the URL bar). Either term is fine when writing Litmus test cases.
  • 3. Menu Bar. Includes the items at the top of the browser such as File | Edit | View | History | Bookmarks | Tools | Help
  • 4. Toolbar. Includes a set of icons that allow you to navigate. There is also a Bookmarks Toolbar located below the main toolbar where you can add individual bookmarks, bookmark folders and RSS feeds.
  • 5. Context Menu. You get a context menu when you right click on the content area, or a mail folder, etc.

Bottom Line: Don't get too hung up in terminology. A Mozilla QA person can help you navigate these waters.

III. OVERALL CAVEATS FOR WRITING TEST CASES FOR LITMUS

  • 1. Be concise with your wording
  • 2. Make the steps easy to follow
  • 3. Be descriptive if necessary (See Example 2 for how being descriptive can help)
  • 4. Don't get hung up on terminology
  • 5. Have a clear set of expected results
  • 6. Write your test cases so someone unfamiliar with a browser or email client could understand them.
  • 7. When writing, keep your audience in mind.

IV. WRITING TEST CASES FOR LITMUS

The set of steps that you will create when writing a testcase are similar to the "Steps to Reproduce" that you might use when filing a bug (some of you might be familiar with this if you use Bugzilla). When writing a test case, here are some helpful guidelines to use: TEST CASE TITLE:

When creating a title for a test case, try to drill down and be as concise as possible.

Good examples would be:

  • Adding a Bookmark
  • Open Search Using Keyboard Shortcuts
  • Restore Home Page to default

Bad Examples would be:

  • Home Page
  • Help
  • Fonts

The bad examples don't give enough information about what part of help, home page and fonts will be tested. A better phrase for both might be:

  • Setting a Home Page
  • Opening Mozilla Firefox Help
  • Increasing Font Size Using Keyboard commands

TEST CASE STEPS:

  • The minimal set of steps necessary to test the feature. Try not to be too wordy here - emphasis is on being concise!
  • When writing, think about the step by step process that will get you to the end game.
  • When I write test case steps, I often script out the basic steps first and then do several iterations to firm them up.
  • When writing a test case for a new feature, it is best to test all aspects of it first and get an understanding of how it works first before writing the test cases. So make sure to launch all windows and click all radio buttons to understand what is supposed to happen.
  • You can storyboard the screens if that is helpful. One way it so to take screenshots and then write your steps below them.
  • If there are special setup steps, make sure to include them at the top of the test case. An example of this would be: If you are asking a tester to test a feature that requires a fresh profile, make sure to state that at the beginning of the test case. Otherwise the tester may test using a profile that already exists.
  • Remember that Firefox and Thunderbird products run on three platforms (Windows, Mac and Linux), and there may be slight differences in behavior between the various platforms. If you have the ability to check the other platforms when writing test cases, that is great. Otherwise, when you submit your test case note that it has only be written for Windows and a QA team member can tweak it so the verbiage works for the other platforms.

EXPECTED RESULTS:

  • What the application or feature should do when it is invoked by the user.
  • Try to avoid using terms like "Expected behavior should be observed." It is better to phrase the expected results so the person testing will understand what is supposed to happen.

Let's now look at some sample test cases.

SAMPLE TEST CASES:

EXAMPLE 1:

You are asked to write a test case to test the functionality of adding an icon to the Firefox toolbar.

When writing this test case, you will first need to think about the series of steps you would need to take to add a toolbar icon. When flushing this out, you also need to remember there are various ways to invoke the "Customize Toolbar" User Interface, and you might need to consider that fact when writing the testcase.

First, let's start with a minimal set of steps and expected results to get us to the additon of a toolbar icon.

Title: Add an icon to the Firefox toolbar

Steps:

  • 1. Right click on the toolbar, and select "Customize" from the dropdown menu.
  • 2. A "Customize Toolbar" dialog box containing icons/spaces should be shown.
  • 3. Using your mouse, select an individual icon.
  • 4. Drag the icon to the toolbar and place it to the right of the URL bar (between the end of the URL bar and the beginning of the search box).
  • 5. Click "Done" to close the Customize Toolbar dialog box.
  • 6. Restart Firefox.

Expected Results:

  • 1. The icon should be added to the toolbar in the place that you specified.

This is an example of a basic test case. But there are a number of assumptions that are not explicity called out there, such as

• other ways to get to the customize toolbar besides a right click (it is important to test these as well - menu and keyboard commands). It is not always necessary to include these in the test case, but it is helpful since it widens the swath of what is tested since users may come at the feature from different angles. Note that there are test cases in Litmus that do cover the keyboard functionality, so even if you don't include it in your test case all is not lost.

Further, the test case stands on its own as a test to whether you can add a icon, but what about the functionality of the actual icon once it has been added (such as when you click the Print icon, does it actually launch the print dialog)? It is often useful to write test cases that "kill two birds with one stone", and simply adding a line that says "Click on the toolbar icon you added and confirm that it launches the expected functionality" will make the test case solid. Note that this won't apply to every test case, but is just so happens that in this example there is great benefit from the addition of that extra line. Think about this kind of thing when you crafting your test cases. You don't want to include everything in one test case, but at the same time it behooves you to include something that will make the test case useful in more ways than one.

Here is the test case, rewritten to include testing the functionality of the Print icon.

  • 1. Right click on the toolbar, and select "Customize" from the dropdown menu.
  • 2. A "Customize Toolbar" dialog box containing icons/spaces should be shown.
  • 3. Select the Print icon using your mouse.
  • 4. Drag the Print icon to the toolbar and place it to the right of the URL bar (between the end of the URL bar and the beginning of the search box).
  • 5. Click "Done" to close the Customize Toolbar dialog box.
  • 6. Restart Firefox.

Expected Results:

  • 1. The Print icon should be added to the toolbar in the place that you *specified.

2. Clicking on the Print icon should launch the Print dialog box.

Note that is is not necessary in include Step 6, Restart Firefox. But the benefit of this step is verifying that after a browser restart that Firefox is honoring the preference you have set.

Example 2:

Title: Drag/Select a Web Page Steps:

  • 1. View any web page.
  • 2. Drag-select the page.

(Specifically, while holding down the mouse button, drag the mouse pointer downwards from any point in the browser's content region to the bottom of the browser's content region.)

Expected Results:

The window should scroll downwards. Scrolled content should be selected.

How this test case could be improved:

This test case is a good first cut, but could still stand some improvement. For example, in the expected results, will a user definitively be able to identify how content is selected? Usually when the content is selected, it will be highlighted in a color. That would probably be something useful to include.

Also, this test case demands a page that has a scroll window, so it would useful to provide a URL for the user so they can navigate quickly to a page that has enough content to be scrolled.

Creating a Variation of this test case

If I wanted to create a variation of this test case, I might do something like this:

Title: Drag/Select a Web Page using "Select All"

Steps:

  • 1. View any web page.
  • 2. Right click in the content pane and choose "Select All."

Expected Results:

All content on the page should be highlighted. Note: The color of the highlighting may vary depending on your system settings.

Hopefully this has given you a basic framework to use when writing a test case. As noted earlier, the Mozilla QA team is ready, willing and able to help out in our IRC channel should you need assistance.