What is Litmus?

Litmus is the new integrated testcase management and QA tool that is designed to improve workflow, visibility, and turnaround time in the Mozilla QA process. Its developed by Mozilla Corp/Mozilla Foundation. Please visit the Litmus wiki for more information.

See also the Mozilla QA Wiki Site and the Mozilla QA Weblog for Community QA Events

The Basics

Litmus is used by Mozilla QA and also by our great and fantastic Community QA Team. The Tests you run in Litmus help to maintain the Quality of Mozilla Projects and help to find possible Errors and Problems. This documentation shall give you a hand through Litmus.

Litmus is open 24/7 for any test you want to run and is also used to track the Result from our Community Testdays. The Testdays are announced in our Blog. If you don`t know which Builds you should test, or if you have any question about Litmus, just ask during testdays in the #testday channel on irc.mozilla.org, or at anytime in our #qa channel also on irc.

If you are also interested in helping QA, it would be great if you would also join our Bugdays.

Creating a Litmus account

To Track your Results and to do your Test, a Litmus account is required.

Registration Process

1. Go to https://litmus.mozilla.org
2. Click on login
3. Fill out the fields:
   • Email: This is your Login Later. Although Litmus takes steps to prevent spammers from harvesting your email address, spam cannot be eliminated entirely. We recommend using a free web email service (such as Yahoo, Gmail, or similar) and not your primary email address if this concerns you.
     
   • Real Name: This name is used to Display your results.
     
   • IRC Nickname: A lot of Testers use the Mozilla IRC System irc.mozilla.org (see http://irc.mozilla.org) (also called MozNet) for Realtime Chats. Even if you don't not plan to join the MozNet you can enter a unique name her (any unique nickname will do). This name is also used to Display your Results. If you don't enter a Nickname here, your Real Name will be used, if you don't enter a Nickname and Realname, your Email address will used to Display the Results.
   • Password: Please choose a Password for your Litmus Access. If you lost or forgot your Password see here how to restore it.

NOTE: Before you run a test, we suggest to create and use a new and clean profile for the testruns. Please see http://kb.mozillazine.org/Profile_Manager for instructions on how to create a new profile.

Setup a Test

Before Litmus can setup up the test for you, some information about your System (OS, Platform) and the Product (Firefox, Thunderbird, Calendar ....) and also the locale and Build ID is required.

• Product: Here you can choose the Mozilla Product (like Firefox, Thunderbird) you want to test.
• Platform: Here you can choose the Platform (Windows, Linux, Sun, Mac), on which you will run the Test.
• Operating System: Please select your Operation System here (Windows Vista, Linux)
• Branch: Depend on the Product that you want, you have here several Choices. As Example for Firefox, you have Trunk, 1.5 Branch and 2.0 Branch
  • 1.5 Branch: Build from the 1.5.x Tree. This means Builds and Releases from the 1.5 Tree like 1.5.0.0. - 1.5.0.x
  • 2.0 Branch: Builds from the 2.0 Tree. This means Builds and Releases like 2.0.x
  • Trunk: Latests Developer Testing builds, currently Minefield and Granparadiso Builds
• Locale: This means not your physical Location ! You can download Products like Firefox or Thunderbird in more than 30 languages/locales. You need to select here you the locale of your Browser (like en-US, DE, zh-CN for China).
• Build ID: The build ID is a 10-digit number that identifies a Mozilla product build down to the date and hour of the build. By supplying the full, correct build ID, you will be making the job of the Mozilla QA team much easier. There are several different ways to determine the build ID of the build you are testing.
  1. Manually: Provided you have installed Talkback, this is the fail-safe method, and allows you to submit the build ID for products other than the one currently being used to submit the results. The Mozilla Quality Assurance wiki has instructions on how to manually verify the build ID.
     
  2. Nightly Tester Tools: Available for both Firefox and Thunderbird, the Nightly Tester Tools extension adds a number of useful features for testing Mozilla products. This includes a handy display of the build ID of the running build in the title bar. You can download this extension from addons.mozilla.org.

Selecting a Test group

Litmus has several test groups with different testcases, depending on your selection in the System Information:</p>

• Surf's Up: The newest testing group in Litmus. This is intended to be a quick smoketest that should only take about 10 minutes of your time, but will help us test some of the common operations users are doing when they are on the Internet, including things like checking their mail, downloading files, etc.
• Smoketest: This Test covers common operations (Install, Uninstall, Usability). It includes a set of various operations. Its the Standard Test.
• Basic Functional Test (BFT): Its included more testcases than a Smoketest and covers all aspects of the Product. From Installing till Security Features.
• Full Functional Tests (FFT): Got some time? This is the biggest testrun in Litmus. It includes a lot of testcases for all aspects of the Product you have selected. Grab a Snickers™.
• Localization (l10): A test provided for for Localization Builds and this test covers aspect thats important for such builds (like Checks if everything is translated and working like in the en-US Build). This Test is important before a new Release or if you are a Member of a L10n Team.

What do the percentages mean?

The testing coverage percentages are based on Platform (Windows, Mac, Linux) and Build ID. The Community testing coverage displays how much is done by you and other testers, the Personal testing coverage is for your personal information, how much is done in the subgroup by you.

While we appreciate all testing efforts, it is more helpful if you select a testing group or subgroup for testing that does not already have complete testing coverage by the community.

Choosing a subgroup

1. Obviously, the first, best option is to choose subgroups that don't yet have 100% coverage;
2. Failing that, if you have another machine running a different operating system, try running some tests from that machine if the coverage statistics are lower;
3. If we are also testing different locales, download a different locale and use it for testing;
4. Try installing an extension or theme (preferably one of the more popular ones), and then test with it installed;
5. Regardless of coverage statistics, pick a subgroup that interests you and test it anyway; and finally,
6. Pick ANY FUNCTIONAL AREA that interests you and do your own testing around it, regardless of whether this area is covered in Litmus or not. Just be sure to report back to the QA team if you find anything!

How to run a Test and how submit the Results?

Running Tests in Litmus is very easy and you don`t need to be technical expert.

The Main Test Windows of the Testcase you run: Here you can execute this Test, you see which Steps are to perform and whats the expected result.

You have the choice if you want that only one testcase is Displayed, or if all testcases from this Subgroup should be displayed.

• Show all: Expand the List of the testcases, so you get an quick overview over all testcases in this Subgroup.
• View: If you click on View, only one Testcase will be displayed and also the Recent Results for this Testcase.

You have 2 Choices to Submit your Results to the Mozilla QA Team:

• Submit a single result: Submits only the current result of this testcase
• Submit all Results: Submit all your results and comments in this test group.

Also:

1. Steps to Perform: Every Testcase has a description what to do on this Testcase. Please follow this Steps on the testcases. If you have any Feedback, or if you think, the Testcase is unclear or broken, you can leave a Feedback.
2. Expected Result: Please check the Result you see with the Expected Results. If you are unsure, you can contact during Mozilla QA Testdays one of the moderators.
3. Results: You have 4 Choices the Result of your Testcase:
   • Not Run: You did not run this Testcase (this is marked by default)
   • Pass: The Testcase Pass and expected Result match with that what you see
   • Fail: What you see don`t match with the Expected Results
   • Test unclear/broken: The Testcase is broken in some way, has typo`s in it or some features are now missing/renamed in a new version.
4. Comments: On every Testcase, you have the opportunity to leave a comment. This comments help to Litmus (if this comment is about errors in a Litmus Testcase) or helps us to reproduce Problems/Errors in the Software you testing .

Because we get a lot of comments, please try to leave a "useful" comment: Examples:

• Useful: (for an unclear/broken testcase) : "Feature is now called 'Organize Bookmarks' and not 'Manage Bookmarks' like in the Testcase
• Not useful: (for an failed Testcase): "Failed, because don't work"

How do I search for results?

Its currently not possible to search in our own results, but there are some Reporting Tools included:

• Search Result: Here you can as example for Results with comments in a Timespan</p>
• Advanced search: Here you can search results limited by field, limited by date, etc
• Testdays: Here you get the Results from the QA - Testdays
• Statistics: Statistics will display some statistic Data about Litmus and the Top Tester in Litmus.
• Searchbar: With the bar on the bottom of every Litmus Page you can also search quick in some results.

Litmus Usage FAQ

• I Lost my Password for Litmus, how can i restore it?
  • Click on the "Forgot Password" button on the login screen. Barring that, contact a channel moderator in #qa or #testday on irc.mozilla.org. These moderators are normally also Admins in Litmus.
• I want my own Litmus Installation, where can i find information about installing Litmus
  • Please take a look at Litmus:InstallFAQ
• How many testcases do I have to run on a Testday?
  • Testing in Litmus is not a kind of Race. Just run as many tests you and your time can afford. There is no maximum or minimum requirement of tests on Testdays. The Top 15 Tester under Statistics have been using Litmus for a long time. We are thankful to for every test you run. These results will help us to improve the quality of Litmus and our Products, so please test carefully. This is more helpful than just clicking "PASS" on as many testcases as possible without carefully testing.
• During my Tests i found a Bug, where should i report this?
  • You can report in our Bug Tracking Database Bugzilla. Please make sure that you search for an existing Bugreport before. During a Testday or other Community Event, please report this also in the #testday or #qa channel.