QA/Execution/ManualTestingLevels

Mozilla Quality Assurance Manual Testing

Purpose

Explain in general the various levels of manual testing in Mozilla QA. Describe when, why and how these test are done.

A Multiple Level Approach

Smoketest

This level of testing is meant to be a quick check of the most critical features to ensure the product is ready for more in depth testing or ready for more development. For Firefox, a dry, long over due for update, set of approximately 40 test cases.

• In the past, this was run on a daily basis on nightly builds.
• Presently, it is run on release candidates only.
• The future:
  • For Q2 2009, QA will be redefining and somewhat expanding this set of test cases for Mozmill coverage to be run each morning on nightly builds.
  • A scenario based set of test cases (approx. 12 test cases). This Fun Smoketest will encourage community based testers to run test cases of the smoketest nature but doing so in such a way as to do their daily browsing.

Basic Functional Test (BFT)

This level of testing is meant to cover the most basic use cases of nearly every feature of the application. For Firefox, it is a large set of test cases that takes anywhere from 6-10 hours to run depending on the tester.

• In the past and the present, BFT's are run against point releases on Windows only and for Fx next milestones on Win, Mac and Linux.
• The future; redefine what should be covered in the BFT's as the current set in Litmus has become undirected and bloated. Efforts in Q2 2009 will be to clean up this level of testing by defining what features need testing and to what extent that should occur. The intent being to reduce the load level on point release testers, yet insure critical features of Firefox have not regressed.

Full Functional Test (FFT)

This level of testing is meant to cover every function of every feature of the application.

• In the past, this FFT's were run against every point release and milestone. however...
• Presently, this test suite has become so large in test case numbers that it can take a tester 2-3 days to run per platform. With limited personnel, this set of test cases can only be run at critical milestones; usually the first beta and rc candidates in Firefox next.
• For the future; The Mozilla QA team will never grow large enough to maintain such a large (and growing) test suite that is used only a few times per year. We need to take a serious and close look at what test cases in this suite are being sufficiently covered in other testing methods within Mozilla Firefox development.