From MozillaWiki
< QA
Jump to: navigation, search
--------------->>> THIS IS A DRAFT! <<<---------------  

This is the test plan for Mozilla Reference Browser


Intro and Summary

  • This wiki outlines the test requirements for Reference Browser releases.
  • Use this generalized test plan as a starting point for creating test plans, suites and case in: TestRail


  • FAQ(s) - TBD

Feature Lists

  • Mozilla Reference Browser is a web browser reference implementation using Mozilla Android Components
  • Reference Browser is not a product intended to ship to end users. Instead it is a Technology Preview for many new mobile components that ...
  • NOTE: Reference Browser bugs are maintained in Github


  • All features for TBD should be completed by TBD to begin acceptance testing
  • Testing, including acceptance testing, should be completed by TBD.
  • Reference Browser TBD is currently scheduled to be released TBD.


QA Team

Resource Plan

  • Each QA person will be TBD% allocated to this project

Test Strategy


Current Sprint


  1. Label Issues
  2. Create/Update Test Cases
  3. Create/Update Automated Test Cases
  4. Verify Issues


  • Issues should belong to a correct milestone before development work can be started
  • Issues may include new features or bugs


  • Label: "QAReady"
    • When the dev work on a github issue is completed (before/after code freeze), the issue will be closed and marked with 'QAReady' status label
  • TestRail
    • Once an issue has been closed (where applicable), a test case should be created (or updated) in TestRail
    • TestRail type field should be set to: "Functional"
    • TestRail automation field to: "Untriaged"
  • Verification
    • QA will verify the issue, and make sure to outline clear test steps in the corresponding TestRail test case
  • Label: "QAVerified"
    • Upon completion, github issue should be marked with QAVerified, removing QAReady status label if exists
    • After the QA signoff, ideally, all verifiable issues should be marked with QAVerified
  • Label: "testing"
    • NOTE: the meta-label testing is also added by both devs and QA for any bug or feature requiring testing of any kind)
  • Label: "automation-TBD"
    • Once the issue is closed, if suitable for automation, label the it in github as automation-TBD and change the automation dropdown in TestRail to "Suitable"

Current or Post Sprint


  1. Query github for issues labelled "automation-TBD"
  2. Create new automated test(s) to verify proper function of feature based on steps outlined in TestRail
  3. Cross-link TestRail test case within automated test (add URL of kotlin test file in github to Pre-Conditions section)
  4. Cross-link automated test to TestRail test case (paste URL to TestRail test case in comment kotlin file of corresponding test)
  5. Submit automated test PR for review


Once UI tests are passing:

  1. change github label to automated
  2. set TestRail type to: "Automated"
  3. set TestRail automation field to: "Completed"

QA Criteria


Dev team

  • Announces code freeze
  • Date for QA signoff decided (usually 1 week from code freeze)



  • During triage, Dev & QA teams decide whether issue(s) found are release-blocking or not
  • At a minimum, a bug shall be a release blocker if it causes crash/hang on a relatively common use case scenario
  • Dev team will provide fix for release blocker(s) during code freeze
  • QA team (or dev team) will verify fix

SIGN OFF QA will sign off the release when:

  1. (Up-to-date) regression test suite run has been completed
  2. All items under Test_Deliverables has been completed
  3. There are no release blockers

Manual Tests

  • UI test case suite for project: "Reference Browser" will be located in TestRail
  • At a minimum, each P1 UI feature will have corresponding test case
  • Test suite will be executed by Softvision QA for acceptance testing

Automated Tests

  • Automated tests will be mainly used for quick regression check of key UI features
  • Overall, automated tests will be added to the same Github repo, and executed on Google Firebase using TaskCluster or Bitrise build system
  • While the unit tests will be run for each commit, UI Tests (in Espresso/UIAutomator framework) will be executed in master branch only. - TBD
  • The test result will be inspected for new failures
  • Github issue(s) will be raised for tracking

Automated tests are located here.

In addition to regularly run automated tests, QA will also generate screenshots for l10n verification. - TBD

Code Analysis

  • Codecov is added to the github repo (TBD), and will display increase/decrease of unit test code coverage
  • Findbugs addon is added to warn developers against possible code issues, as well as Lint - - TBD

Within Scope

Following tests are within the scope of QA team:

Outside Scope

Following tests are outside the scope of QA team:

  • Unit Tests
  • Performance Tests
  • Localization Tests
  • Beta testing with a wider audience

Test Deliverables

  • Completed TestRail Test suite which covers all features with UI aspects
  • Automated Test suite in master branch, running on every checkin - - TBD
  • Github issue filed for each issue found
  • Manual acceptance test report - - add description?
  • Signoff Decision - - more detail - by whom? where? which communication channel?


  • Since blocking status of trackers is not exposed via ADB log or UI, validation will rely on unit tests
  • Performance of Reference Browser is not measured precisely and will rely on the tester feedback
  • Tester pool for preliminary first release will be extremely small
  • Automated / manual test suites will be built incrementally as new features are added to Github
  • Late features may not be exposed to validation as much as earlier ones
  • Acceptance test (after feature completion date) will be performed with special emphasis on:
    • key features
    • most recently added features

Signoff/Exit Criteria

  • No critical or major UI bugs
  • No easily reproducible crashes