QA/Mobile/ReferenceBrowserTestPlan

From MozillaWiki
< QA
Jump to: navigation, search 
--------------->>> THIS IS A DRAFT! <<<---------------

This is the test plan for Mozilla Reference Browser

Intro/Summary/Notes

Intro and Summary

  • This wiki outlines the test requirements for Reference Browser releases.
  • Use this generalized test plan as a starting point for creating test plans, suites and case in: TestRail

Notes

  • FAQ(s) - TBD

Feature Lists

  • Mozilla Reference Browser is a web browser reference implementation using Mozilla Android Components
  • Reference Browser is not a product intended to ship to end users. Instead it is a Technology Preview for many new mobile components that ...
  • NOTE: Reference Browser bugs are maintained in Github

Schedule

  • All features for TBD should be completed by TBD to begin acceptance testing
  • Testing, including acceptance testing, should be completed by TBD.
  • Reference Browser TBD is currently scheduled to be released TBD.

Personnel

QA Team

Resource Plan

  • Each QA person will be TBD% allocated to this project

Test Strategy

Process

Current Sprint

SUMMARY

  1. Label Issues
  2. Create/Update Test Cases
  3. Create/Update Automated Test Cases
  4. Verify Issues

NOTE

  • Issues should belong to a correct milestone before development work can be started
  • Issues may include new features or bugs

STEPS

  • Label: "QAReady"
    • When the dev work on a github issue is completed (before/after code freeze), the issue will be closed and marked with 'QAReady' status label
  • TestRail
    • Once an issue has been closed (where applicable), a test case should be created (or updated) in TestRail
    • TestRail type field should be set to: "Functional"
    • TestRail automation field to: "Untriaged"
  • Verification
    • QA will verify the issue, and make sure to outline clear test steps in the corresponding TestRail test case
  • Label: "QAVerified"
    • Upon completion, github issue should be marked with QAVerified, removing QAReady status label if exists
    • After the QA signoff, ideally, all verifiable issues should be marked with QAVerified
  • Label: "testing"
    • NOTE: the meta-label testing is also added by both devs and QA for any bug or feature requiring testing of any kind)
  • Label: "automation-TBD"
    • Once the issue is closed, if suitable for automation, label the it in github as automation-TBD and change the automation dropdown in TestRail to "Suitable"

Current or Post Sprint

STEPS

  1. Query github for issues labelled "automation-TBD"
  2. Create new automated test(s) to verify proper function of feature based on steps outlined in TestRail
  3. Cross-link TestRail test case within automated test (add URL of kotlin test file in github to Pre-Conditions section)
  4. Cross-link automated test to TestRail test case (paste URL to TestRail test case in comment kotlin file of corresponding test)
  5. Submit automated test PR for review

FOLLOW-UP

Once UI tests are passing:

  1. change github label to automated
  2. set TestRail type to: "Automated"
  3. set TestRail automation field to: "Completed"

QA Criteria

STEPS

Dev team

  • Announces code freeze
  • Date for QA signoff decided (usually 1 week from code freeze)

QA

ISSUE RESOLUTION

  • During triage, Dev & QA teams decide whether issue(s) found are release-blocking or not
  • At a minimum, a bug shall be a release blocker if it causes crash/hang on a relatively common use case scenario
  • Dev team will provide fix for release blocker(s) during code freeze
  • QA team (or dev team) will verify fix

SIGN OFF QA will sign off the release when:

  1. (Up-to-date) regression test suite run has been completed
  2. All items under Test_Deliverables has been completed
  3. There are no release blockers

Manual Tests

  • UI test case suite for project: "Reference Browser" will be located in TestRail
  • At a minimum, each P1 UI feature will have corresponding test case
  • Test suite will be executed by Softvision QA for acceptance testing

Automated Tests

  • Automated tests will be mainly used for quick regression check of key UI features
  • Overall, automated tests will be added to the same Github repo, and executed on Google Firebase using TaskCluster or Bitrise build system
  • While the unit tests will be run for each commit, UI Tests (in Espresso/UIAutomator framework) will be executed in master branch only. - TBD
  • The test result will be inspected for new failures
  • Github issue(s) will be raised for tracking

Automated tests are located here.

In addition to regularly run automated tests, QA will also generate screenshots for l10n verification. - TBD

Code Analysis

  • Codecov is added to the github repo (TBD), and will display increase/decrease of unit test code coverage
  • Findbugs addon is added to warn developers against possible code issues, as well as Lint - - TBD

Within Scope

Following tests are within the scope of QA team:

Outside Scope

Following tests are outside the scope of QA team:

  • Unit Tests
  • Performance Tests
  • Localization Tests
  • Beta testing with a wider audience

Test Deliverables

  • Completed TestRail Test suite which covers all features with UI aspects
  • Automated Test suite in master branch, running on every checkin - - TBD
  • Github issue filed for each issue found
  • Manual acceptance test report - - add description?
  • Signoff Decision - - more detail - by whom? where? which communication channel?

Dependencies/Risks

  • Since blocking status of trackers is not exposed via ADB log or UI, validation will rely on unit tests
  • Performance of Reference Browser is not measured precisely and will rely on the tester feedback
  • Tester pool for preliminary first release will be extremely small
  • Automated / manual test suites will be built incrementally as new features are added to Github
  • Late features may not be exposed to validation as much as earlier ones
  • Acceptance test (after feature completion date) will be performed with special emphasis on:
    • key features
    • most recently added features

Signoff/Exit Criteria

  • No critical or major UI bugs
  • No easily reproducible crashes
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Mobile/ReferenceBrowserTestPlan&oldid=1207020"