QA/Mobile/FocusAndroidTestPlan

This is the Test Plan for Focus for Android (https://github.com/mozilla-mobile/focus-android)

Intro/Summary/Notes

Intro and Summary

• This wiki outlines the test requirements for Focus for Android V1.
• Use this generalized Test Plan as a starting point for creating Plans, Suites, and Cases

Notes

• FAQs

Feature Lists

• "Must" P1 features (as of May 2017 and subject to change):
  • Provide tracking-free browsing experience
  • Erase browsing history
  • Enable/Disable blocking of different types of trackers
  • Onboarding, we will highlight features that we believe will bring retention
  • Per session blocking
  • Custom tabs (scoped down)
  • Show notification while browsing session is active
  • Create localized completion list
  • Setting to allow screenshots
  • Support app shortcuts API

• P2 (if time permits, in order of priority)
  • Show number of ads/trackers blocked (4)
  • Locale switcher for Android (5)
  • Allow saving images in a website (4)
  • Add "search" action into text selection of third-party apps (3)

• NOTE: Focus for Android bugs are maintained in Github
• (List of complete P1 features in Github)

Schedule

• All features for V1 should be completed by June 9th, 2017 to begin acceptance testing
• Testing, including acceptance testing, should be completed by June 14th, 2017.
• Focus for Android is currently scheduled to be released June 20th, 2017.

Personnel

Program Management

• Jenn Chaulk

Product Management

• Barbara Bermes

Development

• Sebastian Kaspari
• Andrzej Hunt

QA Team

• No-Jun Park
• Aaron Train

Resource Plan

• Each QA person will be 40% allocated to this project

Test Strategy

Manual Tests

UI Test suite for Focus for Android will be located in TestRail. As a minimum, each of the P1 features will have corresponding test cases. We will use Focus for iOS as a base.

Automated Tests

Automation tests will be mainly used for quick regression check of key UI features. Overall, automation tests will be added to the same Github repo, and executed on BuddyBuild CI system. While the unit tests will be run for each commit, UI Tests (in Espresso framework) will be executed in master branch only. The test result will be inspected for new failures, and github issue will be raised for tracking.

Following automation test cases will be added by end of the May:

• Accessing badly formed URL
• Accessing each of pull-down menu items
• Accessing simple webpages
• Changing search engine
• Checking for overall appearance
• Deleting browsing history
• Quick check of URL autocompletion
• Quick check of webview data deletion
• Quick check of Onboarding dialog (Pending)
• Custom tabs (Pending)
• Check notification while browsing session is active (Pending)

Automation tests are located here.

In addition to regularly run automated tests, QA will also generate screenshots for l10n verification.

Code Analysis

• Codecov is added to the github repo, and will display increase/decrease of unit test code coverage
• Findbugs addon is added to warn developers against possible code issues, as well as Lint.

Outside Scope

Following tests are outside the scope of the QA:

• Unit Tests
• Performance Tests
• Localization Tests
• Beta testing with a wider audience

Test Deliverables

• Completed TestRail Test suite which covers all P1 features with UI aspects
• Automated Test suite in master branch, running on every checkin
• Github issues for every issue found
• Manual acceptance test report
• Signoff Decision

Testing Days

June 14th 2017:

• Ran Functional test suites on both Android phone (Samsung Galaxy S6 EDGE - Android 7.0) and tablet (Nexus 9 - Android 6.0.1)
  • Test rail suite for Nexus 9 Android 6.0.1
    
  • Test rail suite for Samsung Galaxy S6 EDGE Android 7.0
• 2 New issues added to the mozilla-mobile/focus-android github section
  • Screenshot block notification not present on tablet devices - this one is a little bit confusing for the user
    
  • Focus Crashes if you erase history from android bar while site loading - this one, IMO should be resolved ASAP since it crashes the App pretty easily
    

July 14th, 2017 (Ioana Chiorean):

• Ran Functional test suite on:
  • Nexus 7 - Android 5.1.2
  • Pixel - Android 7.1.2
  • Nexus 6P - Android 8.0
• New issues added to the mozilla-mobile/focus-android github section
  • Thumbnail in Recent Apps shows settings menu #924
  • White screen while playing Vimeo videos #925
  • Vimeo.com displayed blank on Nexus 7 ( 5.1.2) #926
• Issues investigated/commented the mozilla-mobile/focus-android github section
  • Filter out Focus in the BottomSheet when "Open with..." is clicked #894

Dependencies/Risks

• Since the blocking status of trackers is not exposed via adb log or UI, its validation will rely on unit tests
• The performance of Focus for Android is not measured precisely, will rely on the tester feedback
• The tester pool for a preliminary first release will be extremely small
• The automation/manual test suites will be built incrementally as new features are added to Github, and the late features may not be exposed to validation as much as earlier ones. The acceptance test after the feature completion date will be performed with the special emphasis on 1. key features and 2. most recently added features

Signoff/Exit Criteria

• No open v1.0 bugs, which impacts the security and key operations of the app
• No critical or major UI bugs
• No easily reproducible crashes