This is the Test Plan for Lockbox for iOS

Intro/Summary/Notes

Intro and Summary

• This wiki outlines the test requirements for Lockbox releases.
• Use this generalized Test Plan as a starting point for creating Plans, Suites, and Cases

Daily standup notes

• If you want to follow along on daily progress and updates check here

Feature Document

• For the list of features and its status, please refer to here

Planning Board

• Waffle

UI/UX Designs

• Invision

Milestones

• Lockbox Milestone Schedule & Checklist here

Issues

• The Lockbox for iOS bugs are maintained in Github

Schedule

• May 25: feature complete
• June 1: app store submission
• June 27: app live in store and Test Pilot

QA Team

• Catalin Suciu
• Simon Basca

Test Strategy

Manual Tests

UI Test suite for Lockbox for iOS will be located in TestRail. As a minimum, each of the P1 UI features will have corresponding test cases. The test suite will be executed by Softvision for the acceptance testing.

Automated Tests

Automation tests will be mainly used for quick regression check of key UI features. Overall, automation tests will be added to the same Github repo, and executed on [1].

Automation tests are located here.

Code Analysis

• Codecov is added to the github repo, and will display increase/decrease of unit test code coverage

Outside Scope

Following tests are outside the scope of the QA:

• Unit Tests
• Performance Tests
• Localization Tests

Test Deliverables

• Completed TestRail Test suite which covers all features with UI aspects
• Github issues for every issue found
• Manual acceptance test report
• Signoff Decision

Testing Days

April 19th 2018

• Exploratory testing on master master 614cc5c
• Devices:
  • iPhone SE (iOS 10.3)
  • iPhone 6 (iOS 10.3)
• New issues filed in Github
  • Settings button not displayed on iOS 10.3 #278
  • Re-installing the app after signing out will automatically re-sign in the account when the app is launched #279

April 23th, 2018

• Exploratory testing on master 5c5d018
• Devices:
  • iPhone SE (iOS 10.3)
  • iPhone 8 Plus (iOS 11.3)
• New issues filed in Github
  • Sign in fails with "Type error" after canceling the sign in confirmation dialog #924
  • Sign in confirmation dialog is never dismissed on iPhone SE/iOS 10.3.3 #293

May 8th, 2018

• Exploratory testing on master 3ffd17f
• Devices:
  • iPhone SE (iOS 10.3)
  • iPhone 6 (iOS 10.3)
• New issue filed in Github
  • Settings" button is cut off in the "Account" page #337

May 10th, 2018

• Exploratory testing on master 3ffd17f
• Devices:
  • iPhone 7 (iOS 11.3)
  • iPhone 6 Plus (iOS 11.1)
• New issue filed in Github
  • Settings page is automatically dimissed when the logins list is populated #347

May 14th, 2018

• Exploratory testing on master 45173f6 and TestFlight build 1.0 (1054)
• Devices:
  • iPhone 6 (iOS 10.3)
  • iPhone 8 Plus (iOS 11.3)
  • iPhone SE (iOS 10.3)
• New issues filed in Github
  • Lockbox 1.0 (1054) crashes at startup #359
  • The sign in/up email confirmation dialog is not displayed #360
  • "Open websites in" - Selected browser is not unselected after uninstalling it #359

May 16th, 2018

• Exploratory testing on master and TestFlight build 1.0 (1090)
• Devices:
  • iPhone 6 Plus (iOS 11.1)
  • iPhone 7 (iOS 11.4beta5)
  • iPhone SE (iOS 10.3)
• New issues filed in Github
  • "Firefox Lockbox" title not centered on small screen devices #372
  • The passcode dots are barely visible on iOS 10.3 #373
  • "Open websites in" - Selected browser is not unselected after uninstalling it #359

May 17th, 2018

• Functional testing on build 1.0 (1128)
  • TestRail - iPhone 7 (iOS 11.4beta5)
  • TestRail - iPhone SE (iOS 10.3)
• New issues filed in Github
  • The logins become visible when interacting with the search/filter keyboard #376
  • Authorization code is requested a second time after inputting the credentials #394
  • The app is incorrectly uninstalled after tapping "Lock Now" #393
  • Logins screen flickers after searching for a login, backgrounding and restoring the app #377

May 18th/21st, 2018

• Functional testing on build 1.0 (1189)
  • TestRail - iPhone 7 (iOS 11.4beta5)
  • TestRail - iPhone SE (iOS 10.3)
• New issue filed in Github
  • The app auto-locks while using it #405

May 24th/25th, 2018

• Functional testing on build 1.0 (1280)
  • TestRail - iPhone 8 Plus (iOS 11.3.1)
  • TestRail - iPhone SE (iOS 10.3)
  • TestRail - iPhone 7 (iOS 11.4beta6)
  • TestRail - iPhone 6 (iOS 10.3)
• New issue filed in Github
  • The Welcome screen is briefly displayed after confirming a sign in #433

May 29th, 2018

• Functional testing on build 1.0 (1343)
  • TestRail - iPhone 8 Plus (iOS 11.3.1)
  • TestRail - iPhone 6 (iOS 10.3)

• Exploratory testing
  • iPhone SE (iOS 10.3)
  • iPhone X (iOS 11.3)

• New issues filed in Github
  • The app won't start after turning on the passcode in iOS settings #451
  • "Auto Lock" option is not listed in settings on devices that don't have a passcode set #450
  • The app auto-launches after force closing it while "Auto Lock" is set to "On app exit" #449
  • No spacing between "Settings" and "Open Websites in" strings on small screens #448
  • "Sign In" string not centered on iOS 10.3 #447
  • The Welcome screen is briefly displayed after confirming a sign in #433
  • "Pull to refresh" doesn't work when the logins list is empty #445

May 31st, 2018

• Exploratory testing
  • iPhone 8 Plus (iOS 11.4)

• New issues filed in Github
  • Logins are missing after auto-lock #462
  • Sorting selection gets reset after ~5 minutes #461

June 4th, 2018

• Exploratory testing
  • iPhone 8 Plus (iOS 11.4)

• Functional testing on build 1.0 (1387)
  • TestRail - iPhone 7 (iOS 11.4)
  • TestRail - iPhone SE (iOS 10.3)

• New issues filed in Github
  • Incorrect prompt to unlock when the passcode is disabled and the app auto-locks #467

June 26th, 2018

• Exploratory testing
  • iPhone X (11.3)

• Functional testing on build 1.1 (1490)
  • TestRail - iPhone 6 (iOS 10.3)
  • TestRail - iPhone SE (iOS 10.3)
  • TestRail - iPhone X (11.3)

• New issues filed in Github
  • The "Welcome to Lockbox" dialog is automatically dismissed #500
  • Welcome page layout breaks after tapping on "Learn more" link #502
  • "Learn more" link is too bottom aligned on iPhone SE link #501

• Important issues
  • Incorrect prompt to unlock when the passcode is disabled and the app auto-locks #467

July 2nd, 2018

• Exploratory testing
  • iPhone X (11.3)

• Functional testing on build 1.1 (1572)
  • TestRail - iPhone 6S (iOS 12b2)
  • TestRail - iPhone X (11.3)

• New issues filed in Github
  • None

July 24th, 2018

• Functional testing on master build (6b50a17).
  • TestRail - iPhone X (11.3)
  • TestRail - iPhone SE (10.3)
  • TestRail - iPhone 8 (11.4)

• New issues filed in Github
  • Logins are not displayed after restarting the app #571
  • The app auto locks immediately after unlocking it #573
  • "Syncing your entries" text cut off on iPhone SE (10.3) #570

August 1st, 2018

• Functional testing on v.1.1.1.1717.
  • TestRail - iPhone X (11.4)
  • TestRail - iPhone SE (10.3)

• New issues filed in Github
  • Sometimes hitting the "Disconnect" button does not disconnect the account #595

August 28th/29th, 2018

• Exploratory testing
  • iPhone SE (10.3)

• Functional testing on build 1.2 (2018)
  • TestRail - iPhone X (iOS 11.4.1)

• New issues filed in Github
  • "No matching entries" after Disconnect/Reconnect #653
  • The picture and display name of the previous sign in are displayed in the "Account" page for a newly created FxA #655
  • No longer able to sign in with authorization code #662
  • The user is taken back to the "Welcome screen" when force quitting the app while in the "confirmation" state #663

December 20th, 2018

• Exploratory testing
  • iPhone 8 (11.4)

• Functional testing on build 1.3 (2018)
  • - iPhone XR (iOS 12.1)
  • - iPhone SE (iOS 10.3)

• New issues filed in Github
  • The loading circle bar is still displayed in entries list after a lock button interrupt #837

March 1st, 2018

• Exploratory testing
  • iPhone XS (11.4)

• Feature testing:
  • iPad Change Inventory and Visual Assets
  • Design Changes for Search UX
  • search UI/component used in the app and the AutoFill experience

• Functional testing on build 1.3 (2018)
  • - iPhone XS MAX (11.4)
  • - iPhone SE (10.3)
  • - iPhone 7 (12.1)
  • - iPad Air 2 (11.4)
  • - iPad Mini 4 (12.2b3)

• New issues filed in Github
  • [2] [iPad] Lockbox crashes after tapping the sort button on iPad
  • [3] Performance issues
  • [4] "Sort entries" screen is shown on Welcome Screen after device autolocks
  • [5] No credentials are displayed in the list when using autofill
  • [6] The previously visited page will be opened after specific steps

Signoff/Exit Criteria

• No critical or major UI bugs
• No easily reproducible crashes