QA/Fennec/Photon-Mobile

Last updated: 2017/10/27

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name	Title	Department	Approval Date	Method
	QA Manager	Product Integrity	Date	Email
	Software Engineer	Engineering	Date	Email
	EPM	Product Management	Date	Email

Overview

1. TOOLBAR
2. SEARCH SUGGESTION
3. TAB TRAY
4. EMPTY SCREEN
5. SPLASH SCREEN
6. ONBOARDING
7. MISC. UI REFRESH

Purpose

• The purpose of this wiki is to serve as a general test plan for verifying the Android UI for Photon Mobile, on Android devices.
• Only UI changes are made in this feature. The user flows and functionality will not be affected.
• Testing responsibilities will be shared between Ioana Chiorean and Oana Horvath.
• Full UI test cases will be created and executed.

Scope

This wiki details the testing that will be performed by the Fennec QA project team for the Android Photon UI project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

• What will be tested
• How testing will be performed

Ownership

• Developer contact:
• QA:
  • email or ping Ioana Chiorean
  • email or ping Oana Horvath

Overall Status

[LANDED] Nightly 57.0a1

[LANDED] Beta: Beta 57

Testing Summary

Scope of testing

All Photon-UI related bugs will be covered and verified, on multiple Android devices, with the supported OS.

Environments

Testing will be performed on the following Android devices:

• Phones
  • 4.x
  • 5.x
  • 6.x
  • 7.x
• Tablet
  • 4.x
  • 5.x
  • 6.x
  • 7.x

Specifications

• bug 1355774 - (meta) Photon Mobile on Firefox for Android
• Phone UX specs: https://drive.google.com/file/d/0B-9PIePlQZRlRTJNZTl0U2ZXM0U/view
• Tablet UX specs: https://drive.google.com/file/d/0B-9PIePlQZRlSFdzc1BXVlA5bDA/view
• https://docs.google.com/presentation/d/1gLfgorSUiGJn6FDbs0x-s4-oqJctZOyWl7kVQmceR_g/edit#slide=id.p

Acceptance Criteria

• All Test cases have passed and all P1 bugs are fixed.

Testing details

Test Cases

• TestRail test cases
• Google docs - test cases

Testing days

Date: 07-17-2017

• Test plan creation
• Started writing test cases

Date: 07-20-2017

• Writing test cases

Date: 07-27-2017

• Added new test cases

Date: 08-25-2017

• Mid-nightly sign off sent: Green.

Date: 09-14-2017

• Added test cases in TestRail:
  • https://testrail.stage.mozaws.net/index.php?/suites/view/254&group_by=cases:section_id&group_order=asc&group_id=5731
• Ran tests on Nightly 57, with Lenovo Yoga Tablet 2 (Android 4.4.2):
  • https://testrail.stage.mozaws.net/index.php?/runs/view/6184

Date: 09-15-2017

• Ran tests on Nightly 57 with:
  • Asus ZenPad 8.0 Z380KL (Android 6.0.1): https://testrail.stage.mozaws.net/index.php?/runs/view/6214
  • LG G4 (Android 5.1): https://testrail.stage.mozaws.net/index.php?/runs/view/6223
  • HTC 10 (Android 7.0.1): https://testrail.stage.mozaws.net/index.php?/runs/view/6224
• Verified:
  • Bug 1394310 - (photon) Change primary color from orange to blue
  • Bug 1394404 - (photon) Custom tabs - the loading indicator doesn't fully load and it has different colors
  • Bug 1395917 - (photon) The reading list folder icon is blue on Android 6 devices
• Logged bug:
  • Bug 1400164 - (photon) New tabs open with the close button invisible, on tablets
• Sent preliminary pre-release sign-off

Date: 09-18-2017

• Ran tests on Nightly 57 with:
  • Huawei MediaPad M2 (Android 5.1.1): https://testrail.stage.mozaws.net/index.php?/runs/view/6228
  • LG G4 (Android 6.0): https://testrail.stage.mozaws.net/index.php?/runs/view/6267
• Verified:
  • Bug 1393072 - (photon) Small tablets: in private mode, the bookmark & share icons are not visible
  • Bug 1394378 - (photon) Android 4 - Highlight of awesomescreen tabs is blue in private mode
  • Bug 1395920 - (photon) Remove vertical dividers in tabs tray page - Android 4&5
  • Bug 1366669 - (photon) update icons
  • Bug 1387349 - (photon) follow up with icons update
  • Bug 1366672 - (photon) Page loading indicator
• Logged bug:
  • Bug 1400804 - (photon) First run tour - the Sync button is not visible, or squished, in landscape
  • Bug 1400858 - [RTL] Tab titles not displayed for some webpages

Date: 09-28-2017

• Ran tests on Beta 57 with:
  • Huawei Honor 8 (Android 6.0): https://testrail.stage.mozaws.net/index.php?/runs/view/6461
  • Huawei Honor 5X (Android 5.1.1): https://testrail.stage.mozaws.net/index.php?/runs/view/6463
• Logged bugs:
  • Bug 1403860 - Crash in system@framework@boot-framework.art@0x46909a
  • Bug 1403980 - Splash screen displayed in new tab opened from the search widget

Date: 10-06-2017

• Ran tests on Beta 57 with:
  • Galaxy Tab S3 (Android 7.0): https://testrail.stage.mozaws.net/index.php?/runs/view/6556
• Logged bug:
  • Bug 1406329 - (photon) Tablet - After removing a theme, the tab highlight remains while switching tabs

Date: 10-23-2017

• Ran full suite on Beta 57: https://testrail.stage.mozaws.net/index.php?/runs/view/6765
• Verified bugs:
  • Bug 1406828 - (photon) (RTL) Back/forward buttons aren't Photon style on 4.2 tablet when RTL is enabled
  • Bug 1403653 - White status bar may clash visually with lightweight themes

Date: 10-24-2017

• Verified bug:
  • Bug 1410338 - [Photon] Incorrect color for History Panel message
• Pre-Release sign off sent: Green

Bug Work

Tracking bugs

• Bugs to track the feature
  • Meta bug: bug 1355774 - (meta) Photon Mobile on Firefox for Android

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1

Signoff

Criteria

• All [Test cases] have passed and all major bugs are fixed.
• All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

• Completed
• Full Test suite
• TestRail Test Runs

Beta testing

• Completed
• Full Test suite
• Test Runs

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)	Done
Testing Infrastructure setup	Done	Nightly builds
Test Plan Creation	Done
Test Cases Creation	Done
Automation Coverage	N/A
Performance Testing	N/A
All Defects Logged	In progress
Critical/Blockers Fixed and Verified	Done
Metrics/Telemetry
Basic/Core functionality Nightly testing	Done
QA mid-Nightly Signoff	Email sent	Green (8/25/2017)
QA Nightly - Full Testing	Done
QA pre-Beta Signoff	Email sent	Status: Yellow (9/19/2017)
QA Beta - Full Testing	Done
QA pre-Release Signoff	Email sent	Status: Green (10/24/2017)