QA/Fennec/Photon-Mobile

From MozillaWiki
< QA‎ | Fennec
Jump to: navigation, search
Last updated: 2017/10/27

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
QA Manager Product Integrity Date Email
Software Engineer Engineering Date Email
EPM Product Management Date Email

Overview

1. TOOLBAR
2. SEARCH SUGGESTION
3. TAB TRAY
4. EMPTY SCREEN
5. SPLASH SCREEN
6. ONBOARDING
7. MISC. UI REFRESH

Purpose

  • The purpose of this wiki is to serve as a general test plan for verifying the Android UI for Photon Mobile, on Android devices.
  • Only UI changes are made in this feature. The user flows and functionality will not be affected.
  • Testing responsibilities will be shared between Ioana Chiorean and Oana Horvath.
  • Full UI test cases will be created and executed.

Scope

This wiki details the testing that will be performed by the Fennec QA project team for the Android Photon UI project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed

Ownership

Overall Status

[LANDED] Nightly 57.0a1

[LANDED] Beta: Beta 57

Testing Summary

Scope of testing

All Photon-UI related bugs will be covered and verified, on multiple Android devices, with the supported OS.

Environments

Testing will be performed on the following Android devices:

  • Phones
    • 4.x
    • 5.x
    • 6.x
    • 7.x
  • Tablet
    • 4.x
    • 5.x
    • 6.x
    • 7.x

Specifications

Acceptance Criteria

  • All Test cases have passed and all P1 bugs are fixed.

Testing details

Test Cases

Testing days

Date: 07-17-2017

  • Test plan creation
  • Started writing test cases

Date: 07-20-2017

  • Writing test cases

Date: 07-27-2017

  • Added new test cases

Date: 08-25-2017

  • Mid-nightly sign off sent: Green.

Date: 09-14-2017

Date: 09-15-2017

Date: 09-18-2017

  • Ran tests on Nightly 57 with:
  • Verified:
    • Bug 1393072 - (photon) Small tablets: in private mode, the bookmark & share icons are not visible
    • Bug 1394378 - (photon) Android 4 - Highlight of awesomescreen tabs is blue in private mode
    • Bug 1395920 - (photon) Remove vertical dividers in tabs tray page - Android 4&5
    • Bug 1366669 - (photon) update icons
    • Bug 1387349 - (photon) follow up with icons update
    • Bug 1366672 - (photon) Page loading indicator
  • Logged bug:
    • Bug 1400804 - (photon) First run tour - the Sync button is not visible, or squished, in landscape
    • Bug 1400858 - [RTL] Tab titles not displayed for some webpages


Date: 09-28-2017

Date: 10-06-2017

Date: 10-23-2017

Date: 10-24-2017

  • Verified bug:
    • Bug 1410338 - [Photon] Incorrect color for History Panel message
  • Pre-Release sign off sent: Green

Bug Work

Tracking bugs

  • Bugs to track the feature
    • Meta bug: bug 1355774 - (meta) Photon Mobile on Firefox for Android
Full Query
ID Priority Summary Status Resolution Assigned to Whiteboard
1361545 -- Add a Gradle flavor for developing Photon for Firefox for Android RESOLVED FIXED Nick Alexander :nalexander [he/him]
1361667 P1 Android Photon Mobile prototyping: search suggestion RESOLVED WORKSFORME
1361670 P1 Android Photon Mobile prototyping: progress bar, tool bar RESOLVED WORKSFORME
1361672 P1 Android Photon Mobile prototyping: tab tray RESOLVED WORKSFORME
1361673 P1 Android Photon Mobile prototyping: splash screen RESOLVED WORKSFORME
1366659 -- (Photon) Follow iOS private tab interaction design VERIFIED FIXED Jing-wei Wu [:jwu]
1366660 -- (photon) Transition for Search suggestion when input keyword into URL bar VERIFIED FIXED Jing-wei Wu [:jwu]
1366661 -- (photon) Light Weight Theme with new awesome bar UI VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.1][MVP]
1366662 -- (photon) Engineering landing plan and implementation so that Photon UIs can be landed incrementally RESOLVED FIXED
1366663 P3 (photon) Lottie library integration RESOLVED INCOMPLETE
1366664 -- (photon) (baseline) New onboarding experience to reflect the new look RESOLVED FIXED Nevin Chen(Not active on Bugzilla) [FNC][SPT57.2][MVP][SP=X, 8]
1366665 P5 (photon) (lottie) New onboarding experience to reflect the new look RESOLVED INCOMPLETE
1366666 -- (photon) Custom Tabs UI refresh RESOLVED INCOMPLETE
1366668 -- (photon) Awesome bar visual refresh for tablet VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.1][MVP]
1366669 -- (photon) update icons VERIFIED FIXED Jing-wei Wu [:jwu]
1366671 -- (photon) Update UI/Content for Empty screens and content for Android VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.1][MVP]
1366672 -- (photon) Page loading indicator VERIFIED FIXED Julian Chu [:walkingice] [FNC][SPT57.2][MVP]
1366673 -- (photon) Tab icon motion for adding a new tab VERIFIED FIXED Jing-wei Wu [:jwu]
1366674 -- (photon) Indicator on splash screen when first time open VERIFIED FIXED Nevin Chen(Not active on Bugzilla)
1366675 -- (photon) In and out Tab tray transition RESOLVED INVALID Nevin Chen(Not active on Bugzilla)
1366676 -- (photon) Change URL text selection timing RESOLVED INCOMPLETE
1366677 P5 (photon) Awesome bar transition RESOLVED INCOMPLETE
1366678 -- (photon) Frequency search result visual refresh VERIFIED FIXED Jing-wei Wu [:jwu]
1366679 -- (photon) Search Suggestions VERIFIED FIXED Jing-wei Wu [:jwu]
1366680 -- (photon) Tab tray UI refresh and consistency VERIFIED FIXED Nevin Chen(Not active on Bugzilla) [FNC][SPT57.1][MVP]
1366703 -- (Photon) Follow up of private tab visual refresh VERIFIED FIXED Jing-wei Wu [:jwu]
1366704 -- (photon) follow up of search suggestion on the visual refresh of private mode VERIFIED FIXED Jing-wei Wu [:jwu]
1372486 -- (photon) Support photon-only resources with mach/gradle build RESOLVED FIXED Jing-wei Wu [:jwu]
1374959 -- (photon) Support detecting Australis/Photon at runtime in Java frontend RESOLVED FIXED Jing-wei Wu [:jwu]
1375351 -- (photon) - Clean up workaround code and resources RESOLVED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.1][INT]
1375448 -- (photon) - Duplicate resource files to different flavor directory RESOLVED FIXED Julian Chu [:walkingice]
1375613 P3 (photon) provide a straightforward way to add a new button (home button) RESOLVED INCOMPLETE
1376995 -- (photon) Update status bar color while switching between normal mode and private mode VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.1][MVP]
1377294 -- (photon) - Let some toolbar views become theme-able RESOLVED FIXED Julian Chu [:walkingice]
1378032 -- (photon) Disable setTabsCounterPrivateMode in tablet VERIFIED FIXED Jing-wei Wu [:jwu]
1378175 -- Splash screen should respect channel VERIFIED FIXED Nevin Chen(Not active on Bugzilla)
1378625 -- (photon) Tab strip visual refresh VERIFIED FIXED Jing-wei Wu [:jwu]
1379552 -- (photon) - Title URL should be faded out in tail RESOLVED DUPLICATE Jing-wei Wu [:jwu] [FNC][SPT57.3][MVP]
1379650 P5 [Meta][Feature] Photon Mobile: TOOLBAR RESOLVED INCOMPLETE
1379652 -- [Meta][Feature] Photon Mobile: SEARCH SUGGESTION RESOLVED INCOMPLETE
1379655 P5 [Meta][Feature] Photon Mobile: TAB TRAY RESOLVED INCOMPLETE
1379656 -- [Meta][Feature] Photon Mobile: EMPTY SCREEN RESOLVED INCOMPLETE
1379657 P5 [Meta][Feature] Photon Mobile: SPLASH SCREEN RESOLVED INCOMPLETE
1379658 -- [Meta] Photon Mobile: Misc. UI Refresh RESOLVED INCOMPLETE
1379660 P5 [Meta] Photon Mobile: Engineering Logistic Work RESOLVED INCOMPLETE
1382980 -- (photon) Incorrect strings in private tab empty screen RESOLVED INVALID
1386975 -- (photon) Enable Photon by default VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.1][INT]
1387349 -- (photon) follow up with icons update VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.1][INT]
1388291 -- (photon) Replacing launcher icon VERIFIED FIXED Sebastian Kaspari (:sebastian; :pocmo)
1388554 P2 "Recently Closed" History panel icon is cropped VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.1][INT]
1388556 P2 Extra space above "Desktop Bookmarks" folder in Bookmarks home panel VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.2][INT]
1388895 P3 Android system status bar should be dark in Reader view dark theme RESOLVED WONTFIX
1389164 -- [photon] Light text in toolbar is unreadable when using dark lightweight theme on Android VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.2][BL]
1389342 -- Weird alignment of device icons in Synced History VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.2][INT]
1389343 P1 Missing/tiny icons for history, bookmark items VERIFIED FIXED Nevin Chen(Not active on Bugzilla) [FNC][SPT58.2][MVP]
1389822 -- (photon) Replacing launcher icon, round 2 RESOLVED WONTFIX Jan Henning [:JanH]
1389935 -- (photon) Replacing launcher icon, round 3 RESOLVED WONTFIX Jan Henning [:JanH]
1390420 -- (photon) follow-up of update UI for empty screens on private tabs tray page VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.2][MVP]
1390735 -- (photon) Support tint color for PageAction icon VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][MVP]
1390791 -- (photon) Adjust text size in tab counter dynamically VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.2][INT]
1391160 -- (photon) Status bar icons are invisible after back from fullscreen mode VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][MVP]
1391177 -- (photon) Visual refinements on mobile photon VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.2][INT]
1391551 P5 (photon) Drop photon skin flavor RESOLVED FIXED Nick Alexander :nalexander [he/him]
1393072 -- (photon) Small tablets: in private mode, the bookmark & share icons are not visible VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][MVP]
1393084 -- (photon) Page text selection handlers should be updated VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][MVP]
1393425 P3 (photon) - Make scrollbar invisible for Welcome Screen RESOLVED INCOMPLETE
1393426 -- (photon) Scrolling in welcome screen is jumpy RESOLVED INCOMPLETE
1393754 -- (photon) Status bar color does not change on all devices RESOLVED INVALID [FNC][SPT57.3][MVP]
1394257 -- Addon's menu items missing from phone but are available on tablet RESOLVED INVALID
1394310 -- (photon) Change primary color from orange to blue VERIFIED FIXED Jing-wei Wu [:jwu]
1394361 -- (photon) tab title is cropped and not align to close button VERIFIED FIXED Nevin Chen(Not active on Bugzilla) [FNC][SPT57.3][INT]
1394378 -- (photon) Android 4 - Highlight of awesomescreen tabs is blue in private mode VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1394404 P2 (photon) Custom tabs - the loading indicator doesn't fully load and it has different colors VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1395917 -- (photon) The reading list folder icon is blue on Android 6 devices VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1395920 -- (photon) Remove vertical dividers in tabs tray page - Android 4&5 VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1396324 P2 Various intermittent crashes in framework code, concentrating on testSessionOOMSave RESOLVED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1396741 P3 [Meta] Photon Mobile: ONBOARDING RESOLVED INCOMPLETE
1398532 P5 Investigate why Activity Stream home panel (possibly in combination with Photon toolbar changes) makes testSessionOOMSave so crashy RESOLVED WONTFIX [MobileAS]
1398655 -- (photon) Remove unused color resources in colors.xml RESOLVED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1398657 -- (photon) Remove unused or redundant drawables RESOLVED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1399378 P1 (photon) update the logo for 57 beta / release VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.1][INT]
1399384 -- (photon) follow up of visual refinements RESOLVED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1400159 P1 Notification area icon uses old shape RESOLVED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1400164 P1 (photon) New tabs open with the close button invisible, on tablets VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1400508 P3 Statusbar text not visible in firefox 57 android on some Android ROMs RESOLVED INCOMPLETE Jing-wei Wu [:jwu]
1400592 P2 Firefox for Android should have a darker theme RESOLVED DUPLICATE
1400804 P1 (photon) First run tour - the Sync button is not visible, or squished, in landscape VERIFIED FIXED Nevin Chen(Not active on Bugzilla) [FNC][SPT58.1][INT]
1400858 P1 [RTL] Tab titles not displayed for some webpages RESOLVED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.1][INT]
1401457 -- (photon) logo in search activity isn't updated yet VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.1][INT]
1401925 -- (photon) Nightly icons on tabs and search widget need to be changed RESOLVED DUPLICATE
1402594 P1 Progress bar animation on loading scrolled website page is visible in wrong place VERIFIED FIXED Randall Barker [:rbarker]
1402598 P3 Themes/Appearances/Personas look faded and bleached VERIFIED WONTFIX
1402601 P5 Height of Address Bar is to high by 16px VERIFIED WONTFIX
1403132 P1 Orange color when tapping "Get started" button VERIFIED FIXED Nevin Chen(Not active on Bugzilla) [FNC][SPT58.2][MVP]
1403596 -- Buttons misaligned between URL bar and tabs tray VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.2][MVP]
1403653 P2 White status bar may clash visually with lightweight themes VERIFIED FIXED Jan Henning [:JanH]
1403728 P1 Additional margin at top and bottom of tabs tray in non-compact portrait mode VERIFIED FIXED Nevin Chen(Not active on Bugzilla) [FNC][SPT58.2][MVP]
1403980 -- Splash screen displayed in new tab opened from the search widget RESOLVED FIXED Nevin Chen(Not active on Bugzilla) [FNC][SPT58.2][MVP]
1405287 -- Change Splash Screen image VERIFIED FIXED Nevin Chen(Not active on Bugzilla) [FNC][SPT58.2][INT]
1405710 -- Find In Page does not have Photon theme VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.2][INT]
1405954 -- (photon) Color and dimension adjustment for URL bar and status bar VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.2][INT]
1406329 P2 (photon) Tablet - After removing a theme, the tab highlight remains while switching tabs RESOLVED INCOMPLETE
1406338 -- Show favicon in tabs tray page VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.2][INT]
1406828 -- (photon) (RTL) Back/forward buttons aren't Photon style on 4.2 tablet when RTL is enabled VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.2][INT]
1406874 -- (photon) Show color indicator for the selected tab strip on tablet VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.2][INT]
1407923 -- (photon) Adjust light weight theme layout/color for Photon style VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.2][INT]
1410338 P1 [Photon] Incorrect color for History Panel message VERIFIED FIXED Nevin Chen(Not active on Bugzilla) [FNC][SPT58.3][INT]
1411896 P5 The 3 dots menu in tabs tray is dark gray on Samsung Galaxy Note 4 RESOLVED INCOMPLETE
1412518 -- Low contrast/badly readable text in time and date pickers with Photon colour scheme VERIFIED FIXED Petru-Mugurel Lingurar [:petru]
1414309 P1 Choosing a light theme results in hard to read text in the Private Browsing Awesomebar VERIFIED FIXED Jan Henning [:JanH]
1414503 P3 When using a coloured theme, it's obvious that Awesomebar icons pop instantly in and out of view, which is ugly. Perhaps they should animate RESOLVED INCOMPLETE
1430546 P2 Orange sync icon on about:home RESOLVED WORKSFORME Brad Arant

112 Total; 0 Open (0%); 52 Resolved (46.43%); 60 Verified (53.57%);


Signoff

Criteria

  • All [Test cases] have passed and all major bugs are fixed.
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

Beta testing

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) Done
Testing Infrastructure setup Done Nightly builds
Test Plan Creation Done
Test Cases Creation Done
Automation Coverage N/A
Performance Testing N/A
All Defects Logged In progress
Critical/Blockers Fixed and Verified Done
Metrics/Telemetry
Basic/Core functionality Nightly testing Done
QA mid-Nightly Signoff Email sent Green (8/25/2017)
QA Nightly - Full Testing Done
QA pre-Beta Signoff Email sent Status: Yellow (9/19/2017)
QA Beta - Full Testing Done
QA pre-Release Signoff Email sent Status: Green (10/24/2017)