QA/Fennec/Photon-Mobile

From MozillaWiki
< QA‎ | Fennec
Jump to: navigation, search
Last updated: 2017/10/27

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
QA Manager Product Integrity Date Email
Software Engineer Engineering Date Email
EPM Product Management Date Email

Overview

1. TOOLBAR
2. SEARCH SUGGESTION
3. TAB TRAY
4. EMPTY SCREEN
5. SPLASH SCREEN
6. ONBOARDING
7. MISC. UI REFRESH

Purpose

  • The purpose of this wiki is to serve as a general test plan for verifying the Android UI for Photon Mobile, on Android devices.
  • Only UI changes are made in this feature. The user flows and functionality will not be affected.
  • Testing responsibilities will be shared between Ioana Chiorean and Oana Horvath.
  • Full UI test cases will be created and executed.

Scope

This wiki details the testing that will be performed by the Fennec QA project team for the Android Photon UI project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed

Ownership

Overall Status

[LANDED] Nightly 57.0a1

[LANDED] Beta: Beta 57

Testing Summary

Scope of testing

All Photon-UI related bugs will be covered and verified, on multiple Android devices, with the supported OS.

Environments

Testing will be performed on the following Android devices:

  • Phones
    • 4.x
    • 5.x
    • 6.x
    • 7.x
  • Tablet
    • 4.x
    • 5.x
    • 6.x
    • 7.x

Specifications

Acceptance Criteria

  • All Test cases have passed and all P1 bugs are fixed.

Testing details

Test Cases

Testing days

Date: 07-17-2017

  • Test plan creation
  • Started writing test cases

Date: 07-20-2017

  • Writing test cases

Date: 07-27-2017

  • Added new test cases

Date: 08-25-2017

  • Mid-nightly sign off sent: Green.

Date: 09-14-2017

Date: 09-15-2017

Date: 09-18-2017

  • Ran tests on Nightly 57 with:
  • Verified:
    • Bug 1393072 - (photon) Small tablets: in private mode, the bookmark & share icons are not visible
    • Bug 1394378 - (photon) Android 4 - Highlight of awesomescreen tabs is blue in private mode
    • Bug 1395920 - (photon) Remove vertical dividers in tabs tray page - Android 4&5
    • Bug 1366669 - (photon) update icons
    • Bug 1387349 - (photon) follow up with icons update
    • Bug 1366672 - (photon) Page loading indicator
  • Logged bug:
    • Bug 1400804 - (photon) First run tour - the Sync button is not visible, or squished, in landscape
    • Bug 1400858 - [RTL] Tab titles not displayed for some webpages


Date: 09-28-2017

Date: 10-06-2017

Date: 10-23-2017

Date: 10-24-2017

  • Verified bug:
    • Bug 1410338 - [Photon] Incorrect color for History Panel message
  • Pre-Release sign off sent: Green

Bug Work

Tracking bugs

  • Bugs to track the feature
    • Meta bug: bug 1355774 - (meta) Photon Mobile on Firefox for Android
Full Query
ID Priority Summary Status Resolution Assigned to Whiteboard
1361545 -- Add a Gradle flavor for developing Photon for Firefox for Android RESOLVED FIXED Nick Alexander :nalexander (less responsive until Jan 3, 2018)
1361667 P1 Android Photon Mobile prototyping: search suggestion RESOLVED WORKSFORME
1361670 P1 Android Photon Mobile prototyping: progress bar, tool bar RESOLVED WORKSFORME
1361672 P1 Android Photon Mobile prototyping: tab tray RESOLVED WORKSFORME
1361673 P1 Android Photon Mobile prototyping: splash screen RESOLVED WORKSFORME
1366659 -- (Photon) Follow iOS private tab interaction design VERIFIED FIXED Jing-wei Wu [:jwu]
1366660 -- (photon) Transition for Search suggestion when input keyword into URL bar VERIFIED FIXED Jing-wei Wu [:jwu]
1366661 -- (photon) Light Weight Theme with new awesome bar UI VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.1][MVP]
1366662 -- (photon) Engineering landing plan and implementation so that Photon UIs can be landed incrementally RESOLVED FIXED
1366663 P2 (photon) Lottie library integration NEW
1366664 -- (photon) (baseline) New onboarding experience to reflect the new look RESOLVED FIXED Nevin Chen [:nechen] [FNC][SPT57.2][MVP][SP=X, 8]
1366665 -- (photon) (lottie) New onboarding experience to reflect the new look NEW
1366666 -- (photon) Custom Tabs UI refresh NEW
1366668 -- (photon) Awesome bar visual refresh for tablet VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.1][MVP]
1366669 -- (photon) update icons VERIFIED FIXED Jing-wei Wu [:jwu]
1366671 -- (photon) Update UI/Content for Empty screens and content for Android VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.1][MVP]
1366672 -- (photon) Page loading indicator VERIFIED FIXED Julian Chu [:walkingice] [FNC][SPT57.2][MVP]
1366673 -- (photon) Tab icon motion for adding a new tab VERIFIED FIXED Jing-wei Wu [:jwu]
1366674 -- (photon) Indicator on splash screen when first time open VERIFIED FIXED Nevin Chen [:nechen]
1366675 -- (photon) In and out Tab tray transition RESOLVED INVALID Nevin Chen [:nechen]
1366676 -- (photon) Change URL text selection timing NEW
1366677 -- (photon) Awesome bar transition NEW
1366678 -- (photon) Frequency search result visual refresh VERIFIED FIXED Jing-wei Wu [:jwu]
1366679 -- (photon) Search Suggestions VERIFIED FIXED Jing-wei Wu [:jwu]
1366680 -- (photon) Tab tray UI refresh and consistency VERIFIED FIXED Nevin Chen [:nechen] [FNC][SPT57.1][MVP]
1366703 -- (Photon) Follow up of private tab visual refresh VERIFIED FIXED Jing-wei Wu [:jwu]
1366704 -- (photon) follow up of search suggestion on the visual refresh of private mode VERIFIED FIXED Jing-wei Wu [:jwu]
1372486 -- (photon) Support photon-only resources with mach/gradle build RESOLVED FIXED Jing-wei Wu [:jwu]
1374959 -- (photon) Support detecting Australis/Photon at runtime in Java frontend RESOLVED FIXED Jing-wei Wu [:jwu]
1375351 -- (photon) - Clean up workaround code and resources RESOLVED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.1][INT]
1375448 -- (photon) - Duplicate resource files to different flavor directory RESOLVED FIXED Julian Chu [:walkingice]
1375613 -- (photon) provide a straightforward way to add a new button (home button) NEW
1376995 -- (photon) Update status bar color while switching between normal mode and private mode VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.1][MVP]
1377294 -- (photon) - Let some toolbar views become theme-able RESOLVED FIXED Julian Chu [:walkingice]
1378032 -- (photon) Disable setTabsCounterPrivateMode in tablet VERIFIED FIXED Jing-wei Wu [:jwu]
1378175 -- Splash screen should respect channel VERIFIED FIXED Nevin Chen [:nechen]
1378625 -- (photon) Tab strip visual refresh RESOLVED FIXED Jing-wei Wu [:jwu]
1379552 -- (photon) - Title URL should be faded out in tail RESOLVED DUPLICATE Jing-wei Wu [:jwu] [FNC][SPT57.3][MVP]
1379650 -- [Meta][Feature] Photon Mobile: TOOLBAR NEW
1379652 -- [Meta][Feature] Photon Mobile: SEARCH SUGGESTION NEW
1379655 -- [Meta][Feature] Photon Mobile: TAB TRAY NEW
1379656 -- [Meta][Feature] Photon Mobile: EMPTY SCREEN NEW
1379657 -- [Meta][Feature] Photon Mobile: SPLASH SCREEN NEW
1379658 -- [Meta] Photon Mobile: Misc. UI Refresh NEW
1379660 -- [Meta] Photon Mobile: Engineering Logistic Work NEW
1382980 -- (photon) Incorrect strings in private tab empty screen RESOLVED INVALID
1386975 -- (photon) Enable Photon by default VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.1][INT]
1387349 -- (photon) follow up with icons update VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.1][INT]
1388291 -- (photon) Replacing launcher icon VERIFIED FIXED Sebastian Kaspari (:sebastian)
1388554 P2 "Recently Closed" History panel icon is cropped VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.1][INT]
1388556 P2 Extra space above "Desktop Bookmarks" folder in Bookmarks home panel VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.2][INT]
1388895 P2 Android system status bar should be dark in Reader view dark theme NEW
1389164 -- [photon] Light text in toolbar is unreadable when using dark lightweight theme on Android VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.2][BL]
1389342 -- Weird alignment of device icons in Synced History VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.2][INT]
1389343 P1 Missing/tiny icons for history, bookmark items VERIFIED FIXED Nevin Chen [:nechen] [FNC][SPT58.2][MVP]
1389822 -- (photon) Replacing launcher icon, round 2 RESOLVED WONTFIX Jan Henning [:JanH]
1389935 -- (photon) Replacing launcher icon, round 3 RESOLVED WONTFIX Jan Henning [:JanH]
1390420 -- (photon) follow-up of update UI for empty screens on private tabs tray page VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.2][MVP]
1390735 -- (photon) Support tint color for PageAction icon VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][MVP]
1390791 -- (photon) Adjust text size in tab counter dynamically VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.2][INT]
1391160 -- (photon) Status bar icons are invisible after back from fullscreen mode VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][MVP]
1391177 -- (photon) Visual refinements on mobile photon VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.2][INT]
1391551 -- (photon) Drop photon skin flavor NEW
1393072 -- (photon) Small tablets: in private mode, the bookmark & share icons are not visible VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][MVP]
1393084 -- (photon) Page text selection handlers should be updated VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][MVP]
1393425 P3 (photon) - Make scrollbar invisible for Welcome Screen NEW
1393426 -- (photon) Scrolling in welcome screen is jumpy NEW
1393754 -- (photon) Status bar color does not change on all devices RESOLVED INVALID [FNC][SPT57.3][MVP]
1394257 -- Addon's menu items missing from phone but are available on tablet RESOLVED INVALID
1394310 -- (photon) Change primary color from orange to blue VERIFIED FIXED Jing-wei Wu [:jwu]
1394361 -- (photon) tab title is cropped and not align to close button VERIFIED FIXED Nevin Chen [:nechen] [FNC][SPT57.3][INT]
1394378 -- (photon) Android 4 - Highlight of awesomescreen tabs is blue in private mode VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1394404 P2 (photon) Custom tabs - the loading indicator doesn't fully load and it has different colors VERIFIED FIXED [FNC][SPT57.3][INT]
1395917 -- (photon) The reading list folder icon is blue on Android 6 devices VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1395920 -- (photon) Remove vertical dividers in tabs tray page - Android 4&5 VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1396324 P2 Various intermittent crashes in framework code, concentrating on testSessionOOMSave RESOLVED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1396741 -- [Meta] Photon Mobile: ONBOARDING NEW
1398532 P2 Investigate why Activity Stream home panel (possibly in combination with Photon toolbar changes) makes testSessionOOMSave so crashy NEW [MobileAS]
1398655 -- (photon) Remove unused color resources in colors.xml RESOLVED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1398657 -- (photon) Remove unused or redundant drawables RESOLVED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1399378 P1 (photon) update the logo for 57 beta / release VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.1][INT]
1399384 -- (photon) follow up of visual refinements RESOLVED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1400159 P1 Notification area icon uses old shape RESOLVED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1400164 P1 (photon) New tabs open with the close button invisible, on tablets VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT57.3][INT]
1400508 P3 Statusbar text not visible in firefox 57 android on some Android ROMs ASSIGNED Jing-wei Wu [:jwu]
1400592 P2 Firefox for Android should have a darker theme RESOLVED DUPLICATE
1400804 P1 (photon) First run tour - the Sync button is not visible, or squished, in landscape VERIFIED FIXED Nevin Chen [:nechen] [FNC][SPT58.1][INT]
1400858 P1 [RTL] Tab titles not displayed for some webpages RESOLVED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.1][INT]
1401457 -- (photon) logo in search activity isn't updated yet VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.1][INT]
1401925 -- (photon) Nightly icons on tabs and search widget need to be changed RESOLVED DUPLICATE
1402594 P1 Progress bar animation on loading scrolled website page is visible in wrong place VERIFIED FIXED Randall Barker [:rbarker]
1402598 P3 Themes/Appearances/Personas look faded and bleached NEW
1402601 P3 Height of Address Bar is to high by 16px NEW
1403132 P1 Orange color when tapping "Get started" button VERIFIED FIXED Nevin Chen [:nechen] [FNC][SPT58.2][MVP]
1403596 -- Buttons misaligned between URL bar and tabs tray VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.2][MVP]
1403653 P2 White status bar may clash visually with lightweight themes VERIFIED FIXED Jan Henning [:JanH]
1403728 P1 Additional margin at top and bottom of tabs tray in non-compact portrait mode VERIFIED FIXED Nevin Chen [:nechen] [FNC][SPT58.2][MVP]
1403980 -- Splash screen displayed in new tab opened from the search widget RESOLVED FIXED Nevin Chen [:nechen] [FNC][SPT58.2][MVP]
1405287 -- Change Splash Screen image VERIFIED FIXED Nevin Chen [:nechen] [FNC][SPT58.2][INT]
1405710 -- Find In Page does not have Photon theme VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.2][INT]
1405954 -- (photon) Color and dimension adjustment for URL bar and status bar VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.2][INT]
1406329 P2 (photon) Tablet - After removing a theme, the tab highlight remains while switching tabs NEW
1406338 -- Show favicon in tabs tray page VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.2][INT]
1406828 -- (photon) (RTL) Back/forward buttons aren't Photon style on 4.2 tablet when RTL is enabled VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.2][INT]
1406874 -- (photon) Show color indicator for the selected tab strip on tablet VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.2][INT]
1407923 -- (photon) Adjust light weight theme layout/color for Photon style VERIFIED FIXED Jing-wei Wu [:jwu] [FNC][SPT58.2][INT]
1410338 P1 [Photon] Incorrect color for History Panel message VERIFIED FIXED Nevin Chen [:nechen] [FNC][SPT58.3][INT]
1411896 -- The 3 dots menu in tabs tray is dark gray on Samsung Galaxy Note 4 NEW
1412518 -- Low contrast/badly readable text in time picker with Photon colour scheme NEW
1414309 P1 Choosing a light theme results in hard to read text in the Private Browsing Awesomebar NEW
1414503 P3 When using a coloured theme, it's obvious that Awesomebar icons pop instantly in and out of view, which is ugly. Perhaps they should animate NEW

111 Total; 27 Open (24.32%); 29 Resolved (26.13%); 55 Verified (49.55%);


Signoff

Criteria

  • All [Test cases] have passed and all major bugs are fixed.
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

Beta testing

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) Done
Testing Infrastructure setup Done Nightly builds
Test Plan Creation Done
Test Cases Creation Done
Automation Coverage N/A
Performance Testing N/A
All Defects Logged In progress
Critical/Blockers Fixed and Verified Done
Metrics/Telemetry
Basic/Core functionality Nightly testing Done
QA mid-Nightly Signoff Email sent Green (8/25/2017)
QA Nightly - Full Testing Done
QA pre-Beta Signoff Email sent Status: Yellow (9/19/2017)
QA Beta - Full Testing Done
QA pre-Release Signoff Email sent Status: Green (10/24/2017)