QA/Fennec/Support of Progressive Apps in Fennec

From MozillaWiki
< QA‎ | Fennec
Jump to: navigation, search
Last updated: 2017/12/07

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
QA Manager Product Integrity Date Email
Software Engineer Engineering Date Email
EPM Product Management Date Email

Overview

Purpose

  • The purpose of this wiki is to serve as a general test plan for verifying that users of Android mobile devices can install and use a progressive web app, as a standalone app.
  • Full functional and UI test cases will be created and executed.

Scope

This wiki details the testing that will be performed by the Fennec QA project team for the PWA project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed
  • Work progress and status

User Stories

* As a user I want to be informed when the current web page is part of a web app I can install so I can benefit from a richer app-like experience
* As a user I want to launch a web app from an icon on my homescreen/launcher/dock/start menu and use it separately from the browser
* As a user I want to switch between web apps using my operating system's task manager
* As a user I want to seamlessly link between web app windows and browser tabs
* As a user I want to persist this site/app to my device and use it even while offline
* As an app developer I want to get my app discovered via web searches because it currently gets buried in app stores across Firefox OS & Firefox
* As an app developer I want to deep link into my app from web search results so that I can drive users to my app via its content
* As an app developer I want to create repeat engagement with my app by adding it to a user's homescreen
* As an app developer I want to create repeat engagement with my app via push notifications

Ownership

  • Feature owner: Wesly Huang, Nicole Yee
  • QA: email or Slack: Oana Horvath

Overall Status

[LANDED] Nightly: Nightly 58 (2017-09-22)

[LANDED] Beta: Beta 58 (2017-11-13)

Testing Summary

Scope of testing

All PWA related bugs will be covered and verified, on multiple Android devices, with the supported OS.

Environments

Testing will be performed on the following Android devices:

  • Phones
    • 4.x
    • 5.x
    • 6.x
    • 7.x
  • Tablet
    • 4.x
    • 5.x
    • 6.x
    • 7.x

Specifications

Acceptance Criteria

  • All [Test cases] have passed and all P1 bugs are fixed.

Testing details

Test Cases

Testing days

Date: 08-07-2017

  • Test plan creation
  • Exploratory testing on Nightly 57

Date: 08-09-2017

  • Ran full tests on Motorola Nexus 6 (Android 7.0)
  • Logged bugs:
    • Bug 1388659 - Permission request for audio and camera on Aliexpress.com
    • Bug 1388750 - Web apps - Apps suddenly stopped responding, running out of memory
    • Bug 1388747 - Web Apps - show the simplified address bar when the user navigates outside the web app's origin
    • Bug 1388760 - Web Apps - Permission request dialogs appear in the browser, not inside the web app
    • Bug 1388757 - Web Apps - Device back button sends the webapp in the background
    • Bug 1388755 - Web Apps - Text selection toolbar not displayed
    • Bug 1388763 - Web Apps - Doorhangers aren't working

Date: 09-27-2017

  • Cleaned up old bugs:
    • Bug 1321320 - Ability to determine whether a url is within an installed app scope
    • Bug 1325021 - Crash in java.lang.IllegalArgumentException: Doorhanger:Add was not registered at org.mozilla.gecko.EventDispatcher.unregisterListener(EventDispatcher.java)
    • Bug 1351169 - Crash in java.lang.NullPointerException: Null native pointer at org.mozilla.gecko.mozglue.GeckoLoader.nativeRun(Native Method)
    • Bug 1347611 - Web Apps (and custom tabs) flash previously opened content
    • Bug 1364044 - Web Apps - Tabs appear in the browser when there are multiple webapps opened
  • Verified:
    • Bug 1363052 - Web Apps - Re-launching a web app with an internal/external page opened, resets is to the homepage
    • Bug 1367077 - Ensure form fill (passwords, etc) working in GeckoView-based custom tabs and web apps
  • Logged bug:
    • Bug 1403566 - Web Apps - Login auto-complete isn't working, with multiple logins saved

Date: 10-05-2017

  • Logged bugs:
    • Bug 1406000 - Web Apps - cannot download PDFs (or other files) from pwa
    • Bug 1406024 - Web Apps - Date-time/ color pickers don't open
    • Bug 1406034 - Web Apps - The aliexpress back button opens Javascript void page

Date: 10-06-2017

Date: 10-25-2017

  • Updated test cases
  • Verified bugs:
    • Bug 1409403 - Update badge icon for PWA
    • Bug 1393672 - PWA Badge and Onboarding - Main workflow
    • Bug 1409191 - The "Add to home screen" prompt doesn't appear to use manifest values
    • Bug 1395841 - implement change for separating “add to home screen” & “add page shortcut”
    • Bug 1409303 - Move PWA setting to Setting - General
  • Updated bugs:
    • Bug 1400605 - Crash in java.lang.NullPointerException: at org.mozilla.gecko.webapps.WebAppActivity.updateTaskAndStatusBar(WebAppActivity.java)
  • Logged bugs:
    • Bug 1411614 - [PWA] The Add to home screen badge disappearing when the app in is background - Don't keep activities ON
    • Bug 1411607 - Should hide Experimental features section when there are no experiments

Date: 10-26-2017

  • Ran full suite on Oneplus Two (Android 6.0.1): https://testrail.stage.mozaws.net/index.php?/runs/view/6830
  • Verified bugs:
    • Bug 1411529 - Context menu functionality in GeckoView based components not working.
    • Bug 1369817 - Implement PWA orientation lock
    • Bug 1406000 - Web Apps - cannot download PDFs (or other files) from pwa
    • Bug 1406024 - GeckoView - Date-time/ color pickers don't open in Custom Tab/PWA if Fennec is already running

Date: 11-16-2017

Ran full tests on Beta 58.0b3 - Huawei Nexus 6P (Android 8.0)

Date: 11-17-2017

Ran full tests on Beta 58.0b3 - Sony Xperia Z5 Premium (Android 6.0.1)

Date: 12-07-2017

Ran full tests on Beta 58.0b9 - Prestigio Grace X5 (Android 4.4.2)

Verified:

  • Bug 1421149 - [PWA] Icons in the "Add to Home screen" overlay should not have outline & dropshadow
  • Bug 1419245 - A2HS badge is shown on site with broken HTTPS

Bug Work

Tracking bugs

  • Bugs to track the feature
    • Meta bug: bug 1212648 - [Meta] Support of Progressive Apps in Fennec
Full Query
ID Priority Summary Status Resolution Whiteboard
826400 -- Use apple-touch-icon for homescreen shortcuts RESOLVED FIXED
968427 P5 Updating Firefox will change the icons for homescreen bookmarks and webapps RESOLVED WONTFIX
1126479 -- Support manifest display: fullscreen display mode RESOLVED FIXED
1156054 -- Implement Push API on Android RESOLVED DUPLICATE
1162150 P3 Experiment with "pull-to-refresh" in fennec NEW [pwa-front-end][Chrome-parity]
1211226 -- Site icon resolution on Android home screen is lower than expected RESOLVED DUPLICATE
1212363 -- Add Telemetry to |Boomark Added| Dialog and its actions RESOLVED FIXED
1232706 -- Promote "Add to home screen" if user visits website often RESOLVED FIXED
1234558 -- Use icon/title from app manifest for "Add to home screen" VERIFIED FIXED
1237045 P3 Allow user to edit home screen shortcut name NEW [pwa-front-end]
1240560 -- When adding website to homes screen, no notification of success/failure is presented VERIFIED FIXED [lang=java][good next bug]
1245497 -- Update FENNEC_GECKOAPP_STARTUP_ACTION to measure homescreen shortcut launches RESOLVED FIXED
1264221 P2 Correctly scale/modify home screen shortcut icons NEW
1285858 -- Support manifest display: standalone display mode NEW [ptototype][fennec 54]
1310986 P3 Expose "Work Offline" as a toggle in the devtools NEW
1328160 -- Prefer short_name from web app manifest for icon titles added to home screen. RESOLVED DUPLICATE [webcompat]
1363052 P1 Web Apps - Re-launching a web app with an internal/external page opened, resets is to the homepage RESOLVED WORKSFORME [pwa-front-end]
1368024 P1 Web Apps - Fall back to 'name' when 'short_name' is missing VERIFIED FIXED [FNC][SPT58.2][MVP][pwa-front-end]
1369815 P3 Implement 'minimal-ui' and 'standalone' matching for display-mode media queries RESOLVED FIXED
1369817 -- Implement PWA orientation lock VERIFIED FIXED
1372926 P1 [Android O] Add to Home screen doesn't work on any version of Firefox on Oreo RESOLVED FIXED
1376263 P1 Installation of many Google PWAs fails UNCONFIRMED
1378421 P3 Prompt user to install PWA RESOLVED DUPLICATE [pwa-front-end]
1379115 P1 Show a badge (in the position of a page action) when a website is installable as a PWA RESOLVED DUPLICATE [FNC][SPT58.3][MVP][pwa-front-end]
1388747 -- Web Apps - show the simplified address bar when the user navigates outside the web app's origin RESOLVED INVALID
1388750 P1 Web apps - Apps suddenly stopped responding, running out of memory REOPENED
1388755 P1 Web Apps - Text selection toolbar not displayed RESOLVED DUPLICATE [pwa-front-end]
1388757 -- Web Apps - Device back button sends the webapp in the background RESOLVED DUPLICATE
1388760 P1 Web Apps - Permission request dialogs appear in the browser, not inside the web app RESOLVED WORKSFORME [pwa-front-end]
1388763 P1 Web Apps - Doorhangers aren't working RESOLVED WORKSFORME [pwa-front-end]
1389051 P2 Improve length of URL bar and use of the space near it NEW [pwa-front-end]
1389236 -- Open external links in PWAs in a Custom Tab VERIFIED FIXED
1391271 P2 Add splash screen for standalone PWA NEW [pwa-front-end]
1393188 -- Back button doesn't go back in PWA VERIFIED FIXED
1393672 P1 PWA Badge and Onboarding - Main workflow VERIFIED FIXED [pwa-front-end][FNC][SPT58.2][INT]
1395570 P2 CustomTab: media notification not displayed NEW
1395839 P1 Disable PWA in private browsing (while keeping page shortcut) RESOLVED DUPLICATE [FNC][SPT58.3][MVP][pwa-front-end]
1395841 P1 implement change for separating “add to home screen” & “add page shortcut” VERIFIED FIXED [FNC][SPT58.3][MVP][pwa-front-end]
1395846 P1 PWA onboarding RESOLVED DUPLICATE [FNC][SPT58.2][MVP][pwa-front-end]
1400334 P2 Crash in java.lang.NullPointerException: Attempt to invoke virtual method ''int java.lang.Integer.intValue()'' on a null object reference at org.mozilla.gecko.webapps.WebAppActivity.onLoadUri(WebAppActivity.java) RESOLVED FIXED
1400605 P1 Crash in java.lang.NullPointerException: at org.mozilla.gecko.webapps.WebAppActivity.updateTaskAndStatusBar(WebAppActivity.java) VERIFIED FIXED
1401996 -- Turn on Progressive Web Apps by default RESOLVED DUPLICATE
1403566 P2 Web Apps - Login auto-complete isn't working, with multiple logins saved RESOLVED FIXED [pwa-front-end][FNC][SPT59.2][INT]
1404311 P3 PWAs and Custom Tabs can't be debugged through WebIDE RESOLVED FIXED
1405081 P2 [geckoview] Add high-level fetch API NEW
1406000 P2 Web Apps - cannot download PDFs (or other files) from pwa RESOLVED WORKSFORME [pwa-front-end]
1406024 P2 GeckoView - Date-time/ color pickers don't open in Custom Tab/PWA if Fennec is already running VERIFIED FIXED [pwa-front-end]
1406034 P3 Web Apps - The aliexpress back button opens Javascript void page NEW [pwa-front-end]
1407961 P1 PWA Badge and Onboarding - Show home after creating PWA shortcut when API >26 VERIFIED FIXED [FNC][SPT58.3][INT][pwa-front-end]
1409191 P1 The "Add to home screen" prompt doesn't appear to use manifest values VERIFIED FIXED [FNC][SPT58.3][MVP][pwa-front-end]
1410235 P1 Only open a custom tab for out-of-scope URLs if Gecko is going to show it RESOLVED FIXED
1410236 P2 Ensure prompts inform Gecko of their result when dismissed RESOLVED FIXED
1410534 -- Fix orientation handling RESOLVED FIXED
1411349 P2 how should push notifications work for standalone web apps? NEW [pwa-front-end]
1411614 P2 [PWA] The Add to home screen badge disappearing when the app in is background - Don't keep activities ON NEW [pwa-front-end]
1412301 P2 [PWA] PWA doesn't always open in standalone mode VERIFIED FIXED
1412678 P2 Progressive Web Apps break fullscreen mode, so do Custom Tabs RESOLVED FIXED
1413384 P3 Progressive web app loading indicator NEW [pwa-front-end]
1413895 P3 [PWA] Don't restore tabs ON - tabs are restored when a PWA is left open NEW
1414219 P3 "Add Page Shorcut" can be performed multiple times on some devices NEW
1417049 P1 Can't install PWAs on Pixel C RESOLVED FIXED
1419000 P1 [PWA] - Keyboard not always triggered when there's a web app in the background RESOLVED WORKSFORME
1419245 P1 A2HS badge is shown on site with broken HTTPS REOPENED [pwa-front-end][FNC][SPT59.2][MVP]
1420148 -- First letter of URL in address bar is hidden RESOLVED INVALID
1421149 P1 [PWA] Icons in the "Add to Home screen" overlay should not have outline & dropshadow RESOLVED FIXED [pwa-front-end][FNC][SPT59.2][MVP]
1423587 -- PWA with expired security certificate should open in browser NEW

66 Total; 19 Open (28.79%); 34 Resolved (51.52%); 13 Verified (19.7%);


Signoff

Criteria

  • All [Test cases] have passed and all major bugs are fixed.
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

Beta testing

  • n/a
  • Full Test suite
  • Link to TestRail Tests Runs

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) Done
Testing Infrastructure setup Done
Test Plan Creation Done
Test Cases Creation Done
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
Basic/Core functionality Nightly testing Done
QA mid-Nightly Signoff Done Status: Yellow
QA Nightly - Full Testing Done
QA pre-Beta Signoff Done Status: Green
QA Beta - Full Testing
QA pre-Release Signoff Email to be sent