QA/Fennec/Support manifest display-standalone display mode

From MozillaWiki
< QA‎ | Fennec
Jump to: navigation, search
Last updated: 2017/06/21

Overview

Purpose

The purpose of this wiki is to serve as a general test plan for verifying that users of Android mobile devices can install and use a progressive web app, as a standalone app.

Ownership

Developer contact:

QA: Oana Horvath (:oanah)

Overall Status

[LANDED] Nightly: In progress

[NOT LANDED] Aurora: n/a

[NOT LANDED] Beta: n/a

Testing Summary

Scope of testing

The testing will focus on verifying on supported devices that progressive web apps (pwa) can be installed as homescreen shortcuts and are opened as standalone apps, with an individual entry in the task manager.

Environments

Testing will be performed on the following Android devices:

  • Phones
    • 4.x
    • 5.x
    • 6.x
    • 7.x
  • Tablet
    • 5.x
    • 6.x
    • 7.x

Specifications

User stories

  • As a user, I want to be able to download a web app so I can launch it from the homescreen, to load instantly, regardless of the network state, without the need of an app store.

Acceptance Criteria

  • All [Test cases] have passed and all bugs were fixed.

Testing details

Test Cases

Testing days

Date: 03-20-2017

  • Started research and studying the documentation.
  • Started creating the Test Plan
  • Started writing Test Cases

Date: 03-21-2017

  • Documentation research and writing test cases

Date: 03-24-2017

  • Documentation research and writing test cases

Date: 03-21-2017

  • Completed the Test Plan
  • Writing Test Cases

Date: 03-30-2017

  • Writing Test Cases
  • Tested on Nightly 55 with LG G4 (Android 5.1)
  • Logged bug: bug 1352099 - [PWA] Web Apps don't always have a title and icon in the task switcher entry

Date: 03-31-2017

  • Verified bugs:
    • Bug 1347154 - Cannot install twitter as a PWA
    • Bug 1347616 - Handle invalid manifests
  • Logged bug:
    • Bug 1352378 - [PWA] Closing the Nightly task crashes if it has web app tabs opened
  • Investigated:
    • Bug 1347585 - Swiping to kill a custom tab leaves the tab open in Fennec

Date: 04-03-2017

  • Writing test cases
  • Tested on Nightly 55 with Motorola Nexus 6 (Android 7.0)
  • Logged bug: Bug 1352991 - Pages loaded in Nightly show in the web app

Date: 04-05-2017

  • Verified bugs:
    • bug 1352726 - Flipboard is not given a title
    • bug 1337341 - Handle the user navigating out the scope of the webapp
    • bug 1352099 - [PWA] Web Apps don't always have a title and icon in the task switcher entry
  • Logged bug:
    • bug 1353714 - Crash @ java.lang.NoClassDefFoundError: android.app.ActivityManager$TaskDescription at org.mozilla.gecko.webapps.WebAppActivity.loadManifest(WebAppActivity.java)

Date: 04-10-2017

  • Added test cases and verified on Nightly 55 with LG G4 (Android 5.1).
  • Verified bug:
    • Bug 1353714 - Crash @ java.lang.NoClassDefFoundError: android.app.ActivityManager$TaskDescription at org.mozilla.gecko.webapps.WebAppActivity.loadManifest(WebAppActivity.java)
  • Logged bugs:
    • Bug 1355094 - WebApps - Remove context menu items that are not needed
    • Bug 1355098 - WebApps - 'Add to homescreen' from the bookmarks/history panels doesn't install the shortcut as a standalone app


Date: 04-11-2017

  • Exploratory Testing on latest Nightly 55.0a1
  • Logged bugs:
    • Bug 1355513 - Allow access to microphone permissions not displayed in web apps

Date: 04-12-2017

  • Exploratory testing on Nightly 55
  • Verified bugs:
    • Bug 1352433 - First run tour in browser and some settings reset after opening a web app
  • Investigated:
    • Bug 1336355 - Launched manifest bookmarks should have their own taskswitcher entry
    • Bug 1355513 - Allow access to microphone and camera permissions not displayed in web apps
  • Added a comment on: Bug 1352099 - [PWA] Web Apps don't always have a title and icon in the task switcher entry
  • Logged bugs:
    • Bug 1355824 - Browser opens a new tab for web apps already running - tabs tray shows wrong info
    • Bug 1355847 - Web apps - Share selected text doesn't work on Android 6+
    • Bug 1355852 - Web Apps - Android 4 & 5 Text selection toolbar not displayed in web apps

Date: 04-13-2017

  • Exploratory testing on latest Nightly 55.0a1
  • Research for relevant add-ons for web apps

Date: 04-18-2017

Date: 04-19-2017

  • Exploratory testing on latest Nightly 55.0a1
  • Logged bug:
    • Bug 1357779 - Web Apps - Some webapps: the external url remains displayed

Date: 04-24-2017

  • Exploratory testing on latest Nightly 55.0a1

Date: 04-26-2017

Functional and exploratory testing on latest Nightly 55.0a1, with Prestigio Grace X5 (Android 4.4.2)

Date: 04-27-2017

Exploratory testing on latest Nightly 55.0a1 Verified bugs:

  • Bug 1355676 - Crash in java.lang.NullPointerException: Attempt to invoke virtual method android.graphics.Bitmap org.mozilla.gecko.icons.decoders.LoadFaviconResult.getBestBitmap(int) on a null object reference at org.mozilla.gecko.BrowserApp.handleMessage(Browser...

Date: 05-04-2017

Verified bugs:

  • Bug 1355824 - Browser opens a new tab for web apps already running - tabs tray shows wrong info
  • Bug 1352991 - Pages loaded in Nightly show in the web app
  • Bug 1353857 - Fix handling of entering/leaving Web App
  • Bug 1355852 - Web Apps - Android 4 & 5 Text selection toolbar not displayed in web apps
  • Bug 1357443 - Web Apps - Can't download files from web apps using Nightly

Date: 05-08-2017

Full testing on Nightly, with:

  • Motorola Motorola Nexus 6 (Android 7.0)
  • Prestigio Grace X5 (Android 4.4.2)

Logged bugs:

  • Bug 1363049 - Web Apps - Web apps are reopened when closing a tab opened from them
  • Bug 1363052 - Web Apps - Re-launching a web app with an external page opened, resets is to the homepage

Date: 05-10-2017

Full testing on Nightly, with:

  • Lenovo A536 (Android 4.4.2)
  • Huawei Honor 5X (Android 5.1.1)

Verified bugs:

  • Bug 1234558 - Use icon/title from app manifest for "Add to home screen"
  • Bug 1360658 - Snack bars not showing up in WebAppActivity

Date: 05-11-2017

Full testing on Nightly, with:

  • Lenovo Yoga Tablet 2 (Android 4.4.2)

Investigated:

  • Bug 1360455 - Web Apps - snapdrop.net cannot be installed as pwa on Huawei/Xiaomi devices

Logged:

  • Bug 1364044 - Web Apps - (Tablets) Tabs appear in the browser when there are multiple webapps opened
  • Bug 1364052 - Web Apps - Some webapps show the internal page url

Date: 05-12-2017

Full testing on Nightly, with:

  • Lenovo Yoga Tablet 2 (Android 4.4.2)

Date: 05-15-2017

Full testing on Nightly, with

  • HTC Nexus 9 (Android 7.1.1)
  • Huawei Honor 5X (Android 5.1.1)

Date: 05-17-2017

Full testing on Nightly, with LG G4 (Android 6)

  • Verified:
    • Bug 1357438 - Web apps - Login autocomplete isn't working
  • Updated:
    • Bug 1364044 - Web Apps - Tabs appear in the browser when there are multiple webapps opened

Date: 05-18-2017

Smoke testing on Nightly, with LG G4 (Android 6.0) Verified Bug 1363359 - standalone manifest install replaces Nightly in Touchwiz Sent the mid-Nightly feature sign-off email.

Date: 05-19-2017

Added PWA feature test cases in TestRail: https://testrail.stage.mozaws.net/index.php?/suites/view/254&group_by=cases:section_id&group_id=3341&group_order=asc

Date: 05-25-2017

Verified: Bug 1363049 - Web Apps - Web apps are reopened when closing a tab opened from them

Date: 05-26-2017

  • Full testing on Asus ZenPad 8.0 Z380KL (Android 6.0.1)
  • Logged bug: Bug 1368024 - Web Apps - Some web apps have an URL instead of title

Bug Work

Tracking bugs

  • Bugs to track the feature
    • Meta bug: bug 1285858 - Support manifest display: standalone display mode
Full Query
ID Priority Summary Status Resolution Assigned to Depends on
1234558 -- Use icon/title from app manifest for "Add to home screen" VERIFIED FIXED Dale Harvey (:daleharvey) 1337345, 826400, 1355676, 1357091, 1360213
1321320 -- Ability to determine whether a url is within an installed app scope VERIFIED FIXED Dale Harvey (:daleharvey)
1325021 P1 Crash in java.lang.IllegalArgumentException: Doorhanger:Add was not registered at org.mozilla.gecko.EventDispatcher.unregisterListener(EventDispatcher.java) VERIFIED FIXED Dylan Roeh (:droeh) (he/him)
1336355 -- Launched manifest bookmarks should have their own taskswitcher entry VERIFIED FIXED Dale Harvey (:daleharvey)
1337341 -- Handle the user navigating out the scope of the webapp VERIFIED FIXED Dale Harvey (:daleharvey) 1348686, 1351739, 1353857, 1360658
1346542 P1 Back button navigation and web content copy/paste broken in restored GeckoApp instance VERIFIED FIXED Eugen Sawin [:esawin] 1351169
1347154 -- Cannot install twitter as a PWA VERIFIED FIXED Dale Harvey (:daleharvey)
1347585 P2 Swiping to kill a custom tab leaves the tab open in Fennec RESOLVED INVALID Dylan Roeh (:droeh) (he/him) 1352997
1347605 -- Web Apps opened from homescreen should be singletons VERIFIED FIXED Dale Harvey (:daleharvey) 1352433
1347611 P2 Web Apps (and custom tabs) flash previously opened content VERIFIED FIXED 1208655
1347616 -- Handle invalid manifests VERIFIED FIXED Dale Harvey (:daleharvey)
1351169 P1 Crash in java.lang.NullPointerException: Null native pointer at org.mozilla.gecko.mozglue.GeckoLoader.nativeRun(Native Method) VERIFIED FIXED Eugen Sawin [:esawin]
1351554 P1 PWA: Collect information on the number of Firefox launches using PWA standalone mode RESOLVED DUPLICATE Nevin Chen(Not active on Bugzilla)
1351555 P1 PWA: Collect information on the number of progressive web apps that’s added to homescreen RESOLVED FIXED Nevin Chen(Not active on Bugzilla)
1351556 P1 PWA: Collect information on the number of users using PWA standalone mode RESOLVED FIXED Nevin Chen(Not active on Bugzilla)
1351739 P1 System for switching between different kinds of GeckoApp RESOLVED FIXED Jan Henning [:JanH] 1325021, 1348686
1351808 P1 Exclude custom tabs from session store RESOLVED FIXED Jan Henning [:JanH] 1352997, 1379374
1351834 -- Set correct tab type when launching WepApp activity from external intent RESOLVED FIXED Dale Harvey (:daleharvey)
1352004 P1 Doorhangers missing background VERIFIED FIXED Nevin Chen(Not active on Bugzilla)
1352099 P1 [PWA] Web Apps don't always have a title and icon in the task switcher entry VERIFIED FIXED Dale Harvey (:daleharvey)
1352394 P1 Crash in java.lang.NullPointerException: Attempt to invoke virtual method ''java.lang.String org.mozilla.gecko.Tab.getURL()'' on a null object reference at org.mozilla.gecko.webapps.WebAppActivity.onNewIntent(WebAppActivity.java) RESOLVED WORKSFORME 1351739
1352433 P1 First run tour in browser and some settings reset after opening a web app VERIFIED FIXED Jan Henning [:JanH]
1352726 -- Flipboard is not given a title VERIFIED FIXED Dale Harvey (:daleharvey)
1352991 P1 Pages loaded in Nightly show in the web app RESOLVED WORKSFORME 1352997
1353714 P1 Crash @ java.lang.NoClassDefFoundError: android.app.ActivityManager$TaskDescription at org.mozilla.gecko.webapps.WebAppActivity.loadManifest(WebAppActivity.java) VERIFIED FIXED Dale Harvey (:daleharvey)
1353857 P2 Fix handling of entering/leaving Web App VERIFIED FIXED Jan Henning [:JanH] 1352997
1355094 -- WebApps - Remove context menu items that are not needed RESOLVED DUPLICATE
1355513 P1 Allow access to microphone and camera permissions not displayed in web apps RESOLVED DUPLICATE Nevin Chen(Not active on Bugzilla)
1355676 P1 Crash in java.lang.NullPointerException: Attempt to invoke virtual method ''android.graphics.Bitmap org.mozilla.gecko.icons.decoders.LoadFaviconResult.getBestBitmap(int)'' on a null object reference at org.mozilla.gecko.BrowserApp.handleMessage(Browser... VERIFIED FIXED Dale Harvey (:daleharvey)
1355824 P1 Browser opens a new tab for web apps already running - tabs tray shows wrong info RESOLVED WORKSFORME 1351808, 1352997
1355847 P1 Web apps - Share selected text doesn't work on Android 6+ RESOLVED DUPLICATE Nevin Chen(Not active on Bugzilla)
1355852 P1 Web Apps - Android 4 & 5 Text selection toolbar not displayed in web apps VERIFIED FIXED Nevin Chen(Not active on Bugzilla)
1357438 P1 Web apps - Login autocomplete isn't working VERIFIED FIXED Dale Harvey (:daleharvey)
1357443 P1 Web Apps - Can't download files from web apps using Nightly RESOLVED WORKSFORME
1360455 P2 Web Apps - snapdrop.net cannot be installed as pwa on Huawei/Xiaomi devices RESOLVED INVALID
1363049 P1 Web Apps - Web apps are reopened when closing a tab opened from them VERIFIED FIXED Jan Henning [:JanH]
1363052 P1 Web Apps - Re-launching a web app with an internal/external page opened, resets is to the homepage RESOLVED WORKSFORME
1363359 P3 manifest with standalone=true has full app context menu in Touchwiz NEW
1364044 P1 Web Apps - Tabs appear in the browser when there are multiple webapps opened VERIFIED FIXED
1364052 P1 Web Apps - Some webapps show the internal page url RESOLVED WORKSFORME
1364471 -- Implement PWA scope attribute for GeckoView-based standalone mode RESOLVED FIXED
1365868 P1 Support minimal context menu functionality in GeckoView-based custom tabs VERIFIED FIXED Nevin Chen(Not active on Bugzilla) 1381983
1367077 P1 Ensure form fill (passwords, etc) working in GeckoView-based custom tabs and web apps RESOLVED FIXED (inactive) Jim Chen [:jchen] [:darchons] 1403566
1368024 P1 Web Apps - Fall back to 'name' when 'short_name' is missing VERIFIED FIXED Nevin Chen(Not active on Bugzilla)
1377320 -- Hook up prompts for progressive web apps RESOLVED FIXED Dylan Roeh (:droeh) (he/him)
1379405 -- Can not dynamically modify the theme color REOPENED
1386154 -- manifest with standalone=true still loads in full browser RESOLVED WORKSFORME
1388747 -- Web Apps - show the simplified address bar when the user navigates outside the web app's origin RESOLVED INVALID
1388750 P5 Web apps - Apps suddenly stopped responding, running out of memory REOPENED Dylan Roeh (:droeh) (he/him)
1388755 P1 Web Apps - Text selection toolbar not displayed RESOLVED DUPLICATE
1388757 -- Web Apps - Device back button sends the webapp in the background RESOLVED DUPLICATE
1388760 P1 Web Apps - Permission request dialogs appear in the browser, not inside the web app RESOLVED WORKSFORME
1388763 P1 Web Apps - Doorhangers aren't working RESOLVED WORKSFORME
1394766 -- Hyperlink navigations with target=_blank remain in standalone browsing context RESOLVED WORKSFORME
1398065 P5 Ability to move a custom tab/web app into the full UI wholesale and without state loss NEW
1398067 P2 Fully support context menu and "tab" related functionality in GeckoView-based custom tabs/web apps NEW 1365868, 1381983
1501397 P3 Add an RDM widget to change display-mode settings NEW

57 Total; 6 Open (10.53%); 27 Resolved (47.37%); 24 Verified (42.11%);


Signoff

Criteria

  • All [Test cases] have passed and all major bugs are fixed.
  • All P1s are fixed.

Results

Nightly testing

  • n/a

Aurora testing

  • n/a

Beta testing

  • n/a