QA/Fennec/Support manifest display-standalone display mode

Last updated: 2017/04/12

Overview

Purpose

The purpose of this wiki is to serve as a general test plan for verifying that users of Android mobile devices can install and use a progressive web app, as a standalone app.

Ownership

Developer contact:

QA: Oana Horvath (:oanah)

Overall Status

[LANDED] Nightly: In progress

[NOT LANDED] Aurora: n/a

[NOT LANDED] Beta: n/a

Testing Summary

Scope of testing

The testing will focus on verifying on supported devices that progressive web apps (pwa) can be installed as homescreen shortcuts and are opened as standalone apps, with an individual entry in the task manager.

Environments

Testing will be performed on the following Android devices:

• Phones
  • 4.x
  • 5.x
  • 6.x
  • 7.x
• Tablet
  • 5.x
  • 6.x
  • 7.x

Specifications

• bug 1285858 - Support manifest display: standalone display mode
• More details about PWAs: https://developers.google.com/web/progressive-web-apps/
• A list of PWAs for testing: https://pwa.rocks/

User stories

• As a user, I want to be able to download a web app so I can launch it from the homescreen, to load instantly, regardless of the network state, without the need of an app store.

Acceptance Criteria

• All [Test cases] have passed and all bugs were fixed.

Testing details

Test Cases

• TestRail test cases
• Google sheet - test cases

Testing days

Date: 03-20-2017

• Started research and studying the documentation.
• Started creating the Test Plan
• Started writing Test Cases

Date: 03-21-2017

• Documentation research and writing test cases

Date: 03-24-2017

• Documentation research and writing test cases

Date: 03-21-2017

• Completed the Test Plan
• Writing Test Cases

Date: 03-30-2017

• Writing Test Cases
• Tested on Nightly 55 with LG G4 (Android 5.1)
• Logged bug: bug 1352099 - [PWA] Web Apps don't always have a title and icon in the task switcher entry

Date: 03-31-2017

• Verified bugs:
  • Bug 1347154 - Cannot install twitter as a PWA
  • Bug 1347616 - Handle invalid manifests
• Logged bug:
  • Bug 1352378 - [PWA] Closing the Nightly task crashes if it has web app tabs opened
• Investigated:
  • Bug 1347585 - Swiping to kill a custom tab leaves the tab open in Fennec

Date: 04-03-2017

• Writing test cases
• Tested on Nightly 55 with Motorola Nexus 6 (Android 7.0)
• Logged bug: Bug 1352991 - Pages loaded in Nightly show in the web app

Date: 04-05-2017

• Verified bugs:
  • bug 1352726 - Flipboard is not given a title
  • bug 1337341 - Handle the user navigating out the scope of the webapp
  • bug 1352099 - [PWA] Web Apps don't always have a title and icon in the task switcher entry
• Logged bug:
  • bug 1353714 - Crash @ java.lang.NoClassDefFoundError: android.app.ActivityManager$TaskDescription at org.mozilla.gecko.webapps.WebAppActivity.loadManifest(WebAppActivity.java)

Date: 04-10-2017

• Added test cases and verified on Nightly 55 with LG G4 (Android 5.1).
• Verified bug:
  • Bug 1353714 - Crash @ java.lang.NoClassDefFoundError: android.app.ActivityManager$TaskDescription at org.mozilla.gecko.webapps.WebAppActivity.loadManifest(WebAppActivity.java)
• Logged bugs:
  • Bug 1355094 - WebApps - Remove context menu items that are not needed
  • Bug 1355098 - WebApps - 'Add to homescreen' from the bookmarks/history panels doesn't install the shortcut as a standalone app

Date: 04-11-2017

• Exploratory Testing on latest Nightly 55.0a1
• Logged bugs:
  • Bug 1355513 - Allow access to microphone permissions not displayed in web apps

Date: 04-12-2017

• Exploratory testing on Nightly 55
• Verified bugs:
  • Bug 1352433 - First run tour in browser and some settings reset after opening a web app
• Investigated:
  • Bug 1336355 - Launched manifest bookmarks should have their own taskswitcher entry
  • Bug 1355513 - Allow access to microphone and camera permissions not displayed in web apps
• Added a comment on: Bug 1352099 - [PWA] Web Apps don't always have a title and icon in the task switcher entry
• Logged bugs:
  • Bug 1355824 - Browser opens a new tab for web apps already running - tabs tray shows wrong info
  • Bug 1355847 - Web apps - Share selected text doesn't work on Android 6+
  • Bug 1355852 - Web Apps - Android 4 & 5 Text selection toolbar not displayed in web apps

Bug Work

Tracking bugs

• bugs to track the feature

Full Query

ID	Summary	Priority	Status
1162150	Experiment with "pull-to-refresh" in fennec	P3	RESOLVED
1234558	Use icon/title from app manifest for "Add to home screen"	--	VERIFIED
1300570	[meta] Separate normal / custom / web app tabs	--	RESOLVED
1321320	Ability to determine whether a url is within an installed app scope	--	VERIFIED
1325021	Crash in java.lang.IllegalArgumentException: Doorhanger:Add was not registered at org.mozilla.gecko.EventDispatcher.unregisterListener(EventDispatcher.java)	P1	VERIFIED
1336355	Launched manifest bookmarks should have their own taskswitcher entry	--	VERIFIED
1337341	Handle the user navigating out the scope of the webapp	--	VERIFIED
1346542	Back button navigation and web content copy/paste broken in restored GeckoApp instance	P1	VERIFIED
1347154	Cannot install twitter as a PWA	--	VERIFIED
1347165	Change context menu "open in new tab" to "open in Firefox tab" when not in browser	P2	RESOLVED
1347585	Swiping to kill a custom tab leaves the tab open in Fennec	P2	RESOLVED
1347605	Web Apps opened from homescreen should be singletons	--	VERIFIED
1347611	Web Apps (and custom tabs) flash previously opened content	P2	VERIFIED
1347616	Handle invalid manifests	--	VERIFIED
1348686	Implement ability to change a tab's tab type	P2	RESOLVED
1351169	Crash in java.lang.NullPointerException: Null native pointer at org.mozilla.gecko.mozglue.GeckoLoader.nativeRun(Native Method)	P1	VERIFIED
1351554	PWA: Collect information on the number of Firefox launches using PWA standalone mode	P1	RESOLVED
1351555	PWA: Collect information on the number of progressive web apps that’s added to homescreen	P1	RESOLVED
1351556	PWA: Collect information on the number of users using PWA standalone mode	P1	RESOLVED
1351739	System for switching between different kinds of GeckoApp	P1	RESOLVED
1351808	Exclude custom tabs from session store	P1	RESOLVED
1351834	Set correct tab type when launching WepApp activity from external intent	--	RESOLVED
1352004	Doorhangers missing background	P1	VERIFIED
1352099	[PWA] Web Apps don't always have a title and icon in the task switcher entry	P1	VERIFIED
1352394	Crash in java.lang.NullPointerException: Attempt to invoke virtual method ''java.lang.String org.mozilla.gecko.Tab.getURL()'' on a null object reference at org.mozilla.gecko.webapps.WebAppActivity.onNewIntent(WebAppActivity.java)	P1	RESOLVED
1352433	First run tour in browser and some settings reset after opening a web app	P1	VERIFIED
1352726	Flipboard is not given a title	--	VERIFIED
1352991	Pages loaded in Nightly show in the web app	P1	RESOLVED
1353714	Crash @ java.lang.NoClassDefFoundError: android.app.ActivityManager$TaskDescription at org.mozilla.gecko.webapps.WebAppActivity.loadManifest(WebAppActivity.java)	P1	VERIFIED
1353857	Fix handling of entering/leaving Web App	P2	VERIFIED
1353868	Move Java-side Web App manifest parsing into a separate class	P2	RESOLVED
1355094	WebApps - Remove context menu items that are not needed	--	RESOLVED
1355098	WebApps - 'Add to homescreen' from the bookmarks/history panels doesn't install the shortcut as a standalone app	P2	RESOLVED
1355513	Allow access to microphone and camera permissions not displayed in web apps	P1	RESOLVED
1355676	Crash in java.lang.NullPointerException: Attempt to invoke virtual method ''android.graphics.Bitmap org.mozilla.gecko.icons.decoders.LoadFaviconResult.getBestBitmap(int)'' on a null object reference at org.mozilla.gecko.BrowserApp.handleMessage(Browser...	P1	VERIFIED
1355824	Browser opens a new tab for web apps already running - tabs tray shows wrong info	P1	RESOLVED

36 Total; 0 Open (0%); 18 Resolved (50%); 18 Verified (50%);

Signoff

Criteria

• All [Test cases] have passed and all major bugs are fixed.

Results

Nightly testing

• n/a

Aurora testing

• n/a

Beta testing

• n/a