QA/Fennec/Support of Progressive Apps in Fennec

From MozillaWiki
< QA‎ | Fennec
Jump to: navigation, search
Last updated: 2017/10/06

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
QA Manager Product Integrity Date Email
Software Engineer Engineering Date Email
EPM Product Management Date Email

Overview

Purpose

  • The purpose of this wiki is to serve as a general test plan for verifying that users of Android mobile devices can install and use a progressive web app, as a standalone app.
  • Full functional and UI test cases will be created and executed.

Scope

This wiki details the testing that will be performed by the Fennec QA project team for the PWA project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed
  • Work progress and status

User Stories

* As a user I want to be informed when the current web page is part of a web app I can install so I can benefit from a richer app-like experience
* As a user I want to launch a web app from an icon on my homescreen/launcher/dock/start menu and use it separately from the browser
* As a user I want to switch between web apps using my operating system's task manager
* As a user I want to seamlessly link between web app windows and browser tabs
* As a user I want to persist this site/app to my device and use it even while offline
* As an app developer I want to get my app discovered via web searches because it currently gets buried in app stores across Firefox OS & Firefox
* As an app developer I want to deep link into my app from web search results so that I can drive users to my app via its content
* As an app developer I want to create repeat engagement with my app by adding it to a user's homescreen
* As an app developer I want to create repeat engagement with my app via push notifications

Ownership

  • Feature owner: Wesly Huang, Nicole Yee
  • QA: email or Slack: Oana Horvath

Overall Status

[LANDED] Nightly: Nightly 58 (2017-09-22)

[NOT LANDED] Beta: n/a

Testing Summary

Scope of testing

All PWA related bugs will be covered and verified, on multiple Android devices, with the supported OS.

Environments

Testing will be performed on the following Android devices:

  • Phones
    • 4.x
    • 5.x
    • 6.x
    • 7.x
  • Tablet
    • 4.x
    • 5.x
    • 6.x
    • 7.x

Specifications

Acceptance Criteria

  • All [Test cases] have passed and all P1 bugs are fixed.

Testing details

Test Cases

Testing days

Date: 08-07-2017

  • Test plan creation
  • Exploratory testing on Nightly 57

Date: 08-09-2017

  • Ran full tests on Motorola Nexus 6 (Android 7.0)
  • Logged bugs:
    • Bug 1388659 - Permission request for audio and camera on Aliexpress.com
    • Bug 1388750 - Web apps - Apps suddenly stopped responding, running out of memory
    • Bug 1388747 - Web Apps - show the simplified address bar when the user navigates outside the web app's origin
    • Bug 1388760 - Web Apps - Permission request dialogs appear in the browser, not inside the web app
    • Bug 1388757 - Web Apps - Device back button sends the webapp in the background
    • Bug 1388755 - Web Apps - Text selection toolbar not displayed
    • Bug 1388763 - Web Apps - Doorhangers aren't working

Date: 09-27-2017

  • Cleaned up old bugs:
    • Bug 1321320 - Ability to determine whether a url is within an installed app scope
    • Bug 1325021 - Crash in java.lang.IllegalArgumentException: Doorhanger:Add was not registered at org.mozilla.gecko.EventDispatcher.unregisterListener(EventDispatcher.java)
    • Bug 1351169 - Crash in java.lang.NullPointerException: Null native pointer at org.mozilla.gecko.mozglue.GeckoLoader.nativeRun(Native Method)
    • Bug 1347611 - Web Apps (and custom tabs) flash previously opened content
    • Bug 1364044 - Web Apps - Tabs appear in the browser when there are multiple webapps opened
  • Verified:
    • Bug 1363052 - Web Apps - Re-launching a web app with an internal/external page opened, resets is to the homepage
    • Bug 1367077 - Ensure form fill (passwords, etc) working in GeckoView-based custom tabs and web apps
  • Logged bug:
    • Bug 1403566 - Web Apps - Login auto-complete isn't working, with multiple logins saved

Date: 10-05-2017

  • Logged bugs:
    • Bug 1406000 - Web Apps - cannot download PDFs (or other files) from pwa
    • Bug 1406024 - Web Apps - Date-time/ color pickers don't open
    • Bug 1406034 - Web Apps - The aliexpress back button opens Javascript void page

Date: 10-06-2017

Bug Work

Tracking bugs

  • Bugs to track the feature
    • Meta bug: bug 1212648 - [Meta] Support of Progressive Apps in Fennec
ID Priority Summary Status Resolution Assigned to Depends on
826400 -- Use apple-touch-icon for homescreen shortcuts RESOLVED FIXED Andrzej Hunt :ahunt 1238656, 1265708
968427 P5 Updating Firefox will change the icons for homescreen bookmarks and webapps RESOLVED WONTFIX
1126479 -- Support manifest display: fullscreen display mode RESOLVED FIXED
1156054 -- Implement Push API on Android RESOLVED DUPLICATE 1151180, 1038811, 1214338
1162150 P2 Experiment with "pull-to-refresh" in fennec NEW
1211226 -- Site icon resolution on Android home screen is lower than expected RESOLVED DUPLICATE
1212363 -- Add Telemetry to |Boomark Added| Dialog and its actions RESOLVED FIXED Mark Capella [:capella]
1232706 -- Promote "Add to home screen" if user visits website often RESOLVED FIXED Sebastian Kaspari (:sebastian) 1269908, 1270034, 1280071, 968427, 1264243, 1265351, 1266383, 1276910, 1277180
1234558 -- Use icon/title from app manifest for "Add to home screen" VERIFIED FIXED Dale Harvey (:daleharvey) 1337345, 826400, 1355676, 1357091, 1360213
1237045 P2 Allow user to edit home screen shortcut name NEW
1240560 -- When adding website to homes screen, no notification of success/failure is presented VERIFIED FIXED Nevin Chen [:nechen]
1245497 -- Update FENNEC_GECKOAPP_STARTUP_ACTION to measure homescreen shortcut launches RESOLVED FIXED :Margaret Leibovic
1264221 P2 Correctly scale/modify home screen shortcut icons NEW 1307621
1285858 -- Support manifest display: standalone display mode NEW 1363359, 1388750, 1398065, 1398067, 1234558, 1321320, 1325021, 1336355, 1337341, 1346542, 1347154, 1347585, 1347605, 1347611, 1347616, 1351169, 1351554, 1351555, 1351556, 1351739, 1351808, 1351834, 1352004, 1352099, 1352394, 1352433, 1352726, 1352991, 1353714, 1353857, 1355094, 1355513, 1355676, 1355824, 1355847, 1355852, 1357438, 1357443, 1360455, 1363049, 1363052, 1364044, 1364052, 1364471, 1365868, 1367077, 1368024, 1377320, 1379405, 1386154, 1388747, 1388755, 1388757, 1388760, 1388763, 1394766
1310986 P3 Expose "Work Offline" as a toggle in the devtools NEW 1156659, 1204115
1321320 -- Ability to determine whether a url is within an installed app scope VERIFIED FIXED Dale Harvey (:daleharvey)
1325021 P1 Crash in java.lang.IllegalArgumentException: Doorhanger:Add was not registered at org.mozilla.gecko.EventDispatcher.unregisterListener(EventDispatcher.java) VERIFIED FIXED Dylan Roeh (:droeh)
1328160 -- Prefer short_name from web app manifest for icon titles added to home screen. RESOLVED DUPLICATE
1336355 -- Launched manifest bookmarks should have their own taskswitcher entry VERIFIED FIXED Dale Harvey (:daleharvey)
1337341 -- Handle the user navigating out the scope of the webapp VERIFIED FIXED Dale Harvey (:daleharvey) 1348686, 1351739, 1353857, 1360658
1346542 P1 Back button navigation and web content copy/paste broken in restored GeckoApp instance VERIFIED FIXED Eugen Sawin [:esawin] 1351169
1347154 -- Cannot install twitter as a PWA VERIFIED FIXED Dale Harvey (:daleharvey)
1347585 P2 Swiping to kill a custom tab leaves the tab open in Fennec RESOLVED INVALID Dylan Roeh (:droeh) 1352997
1347605 -- Web Apps opened from homescreen should be singletons VERIFIED FIXED Dale Harvey (:daleharvey) 1352433
1347611 P2 Web Apps (and custom tabs) flash previously opened content VERIFIED FIXED 1208655
1347616 -- Handle invalid manifests VERIFIED FIXED Dale Harvey (:daleharvey)
1351169 P1 Crash in java.lang.NullPointerException: Null native pointer at org.mozilla.gecko.mozglue.GeckoLoader.nativeRun(Native Method) VERIFIED FIXED Eugen Sawin [:esawin]
1351554 P1 PWA: Collect information on the number of Firefox launches using PWA standalone mode RESOLVED DUPLICATE Nevin Chen [:nechen]
1351555 P1 PWA: Collect information on the number of progressive web apps that’s added to homescreen RESOLVED FIXED Nevin Chen [:nechen]
1351556 P1 PWA: Collect information on the number of users using PWA standalone mode RESOLVED FIXED Nevin Chen [:nechen]
1351739 P1 System for switching between different kinds of GeckoApp RESOLVED FIXED Jan Henning [:JanH] 1325021, 1348686
1351808 P1 Exclude custom tabs from session store RESOLVED FIXED Jan Henning [:JanH] 1352997, 1379374
1351834 -- Set correct tab type when launching WepApp activity from external intent RESOLVED FIXED Dale Harvey (:daleharvey)
1352004 P1 Doorhangers missing background VERIFIED FIXED Nevin Chen [:nechen]
1352099 P1 [PWA] Web Apps don't always have a title and icon in the task switcher entry VERIFIED FIXED Dale Harvey (:daleharvey)
1352394 P1 Crash in java.lang.NullPointerException: Attempt to invoke virtual method ''java.lang.String org.mozilla.gecko.Tab.getURL()'' on a null object reference at org.mozilla.gecko.webapps.WebAppActivity.onNewIntent(WebAppActivity.java) RESOLVED WORKSFORME 1351739
1352433 P1 First run tour in browser and some settings reset after opening a web app VERIFIED FIXED Jan Henning [:JanH]
1352726 -- Flipboard is not given a title VERIFIED FIXED Dale Harvey (:daleharvey)
1352991 P1 Pages loaded in Nightly show in the web app RESOLVED WORKSFORME 1352997
1353714 P1 Crash @ java.lang.NoClassDefFoundError: android.app.ActivityManager$TaskDescription at org.mozilla.gecko.webapps.WebAppActivity.loadManifest(WebAppActivity.java) VERIFIED FIXED Dale Harvey (:daleharvey)
1353857 P2 Fix handling of entering/leaving Web App VERIFIED FIXED Jan Henning [:JanH] 1352997
1355094 -- WebApps - Remove context menu items that are not needed RESOLVED DUPLICATE
1355513 P1 Allow access to microphone and camera permissions not displayed in web apps RESOLVED DUPLICATE Nevin Chen [:nechen]
1355676 P1 Crash in java.lang.NullPointerException: Attempt to invoke virtual method ''android.graphics.Bitmap org.mozilla.gecko.icons.decoders.LoadFaviconResult.getBestBitmap(int)'' on a null object reference at org.mozilla.gecko.BrowserApp.handleMessage(Browser... VERIFIED FIXED Dale Harvey (:daleharvey)
1355824 P1 Browser opens a new tab for web apps already running - tabs tray shows wrong info RESOLVED WORKSFORME 1351808, 1352997
1355847 P1 Web apps - Share selected text doesn't work on Android 6+ RESOLVED DUPLICATE Nevin Chen [:nechen]
1355852 P1 Web Apps - Android 4 & 5 Text selection toolbar not displayed in web apps VERIFIED FIXED Nevin Chen [:nechen]
1357438 P1 Web apps - Login autocomplete isn't working VERIFIED FIXED Dale Harvey (:daleharvey)
1357443 P1 Web Apps - Can't download files from web apps using Nightly RESOLVED WORKSFORME
1360455 P2 Web Apps - snapdrop.net cannot be installed as pwa on Huawei/Xiaomi devices RESOLVED INVALID
1363049 P1 Web Apps - Web apps are reopened when closing a tab opened from them VERIFIED FIXED Jan Henning [:JanH]
1363052 P1 Web Apps - Re-launching a web app with an internal/external page opened, resets is to the homepage RESOLVED WORKSFORME
1363359 P3 manifest with standalone=true has full app context menu in Touchwiz NEW
1364044 P1 Web Apps - Tabs appear in the browser when there are multiple webapps opened VERIFIED FIXED
1364052 P1 Web Apps - Some webapps show the internal page url RESOLVED WORKSFORME
1364471 -- Implement PWA scope attribute for GeckoView-based standalone mode RESOLVED FIXED
1365868 P1 Support minimal context menu functionality in GeckoView-based custom tabs RESOLVED FIXED Nevin Chen [:nechen] 1381983
1367077 P1 Ensure form fill (passwords, etc) working in GeckoView-based custom tabs and web apps RESOLVED FIXED (pto > 10/23) Jim Chen [:jchen] [:darchons] 1403566
1368024 P1 Web Apps - Fall back to 'name' when 'short_name' is missing VERIFIED FIXED Nevin Chen [:nechen]
1369815 P3 Implement 'minimal-ui' and 'standalone' matching for display-mode media queries RESOLVED FIXED James Willcox (:snorp) (jwillcox@mozilla.com)
1369817 -- Implement PWA orientation lock RESOLVED FIXED
1372926 P1 [Android O] Add to Home screen doesn't work on any version of Firefox on Oreo RESOLVED FIXED James Willcox (:snorp) (jwillcox@mozilla.com) 1352015
1376263 P1 Installation of many Google PWAs fails UNCONFIRMED
1377320 -- Hook up prompts for progressive web apps RESOLVED FIXED Dylan Roeh (:droeh)
1378421 P3 Prompt user to install PWA NEW
1379115 P1 Show a badge (in the position of a page action) when a website is installable as a PWA RESOLVED DUPLICATE
1379405 -- Can not dynamically modify the theme color RESOLVED INVALID
1386154 -- manifest with standalone=true still loads in full browser RESOLVED WORKSFORME
1388747 -- Web Apps - show the simplified address bar when the user navigates outside the web app's origin RESOLVED INVALID
1388750 P1 Web apps - Apps suddenly stopped responding, running out of memory REOPENED Dylan Roeh (:droeh)
1388755 P1 Web Apps - Text selection toolbar not displayed RESOLVED DUPLICATE
1388757 -- Web Apps - Device back button sends the webapp in the background RESOLVED DUPLICATE
1388760 P1 Web Apps - Permission request dialogs appear in the browser, not inside the web app RESOLVED WORKSFORME
1388763 P1 Web Apps - Doorhangers aren't working RESOLVED WORKSFORME
1389051 P2 Improve length of URL bar and use of the space near it NEW
1389236 -- Open external links in PWAs in a Custom Tab VERIFIED FIXED Eugen Sawin [:esawin] 1377580
1391271 P2 Add splash screen for standalone PWA NEW
1393188 -- Back button doesn't go back in PWA VERIFIED FIXED
1393672 P1 PWA Badge and Onboarding - Main workflow RESOLVED FIXED Nevin Chen [:nechen]
1394766 -- Hyperlink navigations with target=_blank remain in standalone browsing context RESOLVED WORKSFORME
1395570 P2 CustomTab: media notification not displayed NEW Eugen Sawin [:esawin]
1395839 P1 Disable PWA in private browsing (while keeping page shortcut) RESOLVED DUPLICATE
1395841 P1 implement change for separating “add to home screen” & “add page shortcut” REOPENED Nevin Chen [:nechen]
1395846 P1 PWA onboarding RESOLVED DUPLICATE
1398065 P5 Ability to move a custom tab/web app into the full UI wholesale and without state loss NEW
1398067 P2 Support context menu and "tab" related functionality in GeckoView-based custom tabs/web apps NEW 1365868, 1381983
1400334 P2 Crash in java.lang.NullPointerException: Attempt to invoke virtual method ''int java.lang.Integer.intValue()'' on a null object reference at org.mozilla.gecko.webapps.WebAppActivity.onLoadUri(WebAppActivity.java) RESOLVED FIXED James Willcox (:snorp) (jwillcox@mozilla.com)
1401996 -- Turn on Progressive Web Apps by default NEW
1403566 P2 Web Apps - Login auto-complete isn't working, with multiple logins saved NEW
1404311 P3 PWAs and Custom Tabs can't be debugged through WebIDE RESOLVED FIXED Eugen Sawin [:esawin]
1405081 P2 [geckoview] Add high-level fetch API NEW
1406000 P2 Web Apps - cannot download PDFs (or other files) from pwa NEW Eugen Sawin [:esawin]
1406024 P2 Web Apps - Date-time/ color pickers don't open NEW Eugen Sawin [:esawin]
1406034 -- Web Apps - The aliexpress back button opens Javascript void page NEW
1409191 P1 The "Add to home screen" prompt doesn't appear to use manifest values NEW Nevin Chen [:nechen]

95 Total; 22 Open (23.16%); 47 Resolved (49.47%); 26 Verified (27.37%);


Signoff

Criteria

  • All [Test cases] have passed and all major bugs are fixed.
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

  • n/a
  • Full Test suite
  • Link to TestRail Tests Runs

Beta testing

  • n/a
  • Full Test suite
  • Link to TestRail Tests Runs

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation Done
Test Cases Creation In progress
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
Basic/Core functionality Nightly testing
QA mid-Nightly Signoff Email to be sent
QA Nightly - Full Testing
QA pre-Beta Signoff Email to be sent
QA Beta - Full Testing
QA pre-Release Signoff Email to be sent