Overview

Purpose

The purpose of this wiki is to serve as a general test plan for verifying that RL has moved to bookmarks.

Ownership

Developer contact: Margaret Leibovic (:Margaret)

QA: Teodora Vermesan (:TeoVermesan)

Overall Status

[LANDED] Nightly: first landed on 10-04-2016

[LANDED] Aurora: first landed on 27-04-2016

[LANDED] Beta 48b1: first landed on 08-06-2016

Testing Summary

Scope of testing

The testing will be focus on:

verifying that RL has moved to bookmarks and there users can access it now
verifying that after signing into FxA, users will be able to sync their RL items
verifying that users can read items in RV and save them for later, offline
verifying that users can search for their RL items in the awesome search
verifying that users can access saved RL items in the new RL folder under bookmarks

Environments

Testing will be performed on following devices:

Phones
- 4.x
- 5.x
- 6.0
Tablet
- 4.x
- 5.x
- 6.0

Specifications

Remove reading list panel (and inform users of the change)

Inform users about the migration

->In the same release

Migrate content into a Bookmarks folder (bug 1234315)
Display this message in the RL panel

->In the release after

Remove RL panel (with message) completely

Remove add to/remove from reading list button in main menu

Remove add to/remove from reading list button in main menu

Indicate when an item is saved in Reader View

Indicate when an item is saved in Reader View

Allow users to bookmark reader view pages.
Remove add to reading list button from share overlay

Remove add to reading list button from share overlay

Third time an item is RV'd, use Helper UI to tell users they can bookmark it

Third time an item is RV'd

IF the user never bookmarks an item after going into RV 3 separate times, also a helper UI should be shown, informing users
that they can bookmark these things
The big orange button should trigger the menu ans actually bookmark the item as well.

Indicate that the "Reading List" folder is different

Indicate that the "Reading List" folder is different

Create snackbar feedback when user bookmarks a RV item

Snackbar

include offline icon in the snackbar and made these blue 
NOT FIRST TIME:
 User presses "Reader View" icon in URL bar
 User presses Bookmark icon in menu
 Display snackbar

First time a RV item is Bookmarked, use Helper UI to tell users of Offline ability

First time a RV item is Bookmarked

 User RV's an item 
 User Bookmarks said item
 See this helper UI about Bookmarks integration

Items originally in old RL Panel

---
Not RV-able:
 - Just add to Mobile Bookmarks 
Reader View-able: 
 - Also add to new "Reading List"

User stories

As a user, I want to know that the RL has moved to bookmarks and where to access them now
As a user, I want to understand that when signed into FxA, I will be able to sync my RL items now
As a user, I want to read items in RV, and save them for later, offline
As a user, I want to search for my RL items in the awesome search
As an existing RL user, I want to access all by saved RL items in the new RL folder under bookmarks

Acceptance Criteria

Migrating users:
- All existing RL items have been moved over to Bookmarks-> no persistent RL folder, existing RV items will show up under Bookmarks and smart folder under Bookmarks.
- User is presented with information that RL has moved
Migrating and new users:
- First time star RV item, show dialog that it’s offline available
Remove RL panel and appearances of the Reading List (plus) icon (Menu, Contextmenu, Shareoverlay)
Smart folder can't be deleted
Replace Reader View toolbar with floating action button
Link in UI to SUMO article
Copy confirmed

==KPIs / Measurements Used:

Qualitative:
- No negative feedback from users (SUMO/Google Play Store)
- Current % of RL users: we'd need to associate individual users with the telemetry events (mfinkle's script): http://cl.ly/1T36142z2X02
Adoption: If we can't get % of users who use RL, we might be able to watch the adoption rate instead to measure this
- Monitor how many times users have seen the migration slide/info to know how many people actual use the concept of a RL
- Monitor how many times users clicked on the "offline" helper UI to get a baseline to compare it with viewed saved offline RL items
Increase in ratio between stored RL items and loaded RL items (currently 0.75)
Monitor how many people load RV items that are not in RL folder vs. RV items that are in the RL folder: that will help us understand if people understand the concept of read later / save for later
Monitor how many people load RV items in offline vs. online mode to help us understand their mental model of using RV items for offline mode https://bugzilla.mozilla.org/show_bug.cgi?id=1243387
More usage != more RL items, increased frequency of accessing RL folder now in comparison to previous panel view
Funnel to understand how many people actually read saved RV items in offline mode: https://bugzilla.mozilla.org/show_bug.cgi?id=1246718
- User clicks on RV https://bugzilla.mozilla.org/show_bug.cgi?id=1246710
- User saves RV item to RL list (folder) https://bugzilla.mozilla.org/show_bug.cgi?id=1246712
- User loads RV item in offline vs online mode https://bugzilla.mozilla.org/show_bug.cgi?id=1243387

Testing details

Testing days

Nightly

Date:28-03-2016

Removed shortcut to add an item to your reading list from long tapping on reader view icon
Removed "Share" and "Add to reading list" items from reader view toolbar

Removed "Share" and "Add to reading list" items from reader view toolbar

12899964 10153443750366720 1224654605 n.png

Date:11-04-2016

Exploratory Testing using Nexus 6 (Android 6.0) and Nexus 7 (Android 5.1)
Verified bugs:
- Bug 1234316 - Remove reading list panel (and inform users of the change)
- Bug 1234331 - Enable bookmark star for reader view pages
- Bug 1246232 - Remove "Add to Reading List" from context menus when long-pressing on link
- Bug 1240728 - Remove add to reading list button from share overlay
- Bug 1226238 - Remove "Share" and "Add to reading list" items from reader view toolba
- Bug 1234319 - Remove add to/remove from reading list button in main menu
- Bug 1234328 - Indicate when an item is saved in Reader View
New Bugs:
- Bug 1263571 - "Switch-to-tab" should not be displayed in "Reading List" folder when the page is not opened in another tab
- Bug 1263550 - While viewing a page from "Reading List" bookmark panel, back button should take user back to "Reading List" folder
- Bug 1263587 - No ability to add the page in reader view to the "Reading List" folder if the normal page was bookmarked before
- Bug 1263544 - Choosing "Edit" after adding a reader view page as a bookmark has no action

Date:18-04-2016

Bookmarking a page while in reader view, will display a snackbar at the bottom of the screen: "Saved offline | SWITCH". Tapping it, will display the Bookmarks panel.

Reader view

Date:29-04-2016

Remove reading list panel (and inform users of the change)

Remove reading list panel

First time a page is RV's inform users

First time a page is RV's inform users

Create snackbar feedback when user bookmarks a RV item

Create snackbar feedback when user bookmarks a RV item

Bookmarks folder

Bookmarks folder

Reading list folder

Reading list folder

Date:04-05-2016

Update "available offline" icon

Update "available offline" icon

Improve main menu after removing "add to reading list"

Improve main menu after removing "add to reading list"

Show only readercache icon, or readercache and bookmark icon, for offline readermode items in search results

Show only readercache/readercache&bookmark icon

Date:13-05-2016

Exploratory Testing on Moto X (Android 4.4)

Date:18-05-2016

Regression window for: Bug 1272677 - Two parallel lines are displayed in "Aa" floating layout button in reader view
Bug verification:
- Bug 1259078 - Remove the migrated reading-list panel in 49
- Bug 1269001 - Reading List smart folder should indicate # of items inside
Exploratory Testing on Moto X (Android 4.4) and Nexus 6 (Android 6.0)
New bug:
- Bug 1273816 - Remove "Reading List" folder from Bookmarks in guest mode

Date:25-05-2016

Exploratory Testing on ONE A2001 (Android 5.1.1) and Nexus 6 (Android N)
- about:reader URL is not displayed in Reader View

Show only readercache/readercache&bookmark icon

Empty reading list smart folder in bookmarks

Show only readercache/readercache&bookmark icon

Aurora

Date:29-04-2016

Remove reading list panel (and inform users of the change)

Remove reading list panel

First time a page is RV's inform users

First time a page is RV's inform users

Create snackbar feedback when user bookmarks a RV item

Create snackbar feedback when user bookmarks a RV item

Bookmarks folder

Bookmarks folder

Reading list folder

Reading list folder

Date:23-05-2016

Exploratory testing on latest Aurora using Alcatel one Touch (Android 4.1.2)

Date:25-05-2016

Update "available offline" icon

Reading list folder

Show only readercache icon, or readercache and bookmark icon, for offline readermode items in search results

Reading list folder

Beta 48b1

Date:08-06-2016

Reading list panel was removed
First time a page is RV's inform users

First time a page is RV's inform users

Create snackbar feedback when user bookmarks a RV item

Create snackbar feedback when user bookmarks a RV item

Bookmarks panel

Bookmarks folder

Reading list folder

Reading list folder

"available offline" icon updated

Reading list folder

Bug Work

Tracking bugs

bugs to track the migration of RL to Bookmarks

ex:

Full Query

ID	Summary	Priority	Status
1219240	Telemetry probe to measure % of pages that show reader view button	--	RESOLVED
1226238	Remove "Share" and "Add to reading list" items from reader view toolbar	--	RESOLVED
1234315	Migrate reading list items to bookmarks	--	RESOLVED
1234316	Remove reading list panel (and inform users of the change)	--	RESOLVED
1234319	Remove add to/remove from reading list button in main menu	--	RESOLVED
1234328	Indicate when an item is saved in Reader View	--	RESOLVED
1234331	Enable bookmark star for reader view pages	--	RESOLVED
1234335	Remove shortcut to add an item to your reading list from long tapping on reader view icon	--	RESOLVED
1237880	Remove Reading List sync code	--	RESOLVED
1240728	Remove add to reading list button from share overlay	--	RESOLVED
1243382	Create probe to understand how many people view offline Reader View items when actually being offline	--	RESOLVED
1243387	Add telemetry probe to understand how often users load reading list items when offline	--	RESOLVED
1246159	Create histogram probe for disk space used by saved reader view cache	--	RESOLVED
1246232	Remove "Add to Reading List" from context menus when long-pressing on link	--	RESOLVED
1246238	First time a RV item is Bookmarked, use Helper UI to tell users of Offline ability	--	RESOLVED
1246239	Create snackbar feedback when user bookmarks a RV item	--	RESOLVED
1246243	Indicate that the "Reading List" folder is different	--	VERIFIED
1246670	Add telemetry probe to understand how often users load content in reader mode when offline	--	RESOLVED
1246710	Add telemetry probe to understand how often users load content in reader mode	--	RESOLVED
1246712	Add telemetry probe to understand how often users save Reader Mode to Reading List (folder)	--	RESOLVED
1247689	Third time an item is RV'd, use Helper UI to tell users they can bookmark it	--	RESOLVED
1251340	Probe to measure if RV item was opened from root Bookmark list or from smart folder	--	RESOLVED
1251661	Add probe to determine activity of saved and loaded RV items in Bookmark panel	--	RESOLVED
1257345	Implement Readermode/Reading-list smart folder	--	RESOLVED
1258430	Put page into readercache when entering/exiting readermode for a bookmark	--	RESOLVED
1259078	Remove the migrated reading-list panel in 49	--	RESOLVED

26 Total; 0 Open (0%); 25 Resolved (96.15%); 1 Verified (3.85%);

Still issues

[ASSIGNED] Bug 1246159 - Create histogram probe for the # items in the Reader View disk cache
[ASSIGNED] Bug 1267467 - Opening a reading list/reader mode icon should bump its history
[ASSIGNED] Bug 1270880 - Update "how to bookmark" helper UI to "first time" instead of "third time"
[NEW] Bug 1246710 - Add telemetry probe to understand how often users load content in reader modey
[NEW] Bug 1248477 - Create SUMO article to help and explain Migration of RL
[NEW] Bug 1267580 - Empty reading list smart folder in bookmarks

Bug fixed

[RESOLVED FIXED] Bug 1219240 - Telemetry probe to measure % of pages that show reader view button
[RESOLVED FIXED] Bug 1226238 - Remove "Share" and "Add to reading list" items from reader view toolbar

↳ 2016-02-29: verified fixed on Firefox for Android 47.0a1 (2016-02-28) using One A2001 (Android 5.1.1)

[RESOLVED FIXED] Bug 1234315 - Migrate reading list items to bookmarks