QA/Fennec/Prototype content update of websites with infrequent updates
Overview
Purpose
The purpose of this wiki is to serve as a general test plan for verifying that users get notifications when their favorite websites are updated
Ownership
Developer contact: Sebastian Kaspari
QA: Sorina Florean
Overall Status
[LANDED] Nightly 48: Under switchboard experiment
[LANDED] Aurora 48: Under switchboard experiment
[LANDED] Beta 48b1: 2016-06-08- Under switchboard experiment
Testing Summary
Scope of testing
The testing will be focus on verifying that users are allowed to retrieve notifications when websites change
Environments
Testing will be performed on following devices:
- Phones
- 4.x
- 5.x
- 6.x
- Tablet
- 4.x
- 5.x
- 6.x
Specifications
- Allow user to retrieve valuable content infrequent updates for websites the user is interested in
- Screenshots:
Android notification mockup
User stories
- As a user, I want to be notified when an update happened to a website that I use
- As a user, I want to view the updated content on the website I was notified about
- As a user, I want to disable notification for websites I’m not interested in
Acceptance Criteria
- Notification is triggered after website content has been updated
- Simple notification lets user open the browser and navigate to the updated content of the website
- KPIs / Measurements Used:
- Probe to define how many times the notification triggered, to figure out if it’s the right amount, i.e. not too annoying and if it triggered properly based on content updates
- Probe how many users turned off the notification to decide if the feature is annoying people, not needed
- Probe how many users actually opened the browser after receiving the trigger to see if this drives engagement
Testing details
Test Cases
Testing days
Date: 2016-03-24
- Created test plan
Date: 2016-04-04
- Exploratory testing
- On Nightly: Settings -> Notifications
- Updates from medium.com
Date: 2016-04-14
- Exploratory testing on latest Nightly using Asus ZenPad 8 (Android 5.0.2) and Nexus 5 (Android 6.0.1).
- Bookmark articles from medium.com and wordpress.com.
Date: 2016-05-23
Date: 2016-06-17
- Exploratory on 48 Beta 1 with Nexus 5 (Android 6.0.1)
- Verified bug:
- bug 1264868 - (Content notifications) Multiple URLS not opening
Date: 2016-07-12
- Exploratory on 48 Beta 6 with Motorola Razr (Android 4.4.4)
- Exploratory on latest Nightly with LG G4 (Android 5.1)
Date: 2016-07-28
- Verified bug:
- bug 1258812 - Create content notifications experiments
- Websites updates from latest Aurora, 48 Beta 11 and Firefox 48 RC
Bug Work
Tracking bugs
- Meta bug
- bug 1238087 - Create background service for subscribing to website feeds
25 Total; 0 Open (0%); 23 Resolved (92%); 2 Verified (8%);
Still issues
[NEW] Bug 1251339 - Try using supported sites' favicon in content update notification
[NEW] Bug 1253280 - Content notifications: Check for subscribe-able site
[NEW] Bug 1253285 - Content notifications: Update frequency
[NEW] Bug 1253290 - Content notifications: Picking sites to subscribe to
[NEW] Bug 1254637 - Deploy switchboard experiment for content notifications
[NEW] Bug 1257502 - Create SUMO article to help and explain Content Notifications
[NEW] Bug 1261809 - (Content notifications) Open tabs tray when opening multiple URLs?
[NEW] Bug 1262198 - Expanding 'watched' sites from bookmark cue
[NEW] Bug 1263969 - Content notifications: Notification title could be ambiguous
[NEW] Bug 1263971 - Content notifications: allow disabling for certain bookmarks
[NEW] Bug 1271335 - Store URLs of new content in urlannotations
[NEW] Bug 1273014 - "Website Updates" notifications are mysterious and irrelevant (e.g. a young woman's tumblr page with some random selfies)
Bug fixed
[RESOLVED FIXED] Bug 1251317 - Probe to measure usage and activity of content notifications
[RESOLVED FIXED] Bug 1241810 - Create background service for subscribing to website feeds
[RESOLVED FIXED] Bug 1258812 - Create content notifications experiments
[RESOLVED FIXED] Bug 1247788 - Add "Notifications" in Settings
[RESOLVED FIXED] Bug 1261808 - (Content notifications) Re-add telemetry for unsubscribe from feeds
[RESOLVED FIXED] Bug 1261830 - Content notifications: Start telemetry sessions
[RESOLVED FIXED] Bug 1258988 - Content notifications: Add support for Tumblr
[RESOLVED FIXED] Bug 1261836 - Content notifications: Check if new content is already in history
[RESOLVED FIXED] Bug 1262105 - Add link to SUMO article to content notifications preference
[RESOLVED FIXED] Bug 1259127 - Content notifications: Consider adding "Read" action to notification
[RESOLVED FIXED] Bug 1264868 - (Content notifications) Multiple URLS not opening
Signoff
Criteria
- All test cases should be executed
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed
Results
Nightly testing:
- Tools -> Add-ons -> Browse all Firefox Add-ons;
- Download and install Switchboard Experiments; the panel is added to home screen;
- In about:experiments, user can enable/disable experiments:
- whatsnew-notification
- content-notifications-12hrs
- content-notifications-8am
- content-notifications-5pm
- Bookmark articles from sites:
- medium.com
- blogspot.com (Blogger)
- wordpress.com
Aurora testing
- Tools -> Add-ons -> Browse all Firefox Add-ons;
- Download and install Switchboard Experiments: the panel is added to home screen;
- In about:experiments, user can enable/disable experiments:
- whatsnew-notification
- content-notifications-12hrs
- content-notifications-8am
- content-notifications-5pm
- Bookmark articles from sites:
- medium.com
- blogspot.com (Blogger)
- wordpress.com
Beta testing
- Tools -> Add-ons -> Browse all Firefox Add-ons;
- Download and install Switchboard Experiments: the panel is added to home screen;
- In about:experiments, user can enable/disable experiments:
- whatsnew-notification
- content-notifications-12hrs
- content-notifications-8am
- content-notifications-5pm
- Bookmark articles from sites:
- medium.com
- blogspot.com (Blogger)
- wordpress.com
