QA/Fennec/LeanPlum Integration iOS

Overview

Purpose

The purpose of this wiki is to serve as a general test plan for verifying that we can create MVP Triggers and Deeplinks for Mobile Marketing Automation

Ownership

Developer contact: Susheel Daswani [slack - @sdaswani]

QA: Simion Basca [slack - @sbasca]

Overall Status

[LANDED] Master 10-08-2017

[NOT LANDED] Beta 9.0

Testing Summary

Scope of testing

• The testing will be focus on
  • P0 trigger: open Firefox 2 times during the last 7 days
  • P0 deeplink: push the dialog when user open Firefox

• P1 Triggers:
  • Loads articles
  • Visiting a website
  • Download videos or any other media
  • SavePassword
  • SaveBookmark
  • LoadBookmark
  • Interact with search url area
  • When user take a screenshot
  • When user “Clear Private Data”

• P1 Deep links:
  • Link to Set Default Browser settings
  • Link to specific Add-on page
  • Link to sync signup/sign in
  • Link to default search engine settings
  • Link to “Save as PDF” feature
  • Take user directly to a Sign up for a newsletter

Environments

Testing will be performed on following devices:

• Phones
  • iPhone 6S -> iOS 10.3.3
• Tablet
  • iPad Air 2 -> 10.3.3

Specifications

• Doc: https://docs.google.com/document/d/1VODMs1FIK8NpxpbeUFZDX7LXDI2pQ6wYASkJ0lIc_qk/edit
• In-app message is to inform/promote users of browser features that may be contextually relevant to them to increase engagement/retention
• Onboarding is to optimize the first Run experience per market and promote specific features
• Push notification is to reactive user’s Lifecycle journey

User stories

• As Mozilla, we want to control the target users we push messages to by country and by locale
• As Mozilla, we want to push in app messages to users based on event triggers
• As a user, when I click on the push notification or in app message, I want to be taken directly to the page where I can take immediate actions
• As Mozilla, we want to change the onboarding feature introduction by users/countries so we can perform A/B testing
• As Mozilla, we want to push notifications (if we chooses to) to users even when Firefox is not running
• As a user, I want to stop receiving push notifications from Mozilla

Testing details

Test Cases

• Test Case LeanPlum

Testing days

Date:2017-05-24

• Created Test Plan

Date:2017-05-25

• Created Test Cases - initial draft

Date:2017-05-31

• Tested on Nightly 55 with Nexus 5 (Android 6.0.1), Nexus 6 (Android 7.0) & Huawei Honor 5X (Android 5.1.1)
• Logged Bugs:
  • bug 1368970 - [LeanPlum] - Tapping on device back button doesn't dismiss the prompt
  • bug 1368995 - [LeanPlum] - Prompt not shown when download a media content
  • bug 1369013 - [LeanPlum] - Keyboard not displayed when going back from the Search Engine trigger
  • bug 1369006 - [LeanPlum] - Default Browser is still triggered even if Firefox is set as the default browser
  • bug 1368971 - [LeanPlum] - Save as PDF infinite loop

Date:2017-06-06

• Tested on Nightly 55 with Nexus 5 (Android 6.0.1), Nexus 6 (Android 7.0), LG G4 (Android 5.1), Oneplus Two (Android 6.0.1), Samsung Galaxy Note 4 (Android 5.0.1) & Samsung Galaxy S4 (Android 4.4.2)
• Logged Bugs:
  • bug 1370515 - [LeanPlum] - Promote Sync triggered on account creation/login

Date:2017-07-21

• Updated Test Cases - Waiting for approval from Nevin

Date:2017-07-31

• Ran Test Cases
• Logged Bugs:
  • bug 1385853 - [LeanPlum] - Default browser is still triggered if app was set as default from another app
  • bug 1385819 - [LeanPlum] - Focus promotion still displayed after installing Focus

Bug Work

Tracking bugs

• Meta bug
  • bug 1337434 - [Meta] iOS Leanplum Integration

Full Query

ID	Priority	Summary	Status	Assigned to
1337436	--	Integrate closed source Leanplum SDK	RESOLVED	Stefan Arentz | :st3fan | ⏰ EST | he/him
1337437	--	Integrate open source Leanplum SDK	RESOLVED
1337630	--	Leanplum iOS - First Run (market and feature specific)	RESOLVED
1337631	--	Leanplum iOS - Add-Ons/App promotion	RESOLVED
1337632	--	Leanplum iOS - Sync Promotion	RESOLVED
1337634	--	Leanplum iOS - Mail app Promotion	RESOLVED
1337930	--	iOS Leanplum - Data collection documentation	RESOLVED
1351446	P1	LeanPlum Phase I Integration	RESOLVED	Bryan Munar (:bkmunar)
1391687	--	Send an event when the user saves an image	VERIFIED
1391690	--	"E_Opened_New_Tab" is not triggered when opening a new tab via '+' symbol on iPad	VERIFIED	Bryan Munar (:bkmunar)
1391692	--	Send an event when the user bookmarks on the new tab page	VERIFIED
1393457	--	"E_First_Run" is not triggered on when launching Firefox after a clean install	RESOLVED
1393458	P1	"E_First_Run" and "E_Second_Run" is not triggered after launching Firefox a second time	VERIFIED	Bryan Munar (:bkmunar)
1393459	--	"E_Opened_App" is not triggered when launching Firefox	RESOLVED
1393466	P2	"E_Opened_New_Tab" not triggered when opening a new tab via menu from Tabs Tray	VERIFIED	Bryan Munar (:bkmunar)
1393473	P3	LeanPlum: "Opened Telephone Link" is not triggered when opening a telephone link	RESOLVED
1393492	P1	"Mailto Is Default" reads "False" on clean install and unregistered device and it should read "True"	VERIFIED
1393506	--	'Push Pre-Permission' message can be sent only once	RESOLVED
1393729	P1	Unable to register device after changing the Region and Language to DE	VERIFIED	Farhan Patel [:farhan]
1393765	--	Firefox will crash when trying to open the deeplink for the History page	RESOLVED
1393766	P3	Deep Link to Sync settings not firing correctly	NEW
1394469	P1	Push notification misbehave after sending a register for push from Leanplum	VERIFIED	James Hugman [:jhugman] [@jhugman]
1395124	P1	Leanplum events (E_First_Run and E_Second_Run) not firing correctly when first launching the app	VERIFIED	Farhan Patel [:farhan]

23 Total; 1 Open (4.35%); 13 Resolved (56.52%); 9 Verified (39.13%);

Signoff

Criteria

• All test cases should be executed
• All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed

Results

Nightly testing

• in progress

Beta testing

• n/a