QA/Fennec/LeanPlum

Overview

Purpose

The purpose of this wiki is to serve as a general test plan for verifying that we can create MVP Triggers and Deeplinks for Mobile Marketing Automation

Ownership

Developer contact: Nevin Chen [:nechen]

QA: Bogdan Surd [:bogdan] Sorina Florean [:sorina]

Overall Status

[LANDED] Nightly 56: 31-05-2017

[LANDED] Beta 56:

Push Notifications:

[LANDED] Nightly 57: 21-08-2017

Testing Summary

Scope of testing

• The testing will be focus on
  • P0 trigger: open Firefox 2 times during the last 7 days
  • P0 deeplink: push the dialog when user open Firefox

• P1 Triggers:
  • Loads articles
  • Visiting a website
  • Download videos or any other media
  • SavePassword
  • SaveBookmark
  • LoadBookmark
  • Interact with search url area
  • When user take a screenshot
  • When user “Clear Private Data”

• P1 Deep links:
  • Link to Set Default Browser settings
  • Link to specific Add-on page
  • Link to sync signup/sign in
  • Link to default search engine settings
  • Link to “Save as PDF” feature
  • Take user directly to a Sign up for a newsletter

Environments

Testing will be performed on following devices:

• Phones
  • Android 4 -> Android 7
• Tablet
  • Android 4 -> Android 7

Specifications

• Doc: https://docs.google.com/document/d/19tKqSi8pQF2P21yLFfLtndmSKDjOtkYVdM1yWAWAmX8/edit#heading=h.e5cqukfqdea6
• In-app message is to inform/promote users of browser features that may be contextually relevant to them to increase engagement/retention
• Onboarding is to optimize the first Run experience per market and promote specific features
• Push notification is to reactive user’s Lifecycle journey

User stories

• As Mozilla, we want to control the target users we push messages to by country and by locale
• As Mozilla, we want to push in app messages to users based on event triggers
• As a user, when I click on the push notification or in app message, I want to be taken directly to the page where I can take immediate actions
• As Mozilla, we want to change the onboarding feature introduction by users/countries so we can perform A/B testing
• As Mozilla, we want to push notifications (if we chooses to) to users even when Firefox is not running

Testing details

Test Cases

• Test Case LeanPlum

Testing days

Date:2017-05-24

• Created Test Plan

Date:2017-05-25

• Created Test Cases - initial draft

Date:2017-05-31

• Tested on Nightly 55 with Nexus 5 (Android 6.0.1), Nexus 6 (Android 7.0) & Huawei Honor 5X (Android 5.1.1)
• Logged Bugs:
  • bug 1368970 - [LeanPlum] - Tapping on device back button doesn't dismiss the prompt
  • bug 1368995 - [LeanPlum] - Prompt not shown when download a media content
  • bug 1369013 - [LeanPlum] - Keyboard not displayed when going back from the Search Engine trigger
  • bug 1369006 - [LeanPlum] - Default Browser is still triggered even if Firefox is set as the default browser
  • bug 1368971 - [LeanPlum] - Save as PDF infinite loop

Date:2017-06-06

• Tested on Nightly 55 with Nexus 5 (Android 6.0.1), Nexus 6 (Android 7.0), LG G4 (Android 5.1), Oneplus Two (Android 6.0.1), Samsung Galaxy Note 4 (Android 5.0.1) & Samsung Galaxy S4 (Android 4.4.2)
• Logged Bugs:
  • bug 1370515 - [LeanPlum] - Promote Sync triggered on account creation/login

Date:2017-07-21

• Updated Test Cases - Waiting for approval from Nevin

Date:2017-07-31

• Ran Test Cases
• Logged Bugs:
  • bug 1385853 - [LeanPlum] - Default browser is still triggered if app was set as default from another app
  • bug 1385819 - [LeanPlum] - Focus promotion still displayed after installing Focus

Date:2017-09-14

• Ran Test Cases
• Devices: Asus ZenPad 8(Android 6.0.1), Huawei MediaPad M2 (Android 5.1.1), Motorola Nexus 6 (Android 7.1.1)

Date:2017-09-15

• Pre-release SignOff:GREEN, feature looks stable, no new issues found
• REport: https://testrail.stage.mozaws.net/index.php?/reports/view/583

Bug Work

Tracking bugs

• Meta bug
  • bug 1365108 - [meta]

Full Query

ID	Priority	Summary	Status	Assigned to
1376690	--	(Leanplum) Disable Leanplum when users are using private mode.	RESOLVED	Nevin Chen(Not active on Bugzilla)
1380950	--	(Leanplum) Deeplinks should not be triggered through content or other apps	RESOLVED	Nevin Chen(Not active on Bugzilla)

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);

Signoff

Criteria

• All test cases should be executed
• All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed

Results

Nightly testing

• in progress

Beta testing

• in progress