QA/Fennec/LeanPlum
From MozillaWiki
Overview
Purpose
The purpose of this wiki is to serve as a general test plan for verifying that we can create MVP Triggers and Deeplinks for Mobile Marketing Automation
Ownership
Developer contact: Nevin Chen [:nechen]
QA: Bogdan Surd [:bogdan] Sorina Florean [:sorina]
Overall Status
[LANDED] Nightly 56: 31-05-2017
[LANDED] Beta 56:
Push Notifications:
[LANDED] Nightly 57: 21-08-2017
Testing Summary
Scope of testing
- The testing will be focus on
- P0 trigger: open Firefox 2 times during the last 7 days
- P0 deeplink: push the dialog when user open Firefox
- P1 Triggers:
- Loads articles
- Visiting a website
- Download videos or any other media
- SavePassword
- SaveBookmark
- LoadBookmark
- Interact with search url area
- When user take a screenshot
- When user “Clear Private Data”
- P1 Deep links:
- Link to Set Default Browser settings
- Link to specific Add-on page
- Link to sync signup/sign in
- Link to default search engine settings
- Link to “Save as PDF” feature
- Take user directly to a Sign up for a newsletter
Environments
Testing will be performed on following devices:
- Phones
- Android 4 -> Android 7
- Tablet
- Android 4 -> Android 7
Specifications
- Doc: https://docs.google.com/document/d/19tKqSi8pQF2P21yLFfLtndmSKDjOtkYVdM1yWAWAmX8/edit#heading=h.e5cqukfqdea6
- In-app message is to inform/promote users of browser features that may be contextually relevant to them to increase engagement/retention
- Onboarding is to optimize the first Run experience per market and promote specific features
- Push notification is to reactive user’s Lifecycle journey
User stories
- As Mozilla, we want to control the target users we push messages to by country and by locale
- As Mozilla, we want to push in app messages to users based on event triggers
- As a user, when I click on the push notification or in app message, I want to be taken directly to the page where I can take immediate actions
- As Mozilla, we want to change the onboarding feature introduction by users/countries so we can perform A/B testing
- As Mozilla, we want to push notifications (if we chooses to) to users even when Firefox is not running
Acceptance Criteria
Testing details
Test Cases
Testing days
Date:2017-05-24
- Created Test Plan
Date:2017-05-25
- Created Test Cases - initial draft
Date:2017-05-31
- Tested on Nightly 55 with Nexus 5 (Android 6.0.1), Nexus 6 (Android 7.0) & Huawei Honor 5X (Android 5.1.1)
- Logged Bugs:
- bug 1368970 - [LeanPlum] - Tapping on device back button doesn't dismiss the prompt
- bug 1368995 - [LeanPlum] - Prompt not shown when download a media content
- bug 1369013 - [LeanPlum] - Keyboard not displayed when going back from the Search Engine trigger
- bug 1369006 - [LeanPlum] - Default Browser is still triggered even if Firefox is set as the default browser
- bug 1368971 - [LeanPlum] - Save as PDF infinite loop
Date:2017-06-06
- Tested on Nightly 55 with Nexus 5 (Android 6.0.1), Nexus 6 (Android 7.0), LG G4 (Android 5.1), Oneplus Two (Android 6.0.1), Samsung Galaxy Note 4 (Android 5.0.1) & Samsung Galaxy S4 (Android 4.4.2)
- Logged Bugs:
- bug 1370515 - [LeanPlum] - Promote Sync triggered on account creation/login
Date:2017-07-21
- Updated Test Cases - Waiting for approval from Nevin
Date:2017-07-31
- Ran Test Cases
- Logged Bugs:
- bug 1385853 - [LeanPlum] - Default browser is still triggered if app was set as default from another app
- bug 1385819 - [LeanPlum] - Focus promotion still displayed after installing Focus
Date:2017-09-14
- Ran Test Cases
- Devices: Asus ZenPad 8(Android 6.0.1), Huawei MediaPad M2 (Android 5.1.1), Motorola Nexus 6 (Android 7.1.1)
Date:2017-09-15
- Pre-release SignOff:GREEN, feature looks stable, no new issues found
- REport: https://testrail.stage.mozaws.net/index.php?/reports/view/583
Bug Work
Tracking bugs
- Meta bug
- bug 1351571 - [meta] Android Leanplum Integration
45 Total; 0 Open (0%); 41 Resolved (91.11%); 4 Verified (8.89%);
Signoff
Criteria
- All test cases should be executed
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed
Results
Nightly testing
- in progress
Beta testing
- in progress
