QA/Fennec/LeanPlum: Difference between revisions

From MozillaWiki
< QA‎ | Fennec
Jump to navigation Jump to search
No edit summary
Line 61: Line 61:
* As Mozilla, we want to change the onboarding feature introduction by users/countries so we can perform A/B testing
* As Mozilla, we want to change the onboarding feature introduction by users/countries so we can perform A/B testing
* As Mozilla, we want to push notifications (if we chooses to) to users even when Firefox is not running
* As Mozilla, we want to push notifications (if we chooses to) to users even when Firefox is not running
* As a user, I want to stop receiving push notifications from Mozilla


== Acceptance Criteria==
== Acceptance Criteria==
Line 72: Line 71:
===Date:2017-05-24===
===Date:2017-05-24===
* Created Test Plan
* Created Test Plan
===Date:2017-05-25===
===Date:2017-05-25===
* Created [https://docs.google.com/spreadsheets/d/1_ssHYnB5vmfkeCXCg66Tr0dku_oNF7NVvsl1bXojORo/edit#gid=0| Test Cases] - initial draft  
* Created [https://docs.google.com/spreadsheets/d/1_ssHYnB5vmfkeCXCg66Tr0dku_oNF7NVvsl1bXojORo/edit#gid=0| Test Cases] - initial draft  
===Date:2017-05-31===
===Date:2017-05-31===
* Tested on Nightly 55 with Nexus 5 (Android 6.0.1), Nexus 6 (Android 7.0) & Huawei Honor 5X (Android 5.1.1)
* Tested on Nightly 55 with Nexus 5 (Android 6.0.1), Nexus 6 (Android 7.0) & Huawei Honor 5X (Android 5.1.1)
Line 82: Line 83:
**{{Bug|1369006}} - [LeanPlum] - Default Browser is still triggered even if Firefox is set as the default browser
**{{Bug|1369006}} - [LeanPlum] - Default Browser is still triggered even if Firefox is set as the default browser
**{{Bug|1368971}} - [LeanPlum] - Save as PDF infinite loop
**{{Bug|1368971}} - [LeanPlum] - Save as PDF infinite loop
===Date:2017-06-06===
===Date:2017-06-06===
* Tested on Nightly 55 with Nexus 5 (Android 6.0.1), Nexus 6 (Android 7.0), LG G4 (Android 5.1), Oneplus Two (Android 6.0.1), Samsung Galaxy Note 4 (Android 5.0.1) & Samsung Galaxy S4 (Android 4.4.2)
* Tested on Nightly 55 with Nexus 5 (Android 6.0.1), Nexus 6 (Android 7.0), LG G4 (Android 5.1), Oneplus Two (Android 6.0.1), Samsung Galaxy Note 4 (Android 5.0.1) & Samsung Galaxy S4 (Android 4.4.2)
* Logged Bugs:
* Logged Bugs:
**{{Bug|1370515}} - [LeanPlum] - Promote Sync triggered on account creation/login
**{{Bug|1370515}} - [LeanPlum] - Promote Sync triggered on account creation/login
===Date:2017-07-21===
===Date:2017-07-21===
* Updated [https://docs.google.com/spreadsheets/d/1_ssHYnB5vmfkeCXCg66Tr0dku_oNF7NVvsl1bXojORo/edit#gid=487515512| Test Cases] - Waiting for approval from Nevin
* Updated [https://docs.google.com/spreadsheets/d/1_ssHYnB5vmfkeCXCg66Tr0dku_oNF7NVvsl1bXojORo/edit#gid=487515512| Test Cases] - Waiting for approval from Nevin
===Date:2017-07-31===
===Date:2017-07-31===
* Ran [https://docs.google.com/spreadsheets/d/1_ssHYnB5vmfkeCXCg66Tr0dku_oNF7NVvsl1bXojORo/edit#gid=487515512| Test Cases]
* Ran [https://docs.google.com/spreadsheets/d/1_ssHYnB5vmfkeCXCg66Tr0dku_oNF7NVvsl1bXojORo/edit#gid=487515512| Test Cases]
Line 93: Line 97:
**{{Bug|1385853}} - [LeanPlum] - Default browser is still triggered if app was set as default from another app
**{{Bug|1385853}} - [LeanPlum] - Default browser is still triggered if app was set as default from another app
**{{Bug|1385819}} - [LeanPlum] - Focus promotion still displayed after installing Focus
**{{Bug|1385819}} - [LeanPlum] - Focus promotion still displayed after installing Focus
===Date:2017-08-30===
* Created new [https://docs.google.com/spreadsheets/d/1_ssHYnB5vmfkeCXCg66Tr0dku_oNF7NVvsl1bXojORo/edit#gid=1670644626| Test Cases] for push notifications


===Date:2017-08-31===
===Date:2017-09-14===
* Ran [https://docs.google.com/spreadsheets/d/1_ssHYnB5vmfkeCXCg66Tr0dku_oNF7NVvsl1bXojORo/edit#gid=1670644626| Test Cases]
* Ran [https://docs.google.com/spreadsheets/d/1_ssHYnB5vmfkeCXCg66Tr0dku_oNF7NVvsl1bXojORo/edit#gid=487515512| Test Cases]
** Filed Bugs:
* Devices: Asus ZenPad 8(Android 6.0.1), Huawei MediaPad M2 (Android 5.1.1), Motorola Nexus 6 (Android 7.1.1)
*** {{Bug|1395548}} - [LeanPlum] - Campaign A no notifications received
*** {{Bug|1395544}} - [LeanPlum] - Campaign B no push notifications received
*** {{Bug|1395551}} - [LeanPlum] - Campaign C no push notifications received
*** {{Bug|1395552}} - [LeanPlum] - Campaign D no push notifications received
*** {{Bug|1395553}} - [LeanPlum] - Campaign F no push notifications received
 
===Date:2017-09-01===
* Imported Push Notification [https://docs.google.com/spreadsheets/d/1_ssHYnB5vmfkeCXCg66Tr0dku_oNF7NVvsl1bXojORo/edit#gid=1670644626| Test Cases] to [https://testrail.stage.mozaws.net/index.php?/suites/view/254&group_by=cases:section_id&group_order=asc&group_id=5907 TestRail]


===Date:2017-09-04===
===Date:2017-09-15===
* Ran [https://docs.google.com/spreadsheets/d/1_ssHYnB5vmfkeCXCg66Tr0dku_oNF7NVvsl1bXojORo/edit#gid=1670644626| Test Cases]
* Pre-release SignOff:GREEN, feature looks stable, no new issues found
** Filed Bug:
* REport: https://testrail.stage.mozaws.net/index.php?/reports/view/583
*** {{Bug|1396548}} - [LeanPlum] - Push Notifications don't work if the user closes Fennec


===Date:2017-09-13===
* Pre-Beta Sign-Off [https://testrail.stage.mozaws.net/index.php?/reports/view/574 Test Cases] Status: Green
** Filed Bugs:
*** {{Bug|1399388}} - [LeanPlum]: Top Sites panel open instead of Bookmarks Panel
*** {{Bug|1399470}} - [LeanPlum] - Campaign B not received for Note 4


=Bug Work=
=Bug Work=
==Tracking bugs==
==Tracking bugs==
* Meta bug
* Meta bug
**{{Bug|1351571}} - [meta] Android Leanplum Integration
**{{Bug|1365108}} - [meta]  
<onlyinclude>
<onlyinclude>
<bugzilla>
<bugzilla>
{
{
     "blocks": "1351571",
     "blocks": "1365108",
     "include_fields": "id, priority, summary, status, assigned_to",
     "include_fields": "id, priority, summary, status, assigned_to",
     "order": "Status"
     "order": "Status"
Line 143: Line 131:


'''Beta testing'''
'''Beta testing'''
*n/a
*in progress

Revision as of 11:53, 15 September 2017

Overview

Purpose

The purpose of this wiki is to serve as a general test plan for verifying that we can create MVP Triggers and Deeplinks for Mobile Marketing Automation

Ownership

Developer contact: Nevin Chen [:nechen]

QA: Bogdan Surd [:bogdan] Sorina Florean [:sorina]

Overall Status

[LANDED] Nightly 56: 31-05-2017

[LANDED] Beta 56:

Push Notifications:

[LANDED] Nightly 57: 21-08-2017

Testing Summary

Scope of testing

  • The testing will be focus on
    • P0 trigger: open Firefox 2 times during the last 7 days
    • P0 deeplink: push the dialog when user open Firefox
  • P1 Triggers:
    • Loads articles
    • Visiting a website
    • Download videos or any other media
    • SavePassword
    • SaveBookmark
    • LoadBookmark
    • Interact with search url area
    • When user take a screenshot
    • When user “Clear Private Data”
  • P1 Deep links:
    • Link to Set Default Browser settings
    • Link to specific Add-on page
    • Link to sync signup/sign in
    • Link to default search engine settings
    • Link to “Save as PDF” feature
    • Take user directly to a Sign up for a newsletter

Environments

Testing will be performed on following devices:

  • Phones
    • Android 4 -> Android 7
  • Tablet
    • Android 4 -> Android 7

Specifications

User stories

  • As Mozilla, we want to control the target users we push messages to by country and by locale
  • As Mozilla, we want to push in app messages to users based on event triggers
  • As a user, when I click on the push notification or in app message, I want to be taken directly to the page where I can take immediate actions
  • As Mozilla, we want to change the onboarding feature introduction by users/countries so we can perform A/B testing
  • As Mozilla, we want to push notifications (if we chooses to) to users even when Firefox is not running

Acceptance Criteria

Testing details

Test Cases

Testing days

Date:2017-05-24

  • Created Test Plan

Date:2017-05-25

Date:2017-05-31

  • Tested on Nightly 55 with Nexus 5 (Android 6.0.1), Nexus 6 (Android 7.0) & Huawei Honor 5X (Android 5.1.1)
  • Logged Bugs:
    • bug 1368970 - [LeanPlum] - Tapping on device back button doesn't dismiss the prompt
    • bug 1368995 - [LeanPlum] - Prompt not shown when download a media content
    • bug 1369013 - [LeanPlum] - Keyboard not displayed when going back from the Search Engine trigger
    • bug 1369006 - [LeanPlum] - Default Browser is still triggered even if Firefox is set as the default browser
    • bug 1368971 - [LeanPlum] - Save as PDF infinite loop

Date:2017-06-06

  • Tested on Nightly 55 with Nexus 5 (Android 6.0.1), Nexus 6 (Android 7.0), LG G4 (Android 5.1), Oneplus Two (Android 6.0.1), Samsung Galaxy Note 4 (Android 5.0.1) & Samsung Galaxy S4 (Android 4.4.2)
  • Logged Bugs:
    • bug 1370515 - [LeanPlum] - Promote Sync triggered on account creation/login

Date:2017-07-21

  • Updated Test Cases - Waiting for approval from Nevin

Date:2017-07-31

  • Ran Test Cases
  • Logged Bugs:
    • bug 1385853 - [LeanPlum] - Default browser is still triggered if app was set as default from another app
    • bug 1385819 - [LeanPlum] - Focus promotion still displayed after installing Focus

Date:2017-09-14

  • Ran Test Cases
  • Devices: Asus ZenPad 8(Android 6.0.1), Huawei MediaPad M2 (Android 5.1.1), Motorola Nexus 6 (Android 7.1.1)

Date:2017-09-15


Bug Work

Tracking bugs

Full Query
ID Priority Summary Status Assigned to
1376690 -- (Leanplum) Disable Leanplum when users are using private mode. RESOLVED Nevin Chen(Not active on Bugzilla)
1380950 -- (Leanplum) Deeplinks should not be triggered through content or other apps RESOLVED Nevin Chen(Not active on Bugzilla)

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


Signoff

Criteria

  • All test cases should be executed
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed

Results

Nightly testing

  • in progress

Beta testing

  • in progress
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Fennec/LeanPlum&oldid=1180523"