Leanplum Contextual Hints

From MozillaWiki
Jump to: navigation, search

Test Plan (iOS & Android) - July 2017

Goal:

With the initial implementation of the Leanplum SDK into our mobile browser products, we want to start A/B testing contextual hints to increase retention.

Overarching Hypothesis:

We believe that by promoting contextual hints that helps users explore and change settings within the browser, we will drive DAUs and retention.

iOS

iOS Hypothesis 1:

We believe that by promoting Focus to our privacy conscious users inside Firefox, we will increase installs and DAUs in Focus, and increase retention on both apps (We know that users who have Focus and Firefox installed retain better (Adjust)).

iOS Hypothesis 2:

We believe that by promoting Sync to our users inside Firefox, we will increase DAUs and retention as Firefox Account data shows that users with 2 synced devices retain longer.

iOS Hypothesis 3:

We believe that by promoting tips and tricks such as customizing settings to our users inside Firefox, we will increase DAUs and retention.

iOS Hypothesis 4:

We believe that by promoting Pocket to our users inside Firefox, we will increase overall DAUs and retention of the Pocket app.

iOS Hypothesis 5:

We believe that by promoting Pocket Stories to our users on Firefox, we will increase overall DAUs and retention of the Firefox browser.

iOS Hypothesis 6:

We believe that introducing a new user to Firefox’s top features and get them using Firefox across devices will increase overall DAUs and retention of the Firefox browser.

Hyp. # Message Name Trigger Targets
H1 Opened New Tab - Focus Promotion Open new tab Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H2 Open Bookmark - Sync Promotion Open bookmark Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H2 Save Bookmark - Sync Promotion Save bookmark Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H2 Save Login and Password - Sync Promotion Save bookmark Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H3 Search URL Area - Default Search Engine Promotion Interacts with search url area Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H3 Search URL Area - Homepage Setting Promotion Interacts with search url area Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H3 Open New Tab - New Tab Setting Promotion Open new tab Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H4 Open Bookmark - Pocket Promotion Open bookmark Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H5 Click on Pocket Stories - Pocket Promotion Click on Pocket Stories Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H6 Onboarding flow: Welcome slide -> FxAccount Sign up slide -> Search Settings banner -> Night Mode banner -> Bookmark banner -> Private browsing banner First run Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw

Android

Android Hypothesis 1:

We believe that by prompting users to change their default browser to Firefox, we will increase DAUs and retention.

Android Hypothesis 2:

We believe that by promoting add-ons to our users inside Firefox, we will increase DAUs and retention.

Android Hypothesis 3:

We believe that by promoting Sync to our users inside Firefox, we will increase DAUs and retention as Firefox Account data shows that users with 2 synced devices retain longer.

Android Hypothesis 4:

We believe that by promoting tips and tricks like the bookmark list or search engine settings to our users inside Firefox, we will increase DAUs and retention.

Android Hypothesis 5:

We believe that by promoting Focus to our privacy conscious users inside Firefox, we will increase installs and DAUs in Focus, and increase retention on both apps (We know that iOS users who have Focus and Firefox installed retain better (Adjust)).

Android Hypothesis 6:

We believe that introducing a new user to Firefox’s top features and get them using Firefox across devices will increase overall DAUs and retention of the Firefox browser.

Android Hypothesis 7:

We believe by not promoting Pocket content to users who have opted out of Pocket in their Top Sites we are respecting the user choices and creating a better user experience.

Android Hypothesis 8:

We believe by knowing when a user opens up the Firefox mobile app we can better measure the success of campaigns that encourage a user to open up the Firefox mobile app.

Hyp. # Message Name Trigger Targets
H1 Launch app - Default Browser promotion App Start Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
H1 Load Bookmark - Default Browser promotion Load bookmark Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
H2 Launch app - Add-ons promotion App start Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw in US, CA, AU, UK, DE, TW, HK, SG
H5 Open New Tab - Focus promotion Open new tab Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
H4 Save Bookmark - Bookmark List Promotion Save bookmark Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
H1 Log in to FxA - Default Browser Promotion Log in to FxA Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
H1 Add Bookmark 5 times - Default Browser Promotion Add Bookmark 5 times Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
H3 Save Bookmark - Sync Promotion Save bookmark Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
H4 Search URL Area - Default Search Engine Promotion Interact with search url area Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
H6 Onboarding flow: Original default onboarding slides -> Reader mode banner -> Bookmark banner -> Default browser banner First run Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw

For more detail about the triggers and deep links in Android, please go to this page[1]