QA/Fennec/Feature-focused first-run experiment

From MozillaWiki
< QA‎ | Fennec
Jump to: navigation, search

Overview

Purpose

The purpose of this wiki is to serve as a general test plan for verifying that users are introduced to features that are unique and sticky

Ownership

Overall Status

[LANDED] Nightly 49: 2016-05-18

[LANDED] Aurora 49: 2016-06-08

[LANDED] Beta 49: 2016-08-03

Testing Summary

Scope of testing

The testing will be focus on:

  • verifying that first-run is about features that are unique and sticky

Environments

Testing will be performed on following devices:

  • Phones
    • 4.x
    • 5.x
    • 6.x
  • Tablet
    • 4.x
    • 5.x
    • 6.x

Specifications

  • Our belief is, if we can optimize the onboarding flow, specifically the first run experience, we will increase retention. In order to optimize the first run experience, we've ran a previous test (https://mozilla.aha.io/features/FENN-298) to validate the usage of multiple slides. The previous test was focused on testing a rather easy, "warm and fuzzy" story about Firefox and its core features. It's time now to validate which slides will perform the best and therefore we want to now compare this version with a stronger feature-focused slides where being a differentiatior to our competitors will be the key message. All feature are presented independently from each other and don't form a specific story arc (as opposed to the previous test).
    • Tab queue
Screenshot 2016-06-13-18-27-25.png
  • Reader View
Screenshot 2016-06-13-18-27-30.png
  • Sign up for Sync
Screenshot 2016-06-13-18-27-36.png

User stories

  • As a user, I want to be introduced to features that are unique and sticky for me to come back to the app and continue to use the browser as well as the features

Acceptance Criteria

  • No discernible different in first-run browser start-up time

KPI/Metrics:

  • Increased retention [requires A/B testing]
  • Increased engagement

Testing details

Test Cases

Testing days

Date: 2016-06-12

  • Created Test Plan

Date: 2016-06-16

  • Exploratory on latest Aurora with Motorola Razr (Android 4.4.4), LG G4 (Android 5.1) and Nexus 5 (Android 6.0.1)
    • Tab queue Aurora first-run
Screenshot 2016-06-16-12-27-01.png
Screenshot 2016-06-16-12-31-05.png
  • Reader View Aurora first-run
Screenshot 2016-06-16-12-27-13.png
  • Sign up for Sync Aurora first-run
Screenshot 2016-06-16-12-27-27.png
  • Verified bugs:
    • bug 1268647 - Implement new set of feature-focused firstrun panels
    • bug 1276374 - Remove firstrun Content Notifications slide

Date: 2016-07-12

  • Exploratory testing on latest Aurora and Nightly with:
    • LG G4 (Android 5.1)
    • Nexus 5 (Android 6.0.1)
    • Prestigio Grace X5 (Android 4.4.2)
    • Asus Transformer Pad (Android 4.2.1)
    • Asus ZenPad 8 (Android 5.0.2)

Date: 2016-08-08

  • Exploratory testing on Firefox 49 Beta 1 with LG G4 (Android 5.1) and Nexus 5 (Android 6.0.1)
    • Tab queue - first-run
Screenshot 2016-08-08-10-37-47.png
Screenshot 2016-08-08-10-37-53.png
  • Reader View - first-run
Screenshot 2016-08-08-10-38-03.png
  • Sign up for Sync - first-run
Screenshot 2016-08-08-10-38-08.png

Date: 2016-09-19

  • Exploratory testing on Firefox 49 RC with Motorola Razr (Android 4.4.4), Asus ZenPad 8 (Android 5.1) and Nexus 5 (Android 6.0.1)

Bug Work

  • Meta bug
    • bug 1268641 - (feature-onboarding) Feature-based onboarding
Full Query
ID Priority Summary Status Assigned to
1268647 -- Implement new set of feature-focused firstrun panels VERIFIED Chenxia Liu [:liuche]
1270993 -- Confirm that we have a sync probe for "have sync account" RESOLVED
1276374 -- Remove firstrun Content Notifications slide RESOLVED Chenxia Liu [:liuche]
1282968 -- Redash android_events_v1.experiments missing onboarding3-a RESOLVED Chenxia Liu [:liuche]

4 Total; 0 Open (0%); 3 Resolved (75%); 1 Verified (25%);


Still issues

Bug fixed

[RESOLVED FIXED] Bug 1268647 - Implement new set of feature-focused firstrun panels
[RESOLVED FIXED] Bug 1276374 - Remove firstrun Content Notifications slide
[RESOLVED FIXED] Bug 1270993 - Confirm that we have a sync probe for "have sync account"

Signoff

Criteria

  • All test cases should be executed
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed

Results

Nightly testing

  • There were no blockers found while testing this feature.
  • All bugs are fixed and verified

Aurora testing

  • There were no blockers found while testing this feature.
  • All bugs are fixed and verified

Beta testing

  • There were no blockers found while testing this feature.
  • All bugs are fixed and verified