QA/Photon Onboarding Tour Refresh

From MozillaWiki
< QA
Jump to: navigation, search

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
06/27/2017 1.0 Justin Williams Created first draft
08/14/2017 1.1 Justin Williams Updated Draft



When the stub installer runs it detects if you have an old Firefox version installed. If you do you are considered an updated user. If you do not and you have old profile data you are considered a new user. The stub installer will then prompt you with the proper screen (new or updated user) and ask if you want to clean your profile data (this is checked by default). This is done to determine which Photon Onboarding Tour is shown to you once Firefox is installed.

See: for more info


Engineering lead
  Tim Chien
Engineering Team
  Matt Howell
UX Team: 
  Michael Verdi
  Bryant Mao
  Justin Williams

Testing summary

Scope of Testing

In Scope

The Firefox Stub Installer and the Profile Folder.

Out of Scope

Everything outside of the profile folder and the stub installer is out of scope. Fennec is also out of scope.

Requirements for testing


Windows 7 x64: Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz
Windows 10 x64: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz, 3601 Mhz
Windows 8.1 x64: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz, 3601 Mhz

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 Refresh a profile the user does not want to be refreshed. TO-1 2-Moderate 1-Unlikely 3-High 6
RAC-2 Show the incorrect prompt. TO-2 2-Moderate 2-Possible 3-High 12
RAC-3 Could potentially break the installer. Since the new code all runs before the installation starts, if any of it crashes/hangs/otherwise fails unrecoverably, then no installing can happen. TO-3 3-High 1-Unlikely 3-High 9
RAC-4 Currently no automation test coverage 3-High 2-Possible 2-Medium 12


  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Evaluation Criteria Test Type RAC Owners
1 Proper refresh/non-refresh is determined by user Refresh/Do not refresh a profile the user wants This action is determined by the user Manual RAC-1 Eng Team
2 Verifying that the Installer works correctly by showing the correct prompt when determining if the user is a new user/updated user Correct prompt is shown. The installer is not broken and works as expect. New/Updated users are shown the proper prompt Manual RAC-2, RAC-3 Eng Team
3 Verify the changes to the installer are not identifiable to the user. Installer is smooth and user friendly. The installer runs smooth and installs Firefox as expected Manual RAC-3 Eng Team
4 Verify the correct pings are being sent to the server. Ping information is being sent and received correctly. Ping data is shown on about:telemetry and STMO Manual Eng Team

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 03/30/1017 11/14/2017
Study documentation/specs received from developers 08/08/2017 11/14/2017
QA - Test plan creation 06/27/2017 11/14/2017
QA - Test cases/Env preparation 08/15/2017 11/14/2017
QA - Nightly Testing 08/18/2017 08/24/2017
QA - Beta Testing
Release Date 11/14/2017 11/14/2017

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation Refresh Test Case Creation
Test case execution Refresh Test Case Execution
Bugs management Refresh Meta Bug
Trello Refresh Trello Card



Track the dates and build number where feature was merged to Release/Beta

Channel Date feature was merged to channel Build number
Nightly 08/11/2017 20170811100330


Test Areas

Test Areas Covered Details Reviewed By
Mouse-only operation Y used by majority of users when using the stub installer Matt Howell
Display (HiDPI) Y used by a large chunk of users Matt Howell
Usable with a screen reader Y needs to be compatible with different accessibility applications Matt Howell
RTL build testing Y needs to be compatible with all locales Matt Howell
Feature upgrades/downgrades data as expected Y feature is dependent on upgrading Matt Howell
Affects first-run or onboarding Y determines the tour based off of previous update version or profile information Matt Howell

Sign off



  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)


Nightly testing

List of OSes that will be covered by testing
Win 7, 8.1, & 10

  • Link for the tests run

Merge to Beta Sign-off
List of OSes that will be covered by testing
Win 7, 8.1, & 10

  • Link for the tests run
    • Full Test suite

Merge to Release Sign-off
List of OSes that will be covered by testing
Win 7, 8.1, & 10 (x64 & x86)


Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Automation Coverage n/a
Performance Testing n/a
All Defects Logged [DONE]
Critical/Blockers Fixed and Verified [DONE]
Metrics/Telemetry n/a
Basic/Core functionality Nightly testing [DONE]
QA mid-Nightly Signoff [DONE] Email sent
QA Nightly - Full Testing [DONE]
QA pre-Beta Signoff [DONE] Email sent
QA Beta - Full Testing [DONE]
QA pre-Release Signoff [DONE] Email sent

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
Lawrence Mandel / RyanVM QA Reviewer Product Integrity Email
Matt Howell Software Engineer Engineering 09/07/2017 Email
Romain Testard PM Product Management 09/07/2017 Email