QA/Photon Onboarding Tour Refresh
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
Date | Version | Author | Description |
---|---|---|---|
06/27/2017 | 1.0 | Justin Williams | Created first draft |
08/14/2017 | 1.1 | Justin Williams | Updated Draft |
Contents
Overview
Purpose
When the stub installer runs it detects if you have an old Firefox version installed. If you do you are considered an updated user. If you do not and you have old profile data you are considered a new user. The stub installer will then prompt you with the proper screen (new or updated user) and ask if you want to clean your profile data (this is checked by default). This is done to determine which Photon Onboarding Tour is shown to you once Firefox is installed.
See: for more info
Ownership
Engineering lead Tim Chien
Engineering Team Matt Howell
UX Team: Michael Verdi Bryant Mao
QA Justin Williams
Testing summary
Scope of Testing
In Scope
The Firefox Stub Installer and the Profile Folder.
Out of Scope
Everything outside of the profile folder and the stub installer is out of scope. Fennec is also out of scope.
Requirements for testing
Environments
Windows 7 x64: Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz Windows 10 x64: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz, 3601 Mhz Windows 8.1 x64: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz, 3601 Mhz
Test Strategy
Risk Assessment and Coverage
ID | Description / Threat Description | Covered by Test Objective | Magnitude | Probability | Priority | Impact Score |
---|---|---|---|---|---|---|
RAC-1 | Refresh a profile the user does not want to be refreshed. | TO-1 | 2-Moderate | 1-Unlikely | 3-High | 6 |
RAC-2 | Show the incorrect prompt. | TO-2 | 2-Moderate | 2-Possible | 3-High | 12 |
RAC-3 | Could potentially break the installer. Since the new code all runs before the installation starts, if any of it crashes/hangs/otherwise fails unrecoverably, then no installing can happen. | TO-3 | 3-High | 1-Unlikely | 3-High | 9 |
RAC-4 | Currently no automation test coverage | 3-High | 2-Possible | 2-Medium | 12 |
Values:
- Magnitude: 1- Low , 2-Moderate, 3-High
- Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
- Priority: 1 - Low, 2-Medium, 3-High
Impact Score Breakdown:
- An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
- An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
- An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
Ref | Function | Test Objective | Evaluation Criteria | Test Type | RAC | Owners |
---|---|---|---|---|---|---|
1 | Proper refresh/non-refresh is determined by user | Refresh/Do not refresh a profile the user wants | This action is determined by the user | Manual | RAC-1 | Eng Team |
2 | Verifying that the Installer works correctly by showing the correct prompt when determining if the user is a new user/updated user | Correct prompt is shown. | The installer is not broken and works as expect. New/Updated users are shown the proper prompt | Manual | RAC-2, RAC-3 | Eng Team |
3 | Verify the changes to the installer are not identifiable to the user. | Installer is smooth and user friendly. | The installer runs smooth and installs Firefox as expected | Manual | RAC-3 | Eng Team |
4 | Verify the correct pings are being sent to the server. | Ping information is being sent and received correctly. | Ping data is shown on about:telemetry and STMO | Manual | Eng Team |
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | 03/30/1017 | 11/14/2017 |
Study documentation/specs received from developers | 08/08/2017 | 11/14/2017 |
QA - Test plan creation | 06/27/2017 | 11/14/2017 |
QA - Test cases/Env preparation | 08/15/2017 | 11/14/2017 |
QA - Nightly Testing | 08/18/2017 | 08/24/2017 |
QA - Beta Testing | ||
Release Date | 11/14/2017 | 11/14/2017 |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | Refresh Test Case Creation |
Test case execution | Refresh Test Case Execution |
Bugs management | Refresh Meta Bug |
Trello | Refresh Trello Card |
Status
Overview
Track the dates and build number where feature was merged to Release/Beta
Channel | Date feature was merged to channel | Build number |
---|---|---|
Nightly | 08/11/2017 | 20170811100330 |
Beta | ||
Release |
Testcases
Test Areas
Test Areas | Covered | Details | Reviewed By |
---|---|---|---|
UI | |||
Mouse-only operation | Y | used by majority of users when using the stub installer | Matt Howell |
Display (HiDPI) | Y | used by a large chunk of users | Matt Howell |
Usable with a screen reader | Y | needs to be compatible with different accessibility applications | Matt Howell |
RTL build testing | Y | needs to be compatible with all locales | Matt Howell |
Install/Upgrade | |||
Feature upgrades/downgrades data as expected | Y | feature is dependent on upgrading | Matt Howell |
Affects first-run or onboarding | Y | determines the tour based off of previous update version or profile information | Matt Howell |
Sign off
Criteria
Checklist
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
List of OSes that will be covered by testing
Win 7, 8.1, & 10
- Link for the tests run
- Full Test suite, link to TestRail - Tests Runs and Results Refresh Test Run
Merge to Beta Sign-off
List of OSes that will be covered by testing
Win 7, 8.1, & 10
- Link for the tests run
- Full Test suite
Merge to Release Sign-off
List of OSes that will be covered by testing
Win 7, 8.1, & 10 (x64 & x86)
- Link for the tests results
Checklist
Exit Criteria | Status | Notes/Details |
---|---|---|
Testing Prerequisites (specs, use cases) | [DONE] | |
Testing Infrastructure setup | [DONE] | |
Test Plan Creation | [DONE] | |
Test Cases Creation | [DONE] | |
Automation Coverage | n/a | |
Performance Testing | n/a | |
All Defects Logged | [DONE] | |
Critical/Blockers Fixed and Verified | [DONE] | |
Metrics/Telemetry | n/a | |
Basic/Core functionality Nightly testing | [DONE] | |
QA mid-Nightly Signoff | [DONE] | Email sent |
QA Nightly - Full Testing | [DONE] | |
QA pre-Beta Signoff | [DONE] | Email sent |
QA Beta - Full Testing | [DONE] | |
QA pre-Release Signoff | [DONE] | Email sent |
Approvals Required / Received
The following individuals are required to/have approved this Test Plan:
Name | Title | Department | Approval Date | Method |
---|---|---|---|---|
Lawrence Mandel / RyanVM | QA Reviewer | Product Integrity | ||
Matt Howell | Software Engineer | Engineering | 09/07/2017 | |
Romain Testard | PM | Product Management | 09/07/2017 |