QA/Photon Onboarding Tour Refresh
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
| Date | Version | Author | Description |
|---|---|---|---|
| 06/27/2017 | 1.0 | Justin Williams | Created first draft |
| 08/14/2017 | 1.1 | Justin Williams | Updated Draft |
Overview
Purpose
Determines when the last time the user has used Firefox by looking at the profiles data to provide the correct tour.
Scope
The Firefox Stub Installer and the Profile Folder.
Ownership
Engineering lead Justin Dolske
Engineering Team Fred Lin Fischer Liu Rex Lee
UX Team: Michael Verdi Bryant Mao
QA Justin Williams
Testing summary
Scope of Testing
In Scope
The Firefox Stub Installer and the Profile Folder.
Out of Scope
Everything not coupled with the Tour Refresh is out of scope.
Requirements for testing
Environments
Windows 7 & 10 Ubuntu 17.04 Mac 10.12
Test Strategy
Risk Assessment and Coverage
| ID | Description / Threat Description | Covered by Test Objective | Magnitude | Probability | Priority | Impact Score |
|---|---|---|---|---|---|---|
| RAC-1 | Refresh a profile the user does not want to be refreshed. | TO-1 | 2-Moderate | 1-Unlikely | 3-High | 6 |
| RAC-2 | Show the incorrect prompt. | TO-2 | 2-Moderate | 2-Possible | 3-High | 12 |
| RAC-3 | Could potentially break the installer. | TO-3 | 3-High | 1-Unlikely | 3-High | 9 |
Values:
- Magnitude: 1- Low , 2-Moderate, 3-High
- Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
- Priority: 1 - Low, 2-Medium, 3-High
Impact Score Breakdown:
- An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
- An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
- An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
| Ref | Function | Test Objective | Evaluation Criteria | Test Type | RAC | Owners |
|---|---|---|---|---|---|---|
| 1 | Name of the feature or sub-function being tested | The objective the test is trying to demonstrate | The criteria that will be evaluated to demonstrate the test is successful | Manual/ Automation/ Regression/ Performance/ Usability/ Security/ Telemetry | RAC-1, RAC-2, RAC-3 | Eng Team |
| 2 | Repeat for each feature/sub-function | |||||
| 3 |
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
| Project phase | Start Date | End Date |
|---|---|---|
| Start project | 03/30/1017 | 11/14/2017 |
| Study documentation/specs received from developers | 08/08/2017 | 11/14/2017 |
| QA - Test plan creation | 06/27/2017 | 11/14/2017 |
| QA - Test cases/Env preparation | 08/15/2017 | 11/14/2017 |
| QA - Nightly Testing | 08/18/2017 | 08/24/2017 |
| QA - Beta Testing | ||
| Release Date | 11/14/2017 | 11/14/2017 |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | Refresh Test Case Creation |
| Test case execution | Refresh Test Case Execution |
| Bugs management | Refresh Meta Bug |
Status
Overview
Track the dates and build number where feature was merged to Release/Beta
| Channel | Date feature was merged to channel | Build number |
|---|---|---|
| Nightly | 08/11/2017 | 20170811100330 |
| Beta | ||
| Release |
References
Testcases
Test Areas
| Test Areas | Covered | Details | Reviewed By |
|---|---|---|---|
| UI | |||
| Mouse-only operation | Y | used by majority of users when using the stub installer | Matt Howell |
| Display (HiDPI) | Y | Matt Howell | |
| Usable with a screen reader | Y | needs to be compatible with different accessibility applications | Matt Howell |
| RTL build testing | Y | needs to be compatible with all locales | Matt Howell |
| Install/Upgrade | |||
| Feature upgrades/downgrades data as expected | Y | feature is dependent on upgrading | Matt Howell |
| Affects first-run or onboarding | Y | determines the tour based off of previous update version or profile information | Matt Howell |
Test suite
Full Test suite - Refresh Test Suite
Bug Work
Meta bugs: 1268708 - bug summary
Logged bugs ( blocking 1268708 )
12 Total; 0 Open (0%); 7 Resolved (58.33%); 5 Verified (41.67%);
Bug fix verification
| ID | Priority | Component | Assigned to | Summary | Status | Resolution | Target milestone |
|---|---|---|---|---|---|---|---|
| 1353194 | P1 | Add-ons Manager | Andrew Swan [:aswan] | Streamline addon update check at startup for 57 | VERIFIED | FIXED | mozilla57 |
| 1357666 | P1 | General | Verdi [:verdi] Best to slack me | [UX] The UX specs for the returning user experiences | VERIFIED | FIXED | Firefox 55 |
| 1369255 | P1 | Installer | Molly Howell (she/her) (mostly inactive) | Profile clean-up option in stub installer | VERIFIED | FIXED | Firefox 57 |
| 1376558 | P1 | Migration | Fischer [:Fischer] | Should not migrate session for Firefox auto-refresh | VERIFIED | FIXED | Firefox 56 |
| 1381650 | P1 | Installer | gasolin@mozilla.com | Update bottom tag line on the stub installer | VERIFIED | FIXED | Firefox 56 |
5 Total; 0 Open (0%); 0 Resolved (0%); 5 Verified (100%);
Sign off
Criteria
Checklist
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
List of OSes that will be covered by testing
Win 7 & 10, Ubuntu 16.04, and Mac 10.12
- Link for the tests run
- Full Test suite, link to TestRail - Tests Runs and Results Refresh Test Run
Merge to Beta Sign-off
List of OSes that will be covered by testing
Win 7 & 10, Ubuntu 16.04, and Mac 10.12
- Link for the tests run
- Full Test suite
Checklist
| Exit Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | [DONE] | |
| Testing Infrastructure setup | [DONE] | |
| Test Plan Creation | [DONE] | |
| Test Cases Creation | ||
| Automation Coverage | ||
| Performance Testing | ||
| All Defects Logged | ||
| Critical/Blockers Fixed and Verified | ||
| Metrics/Telemetry | ||
| Basic/Core functionality Nightly testing | ||
| QA mid-Nightly Signoff | Email to be sent | |
| QA Nightly - Full Testing | ||
| QA pre-Beta Signoff | Email to be sent | |
| QA Beta - Full Testing | ||
| QA pre-Release Signoff | Email to be sent |