QA/Photon Visual Redesign
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
| Date | Version | Author | Description |
|---|---|---|---|
| 05/5/2017 | 1.0 | Brindusa Tot | Created first draft |
| 08/18/2017 | 1.1 | Ovidiu Boca | Edits to draft |
Contents
Overview
Purpose
The goal of this feature is to change the Firefox UI in following areas:
- Themes
- Tabs
- Primary Toolbar
- Bookmarks Toolbar
- Iconography
- History and Bookmarks sidebars
Scope
The scope of testing will be covering the UI section (Themes, Tabs, Toolbars, Iconography) implementation according to specifications.
Ownership
Engineering lead Justin Dolske
Engineering Team Dao Gottwald Nihanth Subramanya Johann Hofmann Dale Harvey
UX Team: Stephen Horlander Michelle Heubusch
QA Brindusa Tot Ovidiu Boca
Testing summary
Scope of Testing
In Scope
This part of Photon project will redesign all UI components the browser with the new Photon style. In the scope of current testing are following areas in regards to the new look:
- General browser Overview(Normal, Compact, Touch)
- Different themes with their font colors
- Different OS themes: Classic, Aero Basic, Aero Glass
- Title bar, Menu bar, Tab bar
- Url bar with all the buttons, Search bar
- Bookmarks toolbar
- New icons/images within Photon style
- Customization page
- Notification bars (Default, Warning, Stop)
Out of Scope
Mobile devices will not be tested.
Requirements for testing
Environments
Testing will be performed on the following OSes:
- Windows 10 x64 Processor AMD FX (TM) - 8320 Eight-Core Processor 3.50 GHz GPU - ATI Radeon 3000(on board) Antivirus: KAV 10.3.0
- Windows 10 with touch x64 Surface Pro 4 Intel Core i5 6300 CPU 2.4 GHz 2.5 GHz GPU - Intel (R)HD Graphics 520
- Windows 7 x86 Processor AMD FX-8320 3.5 GHz GPU - ATI Radeon 3000(on board) Antivirus: KAV 10.3.0
- Windows 8.1 x64 Processor AMD FX-8320 3.5 GHz GPU - ATI Radeon 3000(on board) Antivirus: KAV 10.3.0
- Mac OS X 10.12 - Processor 2.7 GHz Intel Core i5 GPU - Intel Iris Pro 1536 MB
- Ubuntu 16.04 x64 - Processor Intel® Core™ i5-4590S CPU @ 3.00GHz × 4 GPU - Gallium 0.4 on AMD CEDAR (DRM 2.43.0 / 4.4.0-92-generic, LLVM 4.0.0)
Channel dependent settings (configs) and environment setups
Nightly
text
Beta
text
Post Beta / Release
text
Test Strategy
Risk Assessment and Coverage
| ID | Description / Threat Description | Covered by Test Objective | Magnitude | Probability | Priority | Impact Score |
|---|---|---|---|---|---|---|
| RAC-1 | Over all browser look | TO-1 | 3-High | 2-Possible | 3-High | 18 |
| RAC-2 | Different themes does not match the spec | TO-1 | 3-High | 2-Possible | 3-High | 18 |
| RAC-3 | Different bars(title,Menu, tab bars) look does not match the spec | TO-2 | 3-High | 2-Possible | 3-High | 18 |
| RAC-3 | Buttons/Icons look and position | TO-2 | 2-Moderate | 2-Possible | 3-High | 12 |
Values:
- Magnitude: 1- Low , 2-Moderate, 3-High
- Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
- Priority: 1 - Low, 2-Medium, 3-High
Impact Score Breakdown:
- An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
- An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
- An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
| Ref | Function | Test Objective | Test Type | RAC | Owners |
|---|---|---|---|---|---|
| 1 | Navigation | UI components are correctly displayed while navigating on different pages | Manual | RAC-1, RAC-2, RAC-3, RAC -4 | Eng Team |
| 2 | Browser Themes updates | UI components are correctly displayed when new theme is applied | Manual | RAC-1, RAC-2, RAC-3, RAC -4 | Eng Team |
| 3 | Title bar display | Title bar is correctly display based on specs | Manual | RAC-1, RAC-2, RAC-3, RAC -4 | Eng Team |
| 4 | Menu bar display | Menu bar is correctly display based on specs with all UI elements | Manual | RAC-1, RAC-2, RAC-3, RAC -4 | Eng Team |
| 5 | Tab bar display | Tab bar is correctly display based on specs with all UI elements | Manual | RAC-1, RAC-2, RAC-3, RAC -4 | Eng Team |
| 6 | URL bar display | Url bar is correctly display based on specs with all UI elements | Manual | RAC-1, RAC-2, RAC-3, RAC -4 | Eng Team |
| 7 | Search bar display | Search bar is correctly display based on specs with all UI elements | Manual | RAC-1, RAC-2, RAC-3, RAC -4 | Eng Team |
| 8 | Bookmarks toolbar display | Bookmarks toolbar is correctly display based on specs with all UI elements | Manual | RAC-1, RAC-2, RAC-3, RAC -4 | Eng Team |
| 9 | High Contrast themes on Windows | Testing browser with windows high-contrast themes | Manual/ | RAC-1, RAC-2, RAC-3, RAC -4 | Eng Team |
Builds
This section should contain links for builds with the feature -
- Links for Nightly builds - link
- Links for Beta builds
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
| Project phase | Start Date | End Date |
|---|---|---|
| Start project | 4/13/2017 | |
| Study documentation/specs received from developers | 4/15/2017 | |
| QA - Test plan creation | 5/5/2017 | |
| QA - Test cases/Env preparation | 10/7/2017 | |
| QA - Nightly Testing | ||
| QA - Beta Testing | ||
| Release Date |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | TestRail/ Google docs |
| Test case execution | TestRail |
| Bugs management | Bugzilla |
Status
Overview
* Nightly 56 - bug verification * Nightly 57 - bug verification * For mid-Nightly sign-offs: we will continue with bug verification. This sign-off is only going to consist of a testing/bug report, instead of a formal sign-off - our focus would mainly be on bug verification (plan is to hit the 80% of the fixed bugs)
References
Bug 1325171 - (photon-visual) [meta] Photon - Firefox visual refresh (Theme)]
Testcases
Test Areas
| Test Areas | Covered | Details |
|---|---|---|
| Private Window | Yes | The UI respect the specifications |
| Multi-Process Enabled | No | |
| Multi-process Disabled | No | |
| Theme (high contrast) | Yes | The Themes have the same colors like the one presented in the spec. |
| UI | ||
| Mouse-only operation | Yes | Mouse over buttons works as desired. |
| Keyboard-only operation | No | |
| Display (HiDPI) | Yes | There are no rendering issues on HiDPI. |
| Interaction (scroll, zoom) | Yes | Scroll or zoom doesn't affect the user interface of the browser. |
| Usable with a screen reader | No | |
| Usability and/or discoverability testing | Yes | Is this feature user friendly |
| RTL build testing | Yes |
Test suite
Full Test suite - Link to test rail - in work - will be used for pre-Beta sign-off testing
Bug Work
Meta bug: 1325171 - (photon-visual) [meta Photon - Firefox visual refresh (Theme)]
Logged bugs ( blocking 1325171 )
141 Total; 12 Open (8.51%); 51 Resolved (36.17%); 78 Verified (55.32%);
Bug fix verification
107 Total; 0 Open (0%); 30 Resolved (28.04%); 77 Verified (71.96%);
Sign off
Criteria
Checklist
- All test cases should be executed
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
- Link for the tests run
- Full Test suite, link to TestRail - Tests Runs and Results link
- Daily Smoke, if needed/available
- Regression Test suite, if needed/available
Merge to Beta Sign-off
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite
Checklist
| Exit Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | [DONE] | |
| Testing Infrastructure setup | ||
| Test Plan Creation | IN PROGRESS | |
| Test Cases Creation | IN PROGRESS | |
| Automation Coverage | ||
| Performance Testing | ||
| All Defects Logged | ||
| Critical/Blockers Fixed and Verified | IN PROGRESS | |
| Metrics/Telemetry | ||
| Basic/Core functionality Nightly testing | ||
| QA mid-Nightly Signoff | Email to be sent | |
| QA Nightly - Full Testing | ||
| QA pre-Beta Signoff | Email to be sent | |
| QA Beta - Full Testing | ||
| QA pre-Release Signoff | Email to be sent |
Approvals Required / Received
The following individuals are required to/have approved this Test Plan:
| Name | Title | Department | Approval Date | Method |
|---|---|---|---|---|
| RyanVM | QA Reviewer | Product Integrity | Date | |
| Justin Dolske | Software Engineer | Engineering | Date | |
| Marco Mucci | EPM | Product Management | Date |
