QA/Photon Preferences reorg V2

From MozillaWiki
< QA
Jump to: navigation, search

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
07/11/2017 1.0 Brindusa Tot Created first draft
08/17/2017 1.1 Alin Deac Updated Document

Overview

Purpose

The purpose of this work is to ensure the interface of the about:preference page is more user-friendly, by reorganizing the tabs structure, for a better integrated view.

Scope

This wiki details the testing that will be performed by the project team for the Preferences reorg. V2 project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed

Ownership

Development Team:
Product Manager: Francis Lee
Engineer: Evan Tseng (irc: evanxd)
Engineer: Ricky Chien (irc: rickychien)

Engineering Team - Nightly Testing:
Rares Bologa (irc: RaresB) - PM for QA team
Brindusa Tot (irc: brindusat) - leading QA efforts
Alin Deac (irc: adeac) - QA

Release QA Team - Beta Testing:
Camelia Badau (irc: cbadau) - QA
Bogdan Maris (irc: bogdan_maris) - QA

Testing summary

Scope of Testing

In Scope

This project purpose is to define the new structure of Preferences pages. The focus of this project should be the structure changes of the preferences pages.

The testing effort for this project will be invested in the following areas:

  • functionality: basic and advanced functionality to be verified according to the existing requirements;
  • usability: preferences pages functionality must be easy to use and straightforward;
  • accessibility: preferences pages functionality can be used with High Contrast Theme, Keyboard only or Screen Reader;

In the scope of current testing are following Preferences pages:

  • General
  • Search
  • Privacy & Security
  • Firefox Account

Out of Scope

Testing should NOT focus on visual details of the preferences pages. The scope of Preferences reorg v2 is to change the structure of the preferences pages, and we should focus testing only on this structural changes according to specs.

Requirements for testing

Environments

Full testing will be performed on the following OSes:

  • Windows 10 x64 - on Processor: Intel Core i5 4590s @3.00GHz x4/RAM: 7.7GiB DDR3/Graphics: Gallium 0.4 on AMD CEDAR (DRM 2.43.0 / 4.4.0-92-generic, LLVM 4.0.0)/ Antivirus: Kaspersky Endpoint Security 10 for Windows
  • Windows 7 x64 - on Processor: Intel Core i5 4590s @3.00GHz x4/RAM: 7.7GiB DDR3/Graphics: Gallium 0.4 on AMD CEDAR (DRM 2.43.0 / 4.4.0-92-generic, LLVM 4.0.0)/ Antivirus: Kaspersky Endpoint Security 10 for Windows
  • Mac OS X 10.12.5 - on Processor: 2,8 GHz Intel Core i5/RAM: 8 GB 1867 MHz DDR3/Graphics: Intel Iris Pro Graphics 6200 1536 MB
  • Ubuntu 16.04 - on Processor: Intel Core i5 4590s @3.00GHz x4/RAM: 7.7GiB DDR3/Graphics: Gallium 0.4 on AMD CEDAR (DRM 2.43.0 / 4.4.0-92-generic, LLVM 4.0.0)

Channel dependent settings (configs) and environment setups

Nightly

text

Beta

text

Post Beta / Release

text

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 Pages are not displayed properly TO-1 2-Moderate 1-Unlikely 3-High 6
RAC-1 Search field behavior could be different than the desirable one TO-1 2-Moderate 1-Unlikely 3-High 6
RAC-2 Not all search results are highlighted TO-2 2-Moderate 2-Possible 3-High 12
RAC-2 Preference changes do not work as expected TO-2 2-Moderate 2-Possible 3-High 12
RAC-2 Product appears unprofessional because pages aren’t displayed correctly TO-2 2-Moderate 2-Possible 3-High 12
RAC-3 An indicator may not correctly point the position of the result when it is hiding in a UI component TO-3 2-Moderate 2-Possible 3-High 12

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Test Type RAC Owners
1 General page Updates on the Preferences - General page are displayed correctly Manual RAC-1 Eng Team
2 Search page Updates on the Search- General page are displayed correctly Manual RAC-1 Eng Team
3 Privacy and Security Updates on the Privacy & Security - General page are displayed correctly Manual RAC-1 Eng Team
4 Firefox account Updates on the firefox Account - General page are displayed correctly Manual RAC-1 Eng Team
5 Accessibility Testing for accessibility includes:

- Testing using high Contract themes
- Screen Reader validation
- Keyboard-only

Manual RAC-1 Eng Team

Builds

This section should contain links for builds with the feature -

  • First landed on Nightly in 07/19/2017 - link
  • Links for Beta builds - link

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 07/10/2017
Study documentation/specs received from developers 07/10/2017 07/12/2017
QA - Test plan creation 07/11/2017 07/15/2017
QA - Test cases/Env preparation 07/11/2017 07/18/2017
QA - Nightly Testing 07/19/2017 07/28/2017
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla

Status

Overview

First time landed on Nightly 56 - on 07/19/2017
Track the dates and build number where feature was merged to Release/Beta

References

Testcases

Test Areas

Test Areas Covered Details Reviewed by
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes at least one basic test
Theme (high contrast) Yes
UI
Mouse-only operation Yes
Keyboard-only operation Yes
Display (HiDPI) Yes
Interaction (scroll, zoom) Yes
Usable with a screen reader Yes e.g. with NVDA
Usability and/or discoverability testing Yes Is this feature user friendly
RTL build testing Yes
Help/Support
Help/support interface required No Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) No Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected No
Does sync work across upgrades No
Requires install testing No separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding No Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Does this affect partner builds? Partner build testing  ?? yes/no options, add comment with details about who will lead testing
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration No
Network proxies/autoconfig No
ESR behavior changes No
Locked preferences No
Data Monitoring
Temporary or permanent telemetry monitoring No List of error conditions to monitor
Telemetry correctness testing No
Server integration testing No
Offline and server failure testing No
Load testing No
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? No
Comprehensive API testing No
Permissions No
Testing with existing/popular addons No
Security Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review No
Privilege escalation testing No
Fuzzing No
Web Compatibility depends on the feature
Testing against target sites No
Survey of many sites for compatibility No
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS No
Interaction of this feature with other browser features No

Test suite

Full Test suite - Link to test rail testcases - link

Bug Work

Meta bug: 1365133 - Updated Preferences re-org according to proposal from May 11th

Logged bugs ( blocking 1365133 )
Full Query
ID Priority Component Assigned to Summary Status Target milestone
941279 -- Preferences Redirect blocking is useless without redirect info UNCONFIRMED ---
1370190 P1 Preferences Ricky Chien [:rickychien] (inactive) Updated Preferences tracking protection section based on re-org v2 VERIFIED Firefox 56
1382057 -- Preferences Intermittent browser/components/preferences/in-content-new/tests/browser_applications_selection.js | Uncaught exception - Condition timed out: () => RESOLVED ---
1382059 -- Preferences Intermittent browser/components/preferences/in-content-new/tests/browser_change_app_handler.js | Uncaught exception - Condition timed out: () => win.document.getAnonymousElementByAttribute(ourItem, "class", "actionsMenu") RESOLVED ---
1382134 P1 Preferences Evan Tseng [:evanxd] Search result shows the wrong section titles VERIFIED Firefox 56
1382170 P3 Preferences Tim Guan-tin Chien [:timdream] (please needinfo) New Preferences is too slow RESOLVED Firefox 56
1382660 P1 Preferences Evan Tseng [:evanxd] Active area for some checkboxes has changed since Preferences re-org VERIFIED Firefox 56
1383702 P1 Preferences Evan Tseng [:evanxd] General "Home page" it's not displayed with title case RESOLVED ---
1383709 -- Preferences (Away until Sept 26) Jared Wein [:jaws] (please needinfo? me) Perma failure when 56 merges to beta in browser_notification_open_settings.js | Test timed out - RESOLVED Firefox 56
1383719 P1 Preferences Evan Tseng [:evanxd] General - Updates section - no changes are visible in the latest build VERIFIED Firefox 56
1383741 P1 Preferences Evan Tseng [:evanxd] Perma failure when 56 merge to beta in browser_experiments.js | Test timed out - RESOLVED Firefox 56
1383783 -- Preferences Privacy & Security tab - Tracking Protection - no changes are visible in the latest build RESOLVED ---
1383788 -- Preferences Privacy & Security tab - Data Collection and Use - Checkbox line missing (inconsistency with reorg doc) RESOLVED ---
1383792 P1 Preferences Evan Tseng [:evanxd] The "Manage Account" string on Firefox Account should be "Manage account" VERIFIED Firefox 56
1384068 P1 Preferences Evan Tseng [:evanxd] Wrong Change search settings redirection from about:newtab and about:home after preferences v2 reorg RESOLVED Firefox 56
1384071 -- Preferences General - Browsing section has 5 options instead of 4 RESOLVED ---
1384087 P1 Preferences Ricky Chien [:rickychien] (inactive) Privacy & Security - Address bar - first option should be Browsing history VERIFIED Firefox 56
1384097 P1 Preferences Ricky Chien [:rickychien] (inactive) "Change preferences for search engine suggestions" link should get user to "about:preferences#search" page VERIFIED Firefox 56
1384177 -- Preferences "Do not disturb me" check box should not be present in Permission section RESOLVED ---
1384402 P1 Preferences Ricky Chien [:rickychien] (inactive) Fix broken about:preferences link after reorg v2 RESOLVED Firefox 56
1384470 P1 Preferences Evan Tseng [:evanxd] General " Files and Applications" inconsistency between spec and latest build for the word "Web" VERIFIED Firefox 56
1384490 -- Preferences V2 Preferences General issue - Inconsistency regarding the position of the text in some areas RESOLVED ---
1384523 -- Preferences For consistency, on Preferences pages should use only "sites" and not both "websites" and "sites" RESOLVED ---
1384576 P1 Preferences Ricky Chien [:rickychien] (inactive) Ellipsis are misses on some buttons on Preferences pages VERIFIED Firefox 56
1384812 P1 Preferences Evan Tseng [:evanxd] The Learn more link of the automaticallySubmitCrashesBox checkbox should not align right side and the defaultEngineGroup checkbox should not be full width after Bug 1382660 is landed. VERIFIED Firefox 56
1384813 P1 Preferences Evan Tseng [:evanxd] Going to about:preferences#privacy-reports, there is no header for the content. VERIFIED Firefox 56
1386560 P1 Preferences Evan Tseng [:evanxd] The Firefox Account header should not be highlighted on search result page. VERIFIED Firefox 57
1388745 P1 Preferences Ricky Chien [:rickychien] (inactive) Some search results are not highlighted VERIFIED Firefox 57
1394746 P3 Preferences Many of the preferences sub-dialogs still come with a "Help" button pointing to old structure SUMO documents REOPENED ---
1400595 P1 Preferences Ricky Chien [:rickychien] (inactive) Unable (again) to scroll Preferences if search engine table is not scrollable VERIFIED Firefox 57
1410488 -- Preferences Preference : Preventing sites from redirecting or reloading the page RESOLVED ---
1420129 -- Search Florian Quèze [:florian] Clicking the 'Settings page' link on about:searchreset should go directly to the Search pref pane VERIFIED Firefox 59
1440082 -- General Mike Conley (:mconley) (:⚙️) Revert erroneous string change in about:preferences RESOLVED Firefox 60

33 Total; 2 Open (6.06%); 16 Resolved (48.48%); 15 Verified (45.45%);


Bug fix verification
Full Query
ID Priority Component Assigned to Summary Status Resolution Target milestone
1370190 P1 Preferences Ricky Chien [:rickychien] (inactive) Updated Preferences tracking protection section based on re-org v2 VERIFIED FIXED Firefox 56
1382134 P1 Preferences Evan Tseng [:evanxd] Search result shows the wrong section titles VERIFIED FIXED Firefox 56
1382170 P3 Preferences Tim Guan-tin Chien [:timdream] (please needinfo) New Preferences is too slow RESOLVED FIXED Firefox 56
1382660 P1 Preferences Evan Tseng [:evanxd] Active area for some checkboxes has changed since Preferences re-org VERIFIED FIXED Firefox 56
1383709 -- Preferences (Away until Sept 26) Jared Wein [:jaws] (please needinfo? me) Perma failure when 56 merges to beta in browser_notification_open_settings.js | Test timed out - RESOLVED FIXED Firefox 56
1383719 P1 Preferences Evan Tseng [:evanxd] General - Updates section - no changes are visible in the latest build VERIFIED FIXED Firefox 56
1383741 P1 Preferences Evan Tseng [:evanxd] Perma failure when 56 merge to beta in browser_experiments.js | Test timed out - RESOLVED FIXED Firefox 56
1383792 P1 Preferences Evan Tseng [:evanxd] The "Manage Account" string on Firefox Account should be "Manage account" VERIFIED FIXED Firefox 56
1384087 P1 Preferences Ricky Chien [:rickychien] (inactive) Privacy & Security - Address bar - first option should be Browsing history VERIFIED FIXED Firefox 56
1384097 P1 Preferences Ricky Chien [:rickychien] (inactive) "Change preferences for search engine suggestions" link should get user to "about:preferences#search" page VERIFIED FIXED Firefox 56
1384402 P1 Preferences Ricky Chien [:rickychien] (inactive) Fix broken about:preferences link after reorg v2 RESOLVED FIXED Firefox 56
1384470 P1 Preferences Evan Tseng [:evanxd] General " Files and Applications" inconsistency between spec and latest build for the word "Web" VERIFIED FIXED Firefox 56
1384576 P1 Preferences Ricky Chien [:rickychien] (inactive) Ellipsis are misses on some buttons on Preferences pages VERIFIED FIXED Firefox 56
1384812 P1 Preferences Evan Tseng [:evanxd] The Learn more link of the automaticallySubmitCrashesBox checkbox should not align right side and the defaultEngineGroup checkbox should not be full width after Bug 1382660 is landed. VERIFIED FIXED Firefox 56
1384813 P1 Preferences Evan Tseng [:evanxd] Going to about:preferences#privacy-reports, there is no header for the content. VERIFIED FIXED Firefox 56
1386560 P1 Preferences Evan Tseng [:evanxd] The Firefox Account header should not be highlighted on search result page. VERIFIED FIXED Firefox 57
1388745 P1 Preferences Ricky Chien [:rickychien] (inactive) Some search results are not highlighted VERIFIED FIXED Firefox 57
1400595 P1 Preferences Ricky Chien [:rickychien] (inactive) Unable (again) to scroll Preferences if search engine table is not scrollable VERIFIED FIXED Firefox 57
1420129 -- Search Florian Quèze [:florian] Clicking the 'Settings page' link on about:searchreset should go directly to the Search pref pane VERIFIED FIXED Firefox 59
1440082 -- General Mike Conley (:mconley) (:⚙️) Revert erroneous string change in about:preferences RESOLVED FIXED Firefox 60

20 Total; 0 Open (0%); 5 Resolved (25%); 15 Verified (75%);

Sign off

Criteria

Checklist

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite, link to TestRail - Tests Runs and Results link
    • Daily Smoke, if needed/available
    • Regression Test suite, if needed/available


Merge to Beta Sign-off
List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Pre-Release Sign-Off (2017-09-08)

List of OSes that will be covered by testing

    • Windows 7 x64
    • Windows 10 x64
    • macOS 10.12
    • Ubuntu 16.04 x64
  • Link for the tests run
    • Full Test suite - Tests Runs and Results link


Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Automation Coverage
Performance Testing
All Defects Logged [DONE]
Critical/Blockers Fixed and Verified [DONE]
Metrics/Telemetry
Basic/Core functionality Nightly testing
QA mid-Nightly Signoff
QA Nightly - Full Testing
QA pre-Beta Signoff [DONE]
QA Beta - Full Testing [DONE]
QA pre-Release Signoff [DONE]

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
RyanVM QA Manager Product Integrity
Evan Tseng Software Engineer Engineering Email
Francis Lee EPM Product Management Email