QA/APZ Keyboard

From MozillaWiki
< QA
Jump to: navigation, search

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
Milan Sreckovic Sr. Engineer Manager Engineering Date Email
Botond Ballo Engineer Engineering Date Email
Thomas Elin EPM Product Management Date Email


Revision History


Date Version Author Description
07/31/2017 1.0 Petruta Rasa Created first draft
08/25/2017 1.1 Petruta Rasa Added mid-Nightly test results

Overview

This test plan will track both APZ keyboard and APZ Autoscrolling features.

Purpose

The goal of these features is to offer smoother and fastest scroll behavior using both keyboard keys and middle mouse button.

Scope

  • Testing will focus on
Comparing APZ scrolling with main-thread scrolling
Functionality of different scrolling types across all platforms
Stability of the browser with these features
  • How testing will be performed
Manual

Ownership

• APZ Keyboard Developers:

• APZ Autoscrolling Developers:

• QA

Testing summary

Scope of Testing

In Scope

These features add support for keyboard scrolling and auto-scrolling with APZ. Scrolling with the keyboard will be smoother and the user experience with the browser more fluently. We will focus our testing efforts on long web pages, pages with plugins, google docs, popular web pages.

Out of Scope

Following areas/features are considered out of scope and will not be considered as testing zones to be handled in this test plan:

  1. Scrolling with mouse
  2. Scrollbar drag
  3. High contrast

Requirements for testing

Environments

Full testing will be performed on the following OSes:

  • Windows 7
  • Windows 10
  • Mac OS X 10.11
  • Ubuntu 16.04

Other configuration/environmental setup needed: APZ Keyboard: On by default in Firefox Nightly, "apz.keyboard.enabled" pref can be used to turn it on and off. This can be checked in about:support, next to "Asynchronous Pan/Zoom" under Graphics, with "keyboard enabled".

APZ Autoscrolling: On by default in Firefox Nightly (since 2017-08-18), "apz.autoscrolling.enabled" pref can be used to turn it off

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 Different OSs may have different behaviors with keyboard scrolling. TO-1 2-Moderate 2-Possible 3-High 12
RAC-2 Web pages like Google sheets or pages with SVG may be affected TO-2 3-High 3-Almost Certain 3-High 27
RAC-3 Pages with plugins may cause scrolling issues TO-3 3-High 2-Possible 3-High 18
RAC-4 A different browser zoom than 100% may reveal some issues TO-4 3-High 2-Possible 3-High 18
RAC-5 Fast scrolling with Home / End keys can cause issues TO-5 3-High 2-Possible 3-High 18
RAC-6 Possible regressions TO-6 3-High 2-Possible 3-High 18

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

Ref Function Test Objective Test Type RAC Owners
TO-1 Different OSs have different behaviors with keyboard scrolling To verify that on every OS the behavior is the expected one Manual RAC-1 QA Team
TO-2 Scrolling in Google sheets Issues related to scrolling may appear since Google Sheets are convoluted pages Manual RAC-2 QA Team
TO-3 Scrolling pages with plugins Pages with plugins are not affected by APZ Manual RAC-3 QA Team
TO-4 Use different browser zoom levels Scrolling is not affected by the browser zoom level Manual RAC-4 QA Team
TO-5 Fast scrolling with Home / End keys long web pages No errors can be seen Manual RAC-5 QA Team
TO-6 Keyboard scrolling is improved Scrolling is smoother and CPU is not affected Manual RAC-6 QA Team

Builds

  • Links for 57.0a1 Nightly builds link
  • Links for Firefox 56 Beta 4 builds link
    • Preferences must be set to true in this build

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 2017-07-28 2017-11-14
Study documentation/specs received from developers 2017-07-28 2017-08-04
QA - Test plan creation 2017-07-31 2017-08-11
QA - Test cases/Env preparation 2017-08-07 2017-08-11
QA - Nightly Testing 2017-07-31 2017-09-15
QA - Beta Testing 2017-09-20 2017-11-09
Release Date 2017-11-10 2017-11-14

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla

Status

Overview

apz.keyboard.enabled pref is on by default since 2017-07-22 Nightly 56.0a1
apz.autoscrolling.enabled pref is on by default since 2017-08-18 Nightly 57.0a1

References

APZ Keyboard

Meta bug: bug 1351783 - (apz-keyboard) APZ: Add support for async scrolling using keyboard input bug 1376525 - APZ: Enable "apz.keyboard.enabled" by default

APZ Autoscrolling

Meta bug: bug 1105109 - APZ: Use APZ for autoscrolling bug 1385463 - Enable APZ autoscrolling by default

Testcases

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled No
Theme (high contrast) No
UI
Mouse-only operation No
Keyboard-only operation Yes
Display (HiDPI) No
Interaction (scroll, zoom) Yes
Usable with a screen reader No e.g. with NVDA
Usability and/or discoverability testing Yes Is this feature user friendly
RTL build testing Yes
Help/Support
Help/support interface required No Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) No Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected No
Does sync work across upgrades No
Requires install testing No separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding No Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Does this affect partner builds? Partner build testing No yes/no options, add comment with details about who will lead testing
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration No
Network proxies/autoconfig No
ESR behavior changes No
Locked preferences No
Data Monitoring
Temporary or permanent telemetry monitoring No List of error conditions to monitor
Telemetry correctness testing No
Server integration testing No
Offline and server failure testing No
Load testing No
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? No
Comprehensive API testing No
Permissions No
Testing with existing/popular addons No
Security Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review No
Privilege escalation testing No
Fuzzing No
Web Compatibility depends on the feature
Testing against target sites No
Survey of many sites for compatibility No
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS No
Interaction of this feature with other browser features No

Test suite

Full Test suite link

Bug Work

Meta bug: 1351783 - (apz-keyboard) APZ: Add support for async scrolling using keyboard input

Meta bug: 1105109 - (apz-autoscrolling) APZ: Use APZ for autoscrolling

Logged bugs
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1393424 P3 Panning and Zooming Botond Ballo [:botond] Autoscrolling starts instantly on Print Preview mode VERIFIED mozilla57

1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);

Full Query
ID Priority Component Assigned to Summary Status Target milestone
1393460 P3 Desktop Keyboard scrolling is not smooth on Pinterest NEW ---

1 Total; 1 Open (100%); 0 Resolved (0%); 0 Verified (0%);

Full Query
ID Priority Component Assigned to Summary Status Target milestone
1399495 -- Desktop Benoit Girard (:BenWa) Jerkiness while scrolling on Facebook pages RESOLVED ---

1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);

Full Query
ID Priority Component Assigned to Summary Status Target milestone
1400238 P3 Panning and Zooming Botond Ballo [:botond] [Ubuntu] Autoscroll indicator is placed below mouse pointer in new windows VERIFIED mozilla58

1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);

Full Query
ID Priority Component Assigned to Summary Status Target milestone
1400271 P3 Panning and Zooming [OSX] Black flicker while scrolling with cmd+up/down arrow on www.theswishlife.com RESOLVED ---

1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);

Bug fix verification
Full Query
ID Priority Component Assigned to Summary Status Resolution Target milestone
1357880 P2 Panning and Zooming Ryan Hunt [:rhunt] Telemetry for keyboard APZ RESOLVED FIXED mozilla55
1371527 -- Graphics Ryan Hunt [:rhunt] Create a LayersMessageUtils from GfxMessageUtils RESOLVED FIXED mozilla56
1374682 P3 Panning and Zooming Kartikaya Gupta (email:kats@mozilla.com) (away until Feb-2020) Allow having non-cancelable blocks in InputQueue RESOLVED FIXED mozilla56
1376879 P3 Panning and Zooming Ryan Hunt [:rhunt] Check if we have a retargeted PresShell in FocusTarget RESOLVED FIXED mozilla56
1381736 P3 Panning and Zooming Ryan Hunt [:rhunt] APZ keyboard shortcuts have incorrect modifier masks RESOLVED FIXED mozilla56
1445403 -- Panning and Zooming Kartikaya Gupta (email:kats@mozilla.com) (away until Feb-2020) FocusState isn't really threadsafe RESOLVED FIXED mozilla61

6 Total; 0 Open (0%); 6 Resolved (100%); 0 Verified (0%);

Full Query
ID Priority Component Assigned to Summary Status Resolution Target milestone
1351783 P2 Panning and Zooming Ryan Hunt [:rhunt] APZ: Add support for async scrolling using keyboard input RESOLVED FIXED mozilla56
1376526 P3 Panning and Zooming Ryan Hunt [:rhunt] APZ: Get scroll snapping working with async keyboard scrolling RESOLVED FIXED mozilla56
1376538 P3 Panning and Zooming Ryan Hunt [:rhunt] APZ: Transmit focus updates more frequently RESOLVED FIXED mozilla56
1376549 P3 Panning and Zooming Ryan Hunt [:rhunt] APZ: Dispatch keyboard shortcuts hardcoded in ESM to content RESOLVED FIXED mozilla56
1376997 P3 Panning and Zooming Ryan Hunt [:rhunt] Use the correct focus target in WebRenderBridgeParent::UpdateAPZ RESOLVED FIXED mozilla56
1377280 P3 Panning and Zooming Ryan Hunt [:rhunt] Enable keyboard APZ on nightly RESOLVED FIXED mozilla56
1379280 P3 Panning and Zooming Ryan Hunt [:rhunt] Typing a space in an input field causes page-down scrolling with apz.keyboard.enabled=true VERIFIED FIXED mozilla56
1383365 P3 Panning and Zooming Ryan Hunt [:rhunt] Add a test for keyboard APZ RESOLVED FIXED mozilla56
1385071 P3 Panning and Zooming Ryan Hunt [:rhunt] Allow keyboard APZ with passive listeners, behind a pref RESOLVED FIXED mozilla57
1387551 P3 Panning and Zooming Ryan Hunt [:rhunt] Smooth scrolling occurs regardless of pref with APZ keyboard scrolling enabled VERIFIED FIXED mozilla57
1388466 P3 Panning and Zooming Ryan Hunt [:rhunt] Ensure APZ keyboard handling works properly with compositor resets RESOLVED FIXED mozilla57
1396323 P5 Panning and Zooming Ryan Hunt [:rhunt] Intermittent browser_bug1025195_switchToTabHavingURI_aOpenParams.js | Test timed out | application crashed [@ mozilla::dom::DocGroup::ValidateAccess] (Assertion failure: IsSafeToRun(), at SchedulerGroup.h:74) when Gecko 57 merges to Beta on 2017-09-20 VERIFIED FIXED mozilla57

12 Total; 0 Open (0%); 9 Resolved (75%); 3 Verified (25%);

Sign off

mid-Nightly Sign off (2017-08-25) is GREEN
pre-Beta Sign off (2017-09-15) is GREEN
pre-Release Sign off (2017-10-26) is GREEN

Criteria

Checklist

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Mid-Nightly testing 2017-08-25
Ran the testcases under Win 7 64-bit, Win 10 64-bit, Mac OS X 10.12 and Ubuntu 16.04 64-bit.

  • Tests run report link


Pre-Beta preliminary status 2017-09-08
Recommendation at this point is to "SHIP IT" due to the fact that no major issues were found that could compromise features functionality.

Merge to Beta Sign-off
Ran the testcases under Win 10 64-bit, Mac OS X 10.12 and Ubuntu 14.04 64-bit.

  • Link for the tests run results link


Pre-Release preliminary status 2017-10-20
Recommendation at this point is to "SHIP IT" due to the fact that no major issues were found that could compromise features functionality.

Merge to Release Sign-off
Ran the testcases under Win 10 64-bit, Mac OS X 10.12 and Ubuntu 14.04 64-bit.

  • Link for the tests run results link

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Automation Coverage [DONE] Covered by developers
Performance Testing [DONE] Covered by developers
All Defects Logged [DONE]
Critical/Blockers Fixed and Verified [ON TRACK]
Metrics/Telemetry [DONE] Covered by developers
Basic/Core functionality Nightly testing [DONE]
QA mid-Nightly Signoff [DONE] Email sent on 2017-08-25
QA Nightly - Full Testing [DONE]
QA pre-Beta Signoff [DONE] Email sent on 2017-09-15
QA Beta - Full Testing [DONE]
QA pre-Release Signoff [DONE] Email sent on 2017-10-26