QA/APZ Keyboard
Approvals Required / Received
The following individuals are required to/have approved this Test Plan:
| Name | Title | Department | Approval Date | Method |
|---|---|---|---|---|
| Milan Sreckovic | Sr. Engineer Manager | Engineering | Date | |
| Botond Ballo | Engineer | Engineering | Date | |
| Thomas Elin | EPM | Product Management | Date |
Revision History
| Date | Version | Author | Description |
|---|---|---|---|
| 07/31/2017 | 1.0 | Petruta Rasa | Created first draft |
| 08/25/2017 | 1.1 | Petruta Rasa | Added mid-Nightly test results |
Contents
Overview
This test plan will track both APZ keyboard and APZ Autoscrolling features.
Purpose
The goal of these features is to offer smoother and fastest scroll behavior using both keyboard keys and middle mouse button.
Scope
- Testing will focus on
- Comparing APZ scrolling with main-thread scrolling
- Functionality of different scrolling types across all platforms
- Stability of the browser with these features
- How testing will be performed
- Manual
Ownership
• APZ Keyboard Developers:
• APZ Autoscrolling Developers:
• QA
Testing summary
Scope of Testing
In Scope
These features add support for keyboard scrolling and auto-scrolling with APZ. Scrolling with the keyboard will be smoother and the user experience with the browser more fluently. We will focus our testing efforts on long web pages, pages with plugins, google docs, popular web pages.
Out of Scope
Following areas/features are considered out of scope and will not be considered as testing zones to be handled in this test plan:
- Scrolling with mouse
- Scrollbar drag
- High contrast
Requirements for testing
Environments
Full testing will be performed on the following OSes:
- Windows 7
- Windows 10
- Mac OS X 10.11
- Ubuntu 16.04
Other configuration/environmental setup needed: APZ Keyboard: On by default in Firefox Nightly, "apz.keyboard.enabled" pref can be used to turn it on and off. This can be checked in about:support, next to "Asynchronous Pan/Zoom" under Graphics, with "keyboard enabled".
APZ Autoscrolling: On by default in Firefox Nightly (since 2017-08-18), "apz.autoscrolling.enabled" pref can be used to turn it off
Test Strategy
Risk Assessment and Coverage
| ID | Description / Threat Description | Covered by Test Objective | Magnitude | Probability | Priority | Impact Score |
|---|---|---|---|---|---|---|
| RAC-1 | Different OSs may have different behaviors with keyboard scrolling. | TO-1 | 2-Moderate | 2-Possible | 3-High | 12 |
| RAC-2 | Web pages like Google sheets or pages with SVG may be affected | TO-2 | 3-High | 3-Almost Certain | 3-High | 27 |
| RAC-3 | Pages with plugins may cause scrolling issues | TO-3 | 3-High | 2-Possible | 3-High | 18 |
| RAC-4 | A different browser zoom than 100% may reveal some issues | TO-4 | 3-High | 2-Possible | 3-High | 18 |
| RAC-5 | Fast scrolling with Home / End keys can cause issues | TO-5 | 3-High | 2-Possible | 3-High | 18 |
| RAC-6 | Possible regressions | TO-6 | 3-High | 2-Possible | 3-High | 18 |
Values:
- Magnitude: 1- Low , 2-Moderate, 3-High
- Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
- Priority: 1 - Low, 2-Medium, 3-High
Impact Score Breakdown:
- An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
- An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
- An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
Test Objectives
| Ref | Function | Test Objective | Test Type | RAC | Owners |
|---|---|---|---|---|---|
| TO-1 | Different OSs have different behaviors with keyboard scrolling | To verify that on every OS the behavior is the expected one | Manual | RAC-1 | QA Team |
| TO-2 | Scrolling in Google sheets | Issues related to scrolling may appear since Google Sheets are convoluted pages | Manual | RAC-2 | QA Team |
| TO-3 | Scrolling pages with plugins | Pages with plugins are not affected by APZ | Manual | RAC-3 | QA Team |
| TO-4 | Use different browser zoom levels | Scrolling is not affected by the browser zoom level | Manual | RAC-4 | QA Team |
| TO-5 | Fast scrolling with Home / End keys long web pages | No errors can be seen | Manual | RAC-5 | QA Team |
| TO-6 | Keyboard scrolling is improved | Scrolling is smoother and CPU is not affected | Manual | RAC-6 | QA Team |
Builds
- Links for 57.0a1 Nightly builds link
- Links for Firefox 56 Beta 4 builds link
- Preferences must be set to true in this build
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
| Project phase | Start Date | End Date |
|---|---|---|
| Start project | 2017-07-28 | 2017-11-14 |
| Study documentation/specs received from developers | 2017-07-28 | 2017-08-04 |
| QA - Test plan creation | 2017-07-31 | 2017-08-11 |
| QA - Test cases/Env preparation | 2017-08-07 | 2017-08-11 |
| QA - Nightly Testing | 2017-07-31 | 2017-09-15 |
| QA - Beta Testing | 2017-09-20 | 2017-11-09 |
| Release Date | 2017-11-10 | 2017-11-14 |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | TestRail/ Google docs |
| Test case execution | TestRail |
| Bugs management | Bugzilla |
Status
Overview
apz.keyboard.enabled pref is on by default since 2017-07-22 Nightly 56.0a1
apz.autoscrolling.enabled pref is on by default since 2017-08-18 Nightly 57.0a1
References
APZ Keyboard
Meta bug: bug 1351783 - (apz-keyboard) APZ: Add support for async scrolling using keyboard input bug 1376525 - APZ: Enable "apz.keyboard.enabled" by default
APZ Autoscrolling
Meta bug: bug 1105109 - APZ: Use APZ for autoscrolling bug 1385463 - Enable APZ autoscrolling by default
Testcases
Test Areas
| Test Areas | Covered | Details |
|---|---|---|
| Private Window | Yes | |
| Multi-Process Enabled | Yes | |
| Multi-process Disabled | No | |
| Theme (high contrast) | No | |
| UI | ||
| Mouse-only operation | No | |
| Keyboard-only operation | Yes | |
| Display (HiDPI) | No | |
| Interaction (scroll, zoom) | Yes | |
| Usable with a screen reader | No | e.g. with NVDA |
| Usability and/or discoverability testing | Yes | Is this feature user friendly |
| RTL build testing | Yes | |
| Help/Support | ||
| Help/support interface required | No | Make sure link to support/help page exist and is easy reachable. |
| Support documents planned(written) | No | Make sure support documents are written and are correct. |
| Install/Upgrade | ||
| Feature upgrades/downgrades data as expected | No | |
| Does sync work across upgrades | No | |
| Requires install testing | No | separate feature/application installation needed (not only Firefox) |
| Affects first-run or onboarding | No | Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned. |
| Does this affect partner builds? Partner build testing | No | yes/no options, add comment with details about who will lead testing |
| Enterprise | Raise up the topic to developers to see if they are expecting to work different on ESR builds | |
| Enterprise administration | No | |
| Network proxies/autoconfig | No | |
| ESR behavior changes | No | |
| Locked preferences | No | |
| Data Monitoring | ||
| Temporary or permanent telemetry monitoring | No | List of error conditions to monitor |
| Telemetry correctness testing | No | |
| Server integration testing | No | |
| Offline and server failure testing | No | |
| Load testing | No | |
| Add-ons | If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on. | |
| Addon API required? | No | |
| Comprehensive API testing | No | |
| Permissions | No | |
| Testing with existing/popular addons | No | |
| Security | Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature. | |
| 3rd-party security review | No | |
| Privilege escalation testing | No | |
| Fuzzing | No | |
| Web Compatibility | depends on the feature | |
| Testing against target sites | No | |
| Survey of many sites for compatibility | No | |
| Interoperability | depends on the feature | |
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | No | |
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | No | |
| Interaction of this feature with other browser features | No |
Test suite
Full Test suite link
Bug Work
Meta bug: 1351783 - (apz-keyboard) APZ: Add support for async scrolling using keyboard input
Meta bug: 1105109 - (apz-autoscrolling) APZ: Use APZ for autoscrolling
Logged bugs
| ID | Priority | Component | Assigned to | Summary | Status | Target milestone |
|---|---|---|---|---|---|---|
| 1393424 | P3 | Panning and Zooming | Botond Ballo [:botond] | Autoscrolling starts instantly on Print Preview mode | VERIFIED | mozilla57 |
1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);
| ID | Priority | Component | Assigned to | Summary | Status | Target milestone |
|---|---|---|---|---|---|---|
| 1393460 | P3 | Site Reports | Keyboard scrolling is not smooth on Pinterest | RESOLVED | --- |
1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);
| ID | Priority | Component | Assigned to | Summary | Status | Target milestone |
|---|---|---|---|---|---|---|
| 1399495 | -- | Site Reports | Benoit Girard (:BenWa) | Jerkiness while scrolling on Facebook pages | RESOLVED | --- |
1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);
| ID | Priority | Component | Assigned to | Summary | Status | Target milestone |
|---|---|---|---|---|---|---|
| 1400238 | P3 | Panning and Zooming | Botond Ballo [:botond] | [Ubuntu] Autoscroll indicator is placed below mouse pointer in new windows | VERIFIED | mozilla58 |
1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);
| ID | Priority | Component | Assigned to | Summary | Status | Target milestone |
|---|---|---|---|---|---|---|
| 1400271 | P3 | Panning and Zooming | [OSX] Black flicker while scrolling with cmd+up/down arrow on www.theswishlife.com | RESOLVED | --- |
1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);
Bug fix verification
| ID | Priority | Component | Assigned to | Summary | Status | Resolution | Target milestone |
|---|---|---|---|---|---|---|---|
| 1357880 | P2 | Panning and Zooming | Ryan Hunt [:rhunt] (on leave until early May) | Telemetry for keyboard APZ | RESOLVED | FIXED | mozilla55 |
| 1371527 | -- | Graphics | Ryan Hunt [:rhunt] (on leave until early May) | Create a LayersMessageUtils from GfxMessageUtils | RESOLVED | FIXED | mozilla56 |
| 1374682 | P3 | Panning and Zooming | Kartikaya Gupta (email:kats@mozilla.staktrace.com) | Allow having non-cancelable blocks in InputQueue | RESOLVED | FIXED | mozilla56 |
| 1376879 | P3 | Panning and Zooming | Ryan Hunt [:rhunt] (on leave until early May) | Check if we have a retargeted PresShell in FocusTarget | RESOLVED | FIXED | mozilla56 |
| 1381736 | P3 | Panning and Zooming | Ryan Hunt [:rhunt] (on leave until early May) | APZ keyboard shortcuts have incorrect modifier masks | RESOLVED | FIXED | mozilla56 |
| 1445403 | -- | Panning and Zooming | Kartikaya Gupta (email:kats@mozilla.staktrace.com) | FocusState isn't really threadsafe | RESOLVED | FIXED | mozilla61 |
6 Total; 0 Open (0%); 6 Resolved (100%); 0 Verified (0%);
| ID | Priority | Component | Assigned to | Summary | Status | Resolution | Target milestone |
|---|---|---|---|---|---|---|---|
| 1351783 | P2 | Panning and Zooming | Ryan Hunt [:rhunt] (on leave until early May) | APZ: Add support for async scrolling using keyboard input | RESOLVED | FIXED | mozilla56 |
| 1376526 | P3 | Panning and Zooming | Ryan Hunt [:rhunt] (on leave until early May) | APZ: Get scroll snapping working with async keyboard scrolling | RESOLVED | FIXED | mozilla56 |
| 1376538 | P3 | Panning and Zooming | Ryan Hunt [:rhunt] (on leave until early May) | APZ: Transmit focus updates more frequently | RESOLVED | FIXED | mozilla56 |
| 1376549 | P3 | Panning and Zooming | Ryan Hunt [:rhunt] (on leave until early May) | APZ: Dispatch keyboard shortcuts hardcoded in ESM to content | RESOLVED | FIXED | mozilla56 |
| 1376997 | P3 | Panning and Zooming | Ryan Hunt [:rhunt] (on leave until early May) | Use the correct focus target in WebRenderBridgeParent::UpdateAPZ | RESOLVED | FIXED | mozilla56 |
| 1377280 | P3 | Panning and Zooming | Ryan Hunt [:rhunt] (on leave until early May) | Enable keyboard APZ on nightly | RESOLVED | FIXED | mozilla56 |
| 1379280 | P3 | Panning and Zooming | Ryan Hunt [:rhunt] (on leave until early May) | Typing a space in an input field causes page-down scrolling with apz.keyboard.enabled=true | VERIFIED | FIXED | mozilla56 |
| 1383365 | P3 | Panning and Zooming | Ryan Hunt [:rhunt] (on leave until early May) | Add a test for keyboard APZ | RESOLVED | FIXED | mozilla56 |
| 1385071 | P3 | Panning and Zooming | Ryan Hunt [:rhunt] (on leave until early May) | Allow keyboard APZ with passive listeners, behind a pref | RESOLVED | FIXED | mozilla57 |
| 1387551 | P3 | Panning and Zooming | Ryan Hunt [:rhunt] (on leave until early May) | Smooth scrolling occurs regardless of pref with APZ keyboard scrolling enabled | VERIFIED | FIXED | mozilla57 |
| 1388466 | P3 | Panning and Zooming | Ryan Hunt [:rhunt] (on leave until early May) | Ensure APZ keyboard handling works properly with compositor resets | RESOLVED | FIXED | mozilla57 |
| 1396323 | P5 | Panning and Zooming | Ryan Hunt [:rhunt] (on leave until early May) | Intermittent browser_bug1025195_switchToTabHavingURI_aOpenParams.js | Test timed out | application crashed [@ mozilla::dom::DocGroup::ValidateAccess] (Assertion failure: IsSafeToRun(), at SchedulerGroup.h:74) when Gecko 57 merges to Beta on 2017-09-20 | VERIFIED | FIXED | mozilla57 |
12 Total; 0 Open (0%); 9 Resolved (75%); 3 Verified (25%);
Sign off
mid-Nightly Sign off (2017-08-25) is GREEN
pre-Beta Sign off (2017-09-15) is GREEN
pre-Release Sign off (2017-10-26) is GREEN
Criteria
Checklist
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Mid-Nightly testing 2017-08-25
Ran the testcases under Win 7 64-bit, Win 10 64-bit, Mac OS X 10.12 and Ubuntu 16.04 64-bit.
- Tests run report link
Pre-Beta preliminary status 2017-09-08
Recommendation at this point is to "SHIP IT" due to the fact that no major issues were found that could compromise features functionality.
Merge to Beta Sign-off
Ran the testcases under Win 10 64-bit, Mac OS X 10.12 and Ubuntu 14.04 64-bit.
- Link for the tests run results link
Pre-Release preliminary status 2017-10-20
Recommendation at this point is to "SHIP IT" due to the fact that no major issues were found that could compromise features functionality.
Merge to Release Sign-off
Ran the testcases under Win 10 64-bit, Mac OS X 10.12 and Ubuntu 14.04 64-bit.
- Link for the tests run results link
Checklist
| Exit Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | [DONE] | |
| Testing Infrastructure setup | [DONE] | |
| Test Plan Creation | [DONE] | |
| Test Cases Creation | [DONE] | |
| Automation Coverage | [DONE] | Covered by developers |
| Performance Testing | [DONE] | Covered by developers |
| All Defects Logged | [DONE] | |
| Critical/Blockers Fixed and Verified | [ON TRACK] | |
| Metrics/Telemetry | [DONE] | Covered by developers |
| Basic/Core functionality Nightly testing | [DONE] | |
| QA mid-Nightly Signoff | [DONE] | Email sent on 2017-08-25 |
| QA Nightly - Full Testing | [DONE] | |
| QA pre-Beta Signoff | [DONE] | Email sent on 2017-09-15 |
| QA Beta - Full Testing | [DONE] | |
| QA pre-Release Signoff | [DONE] | Email sent on 2017-10-26 |
