QA/Photon Animations

From MozillaWiki
< QA
Jump to: navigation, search

Overview

Purpose

The goal of this feature is to make Firefox's UI more user friendly by including animation in following areas:

  • Tab reorder
  • Downloading
  • Doorhanger Drop-down/Submenues
  • Refresh/Stop button
  • Opening/Closing arrow panel
  • Animations run on the compositor
  • Bookmarking
  • Save to Pocket
  • Tab loading indicator
  • Send to Overflow menu
  • Copy URL Confirmation Message
  • Send to Device Confirmation Message

Scope

Testing will cover new animated area of Firefox such as Bookmarking, Downloading, Tab reordering/loading, Doorhanger, and pocket etc.

Ownership

Engineering lead
  Justin Dolske
Engineering Team
  Jared Wein
  Sam Foster
UX Team: 
  Amy Lee
  Eric Pang
QA 
  Stefan Georgiev
  Justin Williams
  Kanchan Kumari

Testing summary

Scope of Testing

In Scope

Testing will be focused on the following areas :

  • Tab reordering animation
  • Download icon animation
  • Refresh/Stop button animation
  • Opening/Closing arrow panel animation
  • Animations run on the compositor
  • Bookmark animation
  • Pocket animation
  • Tab loading indicator
  • Send to Overflow menu
  • Copy URL Confirmation Message
  • Send to Device Confirmation Message

Out of Scope

Mobile Devices will not be tested

Requirements for testing

Environments

Windows 7 x64: Intel(R) Core(TM) i5-47900 CPU @ 3.60GHz ; NVIDIA GeForce GTX 745 ; RAM 16GB
Windows 10 Home x64: Intel(R) Core(TM) i3-7100U CPU @ 2.40GHz; Intel(R) HD Graphics 620 ; RAM 4 GB
Ubuntu 17.04 x64:Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz ; Intel(R) HD Graphics 630 ; RAM 16GB
Mac OS X 10.12 x64: 2.7 GHz Intel Core i5 / 2.9 GHz Intel Core i5 ; Intel Iris Pro Graphics 2.7 GHz / NVIDIA GeForce GT 750M 1GB ; RAM 8GB
All PC's using for testing has installed Kaspersky Endpoint Security 10 software with version 10.3.0.6294 AES256.

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 Buttons may become unclickable due to animations being on top of them. TO-1 2-Moderate 2-Possible 2-Medium 8
RAC-2 Sizes of the items in the toolbar may become distorted TO-1 2-Moderate 2-Possible 2-Medium 8
RAC-3 Animations do not play correctly in RTL locale. TO-1 2-Moderate 2-Possible 2-Medium 8
RAC-4 Changing density (compact to touch) may effect the position of the animations. TO-1 2-Moderate 2-Possible 2-Medium 8
RAC-5 Keyboard navigation only could not trigger the animation. TO-1 2-Moderate 2-Possible 2-Medium 8
RAC-6 Animation do not play correctly using HIgh-Contrast theme. TO-1 2-Moderate 2-Possible 2-Medium 8

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Evaluation Criteria Test Type Owners
1 Tab Reorder Verify tab moving/reordering use photon animation curve Check Tab moving/reordering always slide tabs in the final place and use the photon animation curve in doing so Manual Eng Team
2 Download Verify new implemented animation for downloading Check download icon animation in all download condition such as In progress, completed, and failed etc. Manual Eng Team
3 Doorhanger Drop-Down/Submenues Verify new door hanger submenu panel animation Check panel animation for the doorhanger sub menus as shown here https://bug1374749.bmoattachments.org/attachment.cgi?id=8879674 Manual Eng Team
4 Reload/Stop button Verify reload/stop animation Check reload/stop button should have an animation when changing between states Manual Eng Team
5 Opening/Closing arrow panel Verify new animation for opening/closing the arrow panels Check animation for opening/closing the arrow panels is positioned and shown correctly Manual Eng Team
6 Bookmark Verify new bookmark star animation Check new bookmark star animation has rays that shoot out from star Manual Eng Team
7 Pocket Verify save to pocket animation Check save to pocket should have associated animation Manual Eng Team
8 Tab Loading Indicator Verify tab loading indicator Check Tab load associates animation Manual Eng Team
9 Overflow icon Verify pin to overflow menu animation Check for associated animation when a user right-clicks on a button and chooses to "Send to overflow menu" Manual Eng Team
10 Copy Confirmation Verify “Copy URL” button trigger the associated animation Check for associated animation when “Copy URL” button is pressed Manual Eng Team
11 Send to Device Verify “Send to Device” confirmation message Check for “Send to Device” confirmation message if the User has a Firefox Account enabled Manual Eng Team
12 Opening/Closing bottom panels Verify Theme density animation Check Theme density animation on Windows Manual Eng Team

Builds

This section should contain links for builds with the feature -

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 03/30/2017
Study documentation/specs received from developers 03/30/2017 05/24/2017
QA - Test plan creation 06/21/2017 09/08/2017
QA - Test cases/Env preparation 08/11/2017 09/11/2017
QA - Nightly Testing 09/11/2017 09/15/2017
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail
Test case execution TestRail
Bugs management Bugzilla
Meta Bug BugZilla
Trello Card Trello
Test Suite TestRail
Test Cases GDoc

Status

Overview

Track the dates and build number where feature was merged to Release/Beta
Channel Date feature was merged to channel Build number
Nightly 08/02/2017 20170802100302
Beta
Release

Test Areas

Test Areas Covered Details Reviewed By
Private Window Yes Behavior will be the same Jared Wein
Multi-Process Enabled Yes Default Jared Wein
Theme (high contrast) Yes Jared Wein
UI
Mouse-only operation Yes Jared Wein
Keyboard-only operation Yes Jared Wein
Display (HiDPI) Yes Jared Wein
Usability and/or discoverability testing Yes Jared Wein
RTL build testing Yes Jared Wein

Bug Work

Meta bug: 1349423 - Photon Animation Meta Bug

Logged bugs ( blocking 1349423 )
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1007283 -- XUL Investigate why animated panels don't work on Linux and re-enable them NEW ---
1352063 P1 Theme (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Implement new animation for bookmarking VERIFIED Firefox 56
1352065 P3 Theme [meta] Implement new animation for downloading NEW ---
1352068 P4 Theme Implement new session restore animation NEW ---
1352069 P1 Preferences Jim Porter (:squib) Introduce a pref that allows for disabling animations (toolkit.cosmeticAnimations.enabled) VERIFIED Firefox 55
1352075 P1 XUL Widgets Sam Foster [:sfoster] (he/him) Implement new animation for opening/closing the arrow panels VERIFIED mozilla57
1352117 -- Theme [meta] Various tab animations NEW ---
1352119 P1 General (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Implement new tab loading/progress indicator animation VERIFIED Firefox 57
1352127 P4 General Toolbar button hover and pressed animations NEW ---
1352164 P3 Address Bar Awesomebar styling / animation tweaks NEW ---
1355922 P1 General (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Save to Pocket should have an associated animation VERIFIED Firefox 56
1355924 P1 General (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) The Reload/Stop button should have an animation when changing between states VERIFIED Firefox 56
1355927 P1 Themes Jim Porter (:squib) Notification bars should use the photon animation curve when appearing/disappearing VERIFIED mozilla55
1358215 P1 Theme (not currently active) Ted Mielczarek Add flag to facilitate early landing of photon-animation work ahead of v57 RESOLVED Firefox 55
1359929 P5 General Disable animations entirely on really low-speced hardware? RESOLVED ---
1364221 P1 Graphics: Layers (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Allow frames to be prerendered as long as the area of the frame is less than the limit RESOLVED mozilla55
1369896 P1 Theme [UX] Download button should only appear in the toolbar when there are current or recent downloads RESOLVED ---
1375152 P1 Theme (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Pin to Overflow Menu should have associated animation VERIFIED Firefox 56
1380150 P1 Tabbed Browser (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Flickering between favicon and loading indicator after first load VERIFIED Firefox 57
1380815 P1 Theme (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Fade overflow menu icon to grey after item is pinned VERIFIED Firefox 56
1382894 P4 Toolbars and Customization Toolbar animations such as the 'Save to Pocket' and 'Stop / Reload' do not work properly outside of the nav-bar NEW ---
1384682 -- Bookmarks & History Bookmarking - There should be On Click feedback in the bookmark icon RESOLVED ---
1385913 P1 Theme (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Remove MOZ_PHOTON_ANIMATIONS ifdef once 56 merges to Beta RESOLVED Firefox 57
1386406 P1 Theme (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Audit animation CSS to make sure that each animation-duration is set to 60fps based on step count RESOLVED Firefox 57
1387077 P1 Pocket (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Save to Pocket animation broken after Pocket button moved to URL bar VERIFIED Firefox 57
1390283 P1 Pocket (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Pocket Icon - Colour looks wrong RESOLVED Firefox 57
1391425 P4 General Tab label jumps to fill in space of tab loading progress indicator NEW ---
1392157 P1 Tabbed Browser (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Show a burst across the tab when a page has finished loading VERIFIED Firefox 57
1392685 P1 Pocket (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Pocket'ing items from Activity Stream results in the Pocket icon appearing outside of the URL bar VERIFIED Firefox 57
1393057 P1 Toolbars and Customization (Currently slow to respond) Philipp Sackl [:phlsa] (Firefox UX) please use needinfo Remove transition from toolbar button hover state RESOLVED Firefox 57
1393061 -- Toolbars and Customization Overflow menu doesn't keep its pressed state correctly when menu is open RESOLVED ---
1393421 P4 Pocket Clean up pocket-animation.svg NEW ---
1393870 P1 XUL Widgets Erica Wright [:ewright] Panels anchored to bottom have weird animation VERIFIED mozilla57
1394680 P1 Theme (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) The Reload/Stop animation is inconsistent in dark lightweight themes VERIFIED Firefox 57
1407927 P5 Theme The 'star drop' animation should be applied to the 'bookmarks menu button' instead of or in addition to the 'library button' if 'bookmarks menu button' is added to the toolbar NEW ---
1408007 P1 Theme Sam Foster [:sfoster] (he/him) SVG indicators not changing colours properly for background tabs(other tabs) RESOLVED Firefox 58
1414040 -- Theme At 125% DPI, the static stop-reload icon changing to its animated version isn't smooth RESOLVED ---
1415019 P1 Theme (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Improve background tab loading indicator colors for better contrast and photon color palette compatibility RESOLVED Firefox 59

38 Total; 10 Open (26.32%); 13 Resolved (34.21%); 15 Verified (39.47%);


Bug fix verification
Full Query
ID Priority Component Assigned to Summary Status Resolution Target milestone
1352063 P1 Theme (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Implement new animation for bookmarking VERIFIED FIXED Firefox 56
1352069 P1 Preferences Jim Porter (:squib) Introduce a pref that allows for disabling animations (toolkit.cosmeticAnimations.enabled) VERIFIED FIXED Firefox 55
1352075 P1 XUL Widgets Sam Foster [:sfoster] (he/him) Implement new animation for opening/closing the arrow panels VERIFIED FIXED mozilla57
1352119 P1 General (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Implement new tab loading/progress indicator animation VERIFIED FIXED Firefox 57
1355922 P1 General (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Save to Pocket should have an associated animation VERIFIED FIXED Firefox 56
1355924 P1 General (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) The Reload/Stop button should have an animation when changing between states VERIFIED FIXED Firefox 56
1355927 P1 Themes Jim Porter (:squib) Notification bars should use the photon animation curve when appearing/disappearing VERIFIED FIXED mozilla55
1358215 P1 Theme (not currently active) Ted Mielczarek Add flag to facilitate early landing of photon-animation work ahead of v57 RESOLVED FIXED Firefox 55
1364221 P1 Graphics: Layers (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Allow frames to be prerendered as long as the area of the frame is less than the limit RESOLVED FIXED mozilla55
1369896 P1 Theme [UX] Download button should only appear in the toolbar when there are current or recent downloads RESOLVED FIXED ---
1375152 P1 Theme (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Pin to Overflow Menu should have associated animation VERIFIED FIXED Firefox 56
1380150 P1 Tabbed Browser (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Flickering between favicon and loading indicator after first load VERIFIED FIXED Firefox 57
1380815 P1 Theme (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Fade overflow menu icon to grey after item is pinned VERIFIED FIXED Firefox 56
1385913 P1 Theme (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Remove MOZ_PHOTON_ANIMATIONS ifdef once 56 merges to Beta RESOLVED FIXED Firefox 57
1386406 P1 Theme (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Audit animation CSS to make sure that each animation-duration is set to 60fps based on step count RESOLVED FIXED Firefox 57
1387077 P1 Pocket (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Save to Pocket animation broken after Pocket button moved to URL bar VERIFIED FIXED Firefox 57
1390283 P1 Pocket (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Pocket Icon - Colour looks wrong RESOLVED FIXED Firefox 57
1392157 P1 Tabbed Browser (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Show a burst across the tab when a page has finished loading VERIFIED FIXED Firefox 57
1392685 P1 Pocket (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Pocket'ing items from Activity Stream results in the Pocket icon appearing outside of the URL bar VERIFIED FIXED Firefox 57
1393057 P1 Toolbars and Customization (Currently slow to respond) Philipp Sackl [:phlsa] (Firefox UX) please use needinfo Remove transition from toolbar button hover state RESOLVED FIXED Firefox 57
1393870 P1 XUL Widgets Erica Wright [:ewright] Panels anchored to bottom have weird animation VERIFIED FIXED mozilla57
1394680 P1 Theme (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) The Reload/Stop animation is inconsistent in dark lightweight themes VERIFIED FIXED Firefox 57
1408007 P1 Theme Sam Foster [:sfoster] (he/him) SVG indicators not changing colours properly for background tabs(other tabs) RESOLVED FIXED Firefox 58
1415019 P1 Theme (Limited availability 5/20-5/24) Jared Wein [:jaws] (please needinfo? me) Improve background tab loading indicator colors for better contrast and photon color palette compatibility RESOLVED FIXED Firefox 59

24 Total; 0 Open (0%); 9 Resolved (37.5%); 15 Verified (62.5%);

Sign off

Criteria

Checklist

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing
Win 7 & 10, Ubuntu 16.04, and Mac 10.12


Merge to Beta Sign-off
List of OSes that will be covered by testing
Win 7 & 10, Ubuntu 16.04, and Mac 10.12

  • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
N/A
Performance Testing N/A
All Defects Logged IN PROGRESS
Critical/Blockers Fixed and Verified IN PROGRESS
Metrics/Telemetry
Basic/Core functionality Nightly testing
QA mid-Nightly Signoff [DONE] 08/28/2017
QA Nightly - Full Testing IN PROGRESS
QA pre-Beta Signoff Email to be sent
QA Beta - Full Testing
QA pre-Release Signoff Email to be sent

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
Lawrence Mandel / RyanVM QA Reviewer Product Integrity Email
Jared Wein, Sam Foster Software Engineer Engineering 09/08/2017 Email
Romain Testard, Peter Dolanjski EPM Product Management Email

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
06/21/2017 1.0 Justin Williams Created first draft
08/14/2017 1.1 Kanchan Kumari Edits to draft
08/24/2017 1.2 Stefan Georgiev Edits to draft
09/8/2017 1.3 Stefan Georgiev Final version