QA/CSS Grid Inspector Layout View

From MozillaWiki
< QA
Jump to: navigation, search

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
QA Manager Product Integrity Date Email
Software Engineer Engineering Date Email
EPM Product Management Date Email


Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
2017-06-12 1.0 Bogdan Maris Created first draft
2017-06-22 1.1 Bogdan Maris Updated info (Test Strategy section)
2017-07-07 1.2 Bogdan Maris Minor updates after mid-Nightly sign-off
2017-09-08 1.3 Bogdan Maris Minor updates after pre-Release sign-off

Overview

Grid Layout lets us properly separate the order of elements in the source from their visual presentation. As a designer this means you are free to change the location of page elements as is best for your layout at different breakpoints and not need to compromise a sensible structured document for your responsive design.

Purpose

The purpose of this testing is to make sure that the web developers don't run into issues when using CSS Grid Layout. Note that Jen Simmons is actively testing CSS Grid, giving feedback and logging bugs.

  • The test scope, focus areas and objectives
  • The test responsibilities
  • The test strategy for the levels and types of test for this release
  • The entry and exit criteria
  • The basis of the test estimates
  • Any risks, issues, assumptions and test dependencies
  • The test schedule and major milestones
  • The test deliverables

Scope

This wiki details the testing that will be performed by the project team for the CSS Grid Inspector Layout View project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed

Ownership

PM:
Bryan Clark (irc: clarkbw)
Development:
Gabriel Luong (irc: gl)
Matteo Ferretti (irc: zer0 or matteo)
Patrick Brosset (irc: pbro)
Julian Descottes (irc: jdescottes)
Platform QA Team:
Andrei Vaida (irc: avaida) - Team Lead
Bogdan Maris (irc: bogdan_maris)
Ciprian Georgiu (irc: ciprian_georgiu) - Peer

Testing summary

Scope of Testing

In Scope

Testing efforts will be focused on the impact that Grid overlay has on layout and functionality of its available settings across platforms.

Out of Scope

We consider out of scope testing grid properties in combination with different css/html elements, that will be best covered by a web developer.

Requirements for testing

Environments

OSs that need to be covered:
* Windows 10
* Windows 7
* Ubuntu 16.04
* macOS 10.12

Channel dependent settings (configs) and environment setups

Nightly

  • Feature fully enabled by default, no config or special environment setup required.

Beta

  • Feature fully enabled by default, no config or special environment setup required.

Post Beta / Release

  • tbd

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 Users are encountering visual artifacts when Grid layout is active TO-1 2-Moderate 2-Possible 3-High 12
RAC-2 Artifacts are left behind when grid overlay is toggled (enabled/disabled) TO-1, TO-3, TO-4 3-High 3-Almost Certain 3-High 27
RAC-3 The Grid overlay is not changing alongside web elements when zooming in page TO-1, TO-2 2-Moderate 2-Possible 3-High 12
RAC-4 Users are not able to change the Grid Overlay color based on their choosing TO-4 2-Moderate 2-Possible 2-Medium 8
RAC-5 Learn more link is pointing to an invalid help page 1-Low 1-Unlikely 1-Low 1
RAC-6 Numbers are not displayed for grid lines TO-4 2-Moderate 2-Possible 3-High 12
RAC-7 Grid areas are not displayed when available TO-4 1-Low 2-Possible 2-Medium 4
RAC-8 Grid lines are not extended infinitely TO-4 1-Low 1-Unlikely 1-Low 1

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Evaluation Criteria Test Objective Test Type RAC Owners
TO-1 CSS Grid graphics Verify that no unwanted graphical glitches can be seen Manual RAC-1, RAC-2, RAC-3 Release Team
TO-2 Scroll/zoom when Grid is active Verify that grid overlay is displayed correctly when zooming and scrolling Manual RAC-3 Release Team
TO-3 Overlay Grid is enabled and disabled Verify that enabling/disabling Grid overlay works every time Manual RAC-2 Release Team
TO-4 Grid display settings Verify that all settings for grid overlay work as expected Manual RAC-2, RAC-4, RAC-6, RAC-7, RAC-8 Release Team
TO-5 Grid overlay not available when CSS grid is not used in page Verify that Grid overlay is only used in websites that use CSS Grid, otherwise display message to inform the user Manual Release Team

Builds

This section should contain links for builds with the feature:

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 2017-03-16 2017-09-26
Study documentation/specs received from developers n/a n/a
QA - Test plan creation 2017-06-12 2017-06-22
QA - Test cases/Env preparation 2017-06-12 2017-06-26
QA - Nightly Testing 2017-06-28 2017-07-28
QA - Beta Testing 2017-08-02 2017-09-25
Release Date 2017-09-26

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla

Status

Overview

Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Beta
  • 2017-08-02 merge to Firefox Beta 56. [DONE]
Track the dates and build number where feature was merged to Release
  • 2017-09-26 merge to Firefox Release 56 [IN PROGRESS]

References

Testcases

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes
Theme (high contrast) Yes
UI
Mouse-only operation No The feature can be enabled and user by keyboard/touchscreen as well
Keyboard-only operation No The feature can be enabled and used by mouse/touchscreen as well
Display (HiDPI) Yes
Interaction (scroll, zoom) Yes
Usable with a screen reader No e.g. with NVDA
Usability and/or discoverability testing Yes Is this feature user friendly
RTL build testing Yes
Help/Support
Help/support interface required TBD Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) TBD Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected No
Does sync work across upgrades No
Requires install testing No This is a feature that is included in Firefox, no 3party apps needed
Affects first-run or onboarding No Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Does this affect partner builds? Partner build testing No yes/no options, add comment with details about who will lead testing
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration No Not yet available in ESR will probably be when it reaches 57
Network proxies/autoconfig No Not yet available in ESR will probably be when it reaches 57
ESR behavior changes No Not yet available in ESR will probably be when it reaches 57
Locked preferences No Not yet available in ESR will probably be when it reaches 57
Data Monitoring
Temporary or permanent telemetry monitoring No List of error conditions to monitor
Telemetry correctness testing No No telemetry is available for this feature AFAIK
Server integration testing No
Offline and server failure testing No
Load testing No
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? No
Comprehensive API testing No
Permissions No Feature does not use permissions to work
Testing with existing/popular addons No
Security Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review No
Privilege escalation testing No
Fuzzing No
Web Compatibility depends on the feature
Testing against target sites Yes
Survey of many sites for compatibility No This feature is only available on a handful of websites that use CSS Grid
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS No
Interaction of this feature with other browser features No

Test suite

Full Test suite - Testrail Link 
Smoke Test suite - Gdoc Link - only draft for now.
Regression Test suite - Link with the tests - if available/needed.

Bug Work

Meta bugs:

Logged bugs ( blocking 1333706 )
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1333707 P2 Inspector Matteo Ferretti [:zer0] [:matteo] Grid stuck on page when using back button VERIFIED Firefox 54
1333714 P2 Inspector Julian Descottes [:jdescottes] Grid highlighter doesn't hide when the DOM it is highlighting is removed/reloaded in jsfiddle RESOLVED Firefox 55
1334523 P2 Inspector Grid inside iframe suffers from graphical issues when altering Fx window RESOLVED ---
1335088 P3 Inspector Matteo Ferretti [:zer0] [:matteo] Grid highlighter tool not working after exiting Offline Mode RESOLVED ---
1342026 P3 Inspector Julian Descottes [:jdescottes] Grid Inspector disappears changing the height of developer toggle tools inside Fx RESOLVED ---
1342051 P3 Inspector Julian Descottes [:jdescottes] Grid highlighter can no longer be activated after Inspector close and page refresh VERIFIED Firefox 54
1342056 P3 Inspector [Non-e10s] Grid Inspector changes position after page refresh when page with grid is in iframe RESOLVED ---
1376764 P3 Inspector Matteo Ferretti [:zer0] [:matteo] UI/Scrolling issues with CSS grid after Firefox is resized RESOLVED ---
1376774 -- Inspector Gabriel [:gl] (ΦωΦ) div.content.grid-content checkbox is displayed unchecked in Inspector but appears as enabled on the webpage after refresh VERIFIED Firefox 56
1376775 P3 Inspector Colorpicker remains behind when an overlay grid is active and the color is changed multiple times thus any action on overlays is ignored NEW ---
1377879 P3 Inspector Matteo Ferretti [:zer0] [:matteo] Highlighters can remain behind if Fx is slowly resized RESOLVED Firefox 57
1378306 -- Inspector Patrick Brosset <:pbro> CSS Overlay Grid cannot be unchecked if the same code is run twice VERIFIED Firefox 56
1395561 P2 Inspector Overlay Grid remains active in Rules after it was activated two or more times in Layout NEW ---
1397179 P2 Inspector Area information is not displayed for boxes that don't have the space to show it above it or on left/right extremities NEW ---
1397185 P3 Inspector Area information is displayed way higher than the box is placed when tilting the grid at different angles NEW ---

15 Total; 4 Open (26.67%); 7 Resolved (46.67%); 4 Verified (26.67%);


Bug fix verification
Full Query
ID Priority Component Assigned to Summary Status Resolution Target milestone
15069 P3 Networking jefft File type url not correctly parsed and created VERIFIED FIXED M13

1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);

Sign off

Criteria

Checklist

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite, link to TestRail - Tests Runs and Results link
    • Daily Smoke, if needed/available
    • Regression Test suite, if needed/available


Merge to Beta Sign-off
List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
Basic/Core functionality Nightly testing
QA mid-Nightly Signoff [DONE] Email sent 2017-07-07
QA Nightly - Full Testing
QA pre-Beta Signoff [DONE] Email sent 2017-07-28
QA Beta - Full Testing
QA pre-Release Signoff [DONE] Email sent 2017-09-08