QA/Compact Themes

From MozillaWiki
< QA
Jump to: navigation, search

Compact Themes

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
2017-01-23 1.0 Ciprian Georgiu Created first draft
2017-01-30 1.1 Ciprian Georgiu Small changes in Test Objectives

Overview

Purpose

Detail the purpose of this document. For example:

  • The test scope, focus areas and objectives
  • The test responsibilities
  • The test strategy for the levels and types of test for this release
  • The entry and exit criteria
  • The basis of the test estimates
  • Any risks, issues, assumptions and test dependencies
  • The test schedule and major milestones
  • The test deliverables

Scope

This wiki details the testing that will be performed by the project team for the Compact Themes feature. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • what will be tested,
  • how testing will be performed.

Ownership

Engineering contact(s):

QA contact(s):

Testing summary

Scope of Testing

In Scope

Testing efforts will be focused on confirming that the mechanism for selecting a theme(s), in this case, Compact Light and Compact Dark is working as expected across platforms. Also, it will be pay particular attention to the theme(s) itself and any obvious UI glitches introduced by a specific change.

Out of Scope

Only the new two themes mentioned above will be included in the testing efforts. Any others themes, are not going to be evaluated in order to find whether it satisfies the specified requirements or not.

Requirements for testing

Environments

Full testing will be done on following OSes:

  • Windows 7
  • Windwos 10
  • Ubuntu 16.04 LTS
  • Ubuntu 14.04 LTS
  • Mac OS X 10.11 or greater

About Compact Themes

Mozilla will ship two new lightweight themes starting with Firefox 53 by replacing compact theme with a light color scheme and a dark color scheme. These are based on the current Dev Edition theme and will be enabled by users in both about:addons#appearance and customization mode. The default theme will continue to be Australis.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Evaluation Criteria Test Type Owners
1 The mechanism for selecting Compact Themes is working properly
  • does an applied theme actually get applied (both in about:addons#appearance and customize mode)
  • does it persist on restart without any obvious flicker / flash of unstyled content
  • does the profile migration work properly - if you had the dark dev edition theme applied then it should apply the dark compact theme and if you had light dev edition theme applied it should apply the light compact theme
all scenarios provided must have the right expected result Manual Desktop Team
2 The theme itself
  • any obvious UI glitches introduced by a specific change
  • things to look at:
    • on mouseover, on mouseclick effects for buttons, menu entries, toolbar entries, site identity information, context menu
    • behaviour on different window sizes
    • if font used after the applied theme is still readable in application
    • the alignment of the text is proper
    • positioning of GUI elements for different screen resolution
there should be not UI issues on Compact Themes Manual Desktop Team

Testing approach

  • set up a smoke/acceptance suite that checks the core functionality associated to this feature, along with a few major UI components
  • set up an exploratory section within this suite that checks all the UI components

Builds

This section should contain links for builds with the feature:

The feature is available on Firefox 53 starting with 2017-01-15.

  • Links for Nightly builds - available since 15-01-2017 link

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 2017-01-12 tbd
Study documentation/specs received from developers 2017-01-12 2017-01-24
QA - Test plan creation 2017-01-23 2016-01-26
QA - Test cases/Env preparation 2017-01-23 2017-01-26
QA - Nightly Testing n/a n/a
QA - Aurora Testing 2017-02-13 2017-02-17
QA - Beta Testing 2017-02-28 2017-03-03
Release Date tbd tbd

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation Google docs/Test Rail
Test case execution TestRail
Bugs management Bugzilla

Overview

First released on Nightly in 15-01-2017 starting with build
Track the dates and build number where feature was merged to Aurora
Track the dates and build number where feature was merged to Release/Beta

Risk analysis

  • Identify the high-risk assumptions.
  • Identify existing bugs on the feature with high risk.
  • Identify if other areas are affected by the fix.

References

  • List and links for specs
 - see the official announcement [1]
  • Meta bug: Bug [2]

Test cases can be found also on TestRail at: [3]

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes
UI
Mouse-only operation Yes
Keyboard-only operation Yes
Display (HiDPI) Yes
Interraction (scroll, zoom) Yes
Usability and/or discoverability testing Yes Is this feature user friendly
Does this affect partner builds? Partner build testing Yes yes/no options, add comment with details about who will lead testing
Web Compatibility depends on the feature
Testing against target sites Yes
Survey of many sites for compatibility Yes
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. Yes
Coordinated testing/interop across the Firefox products: Desktop, Android, iOS
Interaction of this feature with other browser features Yes

Test suite

Compact Themes in Firefox - Sign off (Fx53) testsuite in [tba] (need Testrail account)

Bug Work

Tracking bug – meta tracker bug 1331679

Bug fix verification

Bugs associated to compact theme tracker bug

Full Query
ID Summary Status Resolution Priority
1094532 Tabs should have a more visible hover effect RESOLVED DUPLICATE P3
1095836 [Developer Edition] stop/reload/go icon is too dark RESOLVED WORKSFORME P3
1097467 [Linux] dark/light theme: Buttons on tabbar don't get the theme background RESOLVED WORKSFORME P3
1111138 [Windows 7/8] Window frame is slightly taller when using a lightweight theme RESOLVED WONTFIX --
1155958 Devedition back button shouldn't have round corners on Windows 8+ RESOLVED WONTFIX P3
1159103 Dark/Light themes: Fix the menubar foreground color on Windows 8 NEW P3
1165666 [DevEdition] Back button not rendered correctly RESOLVED WORKSFORME P3
1166083 Update back and forward button to use 2x icon in Windows Dev Edition theme RESOLVED WORKSFORME P3
1172553 DevEdition theme has a white titlebar on Yosemite whens tabs aren't in titlebar RESOLVED WORKSFORME P3
1184097 New height of the location and search bars breaks back/fwd buttons in DevEdition on Win10 RESOLVED WORKSFORME --
1186351 [HiDPI][Tablet Mode] Back button is smaller than the URL/search bar in Dev Edition on Windows 10 RESOLVED INVALID P4
1204624 Don't override devedition dark theme toolbar button styling on Windows 10/8 RESOLVED WORKSFORME P4
1208616 [Windows 7 aero glass][Light theme] White text in Menu bar and TabsToolbar is barely visible on light background RESOLVED WONTFIX P2
1229694 [devedition theme] Toolbar button active/open styles are not correctly applied in normal (non-private) windows RESOLVED WORKSFORME --
1303835 The grey text in the location bar doesn't have enough contrast with dark theme NEW P3
1307067 [Linux] bookmarks toolbar buttons should have a hardcoded foreground color in the devedition theme RESOLVED WORKSFORME P3
1317190 [Linux] New tab button in dev edition dark theme always has the darker background RESOLVED FIXED P3
1319034 [Windows 7 classic theme][devedition theme] tab close icons are bright (white) even on light devedition theme (which has light-grey tabs) RESOLVED FIXED P1
1326993 Findbar next/previous buttons look very bad on dark developer edition theme RESOLVED WORKSFORME P3
1339754 Compact themes are not applied after restart if you try to enable them with a complete theme installed RESOLVED WONTFIX P5
1339859 [meta] Very slow width resize action when multiple tabs are pinned using compact themes RESOLVED FIXED --
1340206 Compact Themes makes text on location bar to look blurry/bouncy when hovering with RTL builds VERIFIED FIXED P2
1340473 [Ubuntu] The X button from Developer Toolbar is barely visible with Compact Dark theme applied RESOLVED DUPLICATE --

23 Total; 2 Open (8.7%); 20 Resolved (86.96%); 1 Verified (4.35%);


Bugs associated to (compact-themes) Change the Dev Edition theme(s) to be alternative "compact themes" and let them ride the trains

Full Query
ID Summary Status Resolution Priority
1321911 Hovering over light or dark compact themes only ever previews the dark theme RESOLVED FIXED --
1323619 Create logos and descriptions for compact themes RESOLVED FIXED P3
1323833 Trim the list of recommended themes from 5 to 3 VERIFIED FIXED --
1329207 Change the default theme icon in about:addons / customize mode VERIFIED FIXED P2
1329262 Enable compact themes in mozscreenshots RESOLVED FIXED P2
1331276 Compact theme broken in Private Windows VERIFIED FIXED --
1331369 Switching to Compact Light Theme sometimes wrongly shows message in the add-on manager indicating it requires a restart of Firefox RESOLVED DUPLICATE --
1331449 2.1 - 3.29% tpaint (osx-10-10, windows8-64) regression on push 4bbe0c7e648909d6118e9cc4eea14296569db80b (Fri Jan 13 2017) VERIFIED FIXED --
1332375 Rename leftover devedition/urlbar-history-dropmarker.svg to compacttheme/urlbar-history-dropmarker.svg RESOLVED FIXED P3

9 Total; 0 Open (0%); 5 Resolved (55.56%); 4 Verified (44.44%);


Logged bugs
Full Query
ID Summary Status Resolution Priority
1339754 Compact themes are not applied after restart if you try to enable them with a complete theme installed RESOLVED WONTFIX P5
1339859 [meta] Very slow width resize action when multiple tabs are pinned using compact themes RESOLVED FIXED --
1340206 Compact Themes makes text on location bar to look blurry/bouncy when hovering with RTL builds VERIFIED FIXED P2
1340473 [Ubuntu] The X button from Developer Toolbar is barely visible with Compact Dark theme applied RESOLVED DUPLICATE --
1343792 Different colors in tab border between Compact Light and Dev edition theme RESOLVED WORKSFORME P3
1345473 Changing tab order using Compact themes cause an overlap VERIFIED FIXED P1

6 Total; 0 Open (0%); 4 Resolved (66.67%); 2 Verified (33.33%);


Criteria

Check list

  • All Criteria under each section of Quality Assurance Strategy should be green.
  • All test cases should be executed
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Mid-Aurora, Pre-Beta and Pre-Release Sign-off List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation tbd
Full Functional Tests Execution tbd
Automation Coverage N/A
Performance Testing N/A
All Defects Logged [DONE]
Critical/Blockers Fixed and Verified [DONE]
Metrics/Telemetry N/A
QA Signoff - Nightly Release N/A
QA Aurora - Full Testing [DONE] Sign off email sent on 2017-02-20
QA Signoff - Aurora Release
QA Beta - Full Testing [DONE] Sign off email sent 2017-03-03
QA Signoff - Beta Release [DONE] Sign off email sent 2017-30-03