QA/Windows-Stub-Installer

From MozillaWiki
< QA
Jump to: navigation, search

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
10/05/2016 1.0 Camelia Badau Initial draft

Overview

Purpose

Detail the purpose of this document. For example:

  • The test scope, focus areas and objectives
  • The test responsibilities
  • The test strategy for the levels and types of test for this release
  • The entry and exit criteria
  • The basis of the test estimates
  • Any risks, issues, assumptions and test dependencies
  • The test schedule and major milestones
  • The test deliverables


Scope

This wiki details the testing activities that will be conducted by QA for the Windows Stub Installer. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed

Ownership

Developer contacts: Matt Howell
QA: Camelia Badau
QA Peer: Mihai Boldan

Testing summary

Scope of Testing

In Scope

  • Currently, the stub installer contains branding download links that point to x86 builds for both x86 and x64 builds.

This feature, the Windows Stub Installer will silently choose 32-bit or 64-bit Firefox for the user. If the user is running a 64-bit version of Windows 7/8/8.1/10, then the stub installer will download the 64-bit Firefox package. Otherwise, it will download the 32-bit Firefox package. The Stub Installer will also provide an option for the user to override the 32/64-bit default setting via Options page.

The testing efforts will be invested in the following areas:

  • functionality: basic and advanced functionality to be verified according to the existing requirements;
  • usability: the Windows Stub Installer must be easy to use and straightforward;
  • integration: The QA team will ensure that there are no issues related to Windows Stub Installer on different Windows versions
  • accessibility (for RTL): the Windows Stub Installer feature can be used with High Contrast Theme, Screen Reader and on touch screen devices.

Requirements for testing

The current system requirements to get 64-bit Firefox from the stub installer are:

  • Win64 OS version 7, 8, 8.1, or 10 (i.e. XP and Vista are not supported)


Environments

Full Testing will be performed on:

  • Windows 7 x64/x86
  • Windows 8 x64/x86
  • Windows 8.1 x64/x86
  • Windows 10 x64/x86
  • Windows XP
  • Windows Vista

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Evaluation Criteria Test Type Owners
1 Stub Installer on a 64-bit version of Windows 7/8/8.1/10 Verify the Firefox package that will be downloaded the stub installer will download the 64-bit Firefox package Manual Rel Team
2 Stub Installer on XP, Vista, or 32-bit Windows 7/8/8.1/10 Verify the Firefox package that will be downloaded the stub installer will download the 32-bit Firefox package Manual Rel Team
3 Installation location Verify where the Firefox package (32-bit or 64-bit) in installed The Firefox package is installed to the 32-bit/64-bit Program Files directory/etc. Manual Rel Team
4 Stub Installer panel Verify the checkboxes displayed A new checkbox is added: "Create Desktop and Quick Launch Shortcuts for Firefox" for Windows XP and Vista/"Create Desktop and Task Bar Shortcuts for Firefox" for Windows 7+ Manual Rel Team
5 Stub Installer options Verify the options displayed A new option is added which allows to override the 32/64-bit default Manual Rel Team

Builds

This section should contain links for builds with the feature -

  • Links for Nightly builds: TBD
  • Links for Aurora builds: TBD
  • Links for Beta builds: TBD

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project TBD
Study documentation/specs received from developers 10/04/2016
QA - Test plan creation 10/05/2016
QA - Test cases/Env preparation
QA - Nightly Testing
QA - Aurora Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation Test Rail
Test case execution Test Rail
Bugs management Bugzilla

Status

Overview

First releases to Nightly: 2016-11-24 (Firefox 53)
Track the dates and build number where feature was merged to Aurora:
Track the dates and build number where feature was merged to Release/Beta:

Risk analysis

Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix


References

  • UX Specification
    • The current UX plan is described in this Google doc
  • Tracking Bug
    • Bug 797208 - Stub installer should automatically select 32-bit or 64-bit Firefox at install-time]

Testcases

Overview

  • Testcases can be found here
  • Some testing scenarios:
    • If the user is running a 64-bit version of Windows 7/8/8.1/10, then the stub installer will download the 64-bit Firefox package and install it to the "C:\Program Files"
    • If the user is running XP, Vista, or 32-bit Windows 7/8/8.1/10, then the stub installer will download the 32-bit Firefox package and install it to the "C:\Program Files (x86)"
    • If the user wants to override the 32/64-bit default, there is an option in the Options page
    • A new checkbox is added: "Create Desktop and Quick Launch Shortcuts for Firefox" for Windows XP and Vista/"Create Desktop and Task Bar Shortcuts for Firefox" for Windows 7+
    • for 64-bit version of Windows 7/8/8.1/10, a new drop-down is added: "Version to Install" with "64-bit Nightly" and "32-bit Nightly" options in the list
    • for XP, Vista, or 32-bit Windows 7/8/8.1/10 (the incompatible Windows versions), the "Version to Install" option is not displayed (because the user on an incompatible OS is not able to select 64-bit Firefox)
    • The user is able to select from the "Version to Install" the wanted version
    • The text from the "Version to Install" (64-bit Nightly, 32-bit Nightly) is correctly displayed and is sized properly for the text it contains
    • When the maintenance service is already installed or if the user cannot install it the checkbox is not displayed
    • The case when both 32 and a 64 bit Firefox are installed into non-default directories -> check if the correct install location is shown when switching between 32 and 64 bit
    • RTL locales testing: check that the box and its label correctly swapped positions and "64-bit Nightly" and "32-bit Nightly" options from "Version to Install" are correctly substituted.
    • RTL testing: check that the Stub Installer is correctly displayed and correctly works with High Contrast Theme
    • RTL testing: the stub installer with Screen Reader
    • RTL testing: the stub installer on touch screen devices
    • Verify that the text in the drop-down has the same baseline as the text beside it.
    • There is no excessive padding between the drop-down and the label.
    • If an existing installation is found, its architecture is detected and its location is only used as the default when the same architecture is selected. The directory for an existing 32-bit installation won't be used as the default for a new 64-bit installation.

Note: These are several examples of scenarios and based on these the testcases will be created.

Test Areas

Test Areas Covered Details
Private Window No
Multi-Process Enabled No
Multi-process Disabled No
Theme (high contrast) Yes
UI
Mouse-only operation
Keyboard-only operation
Display (HiDPI) No
Interraction (scroll, zoom) No
Usable with a screen reader Yes for RTL
Usability and/or discoverability testing Yes Is this feature user friendly
Help/Support
Help/support interface required No
Support documents planned(written) No
Install/Upgrade
Feature upgrades/downgrades data as expected Yes
Does sync work across upgrades No
Requires install testing Yes
Affects first-run or onboarding No
Does this affect partner builds? Partner build testing No
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration No
Network proxies/autoconfig No
ESR behavior changes No
Locked preferences No
Data Monitoring
Temporary or permanent telemetry monitoring No
Telemetry correctness testing No
Server integration testing No
Offline and server failure testing No
Load testing No
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? No
Comprehensive API testing No
Permissions No
Testing with existing/popular addons No
Security Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review No
Privilege escalation testing No
Fuzzing No
Web Compatibility depends on the feature
Testing against target sites No
Survey of many sites for compatibility No
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS No
Interaction of this feature with other browser features No

Test suite

Full Test suite - TestSuite
Smoke Test suite - Link with the gdoc, follow the format from  TBC 
Regression Test suite - Link with the gdoc - if available/needed.

Bug Work

Tracking bug: Bug 797208 - Stub installer should automatically select 32-bit or 64-bit Firefox at install-time

Logged bugs ( blocking 797208 )

Full Query
ID Priority Component Assigned to Summary Status Target milestone
1317281 P1 Installer SonicWALL Firewall Appliance with Gateway Anti-Virus is currently blocking the 64-bit stub installer VERIFIED ---
1332063 -- Installer Disable Stub Installer pre-merge RESOLVED ---
1334786 P2 Installer Molly Howell (she/her) (no longer active) Installer textbox is too small to display the sentences correctly VERIFIED Firefox 54

3 Total; 0 Open (0%); 1 Resolved (33.33%); 2 Verified (66.67%);


Bug Verification (blocking 797208)

Full Query
ID Priority Component Assigned to Summary Status Target milestone
1317281 P1 Installer SonicWALL Firewall Appliance with Gateway Anti-Virus is currently blocking the 64-bit stub installer VERIFIED ---
1334786 P2 Installer Molly Howell (she/her) (no longer active) Installer textbox is too small to display the sentences correctly VERIFIED Firefox 54

2 Total; 0 Open (0%); 0 Resolved (0%); 2 Verified (100%);


NEW issues

[NEW] Bug 1339870 - Win 64 stub installer on aurora defaults to installing en-us
[NEW] Bug 1340131 - Stub installer checkboxes spacing inconsistent


Known issues

[NEW] Bug 1319202 - Stub installer checkbox labels can flicker when toggling checkbox
[NEW] Bug 1319479 - Stub installer architecture dropdown text not vertically aligned with label
[NEW] Bug 1319482 - Stub installer option label spacing inconsistent
[NEW] Bug 1319192 - Text cursor misaligned at end of stub installer destination folder box

Sign off

Criteria

Check list

  • All test cases should be executed
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results



Old Sign-Offs

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Full Functional Tests Execution [DONE]
Automation Coverage [N/A]
Performance Testing [N/A]
All Defects Logged
Critical/Blockers Fixed and Verified
Daily Status Report (email/etherpad statuses/ gdoc with results) [N/A]
Metrics/Telemetry [N/A]
QA Signoff - Nightly Release
QA Aurora - Full Testing [DONE]
QA Signoff - Aurora Release [NOT STARTED]
QA Beta - Full Testing [NOT STARTED]
QA Signoff - Beta Release [NOT STARTED]