QA/Windows-Stub-Installer
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
| Date | Version | Author | Description |
|---|---|---|---|
| 10/05/2016 | 1.0 | Camelia Badau | Initial draft |
Contents
Overview
Purpose
Detail the purpose of this document. For example:
- The test scope, focus areas and objectives
- The test responsibilities
- The test strategy for the levels and types of test for this release
- The entry and exit criteria
- The basis of the test estimates
- Any risks, issues, assumptions and test dependencies
- The test schedule and major milestones
- The test deliverables
Scope
This wiki details the testing activities that will be conducted by QA for the Windows Stub Installer. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
- What will be tested
- How testing will be performed
Ownership
Developer contacts: Matt Howell
QA: Camelia Badau
QA Peer: Mihai Boldan
Testing summary
Scope of Testing
In Scope
- Currently, the stub installer contains branding download links that point to x86 builds for both x86 and x64 builds.
This feature, the Windows Stub Installer will silently choose 32-bit or 64-bit Firefox for the user. If the user is running a 64-bit version of Windows 7/8/8.1/10, then the stub installer will download the 64-bit Firefox package. Otherwise, it will download the 32-bit Firefox package. The Stub Installer will also provide an option for the user to override the 32/64-bit default setting via Options page.
The testing efforts will be invested in the following areas:
- functionality: basic and advanced functionality to be verified according to the existing requirements;
- usability: the Windows Stub Installer must be easy to use and straightforward;
- integration: The QA team will ensure that there are no issues related to Windows Stub Installer on different Windows versions
- accessibility (for RTL): the Windows Stub Installer feature can be used with High Contrast Theme, Screen Reader and on touch screen devices.
Requirements for testing
The current system requirements to get 64-bit Firefox from the stub installer are:
- Win64 OS version 7, 8, 8.1, or 10 (i.e. XP and Vista are not supported)
Environments
Full Testing will be performed on:
- Windows 7 x64/x86
- Windows 8 x64/x86
- Windows 8.1 x64/x86
- Windows 10 x64/x86
- Windows XP
- Windows Vista
Test Strategy
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
| Ref | Function | Test Objective | Evaluation Criteria | Test Type | Owners |
|---|---|---|---|---|---|
| 1 | Stub Installer on a 64-bit version of Windows 7/8/8.1/10 | Verify the Firefox package that will be downloaded | the stub installer will download the 64-bit Firefox package | Manual | Rel Team |
| 2 | Stub Installer on XP, Vista, or 32-bit Windows 7/8/8.1/10 | Verify the Firefox package that will be downloaded | the stub installer will download the 32-bit Firefox package | Manual | Rel Team |
| 3 | Installation location | Verify where the Firefox package (32-bit or 64-bit) in installed | The Firefox package is installed to the 32-bit/64-bit Program Files directory/etc. | Manual | Rel Team |
| 4 | Stub Installer panel | Verify the checkboxes displayed | A new checkbox is added: "Create Desktop and Quick Launch Shortcuts for Firefox" for Windows XP and Vista/"Create Desktop and Task Bar Shortcuts for Firefox" for Windows 7+ | Manual | Rel Team |
| 5 | Stub Installer options | Verify the options displayed | A new option is added which allows to override the 32/64-bit default | Manual | Rel Team |
Builds
This section should contain links for builds with the feature -
- Links for Nightly builds: TBD
- Links for Aurora builds: TBD
- Links for Beta builds: TBD
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
| Project phase | Start Date | End Date |
|---|---|---|
| Start project | TBD | |
| Study documentation/specs received from developers | 10/04/2016 | |
| QA - Test plan creation | 10/05/2016 | |
| QA - Test cases/Env preparation | ||
| QA - Nightly Testing | ||
| QA - Aurora Testing | ||
| QA - Beta Testing | ||
| Release Date |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | Test Rail |
| Test case execution | Test Rail |
| Bugs management | Bugzilla |
Status
Overview
First releases to Nightly: 2016-11-24 (Firefox 53) Track the dates and build number where feature was merged to Aurora: Track the dates and build number where feature was merged to Release/Beta:
Risk analysis
Identify the high-risk assumptions Identify existing bugs on the feature with high risk Identify if other areas are affected by the fix
References
- UX Specification
- The current UX plan is described in this Google doc
- Tracking Bug
- Bug 797208 - Stub installer should automatically select 32-bit or 64-bit Firefox at install-time]
Testcases
Overview
- Testcases can be found here
- Some testing scenarios:
- If the user is running a 64-bit version of Windows 7/8/8.1/10, then the stub installer will download the 64-bit Firefox package and install it to the "C:\Program Files"
- If the user is running XP, Vista, or 32-bit Windows 7/8/8.1/10, then the stub installer will download the 32-bit Firefox package and install it to the "C:\Program Files (x86)"
- If the user wants to override the 32/64-bit default, there is an option in the Options page
- A new checkbox is added: "Create Desktop and Quick Launch Shortcuts for Firefox" for Windows XP and Vista/"Create Desktop and Task Bar Shortcuts for Firefox" for Windows 7+
- for 64-bit version of Windows 7/8/8.1/10, a new drop-down is added: "Version to Install" with "64-bit Nightly" and "32-bit Nightly" options in the list
- for XP, Vista, or 32-bit Windows 7/8/8.1/10 (the incompatible Windows versions), the "Version to Install" option is not displayed (because the user on an incompatible OS is not able to select 64-bit Firefox)
- The user is able to select from the "Version to Install" the wanted version
- The text from the "Version to Install" (64-bit Nightly, 32-bit Nightly) is correctly displayed and is sized properly for the text it contains
- When the maintenance service is already installed or if the user cannot install it the checkbox is not displayed
- The case when both 32 and a 64 bit Firefox are installed into non-default directories -> check if the correct install location is shown when switching between 32 and 64 bit
- RTL locales testing: check that the box and its label correctly swapped positions and "64-bit Nightly" and "32-bit Nightly" options from "Version to Install" are correctly substituted.
- RTL testing: check that the Stub Installer is correctly displayed and correctly works with High Contrast Theme
- RTL testing: the stub installer with Screen Reader
- RTL testing: the stub installer on touch screen devices
- Verify that the text in the drop-down has the same baseline as the text beside it.
- There is no excessive padding between the drop-down and the label.
- If an existing installation is found, its architecture is detected and its location is only used as the default when the same architecture is selected. The directory for an existing 32-bit installation won't be used as the default for a new 64-bit installation.
Note: These are several examples of scenarios and based on these the testcases will be created.
Test Areas
| Test Areas | Covered | Details |
|---|---|---|
| Private Window | No | |
| Multi-Process Enabled | No | |
| Multi-process Disabled | No | |
| Theme (high contrast) | Yes | |
| UI | ||
| Mouse-only operation | ||
| Keyboard-only operation | ||
| Display (HiDPI) | No | |
| Interraction (scroll, zoom) | No | |
| Usable with a screen reader | Yes | for RTL |
| Usability and/or discoverability testing | Yes | Is this feature user friendly |
| Help/Support | ||
| Help/support interface required | No | |
| Support documents planned(written) | No | |
| Install/Upgrade | ||
| Feature upgrades/downgrades data as expected | Yes | |
| Does sync work across upgrades | No | |
| Requires install testing | Yes | |
| Affects first-run or onboarding | No | |
| Does this affect partner builds? Partner build testing | No | |
| Enterprise | Raise up the topic to developers to see if they are expecting to work different on ESR builds | |
| Enterprise administration | No | |
| Network proxies/autoconfig | No | |
| ESR behavior changes | No | |
| Locked preferences | No | |
| Data Monitoring | ||
| Temporary or permanent telemetry monitoring | No | |
| Telemetry correctness testing | No | |
| Server integration testing | No | |
| Offline and server failure testing | No | |
| Load testing | No | |
| Add-ons | If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on. | |
| Addon API required? | No | |
| Comprehensive API testing | No | |
| Permissions | No | |
| Testing with existing/popular addons | No | |
| Security | Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature. | |
| 3rd-party security review | No | |
| Privilege escalation testing | No | |
| Fuzzing | No | |
| Web Compatibility | depends on the feature | |
| Testing against target sites | No | |
| Survey of many sites for compatibility | No | |
| Interoperability | depends on the feature | |
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | No | |
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | No | |
| Interaction of this feature with other browser features | No |
Test suite
Full Test suite - TestSuite Smoke Test suite - Link with the gdoc, follow the format from TBC Regression Test suite - Link with the gdoc - if available/needed.
Bug Work
Tracking bug: Bug 797208 - Stub installer should automatically select 32-bit or 64-bit Firefox at install-time
Logged bugs ( blocking 797208 )
| ID | Priority | Component | Assigned to | Summary | Status | Target milestone |
|---|---|---|---|---|---|---|
| 1317281 | P1 | Installer | SonicWALL Firewall Appliance with Gateway Anti-Virus is currently blocking the 64-bit stub installer | VERIFIED | --- | |
| 1332063 | -- | Installer | Disable Stub Installer pre-merge | RESOLVED | --- | |
| 1334786 | P2 | Installer | Molly Howell (she/her) (mostly inactive) | Installer textbox is too small to display the sentences correctly | VERIFIED | Firefox 54 |
3 Total; 0 Open (0%); 1 Resolved (33.33%); 2 Verified (66.67%);
Bug Verification (blocking 797208)
| ID | Priority | Component | Assigned to | Summary | Status | Target milestone |
|---|---|---|---|---|---|---|
| 1317281 | P1 | Installer | SonicWALL Firewall Appliance with Gateway Anti-Virus is currently blocking the 64-bit stub installer | VERIFIED | --- | |
| 1334786 | P2 | Installer | Molly Howell (she/her) (mostly inactive) | Installer textbox is too small to display the sentences correctly | VERIFIED | Firefox 54 |
2 Total; 0 Open (0%); 0 Resolved (0%); 2 Verified (100%);
NEW issues
[NEW] Bug 1339870 - Win 64 stub installer on aurora defaults to installing en-us
[NEW] Bug 1340131 - Stub installer checkboxes spacing inconsistent
Known issues
[NEW] Bug 1319202 - Stub installer checkbox labels can flicker when toggling checkbox
[NEW] Bug 1319479 - Stub installer architecture dropdown text not vertically aligned with label
[NEW] Bug 1319482 - Stub installer option label spacing inconsistent
[NEW] Bug 1319192 - Text cursor misaligned at end of stub installer destination folder box
Sign off
Criteria
Check list
- All test cases should be executed
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
- Mid-Nightly 55.0a1 Sign-off
Old Sign-Offs
- pre-Aurora sign-off
- mid-Aurora sign-off
Checklist
| Exit Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | ||
| Testing Infrastructure setup | ||
| Test Plan Creation | [DONE] | |
| Test Cases Creation | [DONE] | |
| Full Functional Tests Execution | [DONE] | |
| Automation Coverage | [N/A] | |
| Performance Testing | [N/A] | |
| All Defects Logged | ||
| Critical/Blockers Fixed and Verified | ||
| Daily Status Report (email/etherpad statuses/ gdoc with results) | [N/A] | |
| Metrics/Telemetry | [N/A] | |
| QA Signoff - Nightly Release | ||
| QA Aurora - Full Testing | [DONE] | |
| QA Signoff - Aurora Release | [NOT STARTED] | |
| QA Beta - Full Testing | [NOT STARTED] | |
| QA Signoff - Beta Release | [NOT STARTED] |
