QA/Windows Child Mode

From MozillaWiki
< QA
Jump to: navigation, search

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
02/17/2016 1.0 Brindusa Tot Created first draft
06/14/2016 2.0 Iulia Cristescu Updated several TP sections

Overview

Purpose

Detail the purpose of this document. For example:

  • The test scope, focus areas and objectives
  • The test responsibilities
  • The test strategy for the levels and types of test for this release
  • The entry and exit criteria
  • The basis of the test estimates
  • Any risks, issues, assumptions and test dependencies
  • The test schedule and major milestones
  • The test deliverables

Scope

This wiki details the testing that will be performed by the project team for the Firefox Support for Windows Child Mode project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed

Ownership

Developer contacts: David Keeler (irc: keeler)

Engineering contact: Matt Wobensmith (irc: mwobensmith)(irc: matt)


Engineering Team - Nightly Testing:
Rares Bologa (irc: RaresB) - PM for QA team

Brindusa Tot (irc: brindusat) - leading QA efforts

Simona Badau (irc: simonab) - QA

Kamil Jozwiak (irc: kjozwiak) - QA
Desktop Team - Release Testing:
Cornel Ionce (irc: cornel_ionce) - QA

Iulia Cristescu (irc: IuliaC) - QA

Testing summary

Scope of Testing

In Scope

Child mode is a feature of Windows that allows a parent (administrator) to set up a child account in order to monitor internet activity. It does so by using an SSL certificate that allows the operating system to spy or “MiTM” the child’s connections. While this helps the parent keep track of the child’s activity, it presents a fundamental problem for the way Firefox works. Unlike other browsers, Firefox does not use the system’s SSL certificates to verify connections, but instead, relies on its own certificate store. Since this Microsoft certificate is not present here, Firefox will see these connections as untrusted, and the user of the child account will not be able to view secure sites. The solution is to import this Microsoft certificate only when Firefox detects that it is running in child mode. Be aware that the the Microsoft Family Safety feature is only working with other browsers, like Firefox, on Windows 8.1.

The testing efforts will be invested on the following areas:

  • Setup environment - create child account, import Microsoft certificate for Firefox, update Firefox configs
  • Verify Firefox works with parental controls for the account (child or administrator) enabled/disabled
  • Verify parental controls works as on other browsers(IE/Edge) if enabled for the account(web filtering settings, logging of child actions on administator account )

Out of Scope

- Functionality of the Microsoft Family Safety Mode with all available features is not included in the testing efforts. Only Firefox support for the Microsoft Child Mode feature on Windows 8.1 is tested.

Requirements for testing

Environments

The Microsoft Family Safety feature is available on Windows 8, Windows 8.1 and Windows 10, but Microsoft offers support for Firefox only on Windows 8.1 (On Windows 8 there is no MitM certificate; on Windows 10 only the native browsers like Edge and IE are supported).


Testing was divided in 2 big areas and was done on the following OS-es:

  • Child Mode Functionality (Firefox behavior on a child account)
    • on Windows 8.1
  • Firefox Basic Functionality (Firefox behavior on a child account and on an admin/parent account having the preference security.family_safety.mode=2)
    • Windows 8
    • Windows 8.1
    • Windows 10


Other configuration/environmental setup needed:

  • In Firefox configuration settings need to add: security.family_safety.mode with values:
    • 0 - disables the feature entirely
    • 1 - enables detection of Family Safety mode but doesn't import the root (mostly for gathering telemetry/diagnostics)
    • 2 - enables detection and imports the root

Child Mode Support Matrix:

Operating System Child Mode Support Loads Secure Sites [Default Build] Access Secure Sites [Try Build with Feature]
Win 10 x64 NO (Edge Only) PASS (x64 & x86 fx builds) PASS (x64 & x86 fx builds)
Win 10 x86 NO (Edge Only) PASS (x86 fx builds) PASS (x86 fx builds)
Win 8 x64 YES PASS (x64 & x86 fx builds) PASS (x64 & x86 fx builds)
Win 8 x86 YES PASS (x86 fx builds) PASS (x86 fx builds)
Win 8.1 x64 YES FAIL (x64 & x86 builds) PASS (x64 & x86 fx builds)
Win 8.1 x86 YES FAIL (x86 builds) PASS (x86 fx builds)
Win 7 x64 YES (Win Essentials) PASS (x64 & x86 fx builds) PASS (x64 & x86 fx builds)
Win 7 x86 YES (Win Essentials) PASS (x86 fx builds) PASS (x86 fx builds)
Last Test/Check: 3/22/2016

Quick Note: Win 8.1 is the only Windows OS that uses a MITM certificate under Child Mode accounts

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Evaluation Criteria Test Type Owners
1 Setup environment The objective the test is trying to demonstrate Firefox works after setting up Windows child mode account Navigation is allowed Manual Eng Team & Desktop Team
2 Firefox with parental control disabled The objective of test is trying to demonstrate Firefox works with parental control disabled Navigation is allowed to any website, Firefox works normally on both child and administrator Manual Eng Team & Desktop Team
3 Firefox with parental control enabled, no other settings done The objective of test is trying to demonstrate Firefox works with parental control enabled Navigation is allowed, but Adult content is blocked. Other Firefox functions works normally on both child and administrator Manual Eng Team & Desktop Team
4 Firefox with parental control enabled, allow specified sites The objective of test is trying to demonstrate Firefox works with parental control enabled Navigation is allowed only on specified sites. Other Firefox functions works normally on both child and administrator Manual Eng Team & Desktop Team
5 Firefox with parental control enabled, block specified sites The objective of test is trying to demonstrate Firefox works with parental control enabled Navigation is allowed, but blocked sites and Adult content are not allowed. Other Firefox functions works normally on both child and administrator Manual Eng Team & Desktop Team
6 Firefox with parental control enabled, set up different Web Filtering Levels The objective of test is trying to demonstrate Firefox works with parental control enabled Navigation is allowed, depending on the Filtering level. Manual Eng Team & Desktop Team
7 Firefox with parental control enabled, set up Firefox as a restricted app The objective of test is trying to demonstrate Firefox cannot be opened if is set as restricted Firefox cannot be opened if is set as restricted Manual Eng Team & Desktop Team
8 Firefox with parental control enabled, Logging disabled/enabled The objective of this test is to validate logging on child activity As administrator of child account I am able to see child actions Manual Eng Team & Desktop Team
9 Multiple different child accounts The objective of this test is to child mode settings are preserved only on setup child account If there are multiple accounts, we don’t want enabling/disabling child mode in one of them to affect how Firefox behaves in a other account Manual Eng Team & Desktop Team

Builds

This section should contain links for builds with the feature -

  • Try builds with the fix are available to the the TryBuild1, TryBuild2, TryBuild3
  • Links for Nightly builds - available since April 3th 2016 link
  • Links for Aurora builds - available since April 26, 2016 link
  • Links for Beta builds - available since June 6, 2016 link

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 02/16/2016
Study documentation/specs received from developers 02/16/2016 02/19/2016
QA - Test plan creation 02/17/2016 04/01/2016
QA - Test cases/Env preparation 02/19/2016 02/26/2016
QA - Nightly Testing 04/04/2016 04/08/2016
QA - Aurora Testing 04/26/2016; 08/01/2016 06/03/2016; 09/12/2016
QA - Beta Testing 06/08/2016; 09/12/2016 Disabled till Fx 50; 11/07/2016
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation Google docs
Test case execution Google docs
Bugs management Bugzilla

Status

Overview

The support for the Windows Child Mode feature was added in Nightly with Bug 1239166 which landed in 2016-04-02. First available in this build.

Track the dates and build number where feature was merged to Aurora - 48.0a2 (2016-04-26) 
Track the dates and build number where feature was merged to Release/Beta - 48.0b1 build1 (20160606200529)

The feature was disabled by default in Firefox 48 and enabled starting with Firefox 50.

Risk analysis

Microsoft future updates on Family Safety feature, might affect the way Firefox behaves under Child Mode accounts or even on Standard accounts. Insecurity, due to the fact that Microsoft offered support also, for windows 10, but in the meanwhile they dropped it.

References

  • Specs available are:
 Available specs - link
 Child Mode Support Matrix - link
 Windows 8.1 upgrade - link
 Microsoft Child Mode updates - link
 Microsoft Family Safety portal - link

Testcases

Overview

All the test cases are available at the link

Kamil's testing is available at the link

Windows Child Mode 48 sign-off test cases

Windows Child Mode 50 testsuite in Testrail (need Testrail account)

Windows Child Mode - Mid-Aurora Sign off (Fx50) testsuite in Testrail (need Testrail account)

Windows Child Mode - Pre-Beta Sign off (Fx50) testsuite in Testrail (need Testrail account)

Windows Child Mode - Pre-Release Sign off (Fx50) testsuite in Testrail (need Testrail account)

Test Areas

Test Areas Covered Details
Private Window yes
Multi-Process Enabled yes
Multi-process Disabled no
Theme (high contrast) yes
UI
Mouse-only operation no
Keyboard-only operation no
Display (HiDPI) no
Interraction (scroll, zoom) no
Usable with a screen reader no e.g. with NVDA
Usability and/or discoverability testing yes Is this feature user friendly
Help/Support
Help/support interface required no Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) no Make sure support documents are written and are correct. Please see the reference section.
Install/Upgrade
Feature upgrades/downgrades data as expected no
Does sync work across upgrades no
Requires install testing no separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding no Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Does this affect partner builds? Partner build testing no yes/no options, add comment with details about who will lead testing
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration no
Network proxies/autoconfig no
ESR behavior changes no
Locked preferences no
Data Monitoring
Temporary or permanent telemetry monitoring no List of error conditions to monitor
Telemetry correctness testing no
Server integration testing no
Offline and server failure testing no
Load testing no
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? no
Comprehensive API testing no
Permissions no
Testing with existing/popular addons yes 10 top rated and 10 most popular add-ons from AMO.
Security Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review no
Privilege escalation testing no
Fuzzing no
Web Compatibility depends on the feature
Testing against target sites yes 10 top alexa sites
Survey of many sites for compatibility
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. no
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS no
Interaction of this feature with other browser features no

Test suite

Full Test suite - link
Child Mode 48 sign-off test cases
Windows Child Mode - Mid-Aurora Sign off (Fx50) testsuite in Testrail (need Testrail account)
Windows Child Mode - Pre-Beta Sign off (Fx50) testsuite in Testrail (need Testrail account)
Windows Child Mode - Pre-Release Sign off (Fx50) testsuite in Testrail (need Testrail account)

Bug Work

Tracking bug – bug 1259463

Bug fix verification
Bug No Summary Status Firefox Version
123 bug summary NEW Nighly 45
Logged bugs

Bug 1258990 Insecure connection not received when SSL certificate is removed
Bug 1259013 Private Browsing is allowed in Child Mode
Bug 1259028 No warning when downloads are blocked in Child Mode
Bug 1259043 Adult Content is allowed in Child Mode when Video DownloadHelper is installed
Bug 1260098 about:home page not displayed on the Windows Child Mode try build
Bug 1260109 Broken carousel on amazon.com on the Windows Child Mode try build
Bug 1251644 A vertical line is displayed in the right side of YouTube's videos when Quality is toggled
Bug 1255732 Firefox crashes on Vimeo on AMD hardware
Bug 1256345 Vertical lines displayed in QuickTime video controls bar
Bug 1256648 Google logo overlaps when scrolling
Bug 1256661 Yahoo Smart Views folders are not visible
Bug 1256664 Yahoo News Feed title is cut off (due to use of modern flexbox to emulate -webkit-box, and default "flex-shrink:1")
Bug 1262767 NS_ERROR_FAILURE in nsITelemetry.getKeyedHistogramById when opening miniclip.com
Bug 1252842 mlb.com doesn't display videos correctly with Adobe Flash Player installed
Bug 1277240 The Microsoft Family Safety certificate is still imported in the Authorities tab in the Certificate Manager
Bug 1278582 After update to latest FF version - the Microsoft Family Safety is imported when the Child Mode feature is disabled
Bug 1278919 Page is not completely loaded

Sign off

Criteria

Check list

  • All test cases should be executed on Windows 8.1 - Child Mode functionality and Windows 8, 8.1 and 10 - Firefox functionality
  • No Blockers should be acceptable at any point in Aurora, Beta, Release
  • No Critical issues beyond Beta - low level crashes may be considered Major
  • More details about the release criteria on each phase are available at link.

Results

Nightly testing

List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite - link
    • Regression Test suite, if needed/available

Aurora testing

Mid-Aurora Sign-off
List of OSes that will be covered by testing: Windows 8.1 - Child Mode Functionality and Windows 10 - Specific Windows 10 Child Mode Tests

  • Link for the tests run

Pre-Beta Sign-off
List of OSes that will be covered by testing: Windows 10 - Specific Windows 10 Child Mode Tests and Windows 8, 8.1 and 10 - Firefox Basic Functionality

  • Link for the tests run

The feature was disabled by default in Firefox 48 and enabled starting with Firefox 50.

Mid-Aurora Sign-off
List of OSes that will be covered by testing: Windows 8.1 - Smoke, Exploratory and Regression Testing

  • Testrail Link for the tests run (need Testrail account)
  • Link for the Manual Testing Summary

Pre-Beta Sign-off
List of OSes that will be covered by testing: Windows 8.1 and Windows 10 - Smoke and Exploratory Testing

  • Link for the tests run (need Testrail account)
  • Link for the Manual Testing Summary

Pre-Release Sign-off
List of OSes that will be covered by testing: Windows 8.1 and Windows 10 - Smoke and Exploratory Testing

  • Link for the tests run (need Testrail account)

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Full Functional Tests Execution [DONE]
Automation Coverage TBD
Performance Testing N/A
All Defects Logged [DONE]
Critical/Blockers Fixed and Verified [DONE]
Metrics/Telemetry N/A
QA Signoff - Nightly Release [DONE] Conditional SignOff email sent on 04.14.2016
QA Aurora - Full Testing [DONE] (AT RISK) Sign off Email sent on 05/23/2016

(Green) Sign off Email sent on 08/26/2016

QA Signoff - Aurora Release [DONE] (Green) Sign off Email sent on 06/03/2016

(Green) Sign off Email sent on 09/15/2016

QA Beta - Full Testing [DONE]
QA Signoff - Beta Release [DONE] (Yellow) Sign off Email sent on 10/31/2016