QA/Captive Portals

From MozillaWiki
< QA
Jump to: navigation, search

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
10/27/2016 1.0 Adrian Florinescu Initial Draft
11/08/2016 1.1 Adrian Florinescu TP review with Nihanth; updates based on it.
11/21/2016 1.2 Adrian Florinescu Updates
02/01/2017 2.0 Adrian Florinescu Updates: preparing for hand-over to RelQA

Overview

Captive Portal feature covers the detection and implementation of handles for captive portals inside Firefox browser. Firefox is expected to handle the handling of a captive portal page upon detection of such. Certain business logic related to the detection and management of a captive portal is expected to be in place and documented in the Specs section.

Purpose

Detail the purpose of this document. For example:

  • The test scope, focus areas and objectives
  • The test responsibilities
  • The test strategy for the levels and types of test for this release
  • The entry and exit criteria
  • The basis of the test estimates
  • Any risks, issues, assumptions and test dependencies
  • The test schedule and major milestones
  • The test deliverables

Scope

This wiki details the testing that will be performed by the project team for the Captive Portals feature. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed

Ownership

• Program Management:

• Product Management:

• Engineering Management:

• Eng.:

• QA

Testing summary

Scope of Testing

In Scope

  • testing a wifi-connection that has enabled captive portals
  • testing a wan connection which has enabled captive portals (not done)

Out of Scope

  • Due to security issues and since the types of captive portals implementation vary depending on manufacturer, setup, security setups, etc., the tests which will be designed and ran will cover the basic functionality that impacts Firefox browser and not the compatibility with all existing captive portals.

Requirements for testing

Environments

  • standard OS's: Ubuntu, Windows, OSX (versions TBD)
  • Captive Portal compliant and activated - wifi network
  • Wireless connection available

Channel dependent settings (configs) and environment setups

Nightly

Captive Portal feature active by default.

Beta

Captive Portal feature active by default.

Post Beta / Release

Not Active.

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Test Type Risk Assessment and Coverage(RAC) Owners
TO-1 Active Portal detection To verify that the active portal is detected in all expected scenarios Manual RAC-1, RAC-2, RAC-3, RAC-4,RAC-8 Eng Team
TO-2 Active Portal notification bar To verify the behavior of the active portal notification bar Manual RAC-5 Eng Team
TO-3 Active Portal Tab To verify that new tab is opened and it's further behavior according to specs on active detection Manual RAC-6, RAC-7 Eng Team
TO-4 Inactive Portal To verify the behavior of FF when the captive portal is not active anymore Manual - Eng Team
TO-5 Preferences To verify the preference - on/off Manual - Eng Team
TO-6 Error page To verify the Error page Manual - Eng Team
TO-7 Localization To verify that the Notification bar, Error page and any other element is localized Manual - Eng Team

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 Multiple FF windows TO-1 2-Moderate 3-High 2-Medium 12
RAC-2 Multiple network connections (wi-fi x2/ wi-fi - wan) TO-1 1- Low 1-Unlikely 2-Medium 2
RAC-3 Existing proxy settings TO-1 2-Moderate 2-Possible 2-Medium 8
RAC-4 Integration with (Windows) Captive Portal managers TO-1 2-Moderate 3-Almost Certain 2-Medium 12
RAC-5 Notification management against Pop-up blockers TO-2 3-High 2-Possible 3-High 18
RAC-6 (OSX) Multi users TO-3 2-Moderate 2-Possible 2-Medium 8
RAC-7 (OSX) Integration with Captive Network Assistant TO-3 2-Moderate 2-Possible 3-High 12
RAC-8 (Ubuntu)Circumventing Captive portal TO-1 2-Moderate 2-Possible 3-High 12

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Builds

This section should contain links for builds with the feature -

  • Links for Nightly builds
  • Links for Aurora builds
  • Links for Beta builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 24.10.2016
Study documentation/specs received from developers 10.31.2016 11.08.2016
QA - Test plan creation 10.31.2016 11.08.2016
QA - Test cases/Env preparation 10.31.2016
QA - Nightly Testing
QA - Aurora Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla

Status

Overview

Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Aurora
Track the dates and build number where feature was merged to Release/Beta

References

  • List and links for specs
 https://public.etherpad-mozilla.org/p/captive-portal-functionality
 https://wiki.mozilla.org/Necko/CaptivePortal

  • bug 1202680 - [userstory] Automatic Captive Portal detection
Full Query
ID Priority Component Assigned to Summary Status Target milestone
562917 P3 Networking Valentin Gosu [:valentin] (he/him) [meta] implement captive portal detection RESOLVED ---
989192 P2 Address Bar Nihanth Subramanya [:nhnt11] Create portal detection events on network topology changes and user login RESOLVED ---
989193 P3 General Nihanth Subramanya [:nhnt11] Open captive portal automatically in a new tab when detected RESOLVED Firefox 50
989194 P3 General Nihanth Subramanya [:nhnt11] Add captive portal notification bar to all tabs when detected VERIFIED Firefox 52
989195 P3 Address Bar Replace about:newtab and about:home with captive portal if detected RESOLVED ---
989196 P3 Address Bar Do not show open Captive Portal automatically when OSX Captive Network Portal Assistant has been opened RESOLVED ---
989197 P3 Address Bar Nihanth Subramanya [:nhnt11] Add SSL error page when captive portal is detected VERIFIED Firefox 53
1202684 P1 General Panos Astithas (he/him) [:past] (please ni?) Breakdown - Captive Portal login page automatically RESOLVED ---
1279491 P3 General Nihanth Subramanya [:nhnt11] Intermittent timeout in browser_CaptivePortalWatcher.js on all platforms RESOLVED Firefox 50
1280599 -- General Handle captive portal page not loading NEW ---
1282611 P3 General Nihanth Subramanya [:nhnt11] Captive portal not detected at Firefox startup RESOLVED Firefox 51
1303775 P2 Networking Valentin Gosu [:valentin] (he/him) Captive portal tab may stay open after login NEW ---
1313568 -- General Nihanth Subramanya [:nhnt11] Handle captive portal notification bar per-window VERIFIED Firefox 53
1313577 P3 General Forget captive portal tab after origin change NEW ---
1313706 -- General Nihanth Subramanya [:nhnt11] Enable captive portal detection on all channels RESOLVED Firefox 53
1315966 -- General CaptivePortalDetector should propagate login page URL when it's available NEW ---
1315967 P3 Networking CaptivePortalService should provide the URL of the login page if it's available NEW ---
1315968 -- General Investigate how to include information about captive portal state in telemetry data/error reports NEW ---
1315969 -- General Nihanth Subramanya [:nhnt11] Simplify the way we open captive portal tabs RESOLVED ---
1318640 -- General Dão Gottwald [::dao] Closing the captive portal tab should go back to the previously selected tab RESOLVED Firefox 53
1320087 -- Networking Valentin Gosu [:valentin] (he/him) [Captive Portal] - Captive Portal Notification bar and Show Login button enabled although no CP wifi is available anymore VERIFIED mozilla53
1320088 -- Networking Valentin Gosu [:valentin] (he/him) [Captive Portal] Detection fails when FF is open (in another connection) prior to engaging CP able Wifi RESOLVED ---
1320100 -- General [Captive Portal] In full screen the notification bar doesn't appear RESOLVED ---
1320113 -- General [Captive Portal] Notification bar and Show login button translation tracker RESOLVED ---
1320128 -- General Nihanth Subramanya [:nhnt11] [Captive Portal] Notification bar misses border in Ubuntu VERIFIED Firefox 53
1320327 -- General [Captive Portal] Captive Portal notification/Login are shown even though the Captive portal canonical link is pre-loaded NEW ---
1320330 -- Notifications and Alerts [Captive Portal] With Voice Over on, text from Captive Portal notification bar is not read NEW ---
1323514 -- General [Captive portal] Banner shown on tabs when visiting local network sites RESOLVED ---
1329546 -- General Get rid of CaptivePortalWatcher and unify captive portal tests RESOLVED ---
1332271 -- Networking Valentin Gosu [:valentin] (he/him) [Captive Portal] Detection fails when FF is open (in no internet connection) prior to engaging Wifi VERIFIED mozilla54
1332630 -- General [Captive Portal] [Accessibility] Keyboard only doesn't focus the Captive Portal notification bar NEW ---
1333075 -- General Nihanth Subramanya [:nhnt11] [Captive Portal] Captive Portal Notification bar is not added to all opened windows VERIFIED Firefox 54
1578633 P1 Networking Nihanth Subramanya [:nhnt11] Captive portal notification bar does not automatically go away after login VERIFIED mozilla71
1579007 P3 General Nihanth Subramanya [:nhnt11] Revisit Captive Portal UI logic and improve tests ASSIGNED ---

34 Total; 10 Open (29.41%); 16 Resolved (47.06%); 8 Verified (23.53%);


Testcases

Test Areas

Test Areas Covered Details (and why) Reviewed by
Private Window Yes Captive portal detection should work in Private window as well Nihanth Subramanya
Multi-Process Enabled Yes By default Nihanth Subramanya
Multi-process Disabled Yes The Captive Portal detection should be verified on non-e10s Nihanth Subramanya
Theme (high contrast) Yes Because of the notification bar and error page Nihanth Subramanya
UI
Mouse-only operation Yes Captive Portal feature is entirely mouse only Nihanth Subramanya
Keyboard-only operation Yes The Show login button should be accessible by keyboard only Nihanth Subramanya
Display (HiDPI) Yes Because Notification bar and Error page Nihanth Subramanya
Interraction (scroll, zoom) Yes Because of Error page and Notification bar Nihanth Subramanya
Usable with a screen reader Yes The notification bar/Show Login button should be recognizable by screen reader Nihanth Subramanya
Usability and/or discoverability testing Yes Is this feature user friendly Nihanth Subramanya
RTL build testing Yes Because of Error page and Notification bar Nihanth Subramanya
Help/Support
Help/support interface required TBD Learn more still not implemented. Nihanth Subramanya
Support documents planned(written) TBD Learn more still not implemented. Nihanth Subramanya
Install/Upgrade
Feature upgrades/downgrades data as expected No feature doesn't upgrade/downgrade Nihanth Subramanya
Does sync work across upgrades No Not applicable for this feature Nihanth Subramanya
Requires install testing No No separate feature/application installation needed (not only Firefox) Nihanth Subramanya
Affects first-run or onboarding Yes To check first run, since there might be problems having the tour interfere with captive portal Nihanth Subramanya
Does this affect partner builds? Partner build testing No It's a core functionality, it's expected not to matter if the build is partner or not Nihanth Subramanya
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds Nihanth Subramanya
Enterprise administration No For the moment, this isn't in scope. Nihanth Subramanya
Network proxies/autoconfig No For the moment, this isn't in scope. Nihanth Subramanya
ESR behavior changes No For the moment, this isn't in scope. Nihanth Subramanya
Locked preferences No For the moment, this isn't in scope. Nihanth Subramanya
Data Monitoring
Temporary or permanent telemetry monitoring Yes There is plan, but it's not designed to be MVP'ed for 52, to reevaluate further on Nihanth Subramanya
Telemetry correctness testing No See above point. Nihanth Subramanya
Server integration testing No Not in scope. Nihanth Subramanya
Offline and server failure testing No Not in scope. Nihanth Subramanya
Load testing No Not in scope. Nihanth Subramanya
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? No Not in scope Nihanth Subramanya
Comprehensive API testing No Not in scope Nihanth Subramanya
Permissions No Not in scope Nihanth Subramanya
Testing with existing/popular addons Yes Because the Notification bar might be blocked by Pop-up blockers Nihanth Subramanya
Security Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review TBD (how we find out if it needs to have 3rd partyt sec review)? Nihanth Subramanya
Privilege escalation testing No Nihanth Subramanya
Fuzzing No Nihanth Subramanya
Web Compatibility depends on the feature
Testing against target sites No Not significant for this feature. Nihanth Subramanya
Survey of many sites for compatibility No Not significant for this feature. Nihanth Subramanya
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. Yes Yes, because there are OS Captive portal managers for FF Captive Portal detection to be tested against. Nihanth Subramanya
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS TBD AT the moment, there is no plan to have this feature implemented on anything else than Desktop Nihanth Subramanya
Interaction of this feature with other browser features No There are no significant reasons to believe that Captive Portal might influence other FF areas. Nihanth Subramanya

Test suite

Full Test suite - Test Rail

Bug Work

Tracking bug – meta bug Bug 1202680 Automatic Captive Portal detection

NEW bugs
Bug No Summary Status Firefox Version
1280599 Handle captive portal page not loading NEW
1303775 Captive portal tab may stay open after login NEW 54:affected; 53:fix-optional; 52:wontfix
1313577 Forget captive portal tab after origin change NEW 54:affected; 53:fix-optional; 52:wontfix
Bug 1315966 CaptivePortalDetector should propagate login page URL when it's available NEW 52:fix-optional
Bug 1315967 CaptivePortalService should provide the URL of the login page if it's available NEW 54:affected; 53:fix-optional; 52:fix-optional
Bug 1320327 [Captive Portal] Captive Portal notification/Login are shown even though the Captive portal canonical link is pre-loaded NEW 53:affected; 52:wontfix
Bug 1320330 [Captive Portal] With Voice Over on, text from Captive Portal notification bar is not read NEW 53:affected
Bug 1332630 [Captive Portal] [Accessibility] Keyboard only doesn't focus the Captive Portal notification bar NEW 53:affected; 52:affected
Bug 1315968 Investigate how to include information about captive portal state in telemetry data/error reports NEW 54:affected; 53:fix-optional; 52:fix-optional
VERIFIED ON 53, NEED UPLIFT ON 52

Bug 1318640 - Closing the captive portal tab should go back to the previously selected tabs
Bug 1322201 - Ensure correct usage of "login" vs "log in" in captive portal strings

Resolved WONTFIX

Bug 1323514 - [Captive portal] Banner shown on tabs when visiting local network sites
Bug 989195 - Replace about:newtab and about:home with captive portal if detected

Resolved INVALID

Bug 1329546 - Get rid of CaptivePortalWatcher and unify captive portal tests
Bug 1320113 - [Captive Portal] Notification bar and Show login button translation tracker
Bug 1320100 - [Captive Portal] In full screen the notification bar doesn't appear

Resolved WORKSFORME

Bug 989196 - Do not show open Captive Portal automatically when OSX Captive Network Portal Assistant has been opened
Bug 989192 - Create portal detection events on network topology changes and user login

Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Merge to Aurora Sign-off

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) No There is available documentation, but currently it's all centralized into a Etherpad.
Testing Infrastructure setup Yes
Test Plan Creation Yes
Test Cases Creation Yes
Full Functional Tests Execution Yes
Automation Coverage TBD
Performance Testing No
All Defects Logged Yes
Critical/Blockers Fixed and Verified Yes
Metrics/Telemetry
QA Signoff - Nightly Release Email to be sent
QA Aurora - Full Testing Yes
QA Signoff - Aurora Release Yes Email sent
QA Beta - Full Testing
QA Signoff - Beta Release Email to be sent