Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date	Version	Author	Description
10/27/2016	1.0	Adrian Florinescu	Initial Draft
11/08/2016	1.1	Adrian Florinescu	TP review with Nihanth; updates based on it.
11/21/2016	1.2	Adrian Florinescu	Updates
02/01/2017	2.0	Adrian Florinescu	Updates: preparing for hand-over to RelQA

Overview

Captive Portal feature covers the detection and implementation of handles for captive portals inside Firefox browser. Firefox is expected to handle the handling of a captive portal page upon detection of such. Certain business logic related to the detection and management of a captive portal is expected to be in place and documented in the Specs section.

Purpose

Detail the purpose of this document. For example:

• The test scope, focus areas and objectives
• The test responsibilities
• The test strategy for the levels and types of test for this release
• The entry and exit criteria
• The basis of the test estimates
• Any risks, issues, assumptions and test dependencies
• The test schedule and major milestones
• The test deliverables

Scope

This wiki details the testing that will be performed by the project team for the Captive Portals feature. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

• What will be tested
• How testing will be performed

Ownership

• Program Management:

• Erin Lancaster

• Product Management:

• Peter Dolanjski

• Engineering Management:

• Panos Astithas

• Eng.:

• Nihanth Subramanya
• Matthew Noorenberghe

• QA

• Adrian Florinescu
• Brindusa Tot
• Ovidiu Boca

Testing summary

Scope of Testing

In Scope

• testing a wifi-connection that has enabled captive portals
• testing a wan connection which has enabled captive portals (not done)

Out of Scope

• Due to security issues and since the types of captive portals implementation vary depending on manufacturer, setup, security setups, etc., the tests which will be designed and ran will cover the basic functionality that impacts Firefox browser and not the compatibility with all existing captive portals.

Requirements for testing

Environments

• standard OS's: Ubuntu, Windows, OSX (versions TBD)
• Captive Portal compliant and activated - wifi network
• Wireless connection available

Channel dependent settings (configs) and environment setups

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref	Function	Test Objective	Test Type	Risk Assessment and Coverage(RAC)	Owners
TO-1	Active Portal detection	To verify that the active portal is detected in all expected scenarios	Manual	RAC-1, RAC-2, RAC-3, RAC-4,RAC-8	Eng Team
TO-2	Active Portal notification bar	To verify the behavior of the active portal notification bar	Manual	RAC-5	Eng Team
TO-3	Active Portal Tab	To verify that new tab is opened and it's further behavior according to specs on active detection	Manual	RAC-6, RAC-7	Eng Team
TO-4	Inactive Portal	To verify the behavior of FF when the captive portal is not active anymore	Manual	-	Eng Team
TO-5	Preferences	To verify the preference - on/off	Manual	-	Eng Team
TO-6	Error page	To verify the Error page	Manual	-	Eng Team
TO-7	Localization	To verify that the Notification bar, Error page and any other element is localized	Manual	-	Eng Team

Risk Assessment and Coverage

ID	Description / Threat Description	Covered by Test Objective	Magnitude	Probability	Priority	Impact Score
RAC-1	Multiple FF windows	TO-1	2-Moderate	3-High	2-Medium	12
RAC-2	Multiple network connections (wi-fi x2/ wi-fi - wan)	TO-1	1- Low	1-Unlikely	2-Medium	2
RAC-3	Existing proxy settings	TO-1	2-Moderate	2-Possible	2-Medium	8
RAC-4	Integration with (Windows) Captive Portal managers	TO-1	2-Moderate	3-Almost Certain	2-Medium	12
RAC-5	Notification management against Pop-up blockers	TO-2	3-High	2-Possible	3-High	18
RAC-6	(OSX) Multi users	TO-3	2-Moderate	2-Possible	2-Medium	8
RAC-7	(OSX) Integration with Captive Network Assistant	TO-3	2-Moderate	2-Possible	3-High	12
RAC-8	(Ubuntu)Circumventing Captive portal	TO-1	2-Moderate	2-Possible	3-High	12

Values:

• Magnitude: 1- Low , 2-Moderate, 3-High

• Probability: 1-Unlikely, 2-Possible, 3-Almost Certain

• Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

• An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
• An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
• An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Builds

This section should contain links for builds with the feature -

• Links for Nightly builds
• Links for Aurora builds
• Links for Beta builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project	24.10.2016
Study documentation/specs received from developers	10.31.2016	11.08.2016
QA - Test plan creation	10.31.2016	11.08.2016
QA - Test cases/Env preparation	10.31.2016
QA - Nightly Testing
QA - Aurora Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	TestRail/ Google docs
Test case execution	TestRail
Bugs management	Bugzilla

Status

Overview

Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Aurora
Track the dates and build number where feature was merged to Release/Beta

References

• List and links for specs

 https://public.etherpad-mozilla.org/p/captive-portal-functionality
 https://wiki.mozilla.org/Necko/CaptivePortal

• bug 1202680 - [userstory] Automatic Captive Portal detection

Full Query

ID	Priority	Component	Assigned to	Summary	Status	Target milestone
562917	P3	Networking	Valentin Gosu [:valentin] (he/him)	[meta] implement captive portal detection	RESOLVED	---
989192	P2	Address Bar	(inactive) Nihanth Subramanya [:nhnt11]	Create portal detection events on network topology changes and user login	RESOLVED	---
989193	P3	General	(inactive) Nihanth Subramanya [:nhnt11]	Open captive portal automatically in a new tab when detected	RESOLVED	Firefox 50
989194	P3	General	(inactive) Nihanth Subramanya [:nhnt11]	Add captive portal notification bar to all tabs when detected	VERIFIED	Firefox 52
989195	P3	Address Bar		Replace about:newtab and about:home with captive portal if detected	RESOLVED	---
989196	P3	Address Bar		Do not show open Captive Portal automatically when OSX Captive Network Portal Assistant has been opened	RESOLVED	---
989197	P3	Address Bar	(inactive) Nihanth Subramanya [:nhnt11]	Add SSL error page when captive portal is detected	VERIFIED	Firefox 53
1202684	P1	General	Panos Astithas (he/him) [:past] (please ni?)	Breakdown - Captive Portal login page automatically	RESOLVED	---
1279491	P3	General	(inactive) Nihanth Subramanya [:nhnt11]	Intermittent timeout in browser_CaptivePortalWatcher.js on all platforms	RESOLVED	Firefox 50
1280599	--	General		Handle captive portal page not loading	RESOLVED	---
1282611	P3	General	(inactive) Nihanth Subramanya [:nhnt11]	Captive portal not detected at Firefox startup	RESOLVED	Firefox 51
1303775	P1	Networking	:prathiksha	Captive portal tab may stay open after login	RESOLVED	mozilla76
1313568	--	General	(inactive) Nihanth Subramanya [:nhnt11]	Handle captive portal notification bar per-window	VERIFIED	Firefox 53
1313577	P1	General	:prathiksha	Forget references to captive portal tab once login has succeeded or aborted	RESOLVED	Firefox 75
1313706	--	General	(inactive) Nihanth Subramanya [:nhnt11]	Enable captive portal detection on all channels	RESOLVED	Firefox 53
1315966	P5	General		CaptivePortalDetector should propagate login page URL when it's available	NEW	---
1315967	P5	Networking		CaptivePortalService should provide the URL of the login page if it's available	NEW	---
1315968	--	General		Investigate how to include information about captive portal state in telemetry data/error reports	NEW	---
1315969	--	General	(inactive) Nihanth Subramanya [:nhnt11]	Simplify the way we open captive portal tabs	RESOLVED	---
1318640	--	General	Dão Gottwald [:dao]	Closing the captive portal tab should go back to the previously selected tab	RESOLVED	Firefox 53
1320087	--	Networking	Valentin Gosu [:valentin] (he/him)	[Captive Portal] - Captive Portal Notification bar and Show Login button enabled although no CP wifi is available anymore	VERIFIED	mozilla53
1320088	--	Networking	Valentin Gosu [:valentin] (he/him)	[Captive Portal] Detection fails when FF is open (in another connection) prior to engaging CP able Wifi	RESOLVED	---
1320100	--	General		[Captive Portal] In full screen the notification bar doesn't appear	RESOLVED	---
1320113	--	General		[Captive Portal] Notification bar and Show login button translation tracker	RESOLVED	---
1320128	--	General	(inactive) Nihanth Subramanya [:nhnt11]	[Captive Portal] Notification bar misses border in Ubuntu	VERIFIED	Firefox 53
1320327	P5	General		[Captive Portal] Captive Portal notification/Login are shown even though the Captive portal canonical link is pre-loaded	NEW	---
1320330	--	Disability Access		[Captive Portal] With Voice Over on, text from Captive Portal notification bar is not read	RESOLVED	---
1323514	--	General		[Captive portal] Banner shown on tabs when visiting local network sites	RESOLVED	---
1329546	--	General		Get rid of CaptivePortalWatcher and unify captive portal tests	RESOLVED	---
1332271	--	Networking	Valentin Gosu [:valentin] (he/him)	[Captive Portal] Detection fails when FF is open (in no internet connection) prior to engaging Wifi	VERIFIED	mozilla54
1332630	--	Disability Access		[Captive Portal] [Accessibility] Keyboard only doesn't focus the Captive Portal notification bar	NEW	---
1333075	--	General	(inactive) Nihanth Subramanya [:nhnt11]	[Captive Portal] Captive Portal Notification bar is not added to all opened windows	VERIFIED	Firefox 54
1578633	P1	Networking	(inactive) Nihanth Subramanya [:nhnt11]	Captive portal notification bar does not automatically go away after login	VERIFIED	mozilla71
1579007	P3	Networking		Revisit Captive Portal UI logic and improve tests	NEW	---
1622801	P3	General		Forget captive portal tab on origin change before login	NEW	---
1740972	P3	Sync		Edgewave/Iprism firewall server causes Firefox to think it's always behind a captive portal	NEW	---
1747535	P2	Networking		Captive portal redirect not working due to forced https in detection site url	UNCONFIRMED	---

37 Total; 9 Open (24.32%); 20 Resolved (54.05%); 8 Verified (21.62%);

Testcases

Test Areas

Test Areas	Covered	Details (and why)	Reviewed by
Private Window	Yes	Captive portal detection should work in Private window as well	Nihanth Subramanya
Multi-Process Enabled	Yes	By default	Nihanth Subramanya
Multi-process Disabled	Yes	The Captive Portal detection should be verified on non-e10s	Nihanth Subramanya
Theme (high contrast)	Yes	Because of the notification bar and error page	Nihanth Subramanya
UI
Mouse-only operation	Yes	Captive Portal feature is entirely mouse only	Nihanth Subramanya
Keyboard-only operation	Yes	The Show login button should be accessible by keyboard only	Nihanth Subramanya
Display (HiDPI)	Yes	Because Notification bar and Error page	Nihanth Subramanya
Interraction (scroll, zoom)	Yes	Because of Error page and Notification bar	Nihanth Subramanya
Usable with a screen reader	Yes	The notification bar/Show Login button should be recognizable by screen reader	Nihanth Subramanya
Usability and/or discoverability testing	Yes	Is this feature user friendly	Nihanth Subramanya
RTL build testing	Yes	Because of Error page and Notification bar	Nihanth Subramanya
Help/Support
Help/support interface required	TBD	Learn more still not implemented.	Nihanth Subramanya
Support documents planned(written)	TBD	Learn more still not implemented.	Nihanth Subramanya
Install/Upgrade
Feature upgrades/downgrades data as expected	No	feature doesn't upgrade/downgrade	Nihanth Subramanya
Does sync work across upgrades	No	Not applicable for this feature	Nihanth Subramanya
Requires install testing	No	No separate feature/application installation needed (not only Firefox)	Nihanth Subramanya
Affects first-run or onboarding	Yes	To check first run, since there might be problems having the tour interfere with captive portal	Nihanth Subramanya
Does this affect partner builds? Partner build testing	No	It's a core functionality, it's expected not to matter if the build is partner or not	Nihanth Subramanya
Enterprise		Raise up the topic to developers to see if they are expecting to work different on ESR builds	Nihanth Subramanya
Enterprise administration	No	For the moment, this isn't in scope.	Nihanth Subramanya
Network proxies/autoconfig	No	For the moment, this isn't in scope.	Nihanth Subramanya
ESR behavior changes	No	For the moment, this isn't in scope.	Nihanth Subramanya
Locked preferences	No	For the moment, this isn't in scope.	Nihanth Subramanya
Data Monitoring
Temporary or permanent telemetry monitoring	Yes	There is plan, but it's not designed to be MVP'ed for 52, to reevaluate further on	Nihanth Subramanya
Telemetry correctness testing	No	See above point.	Nihanth Subramanya
Server integration testing	No	Not in scope.	Nihanth Subramanya
Offline and server failure testing	No	Not in scope.	Nihanth Subramanya
Load testing	No	Not in scope.	Nihanth Subramanya
Add-ons		If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required?	No	Not in scope	Nihanth Subramanya
Comprehensive API testing	No	Not in scope	Nihanth Subramanya
Permissions	No	Not in scope	Nihanth Subramanya
Testing with existing/popular addons	Yes	Because the Notification bar might be blocked by Pop-up blockers	Nihanth Subramanya
Security		Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review	TBD	(how we find out if it needs to have 3rd partyt sec review)?	Nihanth Subramanya
Privilege escalation testing	No		Nihanth Subramanya
Fuzzing	No		Nihanth Subramanya
Web Compatibility		depends on the feature
Testing against target sites	No	Not significant for this feature.	Nihanth Subramanya
Survey of many sites for compatibility	No	Not significant for this feature.	Nihanth Subramanya
Interoperability		depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.	Yes	Yes, because there are OS Captive portal managers for FF Captive Portal detection to be tested against.	Nihanth Subramanya
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS	TBD	AT the moment, there is no plan to have this feature implemented on anything else than Desktop	Nihanth Subramanya
Interaction of this feature with other browser features	No	There are no significant reasons to believe that Captive Portal might influence other FF areas.	Nihanth Subramanya

Test suite

Full Test suite - Test Rail

Bug Work

Tracking bug – meta bug Bug 1202680 Automatic Captive Portal detection

Sign off

Criteria

Check list

• All test cases should be executed
• Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
• All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Merge to Aurora Sign-off

• Link for the tests run
  • Full Test suite

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)	No	There is available documentation, but currently it's all centralized into a Etherpad.
Testing Infrastructure setup	Yes
Test Plan Creation	Yes
Test Cases Creation	Yes
Full Functional Tests Execution	Yes
Automation Coverage	TBD
Performance Testing	No
All Defects Logged	Yes
Critical/Blockers Fixed and Verified	Yes
Metrics/Telemetry
QA Signoff - Nightly Release		Email to be sent
QA Aurora - Full Testing	Yes
QA Signoff - Aurora Release	Yes	Email sent
QA Beta - Full Testing
QA Signoff - Beta Release		Email to be sent