QA/InsecurePasswords
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
Date | Version | Author | Description |
---|---|---|---|
01/05/2017 | 1.0 | Adrian Florinescu | Initial Draft |
02/01/2017 | 2.0 | Adrian Florinescu | Updates: Preparing for hand-over to RelQA |
Contents
Overview
If the user has a saved password on an HTTP page, the Password Manager should not fill it automatically. Instead, the user should have to first type in or select their username. At that point (after user interaction) the Password Manager can fill the password. This change signifies that Firefox moves to a more secure Password manager that uses manual auto-fill versus old automatic auto-fill.
Purpose
Detail the purpose of this document. For example:
- The test scope, focus areas and objectives
- The test responsibilities
- The test strategy for the levels and types of test for this release
- The entry and exit criteria
- The basis of the test estimates
- Any risks, issues, assumptions and test dependencies
- The test schedule and major milestones
- The test deliverables
Scope
This wiki details the testing that will be performed for the Insecure Passwords feature. The scope of the Insecure Passwords feature is as follows:
- refactoring: bug 1296638 Switch toolkit Form Autocomplete popup from using a <xul:tree> to using a <xul:richlistbox
- refactoring: bug 1294502 Combine e10s and non-e10s nsFormAutoComplete implementations
- bug 1217152 Flip prefs to disable login autofill on HTTP and enable the warning on insecure login fields
- any section related to autocomplete of usernames, passwords, data lists or even regular form history.
Ownership
• Program Management:
• Project Management:
• Engineering Management:
• Eng.:
• QA
Testing summary
Scope of Testing
In Scope
- Verify that Sync and Password Manager still function as prior to introduction of Insecure passwords change
- The Password Manager will not auto-fill on the following cases:
-http sites -https sites that have mixed active content -in iframe sites ? (if a https form is loaded into a http site? see https://bugzilla.mozilla.org/show_bug.cgi?id=1217152#c2) -invisible form fields
Out of Scope
Although at this point the Password Manager overall functionality is only partially affected, there should be decided at a later point if the password manager necessitates to be added into the full scope of this test plan.
Requirements for testing
Environments
- standard OSs': Ubuntu, Windows, OSX (versions TBD)
- various http login sites
- various https login sites
- sync accounts that have Password sync enabled and saved users/passwords
Channel dependent settings (configs) and environment setups
Nightly
Currently the two preferences that enable Insecure Passwords are set as default on Nightly 53:
- "security.insecure_field_warning.contextual.enabled" - true
- "signon.autofillForms.http" - false
Aurora
Currently the two preferences that enable Insecure Passwords are set as default on Aurora 52:
- "security.insecure_field_warning.contextual.enabled" - true
- "signon.autofillForms.http" - false
Beta
Beta 51 - N/A
Post Beta / Release
Release 50 - N/A
Test Strategy
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
Ref | Function | Test Objective | Test Type | Risk Assessment and Coverage(RAC) | Owners |
---|---|---|---|---|---|
TO-1 | User Experience expectations | To ensure a positive user experience is balanced with the security expectations | Manual | RAC-1 | Eng Team |
TO-2 | Functional verification | Autocomplete/autofill functional verification | Manual | RAC-3 | Eng Team |
TO-3 | User/pass autocomplete layout verification | To ensure the user/pass autocomplete dropdown lists are behaving as expected | Manual | RAC-1, RAC-2 | Eng Team |
TO-4 | Form history | Ensure the form history functions as expected | Manual | RAC-4 | Eng Team |
TO-5 | Data lists | To ensure that data lists are functioning as expected | Manual | RAC-4 | Eng Team |
TO-6 | Integration with Password manager | To ensure that Password manager in/out are not affected by the autocomplete refactoring | Manual | - | Eng Team |
TO-7 | Preferences | To verify the preferences: - on enables the feature; - off dissables the feature | Manual | - | Eng Team |
TO-8 | Accesibility | To verify accesibility pre-requisites: colors, mouse only, keyboard only | Manual | RAC-5, RAC-6 | Eng Team |
TO-9 | Localization | RTL and strings in general (contextual warning) | Manual | RAC-7 | Eng Team |
TO-10 | Sync integration | To ensure that password manager autofill and contextual warning are integrated with sync | Manual | - | Eng Team |
Risk Assessment and Coverage
ID | Description / Threat Description | Covered by Test Objective | Magnitude | Probability | Priority | Impact Score |
---|---|---|---|---|---|---|
RAC-1 | User Experience | TO-1, TO-03, TO-05 | 3-High | 3-High | 3-High | 27 |
RAC-2 | Autocomplete changes/Contextual warning - effect on usability | TO-03 | 3-High | 3-High | 3-High | 27 |
RAC-3 | Autocomplete/ Contextual behavior is constant e10/non10s | TO-02 | 3-High | 2-Possible | 3-High | 18 |
RAC-4 | Form History/Data List might be affected by the autocomplete refactoring (1296638) | TO-04, TO-05 | 3-High | 2-Possible | 3-High | 18 |
RAC-5 | Backround colors and colors in general are visible and readable under high contrast | TO-08 | 3-High | 3-High | 2-Medium | 18 |
RAC-6 | Screen reader on the warning message? | TO-08 | 1- Low | 2-Possible | 3-High | 6 |
RAC-7 | RTL | TO-09 | 1- Low | 2-Possible | 2-Medium | 4 |
Values:
- Magnitude: 1- Low , 2-Moderate, 3-High
- Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
- Priority: 1 - Low, 2-Medium, 3-High
Impact Score Breakdown:
- An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
- An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
- An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
Builds
This section should contain links for builds with the feature - Insecure Passwords Enabled
- Links for Nightly builds - http://ftp.mozilla.org/pub/firefox/nightly/2017/01/2017-01-05-03-02-29-mozilla-central/
- Links for Aurora builds - http://ftp.mozilla.org/pub/firefox/nightly/2017/01/2017-01-06-00-40-19-mozilla-aurora/
- Links for Beta builds - N/A
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | 27.12.2016 | |
Study documentation/specs received from developers | 12.27.2016 | 01.10.2017 |
QA - Test plan creation | 01.05.2016 | 01.13.2016 |
QA - Test cases/Env preparation | 01.05.2017 | 01.20.2017 |
QA - Nightly Testing | 12.27.2016 | 01.21.2017 |
QA - Aurora Testing | 01.05.2017 | 01.23.2017 |
QA - Beta Testing | N/A | N/A |
Release Date | N/A | N/A |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | TestRail/ Google docs |
Test case execution | TestRail |
Bugs management | Bugzilla |
Status
Overview
Track the dates and build number where feature was released to Nightly Track the dates and build number where feature was merged to Aurora Track the dates and build number where feature was merged to Release/Beta
References
- List and links for specs
-
- bug 1217142 - [userstory] [meta] Insecure Passwords
37 Total; 4 Open (10.81%); 24 Resolved (64.86%); 9 Verified (24.32%);
- bug 1304224 - [meta] Add Contextual Warning to username/password fields on HTTP pages
55 Total; 11 Open (20%); 22 Resolved (40%); 22 Verified (40%);
- bug 1217152 - Flip prefs to disable login autofill on HTTP and enable the warning on insecure login fields
16 Total; 6 Open (37.5%); 5 Resolved (31.25%); 5 Verified (31.25%);
Testcases
Overview
Summary of testing scenarios
Test Areas
Test Areas | Covered | Details (and why) | Reviewed by |
---|---|---|---|
Private Window | No | Altought the private window is affected by the changes, it doesn't enter the scope | :tanvi, :MattN |
Multi-Process Enabled | Yes | Default setting | :tanvi, :MattN |
Multi-process Disabled | Yes | The same functionality as when 10s enabled should be verfied | :tanvi, :MattN |
Theme (high contrast) | Yes | Ensure that the warning is visible and accessibility colors are used | :tanvi, :MattN |
UI | |||
Mouse-only operation | Yes | User facing functionality | :tanvi, :MattN |
Keyboard-only operation | Yes | User facing functionality | :tanvi, :MattN |
Display (HiDPI) | Yes | HiDPI enters the scope of the testing | :tanvi, :MattN |
Interraction (scroll, zoom) | Yes | :tanvi, :MattN | |
Usable with a screen reader | Yes | Technically, the autofill rich text should be accessible, so yes | :tanvi, :MattN |
Usability and/or discoverability testing | Yes | :tanvi, :MattN | |
RTL build testing | Yes | :tanvi, :MattN | |
Help/Support | |||
Help/support interface required | TBD | ||
Support documents planned(written) | Yes | Support page needs updating | :tanvi, :MattN |
Install/Upgrade | |||
Feature upgrades/downgrades data as expected | No | feature doesn't upgrade/downgrade | :tanvi, :MattN |
Does sync work across upgrades | No | not in the scope | :tanvi, :MattN |
Requires install testing | No | not in the scope | :tanvi, :MattN |
Affects first-run or onboarding | No | not in the scope | :tanvi, :MattN |
Does this affect partner builds? Partner build testing | Yes | This pref and feature are going to be defaulted, therefore they will affect partner builds as default | :tanvi, :MattN |
Enterprise | Raise up the topic to developers to see if they are expecting to work different on ESR builds | ||
Enterprise administration | N/A | :tanvi, :MattN | |
Network proxies/autoconfig | N/A | :tanvi, :MattN | |
ESR behavior changes | N/A | :tanvi, :MattN | |
Locked preferences | N/A | :tanvi, :MattN | |
Data Monitoring | |||
Temporary or permanent telemetry monitoring | Yes | Should be monitored | :tanvi, :MattN |
Telemetry correctness testing | Yes | to contact :MattN for details | :tanvi, :MattN |
Server integration testing | No | Not in scope. | :tanvi, :MattN |
Offline and server failure testing | No | Not in scope. | :tanvi, :MattN |
Load testing | No | Not in scope. | :tanvi, :MattN |
Add-ons | If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on. | ||
Addon API required? | No | Not in scope | :tanvi, :MattN |
Comprehensive API testing | No | Not in scope | :tanvi, :MattN |
Permissions | No | Not in scope | :tanvi, :MattN |
Testing with existing/popular addons | Yes | to test with the most popular password managers | :tanvi, :MattN |
Security | Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature. | ||
3rd-party security review | No | :tanvi, :MattN | |
Privilege escalation testing | No | :tanvi, :MattN | |
Fuzzing | No | :tanvi, :MattN | |
Web Compatibility | depends on the feature | ||
Testing against target sites | Yes | :tanvi, :MattN | |
Survey of many sites for compatibility | Yes | :tanvi, :MattN | |
Interoperability | depends on the feature | ||
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | No | :tanvi, :MattN | |
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | No | not implemented at this moment across the platforms | :tanvi, :MattN |
Interaction of this feature with other browser features | Yes | autocomplete refactoring might afect other areas such form history or data lists | :tanvi, :MattN |
Test suite
Full Test suite - TestSuite
Bug Work
Tracking bugs:
- 1304224 meta Add Contextual Warning to username/password fields on HTTP pages
- 1217142 meta Insecure Passwords
- 1217152 Flip prefs to disable login autofill on HTTP and enable the warning on insecure login fields
Bug Work
27 Total; 0 Open (0%); 6 Resolved (22.22%); 21 Verified (77.78%);
24 Total; 0 Open (0%); 15 Resolved (62.5%); 9 Verified (37.5%);
ID | Priority | Component | Assigned to | Summary | Status | Target milestone |
---|---|---|---|---|---|---|
376668 | P1 | Password Manager | Dale Harvey (:daleharvey) | Improve discoverability of login autocompletion (used with multiple accounts) | VERIFIED | mozilla53 |
1289913 | P1 | Password Manager | Sean Lee [:seanlee][:weilonge] | Show autocomplete UI on password fields | VERIFIED | mozilla52 |
1302474 | P1 | Password Manager | Johann Hofmann [:johannh] | Add a pref to disable login autofill on insecure forms | VERIFIED | mozilla52 |
1330561 | P1 | Password Manager | Jared Wein [:jaws] (please needinfo? me) | Autofill/Autocomplete/Insecure warning cannot be opened in username field right away in a new tab | VERIFIED | mozilla67 |
4 Total; 0 Open (0%); 0 Resolved (0%); 4 Verified (100%);
Sign off
Criteria
Check list
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
List of OSes that will be covered by testing
Beta Testing
- Pre-Release sign-off
- List of OSes that will be covered by testing: Windows 10 x64, Windows 7 x64, Ubuntu 16.04 x64 and Mac OS X 10.11.6
- Link for the tests run - https://testrail.stage.mozaws.net/index.php?/plans/view/2569
- Full Test suite - https://testrail.stage.mozaws.net/index.php?/suites/view/415&group_by=cases:section_id&group_order=asc
- List of OSes that will be covered by testing: Windows 10 x64, Windows 7 x64, Ubuntu 16.04 x64 and Mac OS X 10.11.6
Checklist
Exit Criteria | Status | Notes/Details |
---|---|---|
Testing Prerequisites (specs, use cases) | No | AFAIK, the available documentation is not summarized in a doc. (spread into several bugs/metabugs) |
Testing Infrastructure setup | Yes | |
Test Plan Creation | Yes | |
Test Cases Creation | Yes | |
Full Functional Tests Execution | Yes | |
Automation Coverage | TBD | |
Performance Testing | ||
All Defects Logged | Yes | |
Critical/Blockers Fixed and Verified | Yes | |
Metrics/Telemetry | ||
QA Signoff - Nightly Release | No | Feature uplifted to Aurora |
QA Aurora - Full Testing | Yes | |
QA Signoff - Aurora Release | Yes | Email sent |
QA Beta - Full Testing | N/A | |
QA Signoff - Pre-Release Sign off | [DONE] | Email sent 02-27-2017 (GREEN) |