Security/QA/TestPlans/Web Authentication: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
m (Minor update)
 
(31 intermediate revisions by the same user not shown)
Line 7: Line 7:
! Name !! Title !! Department !! Approval Date !! Method
! Name !! Title !! Department !! Approval Date !! Method
|-
|-
| || QA Manager || Product Integrity || Date || Email
| Ryan VanderMuelen || QA Manager || Product Integrity || Date || Email
|-
|-
| JC Jones || Software Engineer || Engineering || Date || Email
| JC Jones || Software Engineer || Engineering || Date || Email
|-
|-
| || EPM || Product Management || Date || Email
| JC Jones || EPM || Product Management (acting) || Date || Email
|}
|}




'''Revision History'''
'''Revision History'''
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded).  The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.


{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
Line 24: Line 22:
|-
|-
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft
|-
| 2017-10-04 || 1.1 || Matt Wobensmith || Sending for review
|-
| 2017-10-04 || 1.2 || Matt Wobensmith || Incorporating review feedback from RyanVM
|}
|}


= Overview =
= Overview =
== Purpose ==
== Purpose ==
Detail the purpose of this document. For example:
Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message.  
* The test scope, focus areas and objectives
 
* The test responsibilities
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.
* The test strategy for the levels and types of test for this release
 
* The entry and exit criteria
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is outlined within the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the spec itself.
* The basis of the test estimates
 
* Any risks, issues, assumptions and test dependencies
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.
* The test schedule and major milestones
 
* The test deliverables
The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements will not change.
 
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.


== Scope ==
== Scope ==
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
The areas of client JavaScript and USB support are the focus of our test effort.
* What will be tested
 
* How testing will be performed
Code integrity
* Unit tests
* Code-level security review
* Fuzzing
 
Functionality
* Manual testing
* Real-world implementations


== Ownership ==
== Ownership ==
This feature is being tested by both Mozilla and one or more third parties.
This feature is being tested by both Mozilla and one or more third parties.
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction
* Yubico is performing smoke tests using hardware keys across a range of hardware and software
* Yubico is performing smoke tests using hardware keys across a range of hardware and software
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown
* The PI Security team has been requested to perform a security review between now and mid-September 2017.
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff  


= Testing summary =  
= Testing summary =  
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it
* Web Authentication, as well as some U2F.
* All JS APIs
* All JS APIs.
* Fuzzing wherever possible
* Fuzzing wherever possible.
* A range of scenario tests that mirror user interaction, including boundary and error cases
* A range of scenario tests that mirror user interaction, including boundary and error cases.
* Some USB hardware, including Yubico keys and a few others given to us.




=== Out of Scope ===
=== Out of Scope ===
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession.
* Software token is unsupported, for now.
* Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us.  
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.
* This feature is not currently supported on Fennec.
* This feature is not currently supported on Fennec.
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.


= Requirements for testing =
= Requirements for testing =
Line 76: Line 91:


  security.webauth.u2f;
  security.webauth.u2f;
security.webauth.u2f_enable_softtoken;
security.webauth.u2f_enable_usbtoken;
  security.webauth.webauthn;
  security.webauth.webauthn;
security.webauth.webauthn_enable_usbtoken;
Optional: to use unsupported soft token, set to true:
  security.webauth.webauthn_enable_softtoken;
  security.webauth.webauthn_enable_softtoken;
security.webauth.webauthn_enable_usbtoken;


=== Nightly ===
=== Nightly ===
Line 103: Line 119:
{| class="wikitable"
{| class="wikitable"
|-
|-
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score  
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Discoverability !! Impact Score  
|-
|-
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6
|-
|-
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3
|-
|-
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4
Line 128: Line 144:
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''


* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''
* '''Discoverability:''' 1 - Low, ''2-Medium'', '''3-High'''


'''Impact Score Breakdown:'''  
'''Impact Score Breakdown:'''  
Line 136: Line 152:


== Test Objectives ==
== Test Objectives ==
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
Verify that the feature works as designed, interacts well with normal use of Firefox, is stable and has secure code.
This could be documented in bullet form or in a table similar to the one below.


{| class="wikitable"
{| class="wikitable"
Line 151: Line 166:


== Builds ==
== Builds ==
This section should contain links for builds with the feature -  
Use latest build of Nightly for your platform from our [https://www.mozilla.org/en-US/firefox/channel/desktop/ product download page].
* Links for Nightly builds
* Links for Beta builds


== Test Execution Schedule ==
== Test Execution Schedule ==
Line 174: Line 187:
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017-09-19 ||  
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
Line 184: Line 197:


== Testing Tools ==
== Testing Tools ==
Detail the tools to be used for testing, for example see the following table:
Testing requires access to Test Rail, as well as physical possession of USB keys.
 
{| class="wikitable" style="width:50%"
{| class="wikitable" style="width:50%"
|-
|-
Line 196: Line 210:
|-
|-
| Bugs management || Bugzilla
| Bugs management || Bugzilla
|-
| Telemetry || SCALARS_SECURITY.WEBAUTHN_USED, WEBAUTHN.CREATE_CREDENTIAL_MS, and WEBAUTHN_GET_ASSERTION_MS
|}
|}


= Status =  
= Status =  
== Overview ==
== Overview ==
* Feature landed, turned off, in Nightly 57 on 15-09-17
* Feature will target Fx58/Fx59.
  Track the dates and build number where feature was released to Nightly
  Track the dates and build number where feature was released to Nightly
  Track the dates and build number where feature was merged to Release/Beta
  Track the dates and build number where feature was merged to Release/Beta


= References =
= References =
Line 211: Line 229:
= Testcases =  
= Testcases =  
== Test Areas ==
== Test Areas ==
{| class="wikitable" style="width:80%"
{| class="wikitable" style="width:80%"
|-
|-
Line 219: Line 238:
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" | yes ||  
|style="text-align:center;" | yes || Test case in Test Rail
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" | yes ||  
|style="text-align:center;" | yes || Test case in Test Rail
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
Line 228: Line 247:
|-
|-
| '''UI'''  
| '''UI'''  
||  ||  
||  || This feature has no UI
|-
|-
| Mouse-only operation   
| Mouse-only operation   
Line 281: Line 300:
|-
|-
| ''' Enterprise '''  
| ''' Enterprise '''  
||  ||  Raise up the topic to developers to see if they are expecting to work different on ESR builds
||  ||  No special support for enterprise - feature is same as on release
|-
|-
| Enterprise administration   
| Enterprise administration   
|style="text-align:center;" | no ||  
|style="text-align:center;" | no || can be turned on/off by pref if desired
|-
|-
| Network proxies/autoconfig   
| Network proxies/autoconfig   
|style="text-align:center;" | no ||  
|style="text-align:center;" | no || n/a
|-
|-
| ESR behavior changes   
| ESR behavior changes   
Line 300: Line 319:
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | no ||
|style="text-align:center;" | yes || see "Testing Tools" [[https://wiki.mozilla.org/Security/QA/TestPlans/Web_Authentication#Testing_Tools section]]
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | no ||  
|style="text-align:center;" | yes || see "Testing Tools" [[https://wiki.mozilla.org/Security/QA/TestPlans/Web_Authentication#Testing_Tools section]]
|-
|-
| Server integration testing   
| Server integration testing   
Line 316: Line 335:
|-
|-
| ''' Add-ons '''  
| ''' Add-ons '''  
||  || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
||  || No additional support for add-ons at this time.
|-
|-
| Addon API required?   
| Addon API required?   
Line 328: Line 347:
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" | no ||  
|style="text-align:center;" | no ||


|-
|-
| ''' Security '''  
| ''' Security '''  
||  || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
||  ||  
|-
|-
| 3rd-party security review   
| 3rd-party security review   
Line 351: Line 370:
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites
|style="text-align:center;" | no || If we support U2F, we can try to find U2F-enabled sites, but otherwise this is a new feature


|-
|-
Line 368: Line 387:


== Test suite ==
== Test suite ==
  Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]
  Full Test suite - Link to test rail [https://testrail.stage.mozaws.net/index.php?/suites/overview/49 link]
  Smoke Test suite - Link with the tests - if available/needed.
  Smoke Test suite - see above.
Regression Test suite - Link with the tests - if available/needed.


= Bug Work =
= Bug Work =
Line 379: Line 397:
<bugzilla>
<bugzilla>
     {
     {
         "blocks":[1395406],
         "id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
     }
     }
</bugzilla>
</bugzilla>
</div>
</div>
</div>


Line 429: Line 445:
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |  
|style="text-align:center;" | complete ||  
| style="text-align:center;" |  
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" |   ||  
|style="text-align:center;" | complete ||  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" | first draft complete  ||  
|style="text-align:center;" | complete  ||  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" | complete ||  
|style="text-align:center;" | complete ||  
|-
|-
|  Automation Coverage ||
|  Automation Coverage  
|style="text-align:center;" |  
|style="text-align:center;" |  n/a ||  
|-
|-
|  Performance Testing  
|  Performance Testing  
|style="text-align:center;" |  ||  
|style="text-align:center;" |  n/a ||  
|-
|-
|  All Defects Logged || ||  
|  All Defects Logged  
|style="text-align:center;" | complete ||  
|-
|-
|  Critical/Blockers Fixed and Verified || ||  
|  Critical/Blockers Fixed and Verified  
|style="text-align:center;" | complete ||  
|-
|-
|  Metrics/Telemetry||
|  Metrics/Telemetry
|style="text-align:center;" |  
|style="text-align:center;" | n/a  ||  
|-
|-
|  Basic/Core functionality Nightly testing
|  Basic/Core functionality Nightly testing

Latest revision as of 21:43, 21 February 2018

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
Ryan VanderMuelen QA Manager Product Integrity Date Email
JC Jones Software Engineer Engineering Date Email
JC Jones EPM Product Management (acting) Date Email


Revision History

Date Version Author Description
2017-08-16 1.0 Matt Wobensmith Created first draft
2017-10-04 1.1 Matt Wobensmith Sending for review
2017-10-04 1.2 Matt Wobensmith Incorporating review feedback from RyanVM

Overview

Purpose

Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message.

What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.

The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is outlined within the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the spec itself.

We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.

The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements will not change.

The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.

Scope

The areas of client JavaScript and USB support are the focus of our test effort.

Code integrity

  • Unit tests
  • Code-level security review
  • Fuzzing

Functionality

  • Manual testing
  • Real-world implementations

Ownership

This feature is being tested by both Mozilla and one or more third parties.

  • Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests
  • JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction
  • Yubico is performing smoke tests using hardware keys across a range of hardware and software
  • Adam Powers (FIDO) is creating tests for the web-platform-test suite
  • The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown
  • The PI Security team has been requested to perform a security review of both JS API and Rust USB library
  • Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff

Testing summary

Scope of Testing

In Scope

  • Web Authentication, as well as some U2F.
  • All JS APIs.
  • Fuzzing wherever possible.
  • A range of scenario tests that mirror user interaction, including boundary and error cases.
  • Some USB hardware, including Yubico keys and a few others given to us.


Out of Scope

  • Software token is unsupported, for now.
  • Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us.
  • Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.
  • This feature is not currently supported on Fennec.
  • We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.

Requirements for testing

Environments

We support the same OS and hardware configurations that Firefox supports on desktop only.

Channel dependent settings (configs) and environment setups

The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true: 

security.webauth.u2f;
security.webauth.webauthn;
security.webauth.webauthn_enable_usbtoken;

Optional: to use unsupported soft token, set to true: 

security.webauth.webauthn_enable_softtoken;

Nightly

Currently set to false.

Beta

Currently set to false.

Post Beta / Release

Depending on ship decisions, will be set to true.

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Discoverability Impact Score
RAC-1 Incorrect authentication allows security bypass TO-1, TO-2, TO-3 3-High 1-Unlikely 2-Moderate 6
RAC-2 XSS/information leak TO-1, TO-3 3-High 1-Unlikely 1-Low 3
RAC-3 Confined to secure context TO-1, TO-3 2-Moderate 2-Possible 1-Low 4
RAC-4 Incorrectly functioning JS API TO-1 3-High 2-Possible 2-Moderate 12
RAC-5 Stability for entire feature TO-1, TO-2 3-High 2-Possible 3-High 18
RAC-6 Interaction with other aspects of normal Firefox usage TO-1, TO-2 3-Moderate 3-Almost Certain 3-High 27
RAC-7 Memory issues in JS API and hardware support code TO-3 3-High 1-Unlikely 2-Moderate 6
RAC-8 Incorrectly functioning hardware TO-2 2-Moderate 1-Unlikely 1-Low 2

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Discoverability: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

Verify that the feature works as designed, interacts well with normal use of Firefox, is stable and has secure code.

Ref Function Test Objective Evaluation Criteria Test Type RAC Owners
TO1 JS API Verify functionality All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens Manual/ Automation / Usability RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 Eng Team, QA
TO2 Hardware support via USB token Verify functionality All tests indicate stable, functional support of USB hardware keys, as above Manual/ Automation / Usability RAC-1, RAC-5, RAC-6, RAC-8 Eng Team, QA
TO3 Stable, secure code Fuzzing and security review All testing and inspection surfaces known security issues Manual/ Security RAC-1, RAC-2, RAC-3, RAC-7 Eng Team, QA, PI Fuzzing + Sec Review

Builds

Use latest build of Nightly for your platform from our product download page.

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 2017-08-01
Study documentation/specs received from developers 2017-08-01
QA - Test plan creation 2017-08-01
QA - Test cases/Env preparation 2017-08-01
QA - Nightly Testing 2017-09-19
QA - Beta Testing
Release Date

Testing Tools

Testing requires access to Test Rail, as well as physical possession of USB keys.

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla
Telemetry SCALARS_SECURITY.WEBAUTHN_USED, WEBAUTHN.CREATE_CREDENTIAL_MS, and WEBAUTHN_GET_ASSERTION_MS

Status

Overview

  • Feature landed, turned off, in Nightly 57 on 15-09-17
  • Feature will target Fx58/Fx59.
Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Release/Beta

References

  • Web Authentication W3C spec
  • Meta bug link
  • Product Integrity Security Assessment link

Testcases

Test Areas

Test Areas Covered Details
Private Window yes Test case
Multi-Process Enabled yes Test case in Test Rail
Multi-process Disabled yes Test case in Test Rail
Theme (high contrast) no n/a
UI This feature has no UI
Mouse-only operation no n/a
Keyboard-only operation no n/a
Display (HiDPI) no n/a
Interaction (scroll, zoom) no n/a
Usable with a screen reader no n/a
Usability and/or discoverability testing no n/a
RTL build testing no n/a
Help/Support
Help/support interface required no
Support documents planned(written) no
Install/Upgrade
Feature upgrades/downgrades data as expected no n/a
Does sync work across upgrades no n/a
Requires install testing no n/a
Affects first-run or onboarding no n/a
Does this affect partner builds? Partner build testing no n/a
Enterprise No special support for enterprise - feature is same as on release
Enterprise administration no can be turned on/off by pref if desired
Network proxies/autoconfig no n/a
ESR behavior changes no
Locked preferences no
Data Monitoring
Temporary or permanent telemetry monitoring yes see "Testing Tools" [section]
Telemetry correctness testing yes see "Testing Tools" [section]
Server integration testing yes If provided by third parties, yes, otherwise no
Offline and server failure testing no
Load testing no
Add-ons No additional support for add-ons at this time.
Addon API required? no
Comprehensive API testing no
Permissions no
Testing with existing/popular addons no
Security
3rd-party security review no In-house security review, yes
Privilege escalation testing yes QA + PI security review
Fuzzing yes Engineering + PI fuzzing team
Web Compatibility depends on the feature
Testing against target sites yes Sample sites are available
Survey of many sites for compatibility no If we support U2F, we can try to find U2F-enabled sites, but otherwise this is a new feature
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. yes This is inherent in the feature, w/r/t hardware keys
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS yes Fennec and Focus support TBD
Interaction of this feature with other browser features yes Largest area of targeted testing by QA

Test suite

Full Test suite - Link to test rail link
Smoke Test suite - see above.

Bug Work

Logged bugs ( blocking 1294514 )
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1395406 P1 DOM: Device Interfaces J.C. Jones [:jcj] (he/they) Crash when using two USB tokens on U2F test site RESOLVED ---
1398268 P2 DOM: Device Interfaces Tim Taubert [:ttaubert] (inactive) [U2F, WebAuthn] Crash when switching between browsers during many verification attempts VERIFIED mozilla59
1399298 P2 DOM: Device Interfaces J.C. Jones [:jcj] (he/they) [WebAuthn] Browser does not recover if USB verification is interrupted when computer goes to sleep RESOLVED ---
1399669 -- DOM: Device Interfaces Tim Taubert [:ttaubert] (inactive) Credential creation test failure on Linux: signature buffer has incorrect number of bytes RESOLVED ---
1400940 P2 DOM: Device Interfaces Tim Taubert [:ttaubert] (inactive) Deadlock after tab switch during verification process RESOLVED mozilla57
1401019 P2 DOM: Device Interfaces Tim Taubert [:ttaubert] (inactive) [U2F] Crash upon signing credential without registering one first RESOLVED mozilla57
1401802 P2 DOM: Device Interfaces J.C. Jones [:jcj] (he/they) [WebAuth] WebIDL missing extension fields RESOLVED ---
1401803 -- DOM: Device Interfaces J.C. Jones [:jcj] (he/they) [WebAuth] Return ArrayBuffer instead of UInt8Array RESOLVED mozilla58
1402114 P2 DOM: Web Authentication J.C. Jones [:jcj] (he/they) [WebAuth] Feature should not be accessible in iframe by default RESOLVED ---
1403330 P2 DOM: Device Interfaces J.C. Jones [:jcj] (he/they) [WebAuth/U2F] Crash when using specific Yubico test key RESOLVED ---

10 Total; 0 Open (0%); 9 Resolved (90%); 1 Verified (10%);

Bug fix verification

Bugzilla query error

Bugzilla API returned an error: 1

Sign off

Criteria

Checklist

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite, link to TestRail - Tests Runs and Results link
    • Daily Smoke, if needed/available
    • Regression Test suite, if needed/available


Merge to Beta Sign-off
List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) complete
Testing Infrastructure setup complete
Test Plan Creation complete
Test Cases Creation complete
Automation Coverage n/a
Performance Testing n/a
All Defects Logged complete
Critical/Blockers Fixed and Verified complete
Metrics/Telemetry n/a
Basic/Core functionality Nightly testing
QA mid-Nightly Signoff Email to be sent
QA Nightly - Full Testing
QA pre-Beta Signoff Email to be sent
QA Beta - Full Testing
QA pre-Release Signoff Email to be sent
Retrieved from "https://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&oldid=1189298"