Security/QA/TestPlans/Web Authentication

From MozillaWiki
Jump to: navigation, search

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
Ryan VanderMuelen QA Manager Product Integrity Date Email
JC Jones Software Engineer Engineering Date Email
JC Jones EPM Product Management (acting) Date Email

Revision History

Date Version Author Description
2017-08-16 1.0 Matt Wobensmith Created first draft
2017-10-04 1.1 Matt Wobensmith Sending for review
2017-10-04 1.2 Matt Wobensmith Incorporating review feedback from RyanVM



Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message.

What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.

The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is outlined within the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the spec itself.

We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.

The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements will not change.

The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.


The areas of client JavaScript and USB support are the focus of our test effort.

Code integrity

  • Unit tests
  • Code-level security review
  • Fuzzing


  • Manual testing
  • Real-world implementations


This feature is being tested by both Mozilla and one or more third parties.

  • Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests
  • JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction
  • Yubico is performing smoke tests using hardware keys across a range of hardware and software
  • Adam Powers (FIDO) is creating tests for the web-platform-test suite
  • The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown
  • The PI Security team has been requested to perform a security review of both JS API and Rust USB library
  • Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff

Testing summary

Scope of Testing

In Scope

  • Web Authentication, as well as some U2F.
  • All JS APIs.
  • Fuzzing wherever possible.
  • A range of scenario tests that mirror user interaction, including boundary and error cases.
  • Some USB hardware, including Yubico keys and a few others given to us.

Out of Scope

  • Software token is unsupported, for now.
  • Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us.
  • Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.
  • This feature is not currently supported on Fennec.
  • We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.

Requirements for testing


We support the same OS and hardware configurations that Firefox supports on desktop only.

Channel dependent settings (configs) and environment setups

The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:


Optional: to use unsupported soft token, set to true:



Currently set to false.


Currently set to false.

Post Beta / Release

Depending on ship decisions, will be set to true.

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Discoverability Impact Score
RAC-1 Incorrect authentication allows security bypass TO-1, TO-2, TO-3 3-High 1-Unlikely 2-Moderate 6
RAC-2 XSS/information leak TO-1, TO-3 3-High 1-Unlikely 1-Low 3
RAC-3 Confined to secure context TO-1, TO-3 2-Moderate 2-Possible 1-Low 4
RAC-4 Incorrectly functioning JS API TO-1 3-High 2-Possible 2-Moderate 12
RAC-5 Stability for entire feature TO-1, TO-2 3-High 2-Possible 3-High 18
RAC-6 Interaction with other aspects of normal Firefox usage TO-1, TO-2 3-Moderate 3-Almost Certain 3-High 27
RAC-7 Memory issues in JS API and hardware support code TO-3 3-High 1-Unlikely 2-Moderate 6
RAC-8 Incorrectly functioning hardware TO-2 2-Moderate 1-Unlikely 1-Low 2


  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Discoverability: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

Verify that the feature works as designed, interacts well with normal use of Firefox, is stable and has secure code.

Ref Function Test Objective Evaluation Criteria Test Type RAC Owners
TO1 JS API Verify functionality All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens Manual/ Automation / Usability RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 Eng Team, QA
TO2 Hardware support via USB token Verify functionality All tests indicate stable, functional support of USB hardware keys, as above Manual/ Automation / Usability RAC-1, RAC-5, RAC-6, RAC-8 Eng Team, QA
TO3 Stable, secure code Fuzzing and security review All testing and inspection surfaces known security issues Manual/ Security RAC-1, RAC-2, RAC-3, RAC-7 Eng Team, QA, PI Fuzzing + Sec Review


Use latest build of Nightly for your platform from our product download page.

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 2017-08-01
Study documentation/specs received from developers 2017-08-01
QA - Test plan creation 2017-08-01
QA - Test cases/Env preparation 2017-08-01
QA - Nightly Testing 2017-09-19
QA - Beta Testing
Release Date

Testing Tools

Testing requires access to Test Rail, as well as physical possession of USB keys.

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla



  • Feature landed, turned off, in Nightly 57 on 15-09-17
  • Feature will target Fx58/Fx59.
Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Release/Beta


  • Web Authentication W3C spec
  • Meta bug link
  • Product Integrity Security Assessment link


Test Areas

Test Areas Covered Details
Private Window yes Test case
Multi-Process Enabled yes Test case in Test Rail
Multi-process Disabled yes Test case in Test Rail
Theme (high contrast) no n/a
UI This feature has no UI
Mouse-only operation no n/a
Keyboard-only operation no n/a
Display (HiDPI) no n/a
Interaction (scroll, zoom) no n/a
Usable with a screen reader no n/a
Usability and/or discoverability testing no n/a
RTL build testing no n/a
Help/support interface required no
Support documents planned(written) no
Feature upgrades/downgrades data as expected no n/a
Does sync work across upgrades no n/a
Requires install testing no n/a
Affects first-run or onboarding no n/a
Does this affect partner builds? Partner build testing no n/a
Enterprise No special support for enterprise - feature is same as on release
Enterprise administration no can be turned on/off by pref if desired
Network proxies/autoconfig no n/a
ESR behavior changes no
Locked preferences no
Data Monitoring
Temporary or permanent telemetry monitoring yes see "Testing Tools" [section]
Telemetry correctness testing yes see "Testing Tools" [section]
Server integration testing yes If provided by third parties, yes, otherwise no
Offline and server failure testing no
Load testing no
Add-ons No additional support for add-ons at this time.
Addon API required? no
Comprehensive API testing no
Permissions no
Testing with existing/popular addons no
3rd-party security review no In-house security review, yes
Privilege escalation testing yes QA + PI security review
Fuzzing yes Engineering + PI fuzzing team
Web Compatibility depends on the feature
Testing against target sites yes Sample sites are available
Survey of many sites for compatibility no If we support U2F, we can try to find U2F-enabled sites, but otherwise this is a new feature
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. yes This is inherent in the feature, w/r/t hardware keys
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS yes Fennec and Focus support TBD
Interaction of this feature with other browser features yes Largest area of targeted testing by QA

Test suite

Full Test suite - Link to test rail link
Smoke Test suite - see above.

Bug Work

Logged bugs ( blocking 1294514 )
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1395406 P1 DOM: Device Interfaces J.C. Jones [:jcj] (he/him) [taking a break] Crash when using two USB tokens on U2F test site RESOLVED ---
1398268 P2 DOM: Device Interfaces Tim Taubert [:ttaubert] (inactive) [U2F, WebAuthn] Crash when switching between browsers during many verification attempts VERIFIED mozilla59
1399298 P2 DOM: Device Interfaces J.C. Jones [:jcj] (he/him) [taking a break] [WebAuthn] Browser does not recover if USB verification is interrupted when computer goes to sleep RESOLVED ---
1399669 -- DOM: Device Interfaces Tim Taubert [:ttaubert] (inactive) Credential creation test failure on Linux: signature buffer has incorrect number of bytes RESOLVED ---
1400940 P2 DOM: Device Interfaces Tim Taubert [:ttaubert] (inactive) Deadlock after tab switch during verification process RESOLVED mozilla57
1401019 P2 DOM: Device Interfaces Tim Taubert [:ttaubert] (inactive) [U2F] Crash upon signing credential without registering one first RESOLVED mozilla57
1401802 P2 DOM: Device Interfaces J.C. Jones [:jcj] (he/him) [taking a break] [WebAuth] WebIDL missing extension fields RESOLVED ---
1401803 -- DOM: Device Interfaces J.C. Jones [:jcj] (he/him) [taking a break] [WebAuth] Return ArrayBuffer instead of UInt8Array RESOLVED mozilla58
1402114 P2 DOM: Web Authentication J.C. Jones [:jcj] (he/him) [taking a break] [WebAuth] Feature should not be accessible in iframe by default RESOLVED ---
1403330 P2 DOM: Device Interfaces J.C. Jones [:jcj] (he/him) [taking a break] [WebAuth/U2F] Crash when using specific Yubico test key RESOLVED ---

10 Total; 0 Open (0%); 9 Resolved (90%); 1 Verified (10%);

Bug fix verification
Full Query
ID Priority Component Assigned to Summary Status Resolution Target milestone
1245527 P3 DOM: Device Interfaces J.C. Jones [:jcj] (he/him) [taking a break] Integrate the FIDO U2F JS API with the u2f-hid-rs library RESOLVED FIXED mozilla57

1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);

Sign off



  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)


Nightly testing

List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite, link to TestRail - Tests Runs and Results link
    • Daily Smoke, if needed/available
    • Regression Test suite, if needed/available

Merge to Beta Sign-off
List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite


Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) complete
Testing Infrastructure setup complete
Test Plan Creation complete
Test Cases Creation complete
Automation Coverage n/a
Performance Testing n/a
All Defects Logged complete
Critical/Blockers Fixed and Verified complete
Metrics/Telemetry n/a
Basic/Core functionality Nightly testing
QA mid-Nightly Signoff Email to be sent
QA Nightly - Full Testing
QA pre-Beta Signoff Email to be sent
QA Beta - Full Testing
QA pre-Release Signoff Email to be sent