Approvals Required / Received
The following individuals are required to/have approved this Test Plan:
|Ryan VanderMuelen||QA Manager||Product Integrity||Date|
|JC Jones||Software Engineer||Engineering||Date|
|JC Jones||EPM||Product Management (acting)||Date|
|2017-08-16||1.0||Matt Wobensmith||Created first draft|
|2017-10-04||1.1||Matt Wobensmith||Sending for review|
|2017-10-04||1.2||Matt Wobensmith||Incorporating review feedback from RyanVM|
- 1 Overview
- 2 Testing summary
- 3 Requirements for testing
- 4 Test Strategy
- 5 Status
- 6 References
- 7 Testcases
- 8 Bug Work
- 9 Sign off
Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message.
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.
The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements will not change.
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.
- Unit tests
- Code-level security review
- Manual testing
- Real-world implementations
This feature is being tested by both Mozilla and one or more third parties.
- Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests
- JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction
- Yubico is performing smoke tests using hardware keys across a range of hardware and software
- Adam Powers (FIDO) is creating tests for the web-platform-test suite
- The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown
- The PI Security team has been requested to perform a security review of both JS API and Rust USB library
- Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff
Scope of Testing
- Web Authentication, as well as some U2F.
- All JS APIs.
- Fuzzing wherever possible.
- A range of scenario tests that mirror user interaction, including boundary and error cases.
- Some USB hardware, including Yubico keys and a few others given to us.
Out of Scope
- Software token is unsupported, for now.
- Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us.
- Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.
- This feature is not currently supported on Fennec.
- We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.
Requirements for testing
We support the same OS and hardware configurations that Firefox supports on desktop only.
Channel dependent settings (configs) and environment setups
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:
security.webauth.u2f; security.webauth.webauthn; security.webauth.webauthn_enable_usbtoken;
Optional: to use unsupported soft token, set to true:
Currently set to false.
Currently set to false.
Post Beta / Release
Depending on ship decisions, will be set to true.
Risk Assessment and Coverage
|ID||Description / Threat Description||Covered by Test Objective||Magnitude||Probability||Discoverability||Impact Score|
|RAC-1||Incorrect authentication allows security bypass||TO-1, TO-2, TO-3||3-High||1-Unlikely||2-Moderate||6|
|RAC-2||XSS/information leak||TO-1, TO-3||3-High||1-Unlikely||1-Low||3|
|RAC-3||Confined to secure context||TO-1, TO-3||2-Moderate||2-Possible||1-Low||4|
|RAC-4||Incorrectly functioning JS API||TO-1||3-High||2-Possible||2-Moderate||12|
|RAC-5||Stability for entire feature||TO-1, TO-2||3-High||2-Possible||3-High||18|
|RAC-6||Interaction with other aspects of normal Firefox usage||TO-1, TO-2||3-Moderate||3-Almost Certain||3-High||27|
|RAC-7||Memory issues in JS API and hardware support code||TO-3||3-High||1-Unlikely||2-Moderate||6|
|RAC-8||Incorrectly functioning hardware||TO-2||2-Moderate||1-Unlikely||1-Low||2|
- Magnitude: 1- Low , 2-Moderate, 3-High
- Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
- Discoverability: 1 - Low, 2-Medium, 3-High
Impact Score Breakdown:
- An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
- An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
- An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
Verify that the feature works as designed, interacts well with normal use of Firefox, is stable and has secure code.
|Ref||Function||Test Objective||Evaluation Criteria||Test Type||RAC||Owners|
|TO1||JS API||Verify functionality||All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens||Manual/ Automation / Usability||RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6||Eng Team, QA|
|TO2||Hardware support via USB token||Verify functionality||All tests indicate stable, functional support of USB hardware keys, as above||Manual/ Automation / Usability||RAC-1, RAC-5, RAC-6, RAC-8||Eng Team, QA|
|TO3||Stable, secure code||Fuzzing and security review||All testing and inspection surfaces known security issues||Manual/ Security||RAC-1, RAC-2, RAC-3, RAC-7||Eng Team, QA, PI Fuzzing + Sec Review|
Use latest build of Nightly for your platform from our product download page.
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
|Project phase||Start Date||End Date|
|Study documentation/specs received from developers||2017-08-01|
|QA - Test plan creation||2017-08-01|
|QA - Test cases/Env preparation||2017-08-01|
|QA - Nightly Testing||2017-09-19|
|QA - Beta Testing|
Testing requires access to Test Rail, as well as physical possession of USB keys.
|Test plan creation||Mozilla wiki|
|Test case creation||TestRail/ Google docs|
|Test case execution||TestRail|
|Telemetry||SCALARS_SECURITY.WEBAUTHN_USED, WEBAUTHN.CREATE_CREDENTIAL_MS, and WEBAUTHN_GET_ASSERTION_MS|
- Feature landed, turned off, in Nightly 57 on 15-09-17
- Feature will target Fx58/Fx59.
Track the dates and build number where feature was released to Nightly Track the dates and build number where feature was merged to Release/Beta
|Private Window||yes||Test case|
|Multi-Process Enabled||yes||Test case in Test Rail|
|Multi-process Disabled||yes||Test case in Test Rail|
|Theme (high contrast)||no||n/a|
|UI||This feature has no UI|
|Interaction (scroll, zoom)||no||n/a|
|Usable with a screen reader||no||n/a|
|Usability and/or discoverability testing||no||n/a|
|RTL build testing||no||n/a|
|Help/support interface required||no|
|Support documents planned(written)||no|
|Feature upgrades/downgrades data as expected||no||n/a|
|Does sync work across upgrades||no||n/a|
|Requires install testing||no||n/a|
|Affects first-run or onboarding||no||n/a|
|Does this affect partner builds? Partner build testing||no||n/a|
|Enterprise||No special support for enterprise - feature is same as on release|
|Enterprise administration||no||can be turned on/off by pref if desired|
|ESR behavior changes||no|
|Temporary or permanent telemetry monitoring||yes||see "Testing Tools" [section]|
|Telemetry correctness testing||yes||see "Testing Tools" [section]|
|Server integration testing||yes||If provided by third parties, yes, otherwise no|
|Offline and server failure testing||no|
|Add-ons||No additional support for add-ons at this time.|
|Addon API required?||no|
|Comprehensive API testing||no|
|Testing with existing/popular addons||no|
|3rd-party security review||no||In-house security review, yes|
|Privilege escalation testing||yes||QA + PI security review|
|Fuzzing||yes||Engineering + PI fuzzing team|
|Web Compatibility||depends on the feature|
|Testing against target sites||yes||Sample sites are available|
|Survey of many sites for compatibility||no||If we support U2F, we can try to find U2F-enabled sites, but otherwise this is a new feature|
|Interoperability||depends on the feature|
|Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.||yes||This is inherent in the feature, w/r/t hardware keys|
|Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS||yes||Fennec and Focus support TBD|
|Interaction of this feature with other browser features||yes||Largest area of targeted testing by QA|
Full Test suite - Link to test rail link Smoke Test suite - see above.
Logged bugs ( blocking 1294514 )
|ID||Priority||Component||Assigned to||Summary||Status||Target milestone|
|1395406||P1||DOM: Device Interfaces||J.C. Jones [:jcj] (he/him) [taking a break]||Crash when using two USB tokens on U2F test site||RESOLVED||---|
|1398268||P2||DOM: Device Interfaces||Tim Taubert [:ttaubert] (inactive)||[U2F, WebAuthn] Crash when switching between browsers during many verification attempts||VERIFIED||mozilla59|
|1399298||P2||DOM: Device Interfaces||J.C. Jones [:jcj] (he/him) [taking a break]||[WebAuthn] Browser does not recover if USB verification is interrupted when computer goes to sleep||RESOLVED||---|
|1399669||--||DOM: Device Interfaces||Tim Taubert [:ttaubert] (inactive)||Credential creation test failure on Linux: signature buffer has incorrect number of bytes||RESOLVED||---|
|1400940||P2||DOM: Device Interfaces||Tim Taubert [:ttaubert] (inactive)||Deadlock after tab switch during verification process||RESOLVED||mozilla57|
|1401019||P2||DOM: Device Interfaces||Tim Taubert [:ttaubert] (inactive)||[U2F] Crash upon signing credential without registering one first||RESOLVED||mozilla57|
|1401802||P2||DOM: Device Interfaces||J.C. Jones [:jcj] (he/him) [taking a break]||[WebAuth] WebIDL missing extension fields||RESOLVED||---|
|1401803||--||DOM: Device Interfaces||J.C. Jones [:jcj] (he/him) [taking a break]||[WebAuth] Return ArrayBuffer instead of UInt8Array||RESOLVED||mozilla58|
|1402114||P2||DOM: Web Authentication||J.C. Jones [:jcj] (he/him) [taking a break]||[WebAuth] Feature should not be accessible in iframe by default||RESOLVED||---|
|1403330||P2||DOM: Device Interfaces||J.C. Jones [:jcj] (he/him) [taking a break]||[WebAuth/U2F] Crash when using specific Yubico test key||RESOLVED||---|
10 Total; 0 Open (0%); 9 Resolved (90%); 1 Verified (10%);
Bug fix verification
|ID||Priority||Component||Assigned to||Summary||Status||Resolution||Target milestone|
|1245527||P3||DOM: Device Interfaces||J.C. Jones [:jcj] (he/him) [taking a break]||Integrate the FIDO U2F JS API with the u2f-hid-rs library||RESOLVED||FIXED||mozilla57|
1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite, link to TestRail - Tests Runs and Results link
- Daily Smoke, if needed/available
- Regression Test suite, if needed/available
Merge to Beta Sign-off
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite
|Testing Prerequisites (specs, use cases)||complete|
|Testing Infrastructure setup||complete|
|Test Plan Creation||complete|
|Test Cases Creation||complete|
|All Defects Logged||complete|
|Critical/Blockers Fixed and Verified||complete|
|Basic/Core functionality Nightly testing|
|QA mid-Nightly Signoff||Email to be sent|
|QA Nightly - Full Testing|
|QA pre-Beta Signoff||Email to be sent|
|QA Beta - Full Testing|
|QA pre-Release Signoff||Email to be sent|