https://wiki.mozilla.org/api.php?action=feedcontributions&user=Mwobensmith&feedformat=atomMozillaWiki - User contributions [en]2024-03-28T12:44:16ZUser contributionsMediaWiki 1.27.4https://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1189298Security/QA/TestPlans/Web Authentication2018-02-21T21:43:27Z<p>Mwobensmith: /* Checklist */</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| Ryan VanderMuelen || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| JC Jones || EPM || Product Management (acting) || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|-<br />
| 2017-10-04 || 1.1 || Matt Wobensmith || Sending for review<br />
|-<br />
| 2017-10-04 || 1.2 || Matt Wobensmith || Incorporating review feedback from RyanVM<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is outlined within the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the spec itself.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.<br />
<br />
The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Discoverability !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Discoverability:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
Verify that the feature works as designed, interacts well with normal use of Firefox, is stable and has secure code.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
Use latest build of Nightly for your platform from our [https://www.mozilla.org/en-US/firefox/channel/desktop/ product download page].<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Testing requires access to Test Rail, as well as physical possession of USB keys.<br />
<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|-<br />
| Telemetry || SCALARS_SECURITY.WEBAUTHN_USED, WEBAUTHN.CREATE_CREDENTIAL_MS, and WEBAUTHN_GET_ASSERTION_MS<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || Test case in Test Rail<br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || Test case in Test Rail<br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || This feature has no UI<br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || No special support for enterprise - feature is same as on release<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || can be turned on/off by pref if desired<br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | yes || see "Testing Tools" [[https://wiki.mozilla.org/Security/QA/TestPlans/Web_Authentication#Testing_Tools section]]<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | yes || see "Testing Tools" [[https://wiki.mozilla.org/Security/QA/TestPlans/Web_Authentication#Testing_Tools section]]<br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || No additional support for add-ons at this time.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || <br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | no || If we support U2F, we can try to find U2F-enabled sites, but otherwise this is a new feature<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail [https://testrail.stage.mozaws.net/index.php?/suites/overview/49 link]<br />
Smoke Test suite - see above.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Test Plan Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage <br />
|style="text-align:center;" | n/a || <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | n/a || <br />
|-<br />
| All Defects Logged <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Critical/Blockers Fixed and Verified <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Metrics/Telemetry<br />
|style="text-align:center;" | n/a || <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1189297Security/QA/TestPlans/Web Authentication2018-02-21T21:35:31Z<p>Mwobensmith: </p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| Ryan VanderMuelen || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| JC Jones || EPM || Product Management (acting) || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|-<br />
| 2017-10-04 || 1.1 || Matt Wobensmith || Sending for review<br />
|-<br />
| 2017-10-04 || 1.2 || Matt Wobensmith || Incorporating review feedback from RyanVM<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is outlined within the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the spec itself.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.<br />
<br />
The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Discoverability !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Discoverability:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
Verify that the feature works as designed, interacts well with normal use of Firefox, is stable and has secure code.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
Use latest build of Nightly for your platform from our [https://www.mozilla.org/en-US/firefox/channel/desktop/ product download page].<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Testing requires access to Test Rail, as well as physical possession of USB keys.<br />
<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|-<br />
| Telemetry || SCALARS_SECURITY.WEBAUTHN_USED, WEBAUTHN.CREATE_CREDENTIAL_MS, and WEBAUTHN_GET_ASSERTION_MS<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || Test case in Test Rail<br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || Test case in Test Rail<br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || This feature has no UI<br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || No special support for enterprise - feature is same as on release<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || can be turned on/off by pref if desired<br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | yes || see "Testing Tools" [[https://wiki.mozilla.org/Security/QA/TestPlans/Web_Authentication#Testing_Tools section]]<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | yes || see "Testing Tools" [[https://wiki.mozilla.org/Security/QA/TestPlans/Web_Authentication#Testing_Tools section]]<br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || No additional support for add-ons at this time.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || <br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | no || If we support U2F, we can try to find U2F-enabled sites, but otherwise this is a new feature<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail [https://testrail.stage.mozaws.net/index.php?/suites/overview/49 link]<br />
Smoke Test suite - see above.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | signed off || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage | n/a |<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | n/a || <br />
|-<br />
| All Defects Logged | complete | || <br />
|-<br />
| Critical/Blockers Fixed and Verified | complete | || <br />
|-<br />
| Metrics/Telemetry| n/a | <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1182814Security/QA/TestPlans/Web Authentication2017-10-24T21:46:42Z<p>Mwobensmith: Minor tweaks</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| Ryan VanderMuelen || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| JC Jones || EPM || Product Management (acting) || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|-<br />
| 2017-10-04 || 1.1 || Matt Wobensmith || Sending for review<br />
|-<br />
| 2017-10-04 || 1.2 || Matt Wobensmith || Incorporating review feedback from RyanVM<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is outlined within the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the spec itself.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.<br />
<br />
The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Discoverability !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Discoverability:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
Verify that the feature works as designed, interacts well with normal use of Firefox, is stable and has secure code.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
Use latest build of Nightly for your platform from our [https://www.mozilla.org/en-US/firefox/channel/desktop/ product download page].<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Testing requires access to Test Rail, as well as physical possession of USB keys.<br />
<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|-<br />
| Telemetry || SCALARS_SECURITY.WEBAUTHN_USED, WEBAUTHN.CREATE_CREDENTIAL_MS, and WEBAUTHN_GET_ASSERTION_MS<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || Test case in Test Rail<br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || Test case in Test Rail<br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || This feature has no UI<br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || No special support for enterprise - feature is same as on release<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || can be turned on/off by pref if desired<br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | yes || see "Testing Tools" [[https://wiki.mozilla.org/Security/QA/TestPlans/Web_Authentication#Testing_Tools section]]<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | yes || see "Testing Tools" [[https://wiki.mozilla.org/Security/QA/TestPlans/Web_Authentication#Testing_Tools section]]<br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || No additional support for add-ons at this time.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || <br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | no || If we support U2F, we can try to find U2F-enabled sites, but otherwise this is a new feature<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail [https://testrail.stage.mozaws.net/index.php?/suites/overview/49 link]<br />
Smoke Test suite - see above.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1182813Security/QA/TestPlans/Web Authentication2017-10-24T21:43:43Z<p>Mwobensmith: Removed boilerplate text</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| Ryan VanderMuelen || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| JC Jones || EPM || Product Management (acting) || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|-<br />
| 2017-10-04 || 1.1 || Matt Wobensmith || Sending for review<br />
|-<br />
| 2017-10-04 || 1.2 || Matt Wobensmith || Incorporating review feedback from RyanVM<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is outlined within the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the spec itself.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.<br />
<br />
The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Discoverability !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Discoverability:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
Verify that the feature works as designed, interacts well with normal use of Firefox, is stable and has secure code.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
Use latest build of Nightly for your platform from our [https://www.mozilla.org/en-US/firefox/channel/desktop/ product download page].<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Testing requires access to Test Rail, as well as physical possession of USB keys.<br />
<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|-<br />
| Telemetry || SCALARS_SECURITY.WEBAUTHN_USED, WEBAUTHN.CREATE_CREDENTIAL_MS, and WEBAUTHN_GET_ASSERTION_MS<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || Test case in Test Rail<br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || Test case in Test Rail<br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || This feature has no UI<br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || No special support for enterprise - feature is same as on release<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || can be turned on/off by pref if desired<br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | yes || see "Testing Tools" [[https://wiki.mozilla.org/Security/QA/TestPlans/Web_Authentication#Testing_Tools section]]<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | yes || see "Testing Tools" [[https://wiki.mozilla.org/Security/QA/TestPlans/Web_Authentication#Testing_Tools section]]<br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || No additional support for add-ons at this time.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | UNKNOWN || Is there a U2F add-on? <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail [https://testrail.stage.mozaws.net/index.php?/suites/overview/49 link]<br />
Smoke Test suite - see above.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1182555Security/QA/TestPlans/Web Authentication2017-10-19T21:31:14Z<p>Mwobensmith: More changes</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| Ryan VanderMuelen || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| JC Jones || EPM || Product Management (acting) || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|-<br />
| 2017-10-04 || 1.1 || Matt Wobensmith || Sending for review<br />
|-<br />
| 2017-10-04 || 1.2 || Matt Wobensmith || Incorporating review feedback from RyanVM<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is outlined within the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the spec itself.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.<br />
<br />
The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Discoverability !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Discoverability:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
Verify that the feature works as designed, interacts well with normal use of Firefox, is stable and has secure code.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
Use latest build of Nightly for your platform from our [https://www.mozilla.org/en-US/firefox/channel/desktop/ product download page].<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Testing requires access to Test Rail, as well as physical possession of USB keys.<br />
<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|-<br />
| Telemetry || SCALARS_SECURITY.WEBAUTHN_USED, WEBAUTHN.CREATE_CREDENTIAL_MS, and WEBAUTHN_GET_ASSERTION_MS<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || Test case in Test Rail<br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || Test case in Test Rail<br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || This feature has no UI<br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || No special support for enterprise - feature is same as on release<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || can be turned on/off by pref if desired<br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | yes || see "Testing Tools" [[https://wiki.mozilla.org/Security/QA/TestPlans/Web_Authentication#Testing_Tools section]]<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | yes || see "Testing Tools" [[https://wiki.mozilla.org/Security/QA/TestPlans/Web_Authentication#Testing_Tools section]]<br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || No additional support for add-ons at this time.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | UNKNOWN || Is there a U2F add-on? <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail [https://testrail.stage.mozaws.net/index.php?/suites/overview/49 link]<br />
Smoke Test suite - see above.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1182554Security/QA/TestPlans/Web Authentication2017-10-19T21:18:16Z<p>Mwobensmith: Edits</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| Ryan VanderMuelen || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| JC Jones || EPM || Product Management (acting) || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|-<br />
| 2017-10-04 || 1.1 || Matt Wobensmith || Sending for review<br />
|-<br />
| 2017-10-04 || 1.2 || Matt Wobensmith || Incorporating review feedback from RyanVM<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is outlined within the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the spec itself.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.<br />
<br />
The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Discoverability !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Discoverability:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
Verify that the feature works as designed, interacts well with normal use of Firefox, is stable and has secure code.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
Use latest build of Nightly for your platform from our [https://www.mozilla.org/en-US/firefox/channel/desktop/ product download page].<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Testing requires access to Test Rail, as well as physical possession of USB keys.<br />
<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|-<br />
| Telemetry || SCALARS_SECURITY.WEBAUTHN_USED, WEBAUTHN.CREATE_CREDENTIAL_MS, and WEBAUTHN_GET_ASSERTION_MS<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || Test case in Test Rail<br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || Test case in Test Rail<br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || This feature has no UI<br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || No special support for enterprise at this time<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | yes || see "Testing Tools" [[https://wiki.mozilla.org/Security/QA/TestPlans/Web_Authentication#Testing_Tools|section]]<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | yes || see "Testing Tools" [[https://wiki.mozilla.org/Security/QA/TestPlans/Web_Authentication#Testing_Tools|section]]<br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail [https://testrail.stage.mozaws.net/index.php?/suites/overview/49 link]<br />
Smoke Test suite - see above.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1182553Security/QA/TestPlans/Web Authentication2017-10-19T21:17:05Z<p>Mwobensmith: Fixing data</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| Ryan VanderMuelen || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| JC Jones || EPM || Product Management (acting) || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|-<br />
| 2017-10-04 || 1.1 || Matt Wobensmith || Sending for review<br />
|-<br />
| 2017-10-04 || 1.2 || Matt Wobensmith || Incorporating review feedback from RyanVM<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is outlined within the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the spec itself.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.<br />
<br />
The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Discoverability !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Discoverability:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
Verify that the feature works as designed, interacts well with normal use of Firefox, is stable and has secure code.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
Use latest build of Nightly for your platform from our [https://www.mozilla.org/en-US/firefox/channel/desktop/ product download page].<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Testing requires access to Test Rail, as well as physical possession of USB keys.<br />
<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|-<br />
| Telemetry || SCALARS_SECURITY.WEBAUTHN_USED, WEBAUTHN.CREATE_CREDENTIAL_MS, and WEBAUTHN_GET_ASSERTION_MS<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || This feature has no UI<br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || No special support for enterprise at this time<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | yes || see "Testing Tools" [[https://wiki.mozilla.org/Security/QA/TestPlans/Web_Authentication#Testing_Tools|section]]<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | yes || see "Testing Tools" [[https://wiki.mozilla.org/Security/QA/TestPlans/Web_Authentication#Testing_Tools|section]]<br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail [https://testrail.stage.mozaws.net/index.php?/suites/overview/49 link]<br />
Smoke Test suite - see above.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1182552Security/QA/TestPlans/Web Authentication2017-10-19T21:12:42Z<p>Mwobensmith: Added telemetry</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| Ryan VanderMuelen || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| JC Jones || EPM || Product Management (acting) || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|-<br />
| 2017-10-04 || 1.1 || Matt Wobensmith || Sending for review<br />
|-<br />
| 2017-10-04 || 1.2 || Matt Wobensmith || Incorporating review feedback from RyanVM<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is outlined within the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the spec itself.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.<br />
<br />
The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Discoverability !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Discoverability:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
Verify that the feature works as designed, interacts well with normal use of Firefox, is stable and has secure code.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
Use latest build of Nightly for your platform from our [https://www.mozilla.org/en-US/firefox/channel/desktop/ product download page].<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Testing requires access to Test Rail, as well as physical possession of USB keys.<br />
<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|-<br />
| Telemetry || SCALARS_SECURITY.WEBAUTHN_USED, WEBAUTHN.CREATE_CREDENTIAL_MS, and WEBAUTHN_GET_ASSERTION_MS<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
<br />
Note: This feature has no UI. Therefore, all test areas that involve UI are marked n/a or not applicable.<br />
<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail [https://testrail.stage.mozaws.net/index.php?/suites/overview/49 link]<br />
Smoke Test suite - see above.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1182551Security/QA/TestPlans/Web Authentication2017-10-19T21:11:15Z<p>Mwobensmith: Integrating feedback</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| Ryan VanderMuelen || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| JC Jones || EPM || Product Management (acting) || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|-<br />
| 2017-10-04 || 1.1 || Matt Wobensmith || Sending for review<br />
|-<br />
| 2017-10-04 || 1.2 || Matt Wobensmith || Incorporating review feedback from RyanVM<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is outlined within the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the spec itself.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.<br />
<br />
The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Discoverability !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Discoverability:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
Verify that the feature works as designed, interacts well with normal use of Firefox, is stable and has secure code.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
Use latest build of Nightly for your platform from our [https://www.mozilla.org/en-US/firefox/channel/desktop/ product download page].<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Testing requires access to Test Rail, as well as physical possession of USB keys.<br />
<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
<br />
Note: This feature has no UI. Therefore, all test areas that involve UI are marked n/a or not applicable.<br />
<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail [https://testrail.stage.mozaws.net/index.php?/suites/overview/49 link]<br />
Smoke Test suite - see above.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181838Security/QA/TestPlans/Web Authentication2017-10-04T22:15:14Z<p>Mwobensmith: UPdated revision history</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| Ryan VanderMuelen || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| (not assigned) || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|-<br />
| 2017-10-04 || 1.1 || Matt Wobensmith || Sending for review<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is outlined within the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the spec itself.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.<br />
<br />
The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
Use latest build of Nightly for your platform from our [https://www.mozilla.org/en-US/firefox/channel/desktop/ product download page].<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail [https://testrail.stage.mozaws.net/index.php?/suites/overview/49 link]<br />
Smoke Test suite - see above.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181837Security/QA/TestPlans/Web Authentication2017-10-04T22:14:04Z<p>Mwobensmith: Minor</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| Ryan VanderMuelen || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| (not assigned) || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is outlined within the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the spec itself.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.<br />
<br />
The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
Use latest build of Nightly for your platform from our [https://www.mozilla.org/en-US/firefox/channel/desktop/ product download page].<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail [https://testrail.stage.mozaws.net/index.php?/suites/overview/49 link]<br />
Smoke Test suite - see above.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181835Security/QA/TestPlans/Web Authentication2017-10-04T22:10:57Z<p>Mwobensmith: Cleaner</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| Ryan VanderMuelen || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is outlined within the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the spec itself.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.<br />
<br />
The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
Use latest build of Nightly for your platform from our [https://www.mozilla.org/en-US/firefox/channel/desktop/ product download page].<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail [https://testrail.stage.mozaws.net/index.php?/suites/overview/49 link]<br />
Smoke Test suite - see above.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181834Security/QA/TestPlans/Web Authentication2017-10-04T22:09:46Z<p>Mwobensmith: Lots of enhancements</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| Ryan VanderMuelen || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication - or "WebAuthN" - is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is outlined within the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the spec itself.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most problems likely to be encountered by everyday Firefox users.<br />
<br />
The exact release of Firefox is dependent on the status of the W3C spec, which is nearing finalization. Regardless, the vast majority of this feature's test requirements is captured here and will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases will be included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico and FIDO have provided us with some USB keys to test with, but the full range of potentially supported keys is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
Use latest build of Nightly for your platform from our [https://www.mozilla.org/en-US/firefox/channel/desktop/ product download page].<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail [https://testrail.stage.mozaws.net/index.php?/suites/overview/49 link]<br />
Smoke Test suite - see above.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181831Security/QA/TestPlans/Web Authentication2017-10-04T21:54:56Z<p>Mwobensmith: Created</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is covered by the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the W3C Web Authentication spec.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most bugs and problems encountered by everyday Firefox users.<br />
<br />
Currently, this feature is slated for Firefox 58, but this dependent on the status of the W3C spec. Regardless, the vast majority of this feature's test requirements is captured here and will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases is included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
Use latest build of Nightly for your platform from our [https://www.mozilla.org/en-US/firefox/channel/desktop/ product download page].<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail [https://testrail.stage.mozaws.net/index.php?/suites/overview/49 link]<br />
Smoke Test suite - see above.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181830Security/QA/TestPlans/Web Authentication2017-10-04T21:52:25Z<p>Mwobensmith: Created</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is covered by the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the W3C Web Authentication spec.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most bugs and problems encountered by everyday Firefox users.<br />
<br />
Currently, this feature is slated for Firefox 58, but this dependent on the status of the W3C spec. Regardless, the vast majority of this feature's test requirements is captured here and will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases is included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
Use latest build of Nightly for your platform from our [https://www.mozilla.org/en-US/firefox/channel/desktop/ product download page].<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181829Security/QA/TestPlans/Web Authentication2017-10-04T21:49:37Z<p>Mwobensmith: Reorder bullets</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is covered by the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the W3C Web Authentication spec.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most bugs and problems encountered by everyday Firefox users.<br />
<br />
Currently, this feature is slated for Firefox 58, but this dependent on the status of the W3C spec. Regardless, the vast majority of this feature's test requirements is captured here and will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases is included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181828Security/QA/TestPlans/Web Authentication2017-10-04T21:48:50Z<p>Mwobensmith: Created</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is covered by the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the W3C Web Authentication spec.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most bugs and problems encountered by everyday Firefox users.<br />
<br />
Currently, this feature is slated for Firefox 58, but this dependent on the status of the W3C spec. Regardless, the vast majority of this feature's test requirements is captured here and will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases is included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
The areas of client JavaScript and USB support are the focus of our test effort.<br />
<br />
Code integrity<br />
* Unit tests<br />
* Code-level security review<br />
* Fuzzing<br />
<br />
Functionality<br />
* Manual testing<br />
* Real-world implementations<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181827Security/QA/TestPlans/Web Authentication2017-10-04T21:43:22Z<p>Mwobensmith: Spelling error, minor enhancements</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication is the proposed W3C standard for creating an interface to validate a local, cryptographically-signed message. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is covered by the W3C spec. Firefox also implements new USB support for interacting with these hardware tokens, which is tangential to our implementation of the W3C Web Authentication spec.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most bugs and problems encountered by everyday Firefox users.<br />
<br />
Currently, this feature is slated for Firefox 58, but this dependent on the status of the W3C spec. Regardless, the vast majority of this feature's test requirements is captured here and will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases is included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181826Security/QA/TestPlans/Web Authentication2017-10-04T21:39:11Z<p>Mwobensmith: Language improvements</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication is the proposed W3C standard for creating an interface to validate a local, crytographically-signed message, with the purpose of enhancing a user's login credentials. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, in addition to typical methods, such as a password.<br />
<br />
The browser is the broker between a web site and the USB device. The site implements the feature in JavaScript, which is the focus of the W3C spec. Firefox also implements a new USB interface for interacting with these hardware tokens, which is tangential to our implementation of the W3C Web Authentication spec.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most bugs and problems encountered by everyday Firefox users.<br />
<br />
Currently, this feature is slated for Firefox 58, but this dependent on the status of the W3C spec. Regardless, the vast majority of this feature's test requirements is captured here and will not change. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is expected that the suite of manual test cases is included in our QA team's build certification passes.<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181825Security/QA/TestPlans/Web Authentication2017-10-04T21:36:28Z<p>Mwobensmith: Intro</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Web Authentication is the proposed W3C standard for creating an interface to validate a local, crytographically-signed message, with the purpose of enhancing a user's login credentials. <br />
<br />
What this means in simple language - for Firefox - is the ability for a user to employ a USB token during a login process as another factor of authentication, on top of typical methods, such as a password.<br />
<br />
The browser is the broker between a supporting web site and the USB device. The site implements the feature in JavaScript, which is the focus of the W3C spec. Firefox also implements a new USB interface for interacting with these hardware tokens, which is tangential to our implementation.<br />
<br />
We are interested in testing both JS API and USB support. In addition, we are most concerned with integration scenarios, which often surface the most bugs and problems encountered by everyday Firefox users.<br />
<br />
Currently, this feature is slated for Firefox 58, but that could change depending on the status of the W3C spec. Regardless, the vast majority of this feature's test requirements is captured here. <br />
<br />
The goal set forth in this document is to outline a test strategy that will be implemented up until the feature has been shipped in a major release of Firefox. At that point, it is hoped that our QA team can run the suite of manual test cases as part of their build certification passes.<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181738Security/QA/TestPlans/Web Authentication2017-10-03T18:09:32Z<p>Mwobensmith: Terminology error</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Unlikely || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181737Security/QA/TestPlans/Web Authentication2017-10-03T18:08:57Z<p>Mwobensmith: Fixed error</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Low || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181379Security/QA/TestPlans/Web Authentication2017-09-27T22:03:51Z<p>Mwobensmith: Added status</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
* Feature landed, turned off, in Nightly 57 on 15-09-17<br />
* Feature will target Fx58/Fx59.<br />
<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181378Security/QA/TestPlans/Web Authentication2017-09-27T21:58:21Z<p>Mwobensmith: Updated date</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | 2017-09-19 || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181377Security/QA/TestPlans/Web Authentication2017-09-27T21:55:40Z<p>Mwobensmith: Edit to pref names</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
Optional: to use unsupported soft token, set to true:<br />
<br />
security.webauth.webauthn_enable_softtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181376Security/QA/TestPlans/Web Authentication2017-09-27T21:53:46Z<p>Mwobensmith: Updated list of items</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as some U2F.<br />
* All JS APIs.<br />
* Fuzzing wherever possible.<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases.<br />
* Some USB hardware, including Yubico keys and a few others given to us.<br />
<br />
<br />
=== Out of Scope ===<br />
* Software token is unsupported, for now.<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. <br />
* Other hardware vendors will need to certify their products on Firefox, as we cannot guarantee coverage on all third party USB tokens.<br />
* This feature is not currently supported on Fennec.<br />
* We will not be shipping U2F on by default, therefore it will not be receiving the full set of tests that WebAuthN has. If that changes, we can easily apply existing WebAuthN test cases to U2F.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181375Security/QA/TestPlans/Web Authentication2017-09-27T21:48:37Z<p>Mwobensmith: /* Ownership */</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* Adam Powers (FIDO) is creating tests for the [https://github.com/w3c/web-platform-tests/tree/master/webauthn web-platform-test suite]<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review of both JS API and Rust USB library<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession.<br />
* This feature is not currently supported on Fennec.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181374Security/QA/TestPlans/Web Authentication2017-09-27T21:45:22Z<p>Mwobensmith: Corrected bug formatting</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession.<br />
* This feature is not currently supported on Fennec.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406,1398268,1399298,1399669,1400940,1401019,1401802,1401803,1402114,1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1181373Security/QA/TestPlans/Web Authentication2017-09-27T21:39:16Z<p>Mwobensmith: Added more bugs</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession.<br />
* This feature is not currently supported on Fennec.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<bugzilla><br />
{<br />
"id":[1398268],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<bugzilla><br />
{<br />
"id":[1399298],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<bugzilla><br />
{<br />
"id":[1399669],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<bugzilla><br />
{<br />
"id":[1400940],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<bugzilla><br />
{<br />
"id":[1401019],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<bugzilla><br />
{<br />
"id":[1401802],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<bugzilla><br />
{<br />
"id":[1401803],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<bugzilla><br />
{<br />
"id":[1402114],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<bugzilla><br />
{<br />
"id":[1403330],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1180145Security/QA/TestPlans/Web Authentication2017-09-08T22:10:03Z<p>Mwobensmith: Fixed bug metadata</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession.<br />
* This feature is not currently supported on Fennec.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"id":[1395406],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<bugzilla><br />
{<br />
"id":[1398268],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1180144Security/QA/TestPlans/Web Authentication2017-09-08T22:08:41Z<p>Mwobensmith: Added bug</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession.<br />
* This feature is not currently supported on Fennec.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<bugzilla><br />
{<br />
"blocks":[1398268],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1180143Security/QA/TestPlans/Web Authentication2017-09-08T22:07:06Z<p>Mwobensmith: Added bug</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession.<br />
* This feature is not currently supported on Fennec.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
},<br />
{<br />
"blocks":[1398268],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1179991Security/QA/TestPlans/Web Authentication2017-09-06T22:43:05Z<p>Mwobensmith: Minor update</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession.<br />
* This feature is not currently supported on Fennec.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | first draft complete || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | complete || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1179990Security/QA/TestPlans/Web Authentication2017-09-06T22:41:09Z<p>Mwobensmith: /* Bug Work */</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession.<br />
* This feature is not currently supported on Fennec.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 1294514] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1179989Security/QA/TestPlans/Web Authentication2017-09-06T22:39:50Z<p>Mwobensmith: Added first bug</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession.<br />
* This feature is not currently supported on Fennec.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1395406],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1179920Security/QA/TestPlans/Web Authentication2017-09-05T21:05:22Z<p>Mwobensmith: Added sec assessment</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession.<br />
* This feature is not currently supported on Fennec.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
* Product Integrity Security Assessment [https://docs.google.com/document/d/1lTm2OD_GtLln608vOzU_mODqqVjO38DPYiEMkGkgy4s link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1383799],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1179651Security/QA/TestPlans/Web Authentication2017-08-30T23:57:03Z<p>Mwobensmith: Fennec edit</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession.<br />
* This feature is not currently supported on Fennec.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports on desktop only.<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1383799],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1179649Security/QA/TestPlans/Web Authentication2017-08-30T23:56:30Z<p>Mwobensmith: Fennec edit</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession.<br />
* This feature is not currently supported on Fennec.<br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports. <br />
<br />
* TBD: What is the behavior on Fennec? <br />
<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1383799],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1179561Security/QA/TestPlans/Web Authentication2017-08-29T22:08:58Z<p>Mwobensmith: /* Test Areas */</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession. <br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports. <br />
<br />
* TBD: What is the behavior on Fennec? <br />
<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || [https://testrail.stage.mozaws.net/index.php?/cases/view/64706&group_by=cases:section_id&group_order=asc&group_id=5844 Test case]<br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1383799],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1179560Security/QA/TestPlans/Web Authentication2017-08-29T22:03:05Z<p>Mwobensmith: Minor update</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession. <br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports. <br />
<br />
* TBD: What is the behavior on Fennec? <br />
<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* Web Authentication W3C [https://www.w3.org/TR/webauthn spec]<br />
<br />
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294514 link]<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1383799],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1179559Security/QA/TestPlans/Web Authentication2017-08-29T22:00:15Z<p>Mwobensmith: /* Testing Tools */</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession. <br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports. <br />
<br />
* TBD: What is the behavior on Fennec? <br />
<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/projects/overview/49 TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* List and links for specs<br />
List and links for available specs - documents, user stories, specifications<br />
* Meta bug<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1383799],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1179558Security/QA/TestPlans/Web Authentication2017-08-29T21:58:56Z<p>Mwobensmith: /* Test Execution Schedule */</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession. <br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports. <br />
<br />
* TBD: What is the behavior on Fennec? <br />
<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | 2017-08-01 || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* List and links for specs<br />
List and links for available specs - documents, user stories, specifications<br />
* Meta bug<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1383799],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1178663Security/QA/TestPlans/Web Authentication2017-08-17T23:19:43Z<p>Mwobensmith: /* Logged bugs ( blocking 12345 ) */</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession. <br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports. <br />
<br />
* TBD: What is the behavior on Fennec? <br />
<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* List and links for specs<br />
List and links for available specs - documents, user stories, specifications<br />
* Meta bug<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[1383799],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1178662Security/QA/TestPlans/Web Authentication2017-08-17T23:18:30Z<p>Mwobensmith: Edits</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession. <br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports. <br />
<br />
* TBD: What is the behavior on Fennec? <br />
<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* List and links for specs<br />
List and links for available specs - documents, user stories, specifications<br />
* Meta bug<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || n/a <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no || n/a<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no || n/a<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, yes, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || In-house security review, yes<br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1178660Security/QA/TestPlans/Web Authentication2017-08-17T23:05:08Z<p>Mwobensmith: Many changes</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession. <br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports. <br />
<br />
* TBD: What is the behavior on Fennec? <br />
<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* List and links for specs<br />
List and links for available specs - documents, user stories, specifications<br />
* Meta bug<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | yes || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | no || <br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | no || <br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | no || <br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | no || <br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | no || e.g. with NVDA<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | no || Is this feature user friendly<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | no || <br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | no || <br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | no || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | no || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | no || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | yes || If provided by third parties, otherwise no<br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | no ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | no ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | no || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | no || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | no || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | no || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | no || <br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | yes || QA + PI security review<br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | yes || Engineering + PI fuzzing team<br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | yes || Sample sites are available<br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | yes || If we support U2F, we can try to find U2F-enabled sites<br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | yes || This is inherent in the feature, w/r/t hardware keys<br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | yes || Fennec and Focus support TBD<br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | yes || Largest area of targeted testing by QA<br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1178659Security/QA/TestPlans/Web Authentication2017-08-17T22:46:10Z<p>Mwobensmith: ratings</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession. <br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports. <br />
<br />
* TBD: What is the behavior on Fennec? <br />
<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 1-Almost Certain || 1-Low || 3<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 1-Low || 4<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 3-High || 2-Possible || 2-Moderate || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 3-High || 2-Possible || 3-High || 18<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 3-Moderate || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 3-High || 1-Unlikely || 2-Moderate || 6<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 1-Unlikely || 1-Low || 2<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* List and links for specs<br />
List and links for available specs - documents, user stories, specifications<br />
* Meta bug<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | || <br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | || <br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | || <br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | ||<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | || <br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | || <br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | || e.g. with NVDA<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | || Is this feature user friendly<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | ||<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.<br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | || Make sure support documents are written and are correct.<br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | ||<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | || <br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)<br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | || yes/no options, add comment with details about who will lead testing<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | || List of error conditions to monitor<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | || <br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | || <br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | || <br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | || <br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | || <br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | || <br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | || <br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1178658Security/QA/TestPlans/Web Authentication2017-08-17T22:37:22Z<p>Mwobensmith: OK</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession. <br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports. <br />
<br />
* TBD: What is the behavior on Fennec? <br />
<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Incorrect authentication allows security bypass || TO-1, TO-2, TO-3 || 2-Moderate || 1-Unlikely || 3-High || 6<br />
|-<br />
| RAC-2 || XSS/information leak || TO-1, TO-3 || 3-High || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-3 || Confined to secure context || TO-1, TO-3 || 2-Moderate || 2-Possible || 3-High || 12<br />
|-<br />
| RAC-4 || Incorrectly functioning JS API || TO-1 || 2-Moderate || 2-Possible || 3-High || 12<br />
|-<br />
| RAC-5 || Stability for entire feature || TO-1, TO-2 || 2-Moderate || 2-Possible || 3-High || 12<br />
|-<br />
| RAC-6 || Interaction with other aspects of normal Firefox usage || TO-1, TO-2 || 2-Moderate || 2-Possible || 3-High || 12<br />
|-<br />
| RAC-7 || Memory issues in JS API and hardware support code || TO-3 || 2-Moderate || 2-Possible || 3-High || 12<br />
|-<br />
| RAC-8 || Incorrectly functioning hardware || TO-2 || 2-Moderate || 2-Possible || 3-High || 12<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* List and links for specs<br />
List and links for available specs - documents, user stories, specifications<br />
* Meta bug<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | || <br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | || <br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | || <br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | ||<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | || <br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | || <br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | || e.g. with NVDA<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | || Is this feature user friendly<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | ||<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.<br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | || Make sure support documents are written and are correct.<br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | ||<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | || <br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)<br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | || yes/no options, add comment with details about who will lead testing<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | || List of error conditions to monitor<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | || <br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | || <br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | || <br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | || <br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | || <br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | || <br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | || <br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1178657Security/QA/TestPlans/Web Authentication2017-08-17T22:33:09Z<p>Mwobensmith: edit</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession. <br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports. <br />
<br />
* TBD: What is the behavior on Fennec? <br />
<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Risk description 1 || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6<br />
|-<br />
| RAC-2 || Risk description 2 || TO-1 || 3-High || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-3 || Risk description 3 || TO-2 || 2-Moderate || 2-Possible || 3-High || 12<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3, RAC-4, RAC-5, RAC-6 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-5, RAC-6, RAC-8 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3, RAC-7 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* List and links for specs<br />
List and links for available specs - documents, user stories, specifications<br />
* Meta bug<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | || <br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | || <br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | || <br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | ||<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | || <br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | || <br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | || e.g. with NVDA<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | || Is this feature user friendly<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | ||<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.<br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | || Make sure support documents are written and are correct.<br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | ||<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | || <br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)<br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | || yes/no options, add comment with details about who will lead testing<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | || List of error conditions to monitor<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | || <br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | || <br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | || <br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | || <br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | || <br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | || <br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | || <br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1178656Security/QA/TestPlans/Web Authentication2017-08-17T22:27:19Z<p>Mwobensmith: Edit</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession. <br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports. <br />
<br />
* TBD: What is the behavior on Fennec? <br />
<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Risk description 1 || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6<br />
|-<br />
| RAC-2 || Risk description 2 || TO-1 || 3-High || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-3 || Risk description 3 || TO-2 || 2-Moderate || 2-Possible || 3-High || 12<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| TO1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3 || Eng Team, QA<br />
|-<br />
| TO2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3 || Eng Team, QA<br />
|-<br />
| TO3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* List and links for specs<br />
List and links for available specs - documents, user stories, specifications<br />
* Meta bug<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | || <br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | || <br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | || <br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | ||<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | || <br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | || <br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | || e.g. with NVDA<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | || Is this feature user friendly<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | ||<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.<br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | || Make sure support documents are written and are correct.<br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | ||<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | || <br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)<br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | || yes/no options, add comment with details about who will lead testing<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | || List of error conditions to monitor<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | || <br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | || <br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | || <br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | || <br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | || <br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | || <br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | || <br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmithhttps://wiki.mozilla.org/index.php?title=Security/QA/TestPlans/Web_Authentication&diff=1178655Security/QA/TestPlans/Web Authentication2017-08-17T22:15:24Z<p>Mwobensmith: More objectives</p>
<hr />
<div>'''Approvals Required / Received'''<br />
<br />
The following individuals are required to/have approved this Test Plan:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Name !! Title !! Department !! Approval Date !! Method<br />
|-<br />
| || QA Manager || Product Integrity || Date || Email<br />
|-<br />
| JC Jones || Software Engineer || Engineering || Date || Email<br />
|-<br />
| || EPM || Product Management || Date || Email<br />
|}<br />
<br />
<br />
'''Revision History'''<br />
<br />
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.<br />
<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Date !! Version !! Author !! Description <br />
|-<br />
| 2017-08-16 || 1.0 || Matt Wobensmith || Created first draft<br />
|}<br />
<br />
= Overview =<br />
== Purpose ==<br />
Detail the purpose of this document. For example:<br />
* The test scope, focus areas and objectives<br />
* The test responsibilities<br />
* The test strategy for the levels and types of test for this release<br />
* The entry and exit criteria<br />
* The basis of the test estimates<br />
* Any risks, issues, assumptions and test dependencies<br />
* The test schedule and major milestones<br />
* The test deliverables<br />
<br />
== Scope ==<br />
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:<br />
* What will be tested<br />
* How testing will be performed<br />
<br />
== Ownership ==<br />
This feature is being tested by both Mozilla and one or more third parties.<br />
* Yubico is performing smoke tests using hardware keys across a range of hardware and software<br />
* JC Jones and Tim Taubert have created unit tests for both JS API and hardware interaction<br />
* The Fuzzing team has been enlisted, initially to test USB interaction, time frame unknown<br />
* The PI Security team has been requested to perform a security review between now and mid-September 2017.<br />
* Matt Wobensmith (QA) is responsible for the entire process, as well as creating manual scenario tests<br />
* Mozilla's QA - most likely SoftVision - will use the manual tests for ongoing build certification post-feature-signoff <br />
<br />
<br />
= Testing summary = <br />
== Scope of Testing ==<br />
=== In Scope ===<br />
* Web Authentication, as well as U2F (both soft token and hardware) if we decide to ship it<br />
* All JS APIs<br />
* Fuzzing wherever possible<br />
* A range of scenario tests that mirror user interaction, including boundary and error cases<br />
<br />
<br />
=== Out of Scope ===<br />
* Yubico has provided us with some USB keys to test with, but the full range of keys plus hardware is not something we have available to us. We are relying on their help but will not be able to replicate their coverage, and will run passes using existing hardware in our possession. <br />
<br />
= Requirements for testing =<br />
== Environments ==<br />
We support the same OS and hardware configurations that Firefox supports. <br />
<br />
* TBD: What is the behavior on Fennec? <br />
<br />
<br />
== Channel dependent settings (configs) and environment setups ==<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
<br />
The feature is controlled by prefs that are gated to channels at the moment. To control this feature, set the following prefs to true:<br />
<br />
security.webauth.u2f;<br />
security.webauth.u2f_enable_softtoken;<br />
security.webauth.u2f_enable_usbtoken;<br />
security.webauth.webauthn;<br />
security.webauth.webauthn_enable_softtoken;<br />
security.webauth.webauthn_enable_usbtoken;<br />
<br />
=== Nightly ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Beta ===<br />
<div class="mw-collapsible-content"><br />
Currently set to false.<br />
</div><br />
<br />
=== Post Beta / Release ===<br />
<div class="mw-collapsible-content"><br />
Depending on ship decisions, will be set to true.<br />
</div><br />
</div><br />
<br />
= Test Strategy = <br />
== Risk Assessment and Coverage ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score <br />
|-<br />
| RAC-1 || Risk description 1 || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6<br />
|-<br />
| RAC-2 || Risk description 2 || TO-1 || 3-High || 3-Almost Certain || 3-High || 27<br />
|-<br />
| RAC-3 || Risk description 3 || TO-2 || 2-Moderate || 2-Possible || 3-High || 12<br />
|}<br />
<br />
'''Values:'''<br />
<br />
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' <br />
<br />
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''<br />
<br />
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''<br />
<br />
'''Impact Score Breakdown:''' <br />
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.<br />
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.<br />
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.<br />
<br />
== Test Objectives ==<br />
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.<br />
This could be documented in bullet form or in a table similar to the one below.<br />
<br />
{| class="wikitable"<br />
|-<br />
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners <br />
|-<br />
| 1 || JS API || Verify functionality || All tests indicate stable, functional API for using Web Authentication and/or U2F with both hardware and software tokens || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3 || Eng Team, QA<br />
|-<br />
| 2 || Hardware support via USB token || Verify functionality || All tests indicate stable, functional support of USB hardware keys, as above || Manual/ Automation / Usability || RAC-1, RAC-2, RAC-3 || Eng Team, QA<br />
|-<br />
| 3 || Stable, secure code || Fuzzing and security review || All testing and inspection surfaces known security issues || Manual/ Security || RAC-1, RAC-2, RAC-3 || Eng Team, QA, PI Fuzzing + Sec Review<br />
|}<br />
<br />
== Builds ==<br />
This section should contain links for builds with the feature - <br />
* Links for Nightly builds<br />
* Links for Beta builds<br />
<br />
== Test Execution Schedule ==<br />
The following table identifies the anticipated testing period available for test execution.<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Project phase !! Start Date !! End Date<br />
|-<br />
| Start project <br />
|style="text-align:center;" | || <br />
|-<br />
| Study documentation/specs received from developers<br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Test plan creation <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Test cases/Env preparation <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Nightly Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA - Beta Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Release Date <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Testing Tools ==<br />
Detail the tools to be used for testing, for example see the following table:<br />
{| class="wikitable" style="width:50%"<br />
|-<br />
! Process !! Tool<br />
|-<br />
| Test plan creation || Mozilla wiki<br />
|-<br />
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs<br />
|-<br />
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]<br />
|-<br />
| Bugs management || Bugzilla<br />
|}<br />
<br />
= Status = <br />
== Overview ==<br />
Track the dates and build number where feature was released to Nightly<br />
Track the dates and build number where feature was merged to Release/Beta<br />
<br />
<br />
= References =<br />
* List and links for specs<br />
List and links for available specs - documents, user stories, specifications<br />
* Meta bug<br />
<br />
= Testcases = <br />
== Test Areas ==<br />
{| class="wikitable" style="width:80%"<br />
|-<br />
! Test Areas !! Covered !! Details<br />
|-<br />
| Private Window <br />
|style="text-align:center;" | || <br />
|-<br />
| Multi-Process Enabled <br />
|style="text-align:center;" | || <br />
|-<br />
| Multi-process Disabled <br />
|style="text-align:center;" | || <br />
|-<br />
| Theme (high contrast) <br />
|style="text-align:center;" | || <br />
|-<br />
| '''UI''' <br />
|| || <br />
|-<br />
| Mouse-only operation <br />
|style="text-align:center;" | || <br />
|-<br />
| Keyboard-only operation <br />
|style="text-align:center;" | ||<br />
|-<br />
| Display (HiDPI) <br />
|style="text-align:center;" | || <br />
|-<br />
| Interaction (scroll, zoom) <br />
|style="text-align:center;" | || <br />
|-<br />
| Usable with a screen reader <br />
|style="text-align:center;" | || e.g. with NVDA<br />
|-<br />
| Usability and/or discoverability testing <br />
|style="text-align:center;" | || Is this feature user friendly<br />
|-<br />
| RTL build testing <br />
|style="text-align:center;" | ||<br />
|-<br />
| '''Help/Support''' <br />
|| || <br />
|-<br />
| Help/support interface required <br />
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.<br />
|-<br />
| Support documents planned(written) <br />
|style="text-align:center;" | || Make sure support documents are written and are correct.<br />
<br />
|-<br />
| '''Install/Upgrade''' <br />
|| || <br />
|-<br />
| Feature upgrades/downgrades data as expected <br />
|style="text-align:center;" | ||<br />
|-<br />
| Does sync work across upgrades <br />
|style="text-align:center;" | || <br />
|-<br />
| Requires install testing <br />
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)<br />
|-<br />
| Affects first-run or onboarding <br />
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.<br />
|-<br />
| Does this affect partner builds? Partner build testing <br />
|style="text-align:center;" | || yes/no options, add comment with details about who will lead testing<br />
<br />
|-<br />
| ''' Enterprise ''' <br />
|| || Raise up the topic to developers to see if they are expecting to work different on ESR builds<br />
|-<br />
| Enterprise administration <br />
|style="text-align:center;" | || <br />
|-<br />
| Network proxies/autoconfig <br />
|style="text-align:center;" | || <br />
|-<br />
| ESR behavior changes <br />
|style="text-align:center;" | || <br />
|-<br />
| Locked preferences <br />
|style="text-align:center;" | ||<br />
<br />
|-<br />
| ''' Data Monitoring ''' <br />
|| || <br />
|-<br />
| Temporary or permanent telemetry monitoring <br />
|style="text-align:center;" | || List of error conditions to monitor<br />
|-<br />
| Telemetry correctness testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Server integration testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Offline and server failure testing <br />
|style="text-align:center;" | ||<br />
|-<br />
| Load testing <br />
|style="text-align:center;" | ||<br />
<br />
|-<br />
| ''' Add-ons ''' <br />
|| || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.<br />
|-<br />
| Addon API required? <br />
|style="text-align:center;" | || <br />
|-<br />
| Comprehensive API testing <br />
|style="text-align:center;" | || <br />
|-<br />
| Permissions <br />
|style="text-align:center;" | || <br />
|-<br />
| Testing with existing/popular addons<br />
|style="text-align:center;" | || <br />
<br />
|-<br />
| ''' Security ''' <br />
|| || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.<br />
|-<br />
| 3rd-party security review <br />
|style="text-align:center;" | || <br />
|-<br />
| Privilege escalation testing<br />
|style="text-align:center;" | || <br />
|-<br />
| Fuzzing <br />
|style="text-align:center;" | || <br />
<br />
|-<br />
| ''' Web Compatibility ''' <br />
|| || depends on the feature<br />
|-<br />
| Testing against target sites <br />
|style="text-align:center;" | || <br />
|-<br />
| Survey of many sites for compatibility <br />
|style="text-align:center;" | || <br />
<br />
|-<br />
| ''' Interoperability ''' <br />
|| || depends on the feature<br />
|-<br />
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. <br />
|style="text-align:center;" | || <br />
|-<br />
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS <br />
|style="text-align:center;" | || <br />
|-<br />
| Interaction of this feature with other browser features <br />
|style="text-align:center;" | || <br />
|}<br />
<br />
== Test suite ==<br />
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]<br />
Smoke Test suite - Link with the tests - if available/needed.<br />
Regression Test suite - Link with the tests - if available/needed.<br />
<br />
= Bug Work =<br />
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======<br />
<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"<br />
}<br />
</bugzilla><br />
<br />
</div><br />
</div><br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"><br />
====== Bug fix verification ======<br />
<div class="mw-collapsible-content"><br />
<bugzilla><br />
{<br />
"blocks":[12345],<br />
"resolution":"FIXED",<br />
"include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"<br />
}<br />
</bugzilla><br />
</div><br />
</div><br />
<br />
= Sign off =<br />
== Criteria ==<br />
Checklist<br />
* All test cases should be executed<br />
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan<br />
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)<br />
<br />
== Results ==<br />
'''Nightly testing'''<br /><br />
<br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]<br />
** Daily Smoke, if needed/available<br />
** Regression Test suite, if needed/available<br />
<br /><br />
<br />
'''Merge to Beta Sign-off'''<br /><br />
List of OSes that will be covered by testing<br /><br />
*Link for the tests run<br />
** Full Test suite<br />
<br />
== Checklist ==<br />
{| class="wikitable" style="width:60%"<br />
|-<br />
! Exit Criteria !! Status !! Notes/Details<br />
|-<br />
| Testing Prerequisites (specs, use cases) <br />
| style="text-align:center;" | <br />
| style="text-align:center;" | <br />
|-<br />
| Testing Infrastructure setup <br />
|style="text-align:center;" | || <br />
|-<br />
| Test Plan Creation <br />
| style="text-align:center;" | || <br />
|-<br />
| Test Cases Creation <br />
|style="text-align:center;" | || <br />
|-<br />
| Automation Coverage ||<br />
|style="text-align:center;" | <br />
|-<br />
| Performance Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| All Defects Logged || || <br />
|-<br />
| Critical/Blockers Fixed and Verified || || <br />
|-<br />
| Metrics/Telemetry|| <br />
|style="text-align:center;" | <br />
|-<br />
| Basic/Core functionality Nightly testing<br />
|style="text-align:center;" | <br />
|style="text-align:center;" | <br />
|-<br />
| QA mid-Nightly Signoff|| <br />
|style="text-align:center;" | Email to be sent <br />
|-<br />
| QA Nightly - Full Testing <br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Beta Signoff|| <br />
|style="text-align:center;"| Email to be sent <br />
|-<br />
| QA Beta - Full Testing<br />
|style="text-align:center;" | || <br />
|-<br />
| QA pre-Release Signoff || <br />
|style="text-align:center;" | Email to be sent <br />
|}</div>Mwobensmith