MozillaWiki

QA/file read access: Difference between revisions

< QA
 
(16 intermediate revisions by 3 users not shown)
Line 1: Line 1:
'''Approvals Required / Received'''
The following individuals are required to/have approved this Test Plan:
{| class="wikitable"
|-
! Name !! Title !! Department !! Approval Date !! Method
|-
|  || QA Manager || Product Integrity || Date || Email
|-
|  || Software Engineer || Engineering || Date || Email
|-
|  || EPM || Product Management || Date || Email
|}
'''Revision History'''
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded).  The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
{| class="wikitable" style="width:60%"
|-
! Date !! Version !! Author !! Description
|-
|style="text-align:center;" |24/05/2017 || style="text-align:center;" | 1.1 || style="text-align:center;"| Mihai Boldan ||style="text-align:center;"| Updated the Test Plan
|-
|style="text-align:center;" |13/06/2017 || style="text-align:center;" | 1.2 || style="text-align:center;"| Mihai Boldan ||style="text-align:center;"| Updated the Test Plan
|-
|style="text-align:center;" |10/07/2017 || style="text-align:center;" | 1.3 || style="text-align:center;"| Mihai Boldan ||style="text-align:center;"| mid-Nightly Sign off(YELLOW)
|-
|style="text-align:center;" |31/07/2017 || style="text-align:center;" | 1.4 || style="text-align:center;"| Mihai Boldan ||style="text-align:center;"| pre-Beta Sign off(YELLOW)
|}
= Overview =
= Overview =
== Purpose ==
== Purpose ==
This page details the testing that will be performed by QA file read access restrictions sand boxing functionality. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
This page details the testing that will be performed by Manual QA for the File read access restrictions sand boxing functionality. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
* Areas of risk
* Areas of risk
* What will be tested
* What will be tested
Line 12: Line 45:


To help ensure the best possible release and long term maintenance of this feature.
To help ensure the best possible release and long term maintenance of this feature.
== Scope ==
This wiki details the testing that will be performed by the project team for the File Read Access Restrictions project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
* What will be tested
** Focus should be on the following:
*** Printing
*** Print to file
*** Clipboard interactions
*** file uploads to sites like google docs and photo sites
*** drag and drop of files into content
*** content file/page saves
*** add-ons:
**** WebExtensions
***** Popular WebExtensions still work
***** WebExtensions-developer use cases loading from home directory (i.e, about:debugging then load both packed and unpacked extensions)   
**** Legacy
*** about:* pages are functional and links within access correctly
*** Flash (file picker)
*** Running browser outside of home directory, i.e., not installing it into proper location such as /Applications (on Mac), or Program Files (Windows)
* How testing will be performed
** Manual
== Ownership ==
Project Manager:
[mailto:jmathies@mozilla.com Jim Mathies] (irc: jimm)
Development:
[mailto:bowen@mozilla.com Bob Owen] (irc: TBD)
[mailto:haftandilian@mozilla.com Haik Aftandilian] (irc: Haik)
Platform QA Team:
[mailto:twalker@mozilla.com Tracy Walker] (irc: TBD)
[mailto:mihai.boldan@softvision.ro Mihai Boldan] (irc: mboldan)
[mailto:bogdan.maris@softvisioninc.eu Bogdan Maris] (irc: bogdan_maris)
[mailto:roxana.leitan@softvision.ro Roxana Leitan] (irc: RoxanaLeitan)
[mailto:ovidiu.boca@softvision.ro Ovidiu Boca] (irc: Ovidiu)
= Testing summary =
== Scope of Testing ==
=== In Scope ===
This section describes what is in scope for manual testing.
* Testing on most used OSs: Windows7x64, Windows10x64, Ubuntu16.04x64 and Mac 10.12
* Testing on different build structures:
** Window 7x64 - Firefox build on 32-bit
** Windows 10x64 - Firefox build on 64-bit
** Mac OS X 10.12 - Firefox build on 64-bit
** Ubuntu 16.04 - Firefox build on 64-bit
* Testing areas:
** Printing
** File uploading
** Drag and Drop
** Add-ons
** Flash content
** Accessibility
** Virtual keyboard support
** Content file/page saves
** about:* pages
** Interoperability
** Downloads
* Test on different environments
=== Out of Scope ===
This section describes what is out of scope for manual testing.
* Testing on a high hardware variety
** Separate test machines were configured for this feature
** Only AMD 760G and ATI Radeon 3000 graphic cards available
* Testing on multiple Linux distribution systems
** Testing will be performed only on Ubuntu OS
= Requirements for testing =
== Environments ==
Full Testing will be performed using Beta build for Desktop with focus on platform specific issues as follows
* Windows
** Roaming windows user
** Running from a network drive
** Profile on network drive
** Profile on symbolic link or junction point
** Windows Users directory on junction point
* Mac OS X
** Profile in symlinked dir
** ~ (home) in symlinked dir
* Linux
** Profile in symlinked dir
** .config in symlinked dir
** ~ (home) in symlinked dir
Note: Read access restrictions on each platform will be slightly different.  When creating test cases, any known differences should be called out in the test case.
== Channel dependent settings (configs) and environment setups ==
<div class="toccolours" style="width:auto">
* The security.sandbox.content.level pref must be set to 3. This is applicable for the three phases of Firefox (Nightly, Beta and Release).
** The pref is set to 3 by default from 56.0a1(22-07-2017)
</div>
= Test Strategy =
== Risk Assessment and Coverage ==
{| class="wikitable"
|-
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score
|-
| RAC-1 || Installing Firefox on a specific environment || Test Cases || 3-High || 2-Possible || 3-High || 18
|-
| RAC-2 || Printing different file types from a specific environment || Test Cases || 2-Moderate || 2-Possible || 2-Moderate || 12
|-
| RAC-3 || Install and run Legacy add-ons || Test Cases || 2-Moderate, || 1-Unlikely || 1 - Low || 2
|-
| RAC-4 || Install and run Web extensions || Test Cases || 3-High || 3-Almost Certain || 3-High || 27
|-
| RAC-5 || Uploading files on different websites || Test Cases || 2-Moderate || 2-Possible || 3-High || 12
|-
| RAC-6 || Downloading different file types, using a specific environment || Test Cases || 2-Moderate || 2-Possible || 3-High || 12
|}
'''Values:'''
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''
'''Impact Score Breakdown:'''
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.


== Risk Analysis ==
== Risk Analysis ==
Line 35: Line 200:
|}
|}


= Testing Strategy =
== Test Objectives ==
== Scope ==
{| class="wikitable" style="width:95%"
|-
! Criteria Description
! Metric
! Test results
! Criteria Met?
! QA Owner
|-
| Manual testing
| Tests passed
| -link to manual test results-
| style="background-color:#CEF2F2;" | TBD (Date status updated)
| Eng + Rel SV teams 
|-
| Unit testing
| Automated tests pass
| [https://treeherder.mozilla.org/perf.html#/graphs perfherder]
| style="background-color:#CEF2F2;" | TBD (Date status updated)
| style="background-color:#CEF2F2;" | TBD
|-
| Security
| Security approves via review
| -Link to security Analysis-
| style="background-color:#CEF2F2;" | TBD (Date status updated)
| style="background-color:#CEF2F2;" | TBD
|-
| Stability
| Crash rate stable
| -link to Stability Analysis-
| style="background-color:#CEF2F2;" | TBD (Date status updated)
| style="background-color:#CEF2F2;" | TBD
|}
 
 
== Builds ==
This section should contain links for builds with the feature -
* [https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ Links for Nightly builds]
* Links for Beta builds - n/a
 
== Test Execution Schedule ==
The following table identifies the anticipated testing period available for test execution.
{| class="wikitable" style="width:60%"
|-
! Project phase !! Start Date !! End Date
|-
| Start project
|style="text-align:center;" | Q1/2016 ||style="text-align:center;"| Q2/2017
|-
| QA - Test plan creation
|style="text-align:center;" | Q2/2017 ||style="text-align:center;"| Q2/2017
|-
| QA - Test cases/Env preparation
|style="text-align:center;" | 22/05/2017 ||style="text-align:center;"| Q2/2017
|-
| QA - Nightly Testing
|style="text-align:center;" | Q2/2017 ||style="text-align:center;"| Q2/2017
|-
| QA - Beta Testing
|style="text-align:center;" | Q2/2017 ||style="text-align:center;"| Q3/2017
|-
| Release Date
|style="text-align:center;" | FF56 ||style="text-align:center;"| 2017-09-26
|}
 
== Testing Tools ==
Detail the tools to be used for testing, for example see the following table:
{| class="wikitable" style="width:50%"
|-
! Process !! Tool
|-
| Test plan creation || Mozilla wiki
|-
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/999&group_by=cases:section_id&group_order=asc TestRail]/ [https://drive.google.com/open?id=0B-cUbHVNThEKdTIxdjhZeVA0VTA Google docs]
|-
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/reports/view/459 on Windows], [https://testrail.stage.mozaws.net/index.php?/reports/view/458 on Mac OS X]
|-
| Bugs management || Bugzilla
|}
 
= Status =
== Overview ==
Track the dates and build number where feature was released to Nightly - security.sandbox.content.level pref set to 3 by default from 56.0a1(22-07-2017)
Track the dates and build number where feature was merged to Release/Beta - TBD
 
= References =
* List and links for specs
  List and links for available specs - documents, user stories, specifications
* Meta bug
= Testcases =  
== Test Areas ==
{| class="wikitable" style="width:80%"
{| class="wikitable" style="width:80%"
|-
|-
Line 45: Line 301:
|-
|-
| Printing  
| Printing  
|style="text-align:center;" | Yes ||
|-
| Clipboard
|style="text-align:center;" | Yes ||  
|style="text-align:center;" | Yes ||  
|-
|-
Line 116: Line 375:
If it's not listed above, it is currently out of scope.
If it's not listed above, it is currently out of scope.


== Objectives ==
== Test suite ==
{| class="wikitable" style="width:95%"
Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/999&group_by=cases:section_id&group_order=asc Test Rail]
Smoke Test suite - TBD
Regression Test suite - TBD
 
= Bug Work =
Meta bug: TBD
 
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Logged bugs ======
 
{| class="wikitable" style="width:80%"
|-
! ID !! Priority !! Component !! Assigned to !! Summary !! Status !! Target milestone
|-
|-
! Criteria Description
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1368690 1368690]
! Metric
| || || || || ||
! Test results
! Criteria Met?
! QA Owner
|-
|-
| Manual testing
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1369670 1369670]
| Tests passed
|| || Security:Process Sandboxing ||Bob Owen||Blank pages are printed with security.sandbox.content.level set to 3 when Users folder is a junction point||VERIFIED FIXED||  
| -link to manual test results-
| style="background-color:#CEF2F2;" | TBD (Date status updated)
| style="background-color:#CEF2F2;" | TBD 
|-
|-
| Unit testing
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1378061 1378061]  
| Automated tests pass
|| || Security:Process Sandboxing ||Bob Owen||Only set user's SID in USER_LIMITED as deny only when not using restricting SIDs.||RESOLVED FIXED||  
| [https://treeherder.mozilla.org/perf.html#/graphs perfherder]
| style="background-color:#CEF2F2;" | TBD (Date status updated)
| style="background-color:#CEF2F2;" | TBD
|-
|-
| Security
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1378141 1378141]
| Security approves via review
|| || Disability Access APIs ||Aaron Klotz||a11y crashes | PBrowserParent::RecvPDocAccessibleConstructor Constructing a top-level PDocAccessible with null COM ||RESOLVED FIXED||  
| -Link to security Analysis-
| style="background-color:#CEF2F2;" | TBD (Date status updated)
| style="background-color:#CEF2F2;" | TBD
|-
|-
| Stability
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1379951 1379951]
| Crash rate stable
|| || Disability Access APIs ||Nobody|| a11y stack overflows at sandbox level 3 ||REOPENED||
| -link to Stability Analysis-
| style="background-color:#CEF2F2;" | TBD (Date status updated)
| style="background-color:#CEF2F2;" | TBD
|}
|}


== Manual Testcases ==
====== Bug fix verification ======
* Focus should be on the following:
 
** Printing
** add-ons:
*** WebExtensions
    **** Popular WebExtensions still work
    **** WebExtensions-developer use cases loading from home directory (i.e, about:debugging then load both packed and unpacked extensions)   
*** Legacy
** file uploads to sites like google docs and photo sites
** about:* pages are functional and links within access correctly
** Flash (file picker)


== Environments ==
= Sign off =
Full Testing will be performed using Nightly for Desktop with focus one platform specific issues
== Criteria ==
* Windows
Checklist
** Accessibility (spot check NVDA)
* All test cases should be executed
** Virtual keyboard support
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
** Roaming windows user
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
** Running from a network drive
** Profile on network drive
** Profile on symbolic link or junction point
** Windows Users directory on junction point
*** especially printing with this
* Mac OS X
* Linux
** Profile in symlinked dir
** .config in symlinked dir
** ~ (home) in symlinked dir
** Various distros and desktop envs, Qt (with themes and GTK style engine) vs GTK
*** At the very least Fedora, Ubuntu and/or KDE or GNOME


Note: Read access restrictions on each platform will be slightly different.  When creating test cases, any known differences should be called out in the test case.
== Results ==
'''Nightly testing'''<br />


= Project Information =
List of OSes that will be covered by testing<br /> - Windows 7x64, Windows 10x64, Mac OS X 10.12
== Ownership ==
*Link for the tests run
Project Manager: [mailto:jmathies@mozilla.com Jim Mathies] <br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/plans/view/4317 on Windows] and [https://testrail.stage.mozaws.net/index.php?/plans/view/4298 on Mac OS X]
Developer contacts: [mailto:bowen@mozilla.com Bob Owen] <br />
** Daily Smoke, if needed/available
QA: [mailto:twalker@mozilla.com Tracy Walker] <br />
** Regression Test suite, if needed/available
QA Peer: TBD (SV)
<br />


== Builds ==
'''Merge to Beta Sign-off'''<br />
* [https://nightly.mozilla.org/ Links for Nightly builds]
List of OSes that will be covered by testing<br />
*Link for the tests run
** Full Test suite


== Schedule ==
== Checklist ==
The following table identifies the anticipated testing period available for test execution.
{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
|-
|-
! Project phase !! Start Date !! End Date
! Exit Criteria !! Status !! Notes/Details
|-
|  Testing Prerequisites (specs, use cases)
| style="text-align:center;" | 
| style="text-align:center;" |
|-
|  Testing Infrastructure setup
|style="text-align:center;" |  ||
|-
|  Test Plan Creation
| style="text-align:center;" |  ||
|-
|  Test Cases Creation
|style="text-align:center;" |  Done ||
|-
|  Automation Coverage ||
|style="text-align:center;" |
|-
|  Performance Testing
|style="text-align:center;" |  ||
|-
|  All Defects Logged || ||
|-
|  Critical/Blockers Fixed and Verified || ||
|-
|  Metrics/Telemetry||
|style="text-align:center;" |
|-
|-
| Start project
| Basic/Core functionality Nightly testing
|style="text-align:center;" | Q1/2016 || Q2/2017
|style="text-align:center;" |
|style="text-align:center;" |
|-
|-
| QA - Test plan creation
| QA mid-Nightly Signoff
|style="text-align:center;" | Q2/2017 || Q2/2017
|style="text-align:center;" | YELLOW|| style="text-align:center;" | 07/07/2017  
|-
|-
| QA - Test cases/Env preparation
| QA Nightly - Full Testing
|style="text-align:center;" | Q2/2017 || Q2/2017
|style="text-align:center;" | || style="text-align:center;" | Email to be sent
|-
|-
| QA - Nightly Testing
| QA pre-Beta Signoff
|style="text-align:center;" | Q2/2017 || Q2/2017
|style="text-align:center;"| YELLOW|| style="text-align:center;" | 26/07/2017  
|-
|-
| QA - Beta Testing  
| QA Beta - Full Testing
|style="text-align:center;" | Q2/2017 || Q3/2017
|style="text-align:center;" | ||  
|-
|-
| Release Date
| QA pre-Release Signoff 
|style="text-align:center;" | FF55 || 2017-08-08
|style="text-align:center;" |GREEN|| style="text-align:center;" | 08/09/2017
|}
|}
== References ==
* develepment bug: TBD

Latest revision as of 12:07, 11 September 2017

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
QA Manager Product Integrity Date Email
Software Engineer Engineering Date Email
EPM Product Management Date Email


Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
24/05/2017 1.1 Mihai Boldan Updated the Test Plan
13/06/2017 1.2 Mihai Boldan Updated the Test Plan
10/07/2017 1.3 Mihai Boldan mid-Nightly Sign off(YELLOW)
31/07/2017 1.4 Mihai Boldan pre-Beta Sign off(YELLOW)

Overview

Purpose

This page details the testing that will be performed by Manual QA for the File read access restrictions sand boxing functionality. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • Areas of risk
  • What will be tested
  • How testing will be performed
    • criteria
    • deliverables
    • ownership
    • schedule
  • What data, if any, will be monitored

To help ensure the best possible release and long term maintenance of this feature.

Scope

This wiki details the testing that will be performed by the project team for the File Read Access Restrictions project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
    • Focus should be on the following:
      • Printing
      • Print to file
      • Clipboard interactions
      • file uploads to sites like google docs and photo sites
      • drag and drop of files into content
      • content file/page saves
      • add-ons:
        • WebExtensions
          • Popular WebExtensions still work
          • WebExtensions-developer use cases loading from home directory (i.e, about:debugging then load both packed and unpacked extensions)
        • Legacy
      • about:* pages are functional and links within access correctly
      • Flash (file picker)
      • Running browser outside of home directory, i.e., not installing it into proper location such as /Applications (on Mac), or Program Files (Windows)
  • How testing will be performed
    • Manual

Ownership

Project Manager:
Jim Mathies (irc: jimm)
Development:
Bob Owen (irc: TBD)
Haik Aftandilian (irc: Haik)

Platform QA Team:
Tracy Walker (irc: TBD)
Mihai Boldan (irc: mboldan)
Bogdan Maris (irc: bogdan_maris)
Roxana Leitan (irc: RoxanaLeitan)
Ovidiu Boca (irc: Ovidiu)

Testing summary

Scope of Testing

In Scope

This section describes what is in scope for manual testing.

  • Testing on most used OSs: Windows7x64, Windows10x64, Ubuntu16.04x64 and Mac 10.12
  • Testing on different build structures:
    • Window 7x64 - Firefox build on 32-bit
    • Windows 10x64 - Firefox build on 64-bit
    • Mac OS X 10.12 - Firefox build on 64-bit
    • Ubuntu 16.04 - Firefox build on 64-bit
  • Testing areas:
    • Printing
    • File uploading
    • Drag and Drop
    • Add-ons
    • Flash content
    • Accessibility
    • Virtual keyboard support
    • Content file/page saves
    • about:* pages
    • Interoperability
    • Downloads
  • Test on different environments

Out of Scope

This section describes what is out of scope for manual testing.

  • Testing on a high hardware variety
    • Separate test machines were configured for this feature
    • Only AMD 760G and ATI Radeon 3000 graphic cards available
  • Testing on multiple Linux distribution systems
    • Testing will be performed only on Ubuntu OS


Requirements for testing

Environments

Full Testing will be performed using Beta build for Desktop with focus on platform specific issues as follows

  • Windows
    • Roaming windows user
    • Running from a network drive
    • Profile on network drive
    • Profile on symbolic link or junction point
    • Windows Users directory on junction point
  • Mac OS X
    • Profile in symlinked dir
    • ~ (home) in symlinked dir
  • Linux
    • Profile in symlinked dir
    • .config in symlinked dir
    • ~ (home) in symlinked dir

Note: Read access restrictions on each platform will be slightly different. When creating test cases, any known differences should be called out in the test case.


Channel dependent settings (configs) and environment setups

  • The security.sandbox.content.level pref must be set to 3. This is applicable for the three phases of Firefox (Nightly, Beta and Release).
    • The pref is set to 3 by default from 56.0a1(22-07-2017)

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 Installing Firefox on a specific environment Test Cases 3-High 2-Possible 3-High 18
RAC-2 Printing different file types from a specific environment Test Cases 2-Moderate 2-Possible 2-Moderate 12
RAC-3 Install and run Legacy add-ons Test Cases 2-Moderate, 1-Unlikely 1 - Low 2
RAC-4 Install and run Web extensions Test Cases 3-High 3-Almost Certain 3-High 27
RAC-5 Uploading files on different websites Test Cases 2-Moderate 2-Possible 3-High 12
RAC-6 Downloading different file types, using a specific environment Test Cases 2-Moderate 2-Possible 3-High 12

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Risk Analysis

Risk area Requirement Status
File Uploading User has ability to upload files TBD
Add-ons File access properly restricted TBD
about: pages no regression in access of about: pages TBD
3rd party hand-offs TBD TBD
Plugins (particularly Flash) File access properly restricted TBD
Profiles (user and Firefox) on "other" drives File access properly restricted TBD
Performance No browser responsiveness regressions TBD
Stability No regressions in crash rate TBD

Test Objectives

Criteria Description Metric Test results Criteria Met? QA Owner
Manual testing Tests passed -link to manual test results- TBD (Date status updated) Eng + Rel SV teams
Unit testing Automated tests pass perfherder TBD (Date status updated) TBD
Security Security approves via review -Link to security Analysis- TBD (Date status updated) TBD
Stability Crash rate stable -link to Stability Analysis- TBD (Date status updated) TBD


Builds

This section should contain links for builds with the feature -

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project Q1/2016 Q2/2017
QA - Test plan creation Q2/2017 Q2/2017
QA - Test cases/Env preparation 22/05/2017 Q2/2017
QA - Nightly Testing Q2/2017 Q2/2017
QA - Beta Testing Q2/2017 Q3/2017
Release Date FF56 2017-09-26

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution on Windows, on Mac OS X
Bugs management Bugzilla

Status

Overview

Track the dates and build number where feature was released to Nightly - security.sandbox.content.level pref set to 3 by default from 56.0a1(22-07-2017)
Track the dates and build number where feature was merged to Release/Beta - TBD

References

  • List and links for specs
 List and links for available specs - documents, user stories, specifications
  • Meta bug

Testcases

Test Areas

Test Area Covered Notes
Multi-Process Enabled Yes
Printing Yes
Clipboard Yes
Flash Plugin Yes
Virtual Keyboard Yes
Accessibility Yes
Private Window Yes
Install/Upgrade
Feature upgrades/downgrades data as expected Yes
Does sync work Yes
Enterprise Profiles on network drives will be affected
Enterprise administration Yes
Network proxies/autoconfig Yes
ESR behavior changes Yes
Data Monitoring
Temporary or permanent telemetry monitoring Yes
Add-ons
Permissions Yes
Testing with existing/popular addons Yes
Web Compatibility Focused on sites with file upload capability
Testing against target sites Yes
Interoperability Focus on third party hand-offs to/from Firefox
Common testing with other clients. Yes

If it's not listed above, it is currently out of scope.

Test suite

Full Test suite - Test Rail
Smoke Test suite - TBD
Regression Test suite - TBD

Bug Work

Meta bug: TBD

Logged bugs
ID Priority Component Assigned to Summary Status Target milestone
1368690
1369670 Security:Process Sandboxing Bob Owen Blank pages are printed with security.sandbox.content.level set to 3 when Users folder is a junction point VERIFIED FIXED
1378061 Security:Process Sandboxing Bob Owen Only set user's SID in USER_LIMITED as deny only when not using restricting SIDs. RESOLVED FIXED
1378141 Disability Access APIs Aaron Klotz PBrowserParent::RecvPDocAccessibleConstructor Constructing a top-level PDocAccessible with null COM RESOLVED FIXED
1379951 Disability Access APIs Nobody a11y stack overflows at sandbox level 3 REOPENED
Bug fix verification

Sign off

Criteria

Checklist

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing
- Windows 7x64, Windows 10x64, Mac OS X 10.12

  • Link for the tests run
    • Full Test suite, link to TestRail - Tests Runs and Results on Windows and on Mac OS X
    • Daily Smoke, if needed/available
    • Regression Test suite, if needed/available


Merge to Beta Sign-off
List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation Done
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
Basic/Core functionality Nightly testing
QA mid-Nightly Signoff YELLOW 07/07/2017
QA Nightly - Full Testing Email to be sent
QA pre-Beta Signoff YELLOW 26/07/2017
QA Beta - Full Testing
QA pre-Release Signoff GREEN 08/09/2017
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/file_read_access&oldid=1180168"