QA/file read access

From MozillaWiki
< QA
Jump to: navigation, search

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
QA Manager Product Integrity Date Email
Software Engineer Engineering Date Email
EPM Product Management Date Email

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
24/05/2017 1.1 Mihai Boldan Updated the Test Plan
13/06/2017 1.2 Mihai Boldan Updated the Test Plan
10/07/2017 1.3 Mihai Boldan mid-Nightly Sign off(YELLOW)
31/07/2017 1.4 Mihai Boldan pre-Beta Sign off(YELLOW)



This page details the testing that will be performed by Manual QA for the File read access restrictions sand boxing functionality. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • Areas of risk
  • What will be tested
  • How testing will be performed
    • criteria
    • deliverables
    • ownership
    • schedule
  • What data, if any, will be monitored

To help ensure the best possible release and long term maintenance of this feature.


This wiki details the testing that will be performed by the project team for the File Read Access Restrictions project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
    • Focus should be on the following:
      • Printing
      • Print to file
      • Clipboard interactions
      • file uploads to sites like google docs and photo sites
      • drag and drop of files into content
      • content file/page saves
      • add-ons:
        • WebExtensions
          • Popular WebExtensions still work
          • WebExtensions-developer use cases loading from home directory (i.e, about:debugging then load both packed and unpacked extensions)
        • Legacy
      • about:* pages are functional and links within access correctly
      • Flash (file picker)
      • Running browser outside of home directory, i.e., not installing it into proper location such as /Applications (on Mac), or Program Files (Windows)
  • How testing will be performed
    • Manual


Project Manager:
Jim Mathies (irc: jimm)
Bob Owen (irc: TBD)
Haik Aftandilian (irc: Haik)
Platform QA Team:
Tracy Walker (irc: TBD)
Mihai Boldan (irc: mboldan)
Bogdan Maris (irc: bogdan_maris)
Roxana Leitan (irc: RoxanaLeitan)
Ovidiu Boca (irc: Ovidiu)

Testing summary

Scope of Testing

In Scope

This section describes what is in scope for manual testing.

  • Testing on most used OSs: Windows7x64, Windows10x64, Ubuntu16.04x64 and Mac 10.12
  • Testing on different build structures:
    • Window 7x64 - Firefox build on 32-bit
    • Windows 10x64 - Firefox build on 64-bit
    • Mac OS X 10.12 - Firefox build on 64-bit
    • Ubuntu 16.04 - Firefox build on 64-bit
  • Testing areas:
    • Printing
    • File uploading
    • Drag and Drop
    • Add-ons
    • Flash content
    • Accessibility
    • Virtual keyboard support
    • Content file/page saves
    • about:* pages
    • Interoperability
    • Downloads
  • Test on different environments

Out of Scope

This section describes what is out of scope for manual testing.

  • Testing on a high hardware variety
    • Separate test machines were configured for this feature
    • Only AMD 760G and ATI Radeon 3000 graphic cards available
  • Testing on multiple Linux distribution systems
    • Testing will be performed only on Ubuntu OS

Requirements for testing


Full Testing will be performed using Beta build for Desktop with focus on platform specific issues as follows

  • Windows
    • Roaming windows user
    • Running from a network drive
    • Profile on network drive
    • Profile on symbolic link or junction point
    • Windows Users directory on junction point
  • Mac OS X
    • Profile in symlinked dir
    • ~ (home) in symlinked dir
  • Linux
    • Profile in symlinked dir
    • .config in symlinked dir
    • ~ (home) in symlinked dir

Note: Read access restrictions on each platform will be slightly different. When creating test cases, any known differences should be called out in the test case.

Channel dependent settings (configs) and environment setups

  • The security.sandbox.content.level pref must be set to 3. This is applicable for the three phases of Firefox (Nightly, Beta and Release).
    • The pref is set to 3 by default from 56.0a1(22-07-2017)

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 Installing Firefox on a specific environment Test Cases 3-High 2-Possible 3-High 18
RAC-2 Printing different file types from a specific environment Test Cases 2-Moderate 2-Possible 2-Moderate 12
RAC-3 Install and run Legacy add-ons Test Cases 2-Moderate, 1-Unlikely 1 - Low 2
RAC-4 Install and run Web extensions Test Cases 3-High 3-Almost Certain 3-High 27
RAC-5 Uploading files on different websites Test Cases 2-Moderate 2-Possible 3-High 12
RAC-6 Downloading different file types, using a specific environment Test Cases 2-Moderate 2-Possible 3-High 12


  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Risk Analysis

Risk area Requirement Status
File Uploading User has ability to upload files TBD
Add-ons File access properly restricted TBD
about: pages no regression in access of about: pages TBD
3rd party hand-offs TBD TBD
Plugins (particularly Flash) File access properly restricted TBD
Profiles (user and Firefox) on "other" drives File access properly restricted TBD
Performance No browser responsiveness regressions TBD
Stability No regressions in crash rate TBD

Test Objectives

Criteria Description Metric Test results Criteria Met? QA Owner
Manual testing Tests passed -link to manual test results- TBD (Date status updated) Eng + Rel SV teams
Unit testing Automated tests pass perfherder TBD (Date status updated) TBD
Security Security approves via review -Link to security Analysis- TBD (Date status updated) TBD
Stability Crash rate stable -link to Stability Analysis- TBD (Date status updated) TBD


This section should contain links for builds with the feature -

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project Q1/2016 Q2/2017
QA - Test plan creation Q2/2017 Q2/2017
QA - Test cases/Env preparation 22/05/2017 Q2/2017
QA - Nightly Testing Q2/2017 Q2/2017
QA - Beta Testing Q2/2017 Q3/2017
Release Date FF56 2017-09-26

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution on Windows, on Mac OS X
Bugs management Bugzilla



Track the dates and build number where feature was released to Nightly - security.sandbox.content.level pref set to 3 by default from 56.0a1(22-07-2017)
Track the dates and build number where feature was merged to Release/Beta - TBD


  • List and links for specs
 List and links for available specs - documents, user stories, specifications
  • Meta bug


Test Areas

Test Area Covered Notes
Multi-Process Enabled Yes
Printing Yes
Clipboard Yes
Flash Plugin Yes
Virtual Keyboard Yes
Accessibility Yes
Private Window Yes
Feature upgrades/downgrades data as expected Yes
Does sync work Yes
Enterprise Profiles on network drives will be affected
Enterprise administration Yes
Network proxies/autoconfig Yes
ESR behavior changes Yes
Data Monitoring
Temporary or permanent telemetry monitoring Yes
Permissions Yes
Testing with existing/popular addons Yes
Web Compatibility Focused on sites with file upload capability
Testing against target sites Yes
Interoperability Focus on third party hand-offs to/from Firefox
Common testing with other clients. Yes

If it's not listed above, it is currently out of scope.

Test suite

Full Test suite - Test Rail
Smoke Test suite - TBD
Regression Test suite - TBD

Bug Work

Meta bug: TBD

Logged bugs
ID Priority Component Assigned to Summary Status Target milestone
1369670 Security:Process Sandboxing Bob Owen Blank pages are printed with security.sandbox.content.level set to 3 when Users folder is a junction point VERIFIED FIXED
1378061 Security:Process Sandboxing Bob Owen Only set user's SID in USER_LIMITED as deny only when not using restricting SIDs. RESOLVED FIXED
1378141 Disability Access APIs Aaron Klotz PBrowserParent::RecvPDocAccessibleConstructor Constructing a top-level PDocAccessible with null COM RESOLVED FIXED
1379951 Disability Access APIs Nobody a11y stack overflows at sandbox level 3 REOPENED
Bug fix verification

Sign off



  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)


Nightly testing

List of OSes that will be covered by testing
- Windows 7x64, Windows 10x64, Mac OS X 10.12

  • Link for the tests run
    • Full Test suite, link to TestRail - Tests Runs and Results on Windows and on Mac OS X
    • Daily Smoke, if needed/available
    • Regression Test suite, if needed/available

Merge to Beta Sign-off
List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite


Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation Done
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Basic/Core functionality Nightly testing
QA mid-Nightly Signoff YELLOW 07/07/2017
QA Nightly - Full Testing Email to be sent
QA pre-Beta Signoff YELLOW 26/07/2017
QA Beta - Full Testing
QA pre-Release Signoff GREEN 08/09/2017