QA/NewStorage

From MozillaWiki
< QA
Jump to: navigation, search

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
03/28/2017 1.0 Roxana Leitan Created first draft

Overview

Websites with Storage API can ask for permissions of storing persistent data. The permission is like the tape on the sealed box so that data stored in the box won't be automatically removed until the user manually make changes in preferences.

Purpose

Detail the purpose of this document. For example:

  • The test scope, focus areas and objectives
  • The test responsibilities
  • The test strategy for the levels and types of test for this release
  • The entry and exit criteria
  • The basis of the test estimates
  • Any risks, issues, assumptions and test dependencies
  • The test schedule and major milestones
  • The test deliverables

Scope

This wiki details the testing that will be performed by the project team for the New Storage project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed

Ownership

Development
DOM Engineers:
Shawn Huang (irc: shawnjohnjr)
Tom Tung (irc: tt)
UX designer:
Mark Liang (irc: mark_liang)
Morpheus Chen (irc: Morpheus)
Tina Hsieh (irc: Tina_Hsieh)
FE Engineers: Fischer Liu (irc: fischer)
EPM: Francis Lee

Engineering QA Team - Nightly Testing:
Rares Bologa (irc: RaresB) - PM for QA team
Brandusa Tot (irc: brindusat) - leading QA efforts
Roxana Leitan (irc: RoxanaLeitan) - QA


Desktop QA Team - Beta Testing:
Andrei Vaida (irc: avaida) - PM for the QA team
Ciprian Georgiu (irc: ciprian_georgiu) - leading QA efforts
Emil Ghitta (irc: emilghitta) - leading QA efforts


Platform QA Team :
Cristian Chiorean (irc: cchiorean) - PM for QA team
Andrei Filip -leading QA efforts
Ionut Budeanu - QA

Testing summary

Scope of Testing

In Scope

  • Permission
  • Full Disk
  • Preferences
  • Changing Status
  • Removing Data

Out of Scope

Following areas/features are considered out of scope and will not be considered as testing zones to be handled in this test plan.

Requirements for testing

Environments

Full testing will be performed on the following OSes:
  • Windows
    • Windows 10
    • Windows 7
  • Mac OS X
  • Ubuntu

Channel dependent settings (configs) and environment setups

Nightly

Currently the two preferences that enable New Storage are set as default on Nightly 55:

  • "dom.storageManager.enabled" - true
  • "browser.storageManager.enabled" - true
  • browser.storageManager.pressureNotification.usageThresholdGB

- Default is 5

  • browser.storageManager.pressureNotification.usageThresholdGB

- Default is 1200000

Beta

text

Post Beta / Release

text

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 User Experience TO-1 3-High 1-Unlikely 2-Medium 6
RAC-2 Preferences options TO-2 3-High 2-Possible 3-High 18
RAC-3 Permission prompt TO-3 2-Moderate 2-Possible 2-Medium 8
RAC-4 Full disk TO-4 3-High 3-Almost Certain 3-High 27
RAC-5 Accesibility TO-5 1-Low 2-Possible 3-High 6
RAC-6 RTL TO-6 1-Low 3-Almost Certain 2-Medium 4

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Evaluation Criteria Test Type RAC Owners
1 Usability To ensure a positive user experience User Experience expectations Manual RAC-1, RAC-2, RAC-3, RAC-5 Eng Team
2 Preferences To verify the existing options of Offline Web Content and User Data The User Data is stored/removed according options Manual RAC-2 Eng Team
3 Permisions To verify the permission pop-ups and basic functions(Allow, Don't Allow) The permissions options work properly Manual RAC-3 Eng Team
4 Full Disk Warning To verify full disk notification bars The notification bars are displayed accordingly Manual RAC-4 Eng Team
5 Accesibility To verify accesibility pre-requisites: colors, mouse only, keyboard only, screen reader New Storage options are usable from accessibility point of view Manual RAC-5 Eng Team
6 Localization New Storage options are properly displayed on locale builds RTL and strings in general (contextual warning) Manual RAC-6 Eng Team

Builds

This section should contain links for builds with the feature -New Storage enabled

  • Links for Nightly builds
  • Links for Beta builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 2017.03.28
Study documentation/specs received from developers 2017.03.28
QA - Test plan creation 2017.03.28
QA - Test cases/Env preparation 2017.04.20
QA - Nightly Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla

Status

Overview

Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Release/Beta

References

 -UX specifications -link
 -Storage specifications -link

Testcases

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes
Theme (high contrast) Yes
UI
Mouse-only operation Yes
Keyboard-only operation Yes
Display (HiDPI) Yes
Interaction (scroll, zoom) Yes
Usable with a screen reader Yes e.g. with NVDA
Usability and/or discoverability testing Yes Is this feature user friendly
RTL build testing Yes
Help/Support
Help/support interface required Yes Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) Yes Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected No
Does sync work across upgrades No
Requires install testing No separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding No Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Does this affect partner builds? Partner build testing No yes/no options, add comment with details about who will lead testing
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration No
Network proxies/autoconfig No
ESR behavior changes No
Locked preferences No
Data Monitoring
Temporary or permanent telemetry monitoring No List of error conditions to monitor
Telemetry correctness testing No
Server integration testing No
Offline and server failure testing No
Load testing No
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? Yes
Comprehensive API testing Yes
Permissions
Testing with existing/popular addons
Security Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review
Privilege escalation testing
Fuzzing
Web Compatibility depends on the feature
Testing against target sites Yes
Survey of many sites for compatibility Yes
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS No
Interaction of this feature with other browser features No

Test suite

Full Test suite - Link to test rail - test cases should be added under Firefox Desktop project link
Smoke Test suite - Link with the tests - if available/needed.
Regression Test suite - Link with the tests - if available/needed.

Bug Work

Meta bug: 1147820 - [meta Improve Storage]

Logged bugs ( blocking 1147820 )
Full Query
ID Priority Component Assigned to Summary Status Target milestone
536509 P3 DOM: Core & HTML :Nika Layzell (on leave) localStorage does not obey "third-party cookies" pref RESOLVED mozilla43
742822 P2 Storage: localStorage & sessionStorage make localStorage and indexedDB share quota mechanisms NEW ---
781984 -- Preferences IndexedDB data is not listed under Preferences->Advanced->Network->Offline Web Content RESOLVED ---
832660 -- Preferences "maintain offline storage" permission(s) confusing and incomplete RESOLVED ---
1102808 P3 Data Sanitization [meta] Clear Recent History / Forget button blind spots NEW ---
1138885 -- General UI for persistent storage RESOLVED ---
1147821 -- Storage: IndexedDB :Nika Layzell (on leave) Only disable IndexedDB in third-party windows when the third-party cookie preference is set RESOLVED mozilla43
1161553 -- General Mark Liang(:mark_liang) [UX] Create design spec for local storage permissions, notifications and preferences RESOLVED ---
1183968 -- DOM: Core & HTML :Nika Layzell (on leave) Make sessionStorage obey the third-party cookies pref RESOLVED ---
1184973 -- DOM: Core & HTML :Nika Layzell (on leave) Add nsContentUtils::StorageAllowedFor* as a unified mechanism for determining storage avaliability RESOLVED mozilla43
1184975 -- Storage: IndexedDB :Nika Layzell (on leave) Update IndexedDB to use common StorageAllowedForWindow logic RESOLVED ---
1184978 -- DOM: Core & HTML :Nika Layzell (on leave) Update CacheStorage to use common StorageAllowedForWindow logic RESOLVED ---
1254428 P3 DOM: Core & HTML [meta] Implement Storage spec REOPENED ---
1268889 P3 DOM: Security Andrea Marchesini [:baku] Implement Clear-Site-Data response header to reset origin storage RESOLVED mozilla62
1296591 P3 DOM: Core & HTML Tom Tung [:tt, :ttung] Bound persistent storage by total quota RESOLVED ---
1296592 P3 Storage: StorageManager Figure out OS interaction when running low on storage NEW ---
1305665 P3 Storage: Quota Manager Consider removing origin groups for storage usage NEW ---
1309118 -- Preferences Fischer [:Fischer] [Meta] Implement Storage management RESOLVED ---
1372592 P2 Preferences Fischer [:Fischer] Should not display non-persistent-storage sites which stores 0 byte in Preferences Site Data section VERIFIED Firefox 57
1372612 -- Preferences The Site Data list is not immediately updated RESOLVED ---
1373184 -- Site Permissions Storage Permission tooltip is not visible on hover in High Contrast mode RESOLVED ---
1375004 -- Preferences Storage permissions are not updated when removed from View Page Info RESOLVED ---
1377104 P2 Preferences Fischer [:Fischer] Should clear all stored site data dynamically VERIFIED Firefox 57
1377852 P2 DOM: Push Notifications Fischer [:Fischer] Do not display the 2nd storage pressure notification when there is one being displayed VERIFIED mozilla57
1383656 P3 DOM: Core & HTML Tweak and analyze the value and find out an appropriate way to generate the padding size for opaque response NEW ---
1384492 P2 Storage: Quota Manager Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor) Notification bar is not displayed if the stored data is more than 50% of disk space RESOLVED mozilla57
1385936 P3 DOM: Push Notifications The notification bar dissapear if click "Learn More" button from the same tab NEW ---
1392593 -- Site Permissions Clearing a website's permission in private window X should not affect the permission status in private window Y and all the other normal windows RESOLVED ---
1400582 P3 Storage: IndexedDB Deleting all history still leaves some traces that can be used to precisely track individual users. RESOLVED ---
1400678 P2 Storage: Quota Manager Andrea Marchesini [:baku] quota storage should be cleared on exit if user has requested cookies to be cleared on exit RESOLVED mozilla62
1400679 P3 Site Identity allow all storage to be blocked on a site-by-site basis with optional click-to-activate RESOLVED ---
1401475 P4 Storage Inspector Storage Inspector lists origins that actually don't store any data (indexedb/localstorage/cache storage) NEW ---
1403441 P2 Storage: Quota Manager Make storage api support history.pushState() NEW ---
1407936 P1 Storage: Quota Manager Andrea Marchesini [:baku] Site data is not completely deleted from profile folder after removing data from Fx site data settings RESOLVED mozilla58
1408406 P2 DOM: Core & HTML Differences in storage sizes between data stored in Firefox and profile folder RESOLVED ---
1421690 P2 Preferences [meta] Storage (Site Data and Cookies) Management V2 NEW ---
1445464 P2 Storage: localStorage & sessionStorage [META]Persist SessionStorage like LocalStorage by default NEW ---

37 Total; 11 Open (29.73%); 23 Resolved (62.16%); 3 Verified (8.11%);


Bug fix verification
Full Query
ID Priority Component Assigned to Summary Status Resolution Target milestone
536509 P3 DOM: Core & HTML :Nika Layzell (on leave) localStorage does not obey "third-party cookies" pref RESOLVED FIXED mozilla43
1147821 -- Storage: IndexedDB :Nika Layzell (on leave) Only disable IndexedDB in third-party windows when the third-party cookie preference is set RESOLVED FIXED mozilla43
1161553 -- General Mark Liang(:mark_liang) [UX] Create design spec for local storage permissions, notifications and preferences RESOLVED FIXED ---
1184973 -- DOM: Core & HTML :Nika Layzell (on leave) Add nsContentUtils::StorageAllowedFor* as a unified mechanism for determining storage avaliability RESOLVED FIXED mozilla43
1268889 P3 DOM: Security Andrea Marchesini [:baku] Implement Clear-Site-Data response header to reset origin storage RESOLVED FIXED mozilla62
1309118 -- Preferences Fischer [:Fischer] [Meta] Implement Storage management RESOLVED FIXED ---
1372592 P2 Preferences Fischer [:Fischer] Should not display non-persistent-storage sites which stores 0 byte in Preferences Site Data section VERIFIED FIXED Firefox 57
1377104 P2 Preferences Fischer [:Fischer] Should clear all stored site data dynamically VERIFIED FIXED Firefox 57
1377852 P2 DOM: Push Notifications Fischer [:Fischer] Do not display the 2nd storage pressure notification when there is one being displayed VERIFIED FIXED mozilla57
1384492 P2 Storage: Quota Manager Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor) Notification bar is not displayed if the stored data is more than 50% of disk space RESOLVED FIXED mozilla57
1400678 P2 Storage: Quota Manager Andrea Marchesini [:baku] quota storage should be cleared on exit if user has requested cookies to be cleared on exit RESOLVED FIXED mozilla62
1407936 P1 Storage: Quota Manager Andrea Marchesini [:baku] Site data is not completely deleted from profile folder after removing data from Fx site data settings RESOLVED FIXED mozilla58

12 Total; 0 Open (0%); 9 Resolved (75%); 3 Verified (25%);

Sign off

Criteria

Checklist

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite, link to TestRail - Tests Runs and Results link
    • Daily Smoke, if needed/available
    • Regression Test suite, if needed/available

Merge to Beta Sign-off List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation Yes
Test Cases Creation Yes
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release GREEN (signed off on 9/13/2017)
QA Signoff - Beta Release GREEN (signed off on 10/27/2017)