MozillaWiki

QA/IntersectionObserver: Difference between revisions

< QA
(Add Android to list of platforms to test)
 
(33 intermediate revisions by 2 users not shown)
Line 1: Line 1:
'''Revision History'''
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded).  The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
{| class="wikitable" style="width:60%"
|-
! Date !! Version !! Author !! Description
|-
| 01/26/2017 || 1.0 || Tobias Schneider || Created first draft
|}
= Overview =
= Overview =
== Purpose ==
== Purpose ==
Detail the purpose of this document. For example:
* The test scope, focus areas and objectives
* The test responsibilities
* The test strategy for the levels and types of test for this release
* The entry and exit criteria
* The basis of the test estimates
* Any risks, issues, assumptions and test dependencies
* The test schedule and major milestones
* The test deliverables
== Scope ==
This wiki details the testing that will be performed by QA for the Intersection Observer API. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
This wiki details the testing that will be performed by QA for the Intersection Observer API. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
* Areas of risk
* What will be tested
* What will be tested
* How testing will be performed
* How testing will be performed
** criteria
** ownership
** schedule
To help ensure the best possible release and long term maintenance of this feature.


== Ownership ==
== Risk Analysis ==
Developer contacts: [mailto:tschneider@mozilla.com Tobias Schneider] <br />
{| class="wikitable" style="width:95%"
QA: <span style=color:red> TBD </span> <br />
QA Peer: <span style=color:red> TBD </span>
= Testing summary =
== Scope of Testing ==
=== In Scope ===
Detail what is in scope from a testing perspective for the project team.
 
=== Out of Scope ===
Detail what is out of scope from a testing perspective for the project team. Note: if usability testing is not in the scope of testing feature.
 
= Requirements for testing =
== Environments ==
Full Testing will be performed using Nightly for Desktop builds on:
 
Windows <br />
Mac OS X <br />
Linux <br />
 
Full Testing will be performed using Nightly for Mobile builds on:
 
Android  <br />
 
== Channel dependent settings (configs) and environment setups ==
 
=== Nightly ===
dom.IntersectionObserver.enabled needs to be set to "true" in about:config <br />
If any kind of Ad-blocker is installed/in use it needs to be uninstalled/disabled
 
= Test Strategy =
== Test Objectives ==
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
This could be documented in bullet form or in a table similar to the one below.
 
{| class="wikitable"
|-
|-
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! Owners
! Risk area !! Requirement !! Status
|-
|-
| 1 || Name of the feature or sub-function being tested || The objective the test is trying to demonstrate || The criteria that will be evaluated to demonstrate the test is successful || Manual/ Automation/ Regression/ Performance/ Usability/ Security/ Telemetry || Eng Team
| Ad intensive web pages || No regression in interaction performance || style="background-color:#CCFF99;" | PASS on Nightly 54
|-
|-
| 2 || Repeat for each feature/sub-function || || || ||
| Security || Fuzz testing finds no security holes || style="background-color:#CCFF99;" | PASS on Nightly 54
|-
|-
| 3 || || || || ||
| Stability || Crash rate does not increase || style="background-color:#CEF2F2;" | TBD
|}
|}


 
= Testing Strategy =
== Builds ==
== Scope ==
This section should contain links for builds with the feature
{| class="wikitable" style="width:80%"
* [https://nightly.mozilla.org/ Links for Nightly builds]
|-
 
! Test Areas !! Covered
== Test Execution Schedule ==
|-
The following table identifies the anticipated testing period available for test execution.
| Private Window
{| class="wikitable" style="width:60%"
|style="text-align:center;" | Yes
|-
| Multi-Process Enabled
|style="text-align:center;" |  Yes
|-
| Multi-process Disabled
|style="text-align:center;" |  Yes
|-
| Stability
|style="text-align:center;" | Yes
|-
|-
! Project phase !! Start Date !! End Date
| '''UI'''
|| 
|-
|-
| Start project
| Display (HiDPI)
|style="text-align:center;" | ||
|style="text-align:center;" | Yes
|-
|-
| Study documentation/specs received from developers
| Interraction (scroll, zoom)
|style="text-align:center;" | ||
|style="text-align:center;" | Yes
 
|-
|-
| QA - Test plan creation
| ''' Add-ons '''
|style="text-align:center;" |  ||
||   
|-
|-
| QA - Test cases/Env preparation
| ad-blocking Add-ons must be disabled/removed to test
|style="text-align:center;" |  ||
|style="text-align:center;" |  Yes
 
|-
|-
| QA - Nightly Testing
| ''' Security '''
|style="text-align:center;" |  ||
||   
 
|-
|-
| QA - Aurora Testing
| Fuzzing 
|style="text-align:center;" |  || 
|style="text-align:center;" |  Yes
|-
|-
| QA - Beta Testing
| ''' Web Compatibility '''
|style="text-align:center;" |  ||
||   
|-
|-
| Release Date
| Testing against target sites 
|style="text-align:center;" | ||
|style="text-align:center;" | Yes
|}
|}


== Testing Tools ==
If it's not listed above, it is currently out of scope.
Detail the tools to be used for testing, for example see the following table:
 
{| class="wikitable" style="width:50%"
== Objectives ==
{| class="wikitable" style="width:95%"
|-
|-
! Process !! Tool
! Criteria Description
! Metric
! w/o IO API
! w/ IO API
! Criteria Met?
! QA Owner
|-
|-
| Test plan creation || Mozilla wiki
| Manual testing
| Interaction performance
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Yes on Nightly 54
| Tracy 
|-
|-
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs
| Unit testing
| Automated tests
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Yes on Nightly builds of 54
| Tobias
|-
|-
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]
| Fuzz testing
| Automated tests
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Pass [1]
| style="background-color:#CCFF99;" | Yes on Nightly 54
| Fuzzing team (Raymond)
|-
|-
| Bugs management || Bugzilla
| Stability
| Telemetry/Crash reports
| tbd - telemetry experiment in [https://bugzilla.mozilla.org/show_bug.cgi?id=1341259 bug 1341259]
| tbd - telemetry experiment in [https://bugzilla.mozilla.org/show_bug.cgi?id=1341259 bug 1341259]
| style="background-color:#CEF2F2;" | soon to begin with 10% enabled
| Tobias
|}
|}
[1] [https://bugzilla.mozilla.org/show_bug.cgi?id=1343807 tracking bug] has no bugs hanging off of it


= Status =
== Testcases ==
== Overview ==
Ad-hoc testing of Ad-intensive websites. In general news sites tent to be rich of Ads. Therefore the top news sites on Alexa are a great source for suitable websites.
First releases to Nightly: <span style=color:red> TBD </span>
 
== Risk analysis ==
Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix
= References =
* List and links for specs
** [https://wicg.github.io/IntersectionObserver/] Intersection Observer Spec
** [https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API] Intersection Observer API
 
* Meta bug
** [https://bugzilla.mozilla.org/show_bug.cgi?id=1321865 Bug 1321865] - Enable IntersectionObserver
 
= Testcases =
== Overview ==
Testing should happen on Ad-intensive websites. In general news sites tent to be rich of Ads. Therefore the top news sites on Alexa are a great source for suitable websites.


[http://www.alexa.com/topsites/category/Top/News] The top 500 news sites on the web
[http://www.alexa.com/topsites/category/Top/News] The top 500 news sites on the web
Line 167: Line 136:
* Repeat scenarios above but let the browser untouched for a longer period of time. Then proceed.
* Repeat scenarios above but let the browser untouched for a longer period of time. Then proceed.


== Test Areas ==
== Environments ==
{| class="wikitable" style="width:80%"
Full Testing will be performed using Nightly for Desktop builds on:
|-
* Windows
! Test Areas !! Covered !! Details
* Mac OS X
|-
* Linux
| Private Window
|style="text-align:center;" | ||
|-
| Multi-Process Enabled
|style="text-align:center;" |  ||
|-
| Multi-process Disabled
|style="text-align:center;" |  ||
|-
| Theme (high contrast)
|style="text-align:center;" |  ||
|-
| '''UI'''
||  ||
|-
| Mouse-only operation 
|style="text-align:center;" |  ||
|-
| Keyboard-only operation 
|style="text-align:center;" |  ||
|-
| Display (HiDPI)
|style="text-align:center;" | ||
|-
| Interraction (scroll, zoom)
|style="text-align:center;" | ||
|-
| Usable with a screen reader 
|style="text-align:center;" |  || e.g. with NVDA
|-
| Usability and/or discoverability testing 
|style="text-align:center;" |  || Is this feature user friendly
|-
| RTL build testing 
|style="text-align:center;" |  ||
|-
| '''Help/Support'''
||  ||
|-
| Help/support interface required 
|style="text-align:center;" |  || Make sure link to support/help page exist and is easy reachable.
|-
| Support documents planned(written) 
|style="text-align:center;" |  || Make sure support documents are written and are correct.


|-
Full Testing will be performed using Nightly for Mobile builds on:
| '''Install/Upgrade'''
* Android
||  ||
|-
| Feature upgrades/downgrades data as expected 
|style="text-align:center;" |  ||
|-
| Does sync work across upgrades 
|style="text-align:center;" |  ||
|-
| Requires install testing   
|style="text-align:center;" |  || separate feature/application installation needed (not only Firefox)
|-
| Affects first-run or onboarding   
|style="text-align:center;" |  || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|-
| Does this affect partner builds? Partner build testing 
|style="text-align:center;" |  || yes/no options, add comment with details about who will lead testing


|-
== Channel dependent settings (configs) and environment setups ==
| ''' Enterprise '''
* In Nightly (and DevEd when available)- dom.IntersectionObserver.enabled needs to be set to "true" in about:config
|| ||  Raise up the topic to developers to see if they are expecting to work different on ESR builds
*If any kind of Ad-blocker is installed/in use it needs to be uninstalled/disabled
|-
| Enterprise administration 
|style="text-align:center;" |  ||
|-
| Network proxies/autoconfig 
|style="text-align:center;" |  ||
|-
| ESR behavior changes 
|style="text-align:center;" |  ||
|-
| Locked preferences 
|style="text-align:center;" |  ||


== Manual Testing Results ==
Ad-hoc testing around testcases and scenarios above in regular and private windows, in e10s and non-e10s windows and on normal and HiDPi displays.
=== Nightly ===
{| class="wikitable" style="width:60%"
|-
|-
| ''' Data Monitoring '''
! Platform
||  ||
! IO disabled
! IO enabled
! QA'd by : build ID
|-
|-
| Temporary or permanent telemetry monitoring 
| Windows
|style="text-align:center;" | || List of error conditions to monitor
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Tracy : 20170228030203
|-
|-
| Telemetry correctness testing 
| Mac
|style="text-align:center;" | ||  
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Tracy : 20170228030203
|-
|-
| Server integration testing 
| Linux
|style="text-align:center;" | ||  
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Tracy : 20170301110155
|-
|-
| Offline and server failure testing 
| Android
|style="text-align:center;" | ||
| style="background-color:#CCFF99;" | Pass
|-
| style="background-color:#CCFF99;" | Pass
| Load testing 
| style="background-color:#CCFF99;" | Tracy : 20170301110150
|style="text-align:center;" | ||
|}


=== Aurora ===
{| class="wikitable" style="width:60%"
|-
|-
| ''' Add-ons '''
! Platform
||  || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
! IO disabled
! IO enabled
! QA'd by : build ID
|-
|-
| Addon API required? 
| Windows
|style="text-align:center;" | ||  
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | TBD
|-
|-
| Comprehensive API testing 
| Mac
|style="text-align:center;" | ||  
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | TBD
|-
|-
| Permissions 
| Linux
|style="text-align:center;" | ||  
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | TBD
|-
|-
| Testing with existing/popular addons
| Android
|style="text-align:center;" | ||  
| style="background-color:#CEF2F2;" | (results)
 
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | TBD
|}
=== Beta ===
{| class="wikitable" style="width:60%"
|-
|-
| ''' Security '''
! Platform
||  || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
! IO disabled
|-
! IO enabled
| 3rd-party security review 
! QA'd by : build ID
|style="text-align:center;" |  ||
|-
| Privilege escalation testing
|style="text-align:center;" |  ||
|-
| Fuzzing 
|style="text-align:center;" |  ||
 
|-
|-
| ''' Web Compatibility '''
| Windows
|| || depends on the feature
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | TBD
|-
|-
| Testing against target sites 
| Mac
|style="text-align:center;" | ||  
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | TBD
|-
|-
| Survey of many sites for compatibility 
| Linux
|style="text-align:center;" | ||  
| style="background-color:#CEF2F2;" | (results)
 
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | TBD
|-
|-
| ''' Interoperability  '''
| Android
||  || depends on the feature
| style="background-color:#CEF2F2;" | (results)
|-
| style="background-color:#CEF2F2;" | (results)
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. 
| style="background-color:#CEF2F2;" | TBD
|style="text-align:center;" | ||
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS 
|style="text-align:center;" | ||
|-
| Interaction of this feature with other browser features 
|style="text-align:center;" | ||
|}
|}


== Test suite ==
= Project Information =
Full Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
== Ownership ==
Smoke Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
Developer contacts: [mailto:tschneider@mozilla.com Tobias Schneider] <br />
Regression Test suite - Link with the gdoc - if available/needed.
QA: [mailto:twalker@mozilla.com Tracy Walker] <br />
QA Peer: <span style=color:red> TBD </span>


= Bug Work =
== Builds ==
Tracking bug – meta bug
This section should contain links for builds with the feature
* [https://nightly.mozilla.org/ Links for Nightly builds]


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
== Schedule ==
 
The following table identifies the anticipated testing period available for test execution.
====== Bug fix verification ======
<div class="mw-collapsible-content">
{| class="wikitable" style="width:50%"
|-
! Bug No !! Summary !! Status !! Firefox Verion
|-
|  123 || bug summary || NEW || Nighly 45
|-
| || || ||
|}
</div></div>
 
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
 
====== Logged bugs ======
<div class="mw-collapsible-content">
Bug 111111<br/>
Bug 211111<br/>
</div>
</div>
 
 
= Sign off =
== Criteria ==
Check list
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
 
== Results ==
<span style=color:red> TBC </span>
 
== Checklist ==
{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
|-
|-
! Exit Criteria !! Status !! Notes/Details
! Project phase !! Start Date !! End Date
|-
|-
| Testing Prerequisites (specs, use cases)
| Start project
| style="text-align:center;" |  
|style="text-align:center;" | 20160128 ||   tbd
| style="text-align:center;" |  
|-
|-
| Testing Infrastructure setup
| Study documentation/specs received from developers
|style="text-align:center;" |   ||  
|style="text-align:center;" | 20170201 ||   20170210
|-
|-
| Test Plan Creation
| QA - Test plan creation
| style="text-align:center;" |   ||  
|style="text-align:center;" | 20170210 ||   20170215
|-
|-
| Test Cases Creation
| QA - Nightly Testing
|style="text-align:center;" |   ||  
|style="text-align:center;" | 20170227 ||   20170301
|-
|-
| Full Functional Tests Execution
| Telemetry - Stability Experiment
|style="text-align:center;" |
|style="text-align:center;" | 20170304?? ||   tbd
|style="text-align:center;" |
|-
|-
| Automation Coverage ||
| QA - Aurora Testing
|style="text-align:center;" |  
|style="text-align:center;" | tbd || 
|-
|-
| Performance Testing  
| QA - Beta Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | tbd ||  
|-
|-
| All Defects Logged || ||
| Release Date
|-
|style="text-align:center;" | tbd ||  
|  Critical/Blockers Fixed and Verified || ||
|-
|  Metrics/Telemetry||
|style="text-align:center;" |
|-
|  QA Signoff  - Nightly Release||
|style="text-align:center;" | Email to be sent
|-
|  QA Aurora - Full Testing
|style="text-align:center;" |  ||
|-
|  QA Signoff  - Aurora Release ||
|style="text-align:center;"| Email to be sent
|-
|  QA Beta - Full Testing
|style="text-align:center;" |  ||
|-
|  QA Signoff  - Beta Release ||
|style="text-align:center;" | Email to be sent
|}
|}
== References ==
* List and links for specs
** [https://wicg.github.io/IntersectionObserver/] Intersection Observer Spec
** [https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API] Intersection Observer API
 
* Meta bug
** [https://bugzilla.mozilla.org/show_bug.cgi?id=1321865 Bug 1321865] - Enable IntersectionObserver

Latest revision as of 17:46, 3 March 2017

Overview

Purpose

This wiki details the testing that will be performed by QA for the Intersection Observer API. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • Areas of risk
  • What will be tested
  • How testing will be performed
    • criteria
    • ownership
    • schedule

To help ensure the best possible release and long term maintenance of this feature.

Risk Analysis

Risk area Requirement Status
Ad intensive web pages No regression in interaction performance PASS on Nightly 54
Security Fuzz testing finds no security holes PASS on Nightly 54
Stability Crash rate does not increase TBD

Testing Strategy

Scope

Test Areas Covered
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes
Stability Yes
UI
Display (HiDPI) Yes
Interraction (scroll, zoom) Yes
Add-ons
ad-blocking Add-ons must be disabled/removed to test Yes
Security
Fuzzing Yes
Web Compatibility
Testing against target sites Yes

If it's not listed above, it is currently out of scope.

Objectives

Criteria Description Metric w/o IO API w/ IO API Criteria Met? QA Owner
Manual testing Interaction performance Pass Pass Yes on Nightly 54 Tracy
Unit testing Automated tests Pass Pass Yes on Nightly builds of 54 Tobias
Fuzz testing Automated tests Pass Pass [1] Yes on Nightly 54 Fuzzing team (Raymond)
Stability Telemetry/Crash reports tbd - telemetry experiment in bug 1341259 tbd - telemetry experiment in bug 1341259 soon to begin with 10% enabled Tobias 
[1] tracking bug has no bugs hanging off of it

Testcases

Ad-hoc testing of Ad-intensive websites. In general news sites tent to be rich of Ads. Therefore the top news sites on Alexa are a great source for suitable websites.

[1] The top 500 news sites on the web

List of known sites using the Intersection Observer API:

Possible test scenarios:

  • Open a single news site. Wait till it is fully loaded to make sure all Ads are properly initialized. If there is content to scroll, scroll up and down a couple of times. Click on some articles to trigger a refresh.
  • Open a multiple news sites in different tabs. Wait till every page is fully loaded to make sure all Ads are properly initialized. Switch between tabs from time too time. Close and open new tabs randomly. If there is content to scroll, scroll up and down a couple of times. Click on some articles to trigger a refresh.
  • Repeat scenarios above but let the browser untouched for a longer period of time. Then proceed.

Environments

Full Testing will be performed using Nightly for Desktop builds on:

  • Windows
  • Mac OS X
  • Linux

Full Testing will be performed using Nightly for Mobile builds on:

  • Android

Channel dependent settings (configs) and environment setups

  • In Nightly (and DevEd when available)- dom.IntersectionObserver.enabled needs to be set to "true" in about:config
  • If any kind of Ad-blocker is installed/in use it needs to be uninstalled/disabled

Manual Testing Results

Ad-hoc testing around testcases and scenarios above in regular and private windows, in e10s and non-e10s windows and on normal and HiDPi displays.

Nightly

Platform IO disabled IO enabled QA'd by : build ID
Windows Pass Pass Tracy : 20170228030203
Mac Pass Pass Tracy : 20170228030203
Linux Pass Pass Tracy : 20170301110155
Android Pass Pass Tracy : 20170301110150

Aurora

Platform IO disabled IO enabled QA'd by : build ID
Windows (results) (results) TBD
Mac (results) (results) TBD
Linux (results) (results) TBD
Android (results) (results) TBD

Beta

Platform IO disabled IO enabled QA'd by : build ID
Windows (results) (results) TBD
Mac (results) (results) TBD
Linux (results) (results) TBD
Android (results) (results) TBD

Project Information

Ownership

Developer contacts: Tobias Schneider
QA: Tracy Walker
QA Peer: TBD

Builds

This section should contain links for builds with the feature

Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 20160128 tbd
Study documentation/specs received from developers 20170201 20170210
QA - Test plan creation 20170210 20170215
QA - Nightly Testing 20170227 20170301
Telemetry - Stability Experiment 20170304?? tbd
QA - Aurora Testing tbd
QA - Beta Testing tbd
Release Date tbd

References

  • List and links for specs
    • [2] Intersection Observer Spec
    • [3] Intersection Observer API
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/IntersectionObserver&oldid=1164524"