QA/IntersectionObserver: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
No edit summary
 
(27 intermediate revisions by the same user not shown)
Line 6: Line 6:
* How testing will be performed
* How testing will be performed
** criteria
** criteria
** deliverables
** ownership
** ownership
** schedule
** schedule
Line 16: Line 15:
! Risk area !! Requirement !! Status
! Risk area !! Requirement !! Status
|-
|-
| Ad intensive web pages || No regression in interaction performance || style="background-color:#CEF2F2;" | TBD
| Ad intensive web pages || No regression in interaction performance || style="background-color:#CCFF99;" | PASS on Nightly 54
|-
|-
| Security || Fuzz testing || style="background-color:#CEF2F2;" | TBD
| Security || Fuzz testing finds no security holes || style="background-color:#CCFF99;" | PASS on Nightly 54
|-
|-
| Stability || Crash rate || style="background-color:#CEF2F2;" | TBD
| Stability || Crash rate does not increase || style="background-color:#CEF2F2;" | TBD
|-
| Add-ons || Breaks ad-block type add-ons??|| style="background-color:#CEF2F2;" |  TBD
|}
|}


Line 56: Line 53:
||   
||   
|-
|-
| Interferes with ad-blocking Add-ons?
| ad-blocking Add-ons must be disabled/removed to test
|style="text-align:center;" |  Yes
|style="text-align:center;" |  Yes


Line 76: Line 73:
If it's not listed above, it is currently out of scope.
If it's not listed above, it is currently out of scope.


= Test Strategy =
== Objectives ==
== Objectives ==
{| class="wikitable" style="width:95%"
{| class="wikitable" style="width:95%"
Line 82: Line 78:
! Criteria Description
! Criteria Description
! Metric
! Metric
! without IO API
! w/o IO API
! IO API value
! w/ IO API  
! Criteria Met?
! Criteria Met?
! QA Owner
! QA Owner
Line 89: Line 85:
| Manual testing  
| Manual testing  
| Interaction performance
| Interaction performance
| (test results)
| style="background-color:#CCFF99;" | Pass
| (test results)
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CEF2F2;" | TBD (Date status updated)
| style="background-color:#CCFF99;" | Yes on Nightly 54
| style="background-color:#CEF2F2;" | Tracy   
| Tracy   
|-
|-
| Unit testing
| Unit testing
| Automated tests
| Automated tests
| (test results)
| style="background-color:#CCFF99;" | Pass
| (test results)
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CEF2F2;" | TBD (Date status updated)
| style="background-color:#CCFF99;" | Yes on Nightly builds of 54
| style="background-color:#CEF2F2;" | Tobias
| Tobias
|-
|-
| Fuzz testing
| Fuzz testing
| Automated tests
| Automated tests
| (test results)
| style="background-color:#CCFF99;" | Pass
| (test results)
| style="background-color:#CCFF99;" | Pass [1]
| style="background-color:#CEF2F2;" | TBD (Date status updated)
| style="background-color:#CCFF99;" | Yes on Nightly 54
| style="background-color:#CEF2F2;" | Fuzzing team (tbd)
| Fuzzing team (Raymond)
|-
|-
| Stability
| Stability
| Crash reports
| Telemetry/Crash reports
| telemetry/crash stats
| tbd - telemetry experiment in [https://bugzilla.mozilla.org/show_bug.cgi?id=1341259 bug 1341259]
| no notable stability regression
| tbd - telemetry experiment in [https://bugzilla.mozilla.org/show_bug.cgi?id=1341259 bug 1341259]
| style="background-color:#CEF2F2;" | TBD (Date status updated)
| style="background-color:#CEF2F2;" | soon to begin with 10% enabled
| style="background-color:#CEF2F2;" | Tobias
| Tobias
|}
|}
[1] [https://bugzilla.mozilla.org/show_bug.cgi?id=1343807 tracking bug] has no bugs hanging off of it


== Testcases ==
== Testcases ==
Testing should happen on Ad-intensive websites. In general news sites tent to be rich of Ads. Therefore the top news sites on Alexa are a great source for suitable websites.
Ad-hoc testing of Ad-intensive websites. In general news sites tent to be rich of Ads. Therefore the top news sites on Alexa are a great source for suitable websites.


[http://www.alexa.com/topsites/category/Top/News] The top 500 news sites on the web
[http://www.alexa.com/topsites/category/Top/News] The top 500 news sites on the web
Line 151: Line 148:
* In Nightly  (and DevEd when available)- dom.IntersectionObserver.enabled needs to be set to "true" in about:config
* In Nightly  (and DevEd when available)- dom.IntersectionObserver.enabled needs to be set to "true" in about:config
*If any kind of Ad-blocker is installed/in use it needs to be uninstalled/disabled
*If any kind of Ad-blocker is installed/in use it needs to be uninstalled/disabled
== Manual Testing Results ==
Ad-hoc testing around testcases and scenarios above in regular and private windows, in e10s and non-e10s windows and on normal and HiDPi displays.
=== Nightly ===
{| class="wikitable" style="width:60%"
|-
! Platform
! IO disabled
! IO enabled
! QA'd by : build ID
|-
| Windows
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Tracy : 20170228030203
|-
| Mac
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Tracy : 20170228030203
|-
| Linux
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Tracy : 20170301110155
|-
| Android
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Pass
| style="background-color:#CCFF99;" | Tracy : 20170301110150
|}
=== Aurora ===
{| class="wikitable" style="width:60%"
|-
! Platform
! IO disabled
! IO enabled
! QA'd by : build ID
|-
| Windows
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | TBD
|-
| Mac
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | TBD
|-
| Linux
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | TBD
|-
| Android
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | TBD
|}
=== Beta ===
{| class="wikitable" style="width:60%"
|-
! Platform
! IO disabled
! IO enabled
! QA'd by : build ID
|-
| Windows
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | TBD
|-
| Mac
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | TBD
|-
| Linux
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | TBD
|-
| Android
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | (results)
| style="background-color:#CEF2F2;" | TBD
|}


= Project Information =
= Project Information =
Line 169: Line 254:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | ||  
|style="text-align:center;" | 20160128 ||   tbd
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
|style="text-align:center;" | ||  
|style="text-align:center;" | 20170201 ||   20170210
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | ||  
|style="text-align:center;" | 20170210 ||   20170215
|-
|-
| QA - Test cases/Env preparation
| QA - Nightly Testing
|style="text-align:center;" | ||  
|style="text-align:center;" | 20170227 ||   20170301
|-
|-
| QA - Nightly Testing
| Telemetry - Stability Experiment
|style="text-align:center;" | ||  
|style="text-align:center;" | 20170304?? ||   tbd
|-
|-
| QA - Aurora Testing
| QA - Aurora Testing
|style="text-align:center;" | ||   
|style="text-align:center;" | tbd ||   
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | tbd ||  
|-
|-
| Release Date  
| Release Date  
|style="text-align:center;" | ||  
|style="text-align:center;" | tbd ||  
|}
|}
 
== References ==
== References ==
* List and links for specs
* List and links for specs

Latest revision as of 17:46, 3 March 2017

Overview

Purpose

This wiki details the testing that will be performed by QA for the Intersection Observer API. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • Areas of risk
  • What will be tested
  • How testing will be performed
    • criteria
    • ownership
    • schedule

To help ensure the best possible release and long term maintenance of this feature.

Risk Analysis

Risk area Requirement Status
Ad intensive web pages No regression in interaction performance PASS on Nightly 54
Security Fuzz testing finds no security holes PASS on Nightly 54
Stability Crash rate does not increase TBD

Testing Strategy

Scope

Test Areas Covered
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes
Stability Yes
UI
Display (HiDPI) Yes
Interraction (scroll, zoom) Yes
Add-ons
ad-blocking Add-ons must be disabled/removed to test Yes
Security
Fuzzing Yes
Web Compatibility
Testing against target sites Yes

If it's not listed above, it is currently out of scope.

Objectives

Criteria Description Metric w/o IO API w/ IO API Criteria Met? QA Owner
Manual testing Interaction performance Pass Pass Yes on Nightly 54 Tracy
Unit testing Automated tests Pass Pass Yes on Nightly builds of 54 Tobias
Fuzz testing Automated tests Pass Pass [1] Yes on Nightly 54 Fuzzing team (Raymond)
Stability Telemetry/Crash reports tbd - telemetry experiment in bug 1341259 tbd - telemetry experiment in bug 1341259 soon to begin with 10% enabled Tobias 
[1] tracking bug has no bugs hanging off of it

Testcases

Ad-hoc testing of Ad-intensive websites. In general news sites tent to be rich of Ads. Therefore the top news sites on Alexa are a great source for suitable websites.

[1] The top 500 news sites on the web

List of known sites using the Intersection Observer API:

Possible test scenarios:

  • Open a single news site. Wait till it is fully loaded to make sure all Ads are properly initialized. If there is content to scroll, scroll up and down a couple of times. Click on some articles to trigger a refresh.
  • Open a multiple news sites in different tabs. Wait till every page is fully loaded to make sure all Ads are properly initialized. Switch between tabs from time too time. Close and open new tabs randomly. If there is content to scroll, scroll up and down a couple of times. Click on some articles to trigger a refresh.
  • Repeat scenarios above but let the browser untouched for a longer period of time. Then proceed.

Environments

Full Testing will be performed using Nightly for Desktop builds on:

  • Windows
  • Mac OS X
  • Linux

Full Testing will be performed using Nightly for Mobile builds on:

  • Android

Channel dependent settings (configs) and environment setups

  • In Nightly (and DevEd when available)- dom.IntersectionObserver.enabled needs to be set to "true" in about:config
  • If any kind of Ad-blocker is installed/in use it needs to be uninstalled/disabled

Manual Testing Results

Ad-hoc testing around testcases and scenarios above in regular and private windows, in e10s and non-e10s windows and on normal and HiDPi displays.

Nightly

Platform IO disabled IO enabled QA'd by : build ID
Windows Pass Pass Tracy : 20170228030203
Mac Pass Pass Tracy : 20170228030203
Linux Pass Pass Tracy : 20170301110155
Android Pass Pass Tracy : 20170301110150

Aurora

Platform IO disabled IO enabled QA'd by : build ID
Windows (results) (results) TBD
Mac (results) (results) TBD
Linux (results) (results) TBD
Android (results) (results) TBD

Beta

Platform IO disabled IO enabled QA'd by : build ID
Windows (results) (results) TBD
Mac (results) (results) TBD
Linux (results) (results) TBD
Android (results) (results) TBD

Project Information

Ownership

Developer contacts: Tobias Schneider
QA: Tracy Walker
QA Peer: TBD

Builds

This section should contain links for builds with the feature

Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 20160128 tbd
Study documentation/specs received from developers 20170201 20170210
QA - Test plan creation 20170210 20170215
QA - Nightly Testing 20170227 20170301
Telemetry - Stability Experiment 20170304?? tbd
QA - Aurora Testing tbd
QA - Beta Testing tbd
Release Date tbd

References

  • List and links for specs
    • [2] Intersection Observer Spec
    • [3] Intersection Observer API
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/IntersectionObserver&oldid=1164524"