QA/Async Drawing Test Plan

From MozillaWiki
< QA
Jump to navigation Jump to search

Overview

Purpose

Quality assurance plan to ensure Flash content on Windows with asyncDrawing enabled is ready for public release.

Quality Criteria

Risk area Requirement Status
Flash videos and apps No significant regression in site correctness, video performance or app/gaming performance 49 Total Bugs, 7 Open (14.29%), 2 Blockers remain
General performance Overall performance of asyncDrawing enabled Firefox Adobe Flash Player should not be notably worse than with asyncDrawing not enabled TBD

Testing summary

Scope of Testing

In Scope

The scope of our testing is the async drawing functionality and performance of the most popular sites and games with latest Adobe Flash Player.

  • Integration: Verify the integration with the current browser functionalities and UI;
  • Functionality: Basic and advanced functionality to be verified according to the existing requirements;
  • Performance: Reference, where applicable, observed and collected performance data.

Out of Scope

We will not be testing on obscure web sites nor in conjunction with popular add-ons

Requirements for testing

Environments

Testing will be performed on following OSes:

  • Windows 10 (64bit and 32bit)
  • Windows 7

Quality Assurance Strategy

Test Items

Flash Video and Apps

Criteria Description Metric asyncDrawing disabled asyncDrawing enabled Criteria Met? QA Owner
Manual testing Test cases passed 100%[1] 49 Total Bugs, 7 Open (14.29%), 2 Blockers remain Bug List StephanG

Bug Triage Spreadsheet - Includes results from various SoftVision testing rounds and analysis.

[1] All open bugs triaged against both settings. Any bug that was reproducible in both modes removed from bug lists.

General Performance

Acceptable regression ranges, if any, need to be determined through testing. In general, where slow path bitmap based painting is in use (wmode=windowless) we have always had performance issues. As such those sites do not play into our decision making here. We now convert all flash that defaults or is set to use window mode, to direct. In general we haven't seen major performance issues on these sites.

Criteria Description Metric asyncDrawing disabled asyncDrawing enabled Criteria Met? QA Owner
CPU usage (observed) Average % CPU results results PASS tracy
Memory usage (observed) Peak memory usage results results PASS tracy

Builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project December 2016 June 2017
QA - Test plan creation 20170120 20170207
QA - Test cases/Env preparation 20170120 20170127
QA - Nightly Testing Dec. 2016 Mar. 2017
QA - Beta Testing April 2017 June 2017
Release Date 2017-06-13

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail
Test case execution TestRail
Bugs management Bugzilla

Status

Overview

  • Track the dates and build number where feature was released to Nightly
  • Track the dates and build number where feature was merged to Aurora
  • Track the dates and build number where feature was merged to Release/Beta

References

Early testing tracked here

Testcases

Available on TestRail or Google Doc format

Overview

  • Summary of testing scenarios

Test suite

  • Full Test suite - Test Rail - (google doc)
    • We should make sure the full test suite includes bugs that have whiteboard STR from the list below

Bug Work


  • Bugzilla logged bugs -

Main List

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 403 [1] => Forbidden ) ) 1


Sign off

Criteria

Check list

  • All Criteria under each section of Quality Assurance Strategy should be green.
  • All test cases should be executed
  • All blockers, critical bugs must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) Done
Testing Infrastructure setup None
Test Plan Creation Done
Test Cases Creation Done
Full Functional Tests Execution Done
Automation Coverage N/A
Performance Testing [IN PROGRESS]
All Defects Logged Done
Critical/Blockers Fixed and Verified [IN PROGRESS]
Metrics/Telemetry Beta
QA Signoff - Nightly Release Email to be sent
QA Beta - Full Testing Email to be sent
QA Signoff - Beta Release Email to be sent

Ownership

Product contact:

Engineering contact:
Jim Mathies

QA contact:
Stefan Georgiev
Tracy Walker (IRC: tracy)

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Async_Drawing_Test_Plan&oldid=1172649"