QA/Async Drawing Test Plan

From MozillaWiki
< QA(Redirected from QA/asyncDrawing)
Jump to: navigation, search

Overview

Purpose

Quality assurance plan to ensure Flash content on Windows with asyncDrawing enabled is ready for public release.

Quality Criteria

Risk area Requirement Status
Flash videos and apps No significant regression in site correctness, video performance or app/gaming performance 49 Total Bugs, 7 Open (14.29%), 2 Blockers remain
General performance Overall performance of asyncDrawing enabled Firefox Adobe Flash Player should not be notably worse than with asyncDrawing not enabled TBD

Testing summary

Scope of Testing

In Scope

The scope of our testing is the async drawing functionality and performance of the most popular sites and games with latest Adobe Flash Player.

  • Integration: Verify the integration with the current browser functionalities and UI;
  • Functionality: Basic and advanced functionality to be verified according to the existing requirements;
  • Performance: Reference, where applicable, observed and collected performance data.

Out of Scope

We will not be testing on obscure web sites nor in conjunction with popular add-ons

Requirements for testing

Environments

Testing will be performed on following OSes:

  • Windows 10 (64bit and 32bit)
  • Windows 7

Quality Assurance Strategy

Test Items

Flash Video and Apps

Criteria Description Metric asyncDrawing disabled asyncDrawing enabled Criteria Met? QA Owner
Manual testing Test cases passed 100%[1] 49 Total Bugs, 7 Open (14.29%), 2 Blockers remain Bug List StephanG

Bug Triage Spreadsheet - Includes results from various SoftVision testing rounds and analysis.

[1] All open bugs triaged against both settings. Any bug that was reproducible in both modes removed from bug lists.

General Performance

Acceptable regression ranges, if any, need to be determined through testing. In general, where slow path bitmap based painting is in use (wmode=windowless) we have always had performance issues. As such those sites do not play into our decision making here. We now convert all flash that defaults or is set to use window mode, to direct. In general we haven't seen major performance issues on these sites.

Criteria Description Metric asyncDrawing disabled asyncDrawing enabled Criteria Met? QA Owner
CPU usage (observed) Average % CPU results results PASS tracy
Memory usage (observed) Peak memory usage results results PASS tracy

Builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project December 2016 June 2017
QA - Test plan creation 20170120 20170207
QA - Test cases/Env preparation 20170120 20170127
QA - Nightly Testing Dec. 2016 Mar. 2017
QA - Beta Testing April 2017 June 2017
Release Date 2017-06-13

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail
Test case execution TestRail
Bugs management Bugzilla

Status

Overview

  • Track the dates and build number where feature was released to Nightly
  • Track the dates and build number where feature was merged to Aurora
  • Track the dates and build number where feature was merged to Release/Beta

References

Early testing tracked here

Testcases

Available on TestRail or Google Doc format

Overview

  • Summary of testing scenarios

Test suite

  • Full Test suite - Test Rail - (google doc)
    • We should make sure the full test suite includes bugs that have whiteboard STR from the list below

Bug Work


  • Bugzilla logged bugs -

Main List

Full Query
ID Summary Status Resolution Whiteboard
1217665 Revive async drawing code for plugins RESOLVED FIXED
1232191 Flash crash in @ F1672456644 & @ F_1373596331 RESOLVED FIXED
1240220 Flash 20.0.0.285 (beta) sometimes crashes when HWA is enabled in Flash settings RESOLVED INVALID [fixed in Flash 20.0.0.286 (stable)]
1245200 Plugin crash spike in 45.0b1 RESOLVED WORKSFORME
1271398 [Regression] Problems with Adobe Flash Player 22 Stage3D and Firefox x64 >=v48 RESOLVED FIXED Try using Flash 23 Beta on Firefox 49+
1276020 [e10s] Flash content does not appear RESOLVED FIXED STR in comment 0
1276426 Flash video stops when switching between tabs RESOLVED DUPLICATE hold, STR in comment #0
1306698 When resize content/window, image under the flash plug-in is briefly display REOPENED layout, STR in comment #0
1309306 Async Rendering Plugin 1.0 AddOn-Firefox 49.0.1-Adobe Flash 23.0 - Private Window RESOLVED INCOMPLETE hold
1311374 Video fails to update after exiting full screen and scrolling the video in and out of view RESOLVED WORKSFORME gfx, layout, STR in comment #0
1311524 Buttons are inaccessible in a new window with Flash (poker's website) RESOLVED WORKSFORME gfx
1311740 Since latest update of flash update, Firefox lags and freezes RESOLVED INCOMPLETE repo?
1311867 Flash content fails to paint initially. Content paints when focus is moved to another tab and then returned RESOLVED FIXED gfx, transparency dupe?, testcase in comment #3
1311928 the video player on fivethirtyeight.com would not work. RESOLVED FIXED adobe, STR in comment #0
1311975 Cannot interact with Flash Video Controls on certain site RESOLVED WORKSFORME gfx, STR in comment #0 (32bit only)
1311985 massive performance decrease with certain Kongregate games when using Flash async drawing mode RESOLVED DUPLICATE repo?
1311990 Painting issues with Farmville 2 (asyncdrawing) RESOLVED FIXED gfx, STR in comment #0
1312043 Paint issues with Comedy Central video player (asyncdrawing) RESOLVED FIXED gfx, STR in comment #0
1312208 Flash fails to paint on image upload site with async plugin painting RESOLVED INCOMPLETE hold
1312242 Missing transparency in Adobe Flash objects with wmode="transparent" RESOLVED WORKSFORME layout/gfx, tests, STR in comment #0 with attached test case
1312293 With async plugin painting Flash video on TVA website fails to paint / paints black RESOLVED FIXED gfx, STR in comment #0
1312342 The Flash menus of QQ mail do not display RESOLVED FIXED hold, STR in comment #0
1312369 dom.ipc.plugins.asyncdrawing.enabled changes mouse wheel behavior RESOLVED WONTFIX sitecompat
1312688 Cannot type some special characters in input text fields in Flash/Flex content with various keyboards (Polish, French) RESOLVED DUPLICATE input
1314467 Huge performance regression when Flash object is opened in many tabs RESOLVED WORKSFORME [parity-chrome] [parity-edge][qawanted , help gecko profile]
1314491 Kongregate's Flash game doesn't load with Flash protected mode disabled and plugin asyncdrawing enabled RESOLVED FIXED fixed by bug 1348629
1314935 Flash game would not respond with key / mouse RESOLVED FIXED [DUPEME], STR in comment #0
1315507 Rendering glitch in Adobe Flash objects with wmode="transparent" (hopscrotch) RESOLVED FIXED adobe [fixed by Flash 24.0.0.170 ], STR in comment #0
1316438 Flash plugin doesn't render most of display on Nightly with async painting enabled RESOLVED FIXED [gfx-noted], regresion from bug 1312242, STR in comment #0
1317995 Firefox 50 64-bit Stable channel is unable to play Farmville 2 Facebook Flash game. RESOLVED FIXED [platform-rel-Zynga]
1320631 Comedy Central Flash videos have white borders RESOLVED FIXED STR in comment #0
1323403 Flash Player on http://youngjump.jp/manga/kingdom/ doesn't work when Flash is windowless RESOLVED WORKSFORME [sitewait] [css] [country-jp]
1323750 Flash Stage3D fails on youngjump.jp in 64-bit Firefox RESOLVED DUPLICATE
1325118 Flash crashes in Windows debug builds RESOLVED WORKSFORME
1328039 Flash blinks when I change zoom level or scroll the page RESOLVED WONTFIX
1334415 Rendering glitch when drawing Flash content with async drawing disabled in Firefox 51/52 RESOLVED WORKSFORME
1335733 flash doesn't respond to mouse when window is maximized (Nightly regression) RESOLVED WORKSFORME
1339873 Comedy Central Flash videos playback is choppy RESOLVED WONTFIX
1339875 Video on fivethirtyeight.com plays back poorly RESOLVED WONTFIX
1340276 Force wmode=direct for certain Flash applets RESOLVED FIXED
1342092 Wheel input issues with Farmville 2 RESOLVED DUPLICATE
1344927 "Flight" game on addictinggames.com is not working properly with aSyncDrawing = true: can't click paper airline or Flash crashes RESOLVED WONTFIX
1344940 Scrolling Distorts Flash Game "Pirate the Treasures Return" RESOLVED WORKSFORME
1344941 Live Pool Facebook game is distorted when scrolling RESOLVED DUPLICATE
1345649 Enable plugin async drawing for flash on release channels RESOLVED DUPLICATE
1345988 Flash 64-bit only: Video fast-forward leads to a black screen on many sites... RESOLVED WORKSFORME
1347575 Disabling acceleration in flash breaks flash settings dialog on live.nicovideo.jp NEW
1347925 wmode direct flash reverts to windowed mode if browser hardware acceleration is disabled RESOLVED WONTFIX
1348629 Zero sized Flash content does not load on 64 bit Firefox RESOLVED WORKSFORME
1354154 Flash content enlarged enormously erratically UNCONFIRMED
1359886 Regression April 12th in accelerated Flash async plugin drawing use RESOLVED WORKSFORME
1360666 BigBlueButton Flash does not receive click events after switching tabs when 32-bit async drawing is enabled RESOLVED FIXED
1363876 Enable async drawing for 64-bit builds in 54 and higher RESOLVED FIXED
1366756 Let Flash async drawing rollout for 32-bit builds in 55 RESOLVED FIXED
1388326 VMware vSphere Web Client (Flash) is not working on Firefox 55 32-bit NEW
1391624 Flash Player failed display of content with Adobe Flex RESOLVED DUPLICATE
1392022 55.0.2: Streaming Audio Stutter When Doing Things in Other Tabs NEW
1393352 flash player: Font white on white background NEW
1394105 Firefox 55 - performance of Shockwave Flash content has dropped significantly, with web based flash running very slowly, disabling async drawing solves the issue RESOLVED WORKSFORME
1394408 [meta] tab switching bugs related to async painting NEW
1422021 Scrolling in flash applet moves page NEW
1423481 Tool tip changes color with FF Quantum and Flash Player UNCONFIRMED

62 Total; 9 Open (14.52%); 53 Resolved (85.48%); 0 Verified (0%);


Sign off

Criteria

Check list

  • All Criteria under each section of Quality Assurance Strategy should be green.
  • All test cases should be executed
  • All blockers, critical bugs must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) Done
Testing Infrastructure setup None
Test Plan Creation Done
Test Cases Creation Done
Full Functional Tests Execution Done
Automation Coverage N/A
Performance Testing [IN PROGRESS]
All Defects Logged Done
Critical/Blockers Fixed and Verified [IN PROGRESS]
Metrics/Telemetry Beta
QA Signoff - Nightly Release Email to be sent
QA Beta - Full Testing Email to be sent
QA Signoff - Beta Release Email to be sent

Ownership

Product contact:

Engineering contact:
Jim Mathies

QA contact:
Stefan Georgiev
Tracy Walker (IRC: tracy)