Overview
Purpose
Quality assurance plan to ensure Flash content on Windows with asyncDrawing enabled is ready for public release.
Quality Criteria
| Risk area |
Requirement |
Status
|
| Flash videos and apps |
No significant regression in site correctness, video performance or app/gaming performance |
49 Total Bugs, 7 Open (14.29%), 2 Blockers remain
|
| General performance |
Overall performance of asyncDrawing enabled Firefox Adobe Flash Player should not be notably worse than with asyncDrawing not enabled |
TBD
|
Testing summary
Scope of Testing
In Scope
The scope of our testing is the async drawing functionality and performance of the most popular sites and games with latest Adobe Flash Player.
- Integration: Verify the integration with the current browser functionalities and UI;
- Functionality: Basic and advanced functionality to be verified according to the existing requirements;
- Performance: Reference, where applicable, observed and collected performance data.
Out of Scope
We will not be testing on obscure web sites nor in conjunction with popular add-ons
Requirements for testing
Environments
Testing will be performed on following OSes:
- Windows 10 (64bit and 32bit)
- Windows 7
Quality Assurance Strategy
Test Items
Flash Video and Apps
| Criteria Description
|
Metric
|
asyncDrawing disabled
|
asyncDrawing enabled
|
Criteria Met?
|
QA Owner
|
| Manual testing
|
Test cases passed
|
100%[1]
|
49 Total Bugs, 7 Open (14.29%), 2 Blockers remain
|
Bug List
|
StephanG
|
Bug Triage Spreadsheet - Includes results from various SoftVision testing rounds and analysis.
[1] All open bugs triaged against both settings. Any bug that was reproducible in both modes removed from bug lists.
General Performance
Acceptable regression ranges, if any, need to be determined through testing. In general, where slow path bitmap based painting is in use (wmode=windowless) we have always had performance issues. As such those sites do not play into our decision making here. We now convert all flash that defaults or is set to use window mode, to direct. In general we haven't seen major performance issues on these sites.
| Criteria Description
|
Metric
|
asyncDrawing disabled
|
asyncDrawing enabled
|
Criteria Met?
|
QA Owner
|
| CPU usage (observed)
|
Average % CPU
|
results
|
results
|
PASS
|
tracy
|
| Memory usage (observed)
|
Peak memory usage
|
results
|
results
|
PASS
|
tracy
|
Builds
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
| Project phase |
Start Date |
End Date
|
| Start project
|
December 2016
|
June 2017
|
| QA - Test plan creation
|
20170120
|
20170207
|
| QA - Test cases/Env preparation
|
20170120
|
20170127
|
| QA - Nightly Testing
|
Dec. 2016
|
Mar. 2017
|
| QA - Beta Testing
|
April 2017
|
June 2017
|
| Release Date
|
|
2017-06-13
|
Testing Tools
Detail the tools to be used for testing, for example see the following table:
| Process |
Tool
|
| Test plan creation |
Mozilla wiki
|
| Test case creation |
TestRail
|
| Test case execution |
TestRail
|
| Bugs management |
Bugzilla
|
Status
Overview
- Track the dates and build number where feature was released to Nightly
- Track the dates and build number where feature was merged to Aurora
- Track the dates and build number where feature was merged to Release/Beta
References
Early testing tracked here
Testcases
Available on TestRail or Google Doc format
Overview
- Summary of testing scenarios
Test suite
- Full Test suite - Test Rail - (google doc)
- We should make sure the full test suite includes bugs that have whiteboard STR from the list below
Bug Work
Main List
Full Query
| ID |
Summary |
Status |
Resolution |
Whiteboard |
| 1217665 |
Revive async drawing code for plugins |
RESOLVED |
FIXED |
|
| 1232191 |
Flash crash in @ F1672456644 & @ F_1373596331 |
RESOLVED |
FIXED |
|
| 1240220 |
Flash 20.0.0.285 (beta) sometimes crashes when HWA is enabled in Flash settings |
RESOLVED |
INVALID |
[fixed in Flash 20.0.0.286 (stable)] |
| 1245200 |
Plugin crash spike in 45.0b1 |
RESOLVED |
WORKSFORME |
|
| 1271398 |
[Regression] Problems with Adobe Flash Player 22 Stage3D and Firefox x64 >=v48 |
RESOLVED |
FIXED |
Try using Flash 23 Beta on Firefox 49+ |
| 1276020 |
[e10s] Flash content does not appear |
RESOLVED |
FIXED |
STR in comment 0 |
| 1276426 |
Flash video stops when switching between tabs |
RESOLVED |
DUPLICATE |
hold, STR in comment #0 |
| 1306698 |
When resize content/window, image under the flash plug-in is briefly display |
RESOLVED |
WONTFIX |
layout, STR in comment #0 |
| 1309306 |
Async Rendering Plugin 1.0 AddOn-Firefox 49.0.1-Adobe Flash 23.0 - Private Window |
RESOLVED |
INCOMPLETE |
hold |
| 1311374 |
Video fails to update after exiting full screen and scrolling the video in and out of view |
RESOLVED |
WORKSFORME |
gfx, layout, STR in comment #0 |
| 1311524 |
Buttons are inaccessible in a new window with Flash (poker's website) |
RESOLVED |
WORKSFORME |
gfx |
| 1311740 |
Since latest update of flash update, Firefox lags and freezes |
RESOLVED |
INCOMPLETE |
repo? |
| 1311867 |
Flash content fails to paint initially. Content paints when focus is moved to another tab and then returned |
RESOLVED |
FIXED |
gfx, transparency dupe?, testcase in comment #3 |
| 1311928 |
the video player on fivethirtyeight.com would not work. |
RESOLVED |
FIXED |
adobe, STR in comment #0 |
| 1311975 |
Cannot interact with Flash Video Controls on certain site |
RESOLVED |
WORKSFORME |
gfx, STR in comment #0 (32bit only) |
| 1311985 |
massive performance decrease with certain Kongregate games when using Flash async drawing mode |
RESOLVED |
DUPLICATE |
repo? |
| 1311990 |
Painting issues with Farmville 2 (asyncdrawing) |
RESOLVED |
FIXED |
gfx, STR in comment #0 |
| 1312043 |
Paint issues with Comedy Central video player (asyncdrawing) |
RESOLVED |
FIXED |
gfx, STR in comment #0 |
| 1312208 |
Flash fails to paint on image upload site with async plugin painting |
RESOLVED |
INCOMPLETE |
hold |
| 1312242 |
Missing transparency in Adobe Flash objects with wmode="transparent" |
RESOLVED |
WORKSFORME |
layout/gfx, tests, STR in comment #0 with attached test case |
| 1312293 |
With async plugin painting Flash video on TVA website fails to paint / paints black |
RESOLVED |
FIXED |
gfx, STR in comment #0 |
| 1312342 |
The Flash menus of QQ mail do not display |
RESOLVED |
FIXED |
hold, STR in comment #0 |
| 1312369 |
dom.ipc.plugins.asyncdrawing.enabled changes mouse wheel behavior |
RESOLVED |
WONTFIX |
sitecompat |
| 1312688 |
Cannot type some special characters in input text fields in Flash/Flex content with various keyboards (Polish, French) |
RESOLVED |
DUPLICATE |
input |
| 1314467 |
Huge performance regression when Flash object is opened in many tabs |
RESOLVED |
WORKSFORME |
[parity-chrome] [parity-edge][qawanted , help gecko profile] |
| 1314491 |
Kongregate's Flash game doesn't load with Flash protected mode disabled and plugin asyncdrawing enabled |
RESOLVED |
FIXED |
fixed by bug 1348629 |
| 1314935 |
Flash game would not respond with key / mouse |
RESOLVED |
FIXED |
[DUPEME], STR in comment #0 |
| 1315507 |
Rendering glitch in Adobe Flash objects with wmode="transparent" (hopscrotch) |
RESOLVED |
FIXED |
adobe [fixed by Flash 24.0.0.170 ], STR in comment #0 |
| 1316438 |
Flash plugin doesn't render most of display on Nightly with async painting enabled |
RESOLVED |
FIXED |
[gfx-noted], regresion from bug 1312242, STR in comment #0 |
| 1317995 |
Firefox 50 64-bit Stable channel is unable to play Farmville 2 Facebook Flash game. |
RESOLVED |
FIXED |
[platform-rel-Zynga] |
| 1320631 |
Comedy Central Flash videos have white borders |
RESOLVED |
FIXED |
STR in comment #0 |
| 1323403 |
Flash Player on http://youngjump.jp/manga/kingdom/ doesn't work when Flash is windowless |
RESOLVED |
WORKSFORME |
[sitewait] [css] [country-jp] |
| 1323750 |
Flash Stage3D fails on youngjump.jp in 64-bit Firefox |
RESOLVED |
DUPLICATE |
|
| 1325118 |
Flash crashes in Windows debug builds |
RESOLVED |
WORKSFORME |
|
| 1328039 |
Flash blinks when I change zoom level or scroll the page |
RESOLVED |
WONTFIX |
|
| 1334415 |
Rendering glitch when drawing Flash content with async drawing disabled in Firefox 51/52 |
RESOLVED |
WORKSFORME |
|
| 1335733 |
flash doesn't respond to mouse when window is maximized (Nightly regression) |
RESOLVED |
WORKSFORME |
|
| 1339873 |
Comedy Central Flash videos playback is choppy |
RESOLVED |
WONTFIX |
|
| 1339875 |
Video on fivethirtyeight.com plays back poorly |
RESOLVED |
WONTFIX |
|
| 1340276 |
Force wmode=direct for certain Flash applets |
RESOLVED |
FIXED |
|
| 1342092 |
Wheel input issues with Farmville 2 |
RESOLVED |
DUPLICATE |
|
| 1344927 |
"Flight" game on addictinggames.com is not working properly with aSyncDrawing = true: can't click paper airline or Flash crashes |
RESOLVED |
WONTFIX |
|
| 1344940 |
Scrolling Distorts Flash Game "Pirate the Treasures Return" |
RESOLVED |
WORKSFORME |
|
| 1344941 |
Live Pool Facebook game is distorted when scrolling |
RESOLVED |
DUPLICATE |
|
| 1345649 |
Enable plugin async drawing for flash on release channels |
RESOLVED |
DUPLICATE |
|
| 1345988 |
Flash 64-bit only: Video fast-forward leads to a black screen on many sites... |
RESOLVED |
WORKSFORME |
|
| 1347575 |
Disabling acceleration in flash breaks flash settings dialog on live.nicovideo.jp |
RESOLVED |
WONTFIX |
|
| 1347925 |
wmode direct flash reverts to windowed mode if browser hardware acceleration is disabled |
RESOLVED |
WONTFIX |
|
| 1348629 |
Zero sized Flash content does not load on 64 bit Firefox |
RESOLVED |
WORKSFORME |
|
| 1354154 |
Flash content enlarged enormously erratically |
RESOLVED |
WONTFIX |
|
| 1359886 |
Regression April 12th in accelerated Flash async plugin drawing use |
RESOLVED |
WORKSFORME |
|
| 1360666 |
BigBlueButton Flash does not receive click events after switching tabs when 32-bit async drawing is enabled |
RESOLVED |
FIXED |
|
| 1363876 |
Enable async drawing for 64-bit builds in 54 and higher |
RESOLVED |
FIXED |
|
| 1366756 |
Let Flash async drawing rollout for 32-bit builds in 55 |
RESOLVED |
FIXED |
|
| 1388326 |
VMware vSphere Web Client (Flash) is not working on Firefox 55 32-bit |
RESOLVED |
WONTFIX |
|
| 1391624 |
Flash Player failed display of content with Adobe Flex |
RESOLVED |
DUPLICATE |
|
| 1392022 |
55.0.2: Streaming Audio Stutter When Doing Things in Other Tabs |
RESOLVED |
WONTFIX |
|
| 1393352 |
flash player: Font white on white background |
RESOLVED |
INCOMPLETE |
|
| 1394105 |
Firefox 55 - performance of Shockwave Flash content has dropped significantly, with web based flash running very slowly, disabling async drawing solves the issue |
RESOLVED |
WORKSFORME |
|
| 1394408 |
[meta] tab switching bugs related to async painting |
RESOLVED |
WONTFIX |
|
| 1422021 |
Scrolling in flash applet moves page |
RESOLVED |
WONTFIX |
|
| 1423481 |
Tool tip changes color with FF Quantum and Flash Player |
RESOLVED |
INCOMPLETE |
|
62 Total;
0 Open (0%);
62 Resolved (100%);
0 Verified (0%);
Sign off
Criteria
Check list
- All Criteria under each section of Quality Assurance Strategy should be green.
- All test cases should be executed
- All blockers, critical bugs must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Checklist
| Exit Criteria |
Status |
Notes/Details
|
| Testing Prerequisites (specs, use cases)
|
Done
|
|
| Testing Infrastructure setup
|
None |
|
| Test Plan Creation
|
Done |
|
| Test Cases Creation
|
Done |
|
| Full Functional Tests Execution
|
Done |
|
| Automation Coverage
|
N/A |
|
| Performance Testing
|
[IN PROGRESS] |
|
| All Defects Logged
|
Done |
|
| Critical/Blockers Fixed and Verified
|
[IN PROGRESS] |
|
| Metrics/Telemetry |
|
Beta
|
| QA Signoff - Nightly Release |
|
Email to be sent
|
| QA Beta - Full Testing |
|
Email to be sent
|
| QA Signoff - Beta Release |
|
Email to be sent
|
Ownership
Product contact:
Engineering contact:
Jim Mathies
QA contact:
Stefan Georgiev
Tracy Walker (IRC: tracy)
