QA/Shutdown Decoders
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
| Date | Version | Author | Description |
|---|---|---|---|
| 10/17/2016 | 1.0 | Adrian Florinescu | Initial Draft |
| 03/29/2017 | 1.1 | Adrian Florinescu | Updates |
| 07/21/2017 | 1.2 | Bogdan Maris | Updates |
Contents
Overview
Purpose
Detail the purpose of this document. For example:
- The test scope, focus areas and objectives
- The test responsibilities
- The test strategy for the levels and types of test for this release
- The entry and exit criteria
- The basis of the test estimates
- Any risks, issues, assumptions and test dependencies
- The test schedule and major milestones
- The test deliverables
Scope
This wiki details the testing that will be performed by the project team for the Shutdown Decoders project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
- What will be tested
- How testing will be performed
Ownership
• Product/Program/Development Management:
- Product Manager (Fennec) - Joe Cheng
- Program Manager (Fennec) - Wesley Huang
- Program Manager (Media) - Bobby Chien
- Dev Manager (Media) - Blake Wu
• Engineering Team:
• UX:
• QA
Feature short description
The main purpose of the Shutdown Decoders feature is to reduce CPU/GPU & memory usage, and it does so by suspending the video element's video decoder when the video element is in background tabs or is invisible even in the foreground tab. The mechanism is that, when a video element is invisible, the original video decoder is replaced with a blank video decoder which only produces white frames that have the right resolution and time information. The original video decoder is released and the the CPU/GPU & memory usage is reduced.
Testing summary
Scope of Testing
In Scope
Testing is concentrated around the 2 scheduled development phases as follows:
- Phase 1: verify that a a blank decoder is used to replace the video decoder. In this phase, the mechanism is applied to:
- Videos with both audio and video tracks
- Videos without audio track
- Phase 2: enhance the mechanism and make sure it can apply to all video files, including tainted videos.
Out of Scope
Detail what is out of scope from a testing perspective for the project team. Note: if usability testing is not in the scope of testing feature.
Requirements for testing
Environments
Full testing will be performed on the following OSes:
- Windows 10
- Mac OS X
- Ubuntu
Test Strategy
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
| Ref | Function | Test Objective | Test Type | RAC | Owners |
|---|---|---|---|---|---|
| TO-1 | Video track is paused when tab containing it loses focus. | Video track is paused and CPU/GPU usage decreases | Manual | - | Eng Team |
| TO-2 | Stopped video track is resumed and synced to the sounds track when the tab containing it is focused. | Video is synced to sound and resumed when tab is refocused | Manual | RAC-01 , RAC-03, RAC-04, RAC-09 | Eng Team |
| TO-3 | Video track is paused for all 3 supported formats for <video> html element (mp4, webm, ogg) | Encoding formats are not affecting video/sound capabilities | Manual | RAC-05 | Eng Team |
| TO-4 | Telemetry is recorded related to the recovery of the video-decode-suspend | Telemetry data is recorded | Manual | - | Eng Team |
| TO-5 | High Resolution >480p videos are covered by Shutdown decoder | High Resolution videos are not lagging | Manual | - | Eng Team |
| TO-6 | Video and sound only videos work properly | FF works properly on video only and sound only tracks | Manual | RAC -6 | Eng Team |
| TO-7 | Performance wise, the pause and resume in case of multiple frames/videos is acceptable and doesn't crash the media container | TBD what are the expectations in terms of "acceptable" | Manual | RAC-07 | Eng Team |
Risk Assessment and Coverage
| ID | Description / Threat Description | Covered by Test Objective | Magnitude | Probability | Priority | Impact Score |
|---|---|---|---|---|---|---|
| RAC-1 | Visibility of video element | TO-2 | 2-Moderate | 2-Possible | 2-Medium | 8 |
| RAC-2 | When Video is resumed, the delay is too big | TO-7 | 3-High | 3-Almost Certain | 3-High | 27 |
| RAC-3 | On focus back, the sync between sound and video is not successful | TO-2 | 3-High | 3-Almost Certain | 3-High | 27 |
| RAC-4 | Video decoder is not accessible by the FF process | TO-2 | 1-Unlikely | 2-Possible | 2-Medium | 4 |
| RAC-5 | Not all video formats are supported correctly | TO-3 | 2-Moderate | 2-Possible | 3-High | 12 |
| RAC-6 | Sounds only "videos" are not correctly managed when talking about suspend decoder | TO-6 | 1-Unlikely | 2-Possible | 2-Medium | 4 |
| RAC-7 | Videos from multiple frames are not handled correctly (visibility concern) | TO-8 | 3-High | 2-Possible | 3-High | 12 |
| RAC-8 | Possible Performance issues with multiple tabs or multiple videos - also behavior on High loaded GPU/CPU | TO-8 | 3-High | 3-Almost Certain | 3-High | 27 |
| RAC-9 | Transition effect might not always be user friendly | TO-2 | 3-High | 3-Almost Certain | 3-High | 27 |
Values:
- Magnitude: 1- Low , 2-Moderate, 3-High
- Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
- Priority: 1 - Low, 2-Medium, 3-High
Impact Score Breakdown:
- An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
- An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
- An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
Builds
This section should contain links for builds with the feature -
- Links for Nightly builds
- Links for Aurora builds
- Links for Beta builds
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
| Project phase | Start Date | End Date |
|---|---|---|
| Start project | 10.03.2016 | |
| Study documentation/specs received from developers | 10.11.2016 | 10.17.2016 |
| QA - Test plan creation | 10.18.2016 | 10.20.2016 |
| QA - Test cases/Env preparation | 10.20.2016 | |
| QA - Nightly Testing | ||
| QA - Aurora Testing | ||
| QA - Beta Testing | 06.15.2017 | 07.21.2017 |
| Release Date |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | TestRail |
| Test case execution | TestRail |
| Bugs management | Bugzilla |
Status
Overview
Track the dates and build number where feature was released to Nightly Track the dates and build number where feature was merged to Aurora Track the dates and build number where feature was merged to Release/Beta
References
- List and links for specs
https://wiki.mozilla.org/Firefox/Shutdown_Decoders https://mozilla.invisionapp.com/share/K48PCVSEM
- bug 1293963 - [Meta] Suspend-video-decoder: phase-1 shipping
17 Total; 1 Open (5.88%); 13 Resolved (76.47%); 3 Verified (17.65%);
Testcases
Full testsuites can be found on TestRail at link
Test Areas
| Test Areas | Covered | Details |
|---|---|---|
| Private Window | Yes | |
| Multi-Process Enabled | Yes | |
| Multi-process Disabled | Yes | |
| Theme (high contrast) | Yes | |
| UI | ||
| Mouse-only operation | Yes | |
| Keyboard-only operation | Yes | |
| Display (HiDPI) | Yes | |
| Interraction (scroll, zoom) | No | |
| Usable with a screen reader | No | e.g. with NVDA |
| Usability and/or discoverability testing | Yes | Is this feature user friendly |
| RTL build testing | No | |
| Help/Support | ||
| Help/support interface required | TBD | Make sure link to support/help page exist and is easy reachable. |
| Support documents planned(written) | No | Make sure support documents are written and are correct. |
| Install/Upgrade | ||
| Feature upgrades/downgrades data as expected | No | |
| Does sync work across upgrades | No | |
| Requires install testing | No | separate feature/application installation needed (not only Firefox) |
| Affects first-run or onboarding | TBD | Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned. |
| Does this affect partner builds? Partner build testing | No | yes/no options, add comment with details about who will lead testing |
| Enterprise | Raise up the topic to developers to see if they are expecting to work different on ESR builds | |
| Enterprise administration | No | |
| Network proxies/autoconfig | TBD | |
| ESR behavior changes | No | |
| Locked preferences | TBD | |
| Data Monitoring | ||
| Temporary or permanent telemetry monitoring | Yes | List of error conditions to monitor |
| Telemetry correctness testing | No | |
| Server integration testing | No | |
| Offline and server failure testing | Yes | |
| Load testing | Yes | |
| Add-ons | If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on. | |
| Addon API required? | No | |
| Comprehensive API testing | No | |
| Permissions | No | |
| Testing with existing/popular addons | No | |
| Security | Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature. | |
| 3rd-party security review | TBD | |
| Privilege escalation testing | No | |
| Fuzzing | No | |
| Web Compatibility | depends on the feature | |
| Testing against target sites | Yes | |
| Survey of many sites for compatibility | Yes | |
| Interoperability | depends on the feature | |
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | TBD | |
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | No | |
| Interaction of this feature with other browser features | No |
Bug Work
Meta bug: 1293963- (Phase_1_Shutdown_Decoder) META Suspend-video-decoder: phase-1 shipping
Logged bugs ( blocking 1293963 )
17 Total; 1 Open (5.88%); 13 Resolved (76.47%); 3 Verified (17.65%);
Bug fix verification
12 Total; 0 Open (0%); 9 Resolved (75%); 3 Verified (25%);
Sign off
Criteria
Check list
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
List of OSes that will be covered by testing
- Link for the tests run
Merge to Aurora Sign-off
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite
Checklist
| Exit Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | ||
| Testing Infrastructure setup | ||
| Test Plan Creation | ||
| Test Cases Creation | ||
| Full Functional Tests Execution | ||
| Automation Coverage | ||
| Performance Testing | ||
| All Defects Logged | ||
| Critical/Blockers Fixed and Verified | ||
| Metrics/Telemetry | ||
| [Desktop] SV Pre-Aurora 54 Signoff | Email to be sent | |
| [Desktop] SV Pre-Beta 55 Signoff | GREEN by Apr 24, 2017 | Email to be sent |
| [Desktop] SV Pre-Release 55 Signoff | GREEN in July 21, 2017 | Email sent |
| [Mobile] SV Pre-Aurora 54 Signoff | GREEN by Mar 21, 2017 | Email to be sent |
| [Mobile] SV Pre-Beta 55 Signoff | YELLOW (DO NOT SHIP IT) by May 18, 2017 | Email to be sent |
| [Mobile] SV Pre-Release 55 Signoff | Email to be sent | |
| Release Signoff | Email to be sent |
