Overview

Suspending video element's video decoder, when the video element is in background tabs or is invisible even in the foreground tab, is a Firefox feature that reduces CPU/GPU & memory usage.

The mechanism is that, when a video element is invisible, we replace its original video decoder with blank video decoder which only produces white frames with right resolution and right time information. The original video decoder is released and the black video decoder is light so that we reduce CPU/GPU & memory usage.

The drawback is that, while the suspended-video-element is switched back to be visible again, we should resume its original video decoder. The resuming operation must be asynchronous and might be time-consuming which depends on the resolution of the video file and whether it contains audio tracks or not.

In the prototype (Phase 0), we have enabled this feature on the Firefox Nightly channel for any video element. We also add telemetry to collect needed information, especially on the resuming time. Currently (Phase 1), we are going to enable this feature on the Firefox Release channel for videos that is able to be resumed quickly and the criteria is 1) videos without audio track or 2) videos with both audio and video tracks but with low resolution (480P for now). In the future (Phase 2), our goal is to enable this feature on all videos without observable latency while resuming.

The following is a step-by-step description of suspending decoder working flow.

At the very beginning of the decoding framework, the raw media data is sent to demuxer.
Demuxer helps to separate a combined signal, e.g., a streaming can be separated into audio data and video data by demuxer. After that, audio and video data are sent to audio and video decoder separately.
In data decoding period, audio decoder keeps working as normal because a user may be listening to the music. But, Firefox uses a blank video decoder to replace current video decoder if the video element is invisible.

Key Documents

Issues: Meta Bug - Bug 1276556 [META] Tracking enable of background tab video decoder suspend

Other Resources

Meeting Minute

Status

Phases

Program Status

Milestone	Date	Status
Phase 1		WORKING ON UX SPEC
Phase 2		NOT STARTED

Phase 1: Using a blank video decoder to replace video decoder instead of shutdown decoders directly. In this phase, the mechanism is applied to 1) Low-resolution video (480P) and 2) Video film without audio track
Phase 2: We're going to enhance the mechanism and make sure it can apply to all video files.

Testing

Test cases: Phase 1

Bug Tracking

MVP Scope

bug 1276556 - [META] Tracking enable of background tab video decoder suspend

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Telemetry

Amount of time hidden - measure of user value (Bucket results by resolution; i.e. are 720p videos hidden less often?)

Recovery time - measure of user cost (separate for noisy vs silent videos) (Bucket results by resolution; i.e. do 720p videos take longer to recover?)
- Bug 1294349 - Telemetry to support background video decoder suspend: Recovery time from video-decode-suspended
- VIDEO_SUSPEND_RECOVERY_TIME_MS

Key frame spacing - distribution allows better tuning

Mochitests

Team

Product owner:

Eng: Alastor, Daniel, Gerald, JW, Kaku,

Program Management: Blake, Josh

UX: Mark, Morpheus

QA: SoftVision and William

Reference

Decoder resuming latency

While the suspended-video-element is switched back to be visible again, we should resume its original video decoder. The resuming operation must be asynchronous since we don't want to block the main thread and might be time-consuming which depends on the resolution of the video file and whether it contains audio tracks or not.

Currently, we have no way to boost the resuming time, however, we have telemetries for collecting the needed time of different resolutions on different platforms.

Related telemetries:

VIDEO_SUSPENDED_RECOVERY_TIME_MS

Related bugs:

Bug 1272919 - Video shows the frame that was on screen when I switched from youtube to a different tab for a fraction of a second

Video element as content source

The situation is about the video element itself is used as a content source and its decoder might be suspended at the same time. This issue could be split into two cases:

Case 1: The decoder is suspended AFTER the {cpatureStream(), drawImage(), createImageBitmap()} is called.
Case 2: The decoder is suspended BEFORE the {cpatureStream(), drawImage(), createImageBitmap()} is called.

Case 1 is relatively easy, we could mark the video element as being a content source while {cpatureStream(), drawImage(), createImageBitmap()} is invoked and then its decoder should never be suspended.

Case 2 is rather complicated. The critical issue is that resuming decoder is an async operation with latency. So, while {cpatureStream(), drawImage(), createImageBitmap()} is invoked on an already-suspended-video, there must be several "blank" frames been leaked, even though we resume the decoder immediately. To completely solve this problem, we must make "resuming decoder" a blocking operation, however, it might block the main thread; otherwise, we leak the blank frames.