Firefox/Block Playback

Overview

Block Playback Video/Audio will be blocked if a new tab been opened but never bring to front.

Meta Bug

• Bug 1262053

Other Resources

• Meeting Minute

Known Issues

Related telemetries:

• VIDEO_SUSPENDED_RECOVERY_TIME_MS

Related bugs:

• Bug 1272919 - Video shows the frame that was on screen when I switched from youtube to a different tab for a fraction of a second

Video element as content source

The situation is about the video element itself is used as a content source and its decoder might be suspended at the same time. This issue could be split into two cases:

• Case 1: The decoder is suspended AFTER the {cpatureStream(), drawImage(), createImageBitmap()} is called.
• Case 2: The decoder is suspended BEFORE the {cpatureStream(), drawImage(), createImageBitmap()} is called.

Case 1 is relatively easy, we could mark the video element as being a content source while {cpatureStream(), drawImage(), createImageBitmap()} is invoked and then its decoder should never be suspended.

Case 2 is rather complicated. The critical issue is that resuming decoder is an async operation with latency. So, while {cpatureStream(), drawImage(), createImageBitmap()} is invoked on an already-suspended-video, there must be several "blank" frames been leaked, even though we resume the decoder immediately. To completely solve this problem, we must make "resuming decoder" a blocking operation, however, it might block the main thread; otherwise, we leak the blank frames.

Related bugs:

• Bug 1284389 - Don't suspend video elements captured via mozCaptureStream()
• Bug 1295921 - Don't suspend video decoder for elements as sources to drawImage() and createImageBitmap()

Planning

Goals

• Help users to reduce device resource usage (CPU & Memory)
• Finish tasks across videos
• No impact on current user flow

Feature Plan

• Phase 0: Shutdown decoders when video element is invisible
• Phase 1: Using a blank video decoder to replace video decoder instead of shutdown decoders directly. In this phase, the mechanism is applied to 1) Low-resolution video (480P) and 2) Video film without audio track
• Phase 2: We're going to enhance the mechanism and make sure it can apply to all video files.

Success

• Effective cross-discipline teams solving problems across platforms
• Validation of key assumptions through telemetrics
• video decode suspend in the hands of all our users

Engineering

Block Playback [status: Green]

Current Goals:

• Ship disabling video decoders for ads
• Silent, looping videos where user doesn't really care the exact point

Next Milestone:

Milestones

Optimizations

• Bug 1282710 - Suspend and resume foreground video decoders according to visibility events
  • done.
• Bug 1282012 - Seek to nearest keyframe when resuming videos with no audio
  • Bug 1294657 - Seek to nearest keyframe when resuming videos with no audio - with audio track but muted
    • pending, excluded from phase 1.
  • Bug 1294658 - Seek to nearest keyframe when resuming videos with no audio - with audio track but might be silent
    • pending, excluded from phase 1.
  • Bug 1294656 - Seek to nearest keyframe when resuming videos with no audio - no audio track
    • done.
• Bug 1274919 - Resume suspended video decoders on tab mouse hover.
  • under review.
• Bug 1284389 - Don't suspend video elements captured via mozCaptureStream()
  • WIP.
• Bug 1295921 - Don't suspend video decoder for elements as sources to drawImage() and createImageBitmap()
  • WIP.
• Resume video when keyboard is used to change tags - Needs Bug
  • For example, if it's possible to detect the direction of cycling and videos are within, say, 5 tabs of the current tab, then start decoding again.
  • Need to hook into the tab navigation/change code and alert media elements, like bug 1274919

Telemetry

• Amount of time hidden - measure of user value (Bucket results by resolution; i.e. are 720p videos hidden less often?)
  • Bug 1285419 - Telemetry to support background video decoder suspend: Hidden play time
  • VIDEO_HIDDEN_PLAY_TIME_MS
  • Bug 1287987 - Telemetry to support background video decoder suspend: Percentage hidden/total play time, keyed by audio presence and height ranges
  • VIDEO_HIDDEN_PLAY_TIME_PERCENTAGE
  • Bug 1293145 - Telemetry to support background video decoder suspend: Percentage video-decode-suspended/total play time
  • VIDEO_INFERRED_DECODE_SUSPEND_PERCENTAGE

• Recovery time - measure of user cost (separate for noisy vs silent videos) (Bucket results by resolution; i.e. do 720p videos take longer to recover?)
  • Bug 1294349 - Telemetry to support background video decoder suspend: Recovery time from video-decode-suspended
  • VIDEO_SUSPEND_RECOVERY_TIME_MS

• Key frame spacing - distribution allows better tuning
  • Bug 1289668 - Telemetry to support background video decoder suspend: Inter-keyframe timings
  • VIDEO_INTER_KEYFRAME_AVERAGE_MS
  • VIDEO_INTER_KEYFRAME_AVERAGE_PERCENTAGE

Mochitests

• Bug 1284177 - Add tests for video suspend in background
  • done
• Bug 1294358 - Add test for suspended videos still fire 'ended' event
  • done
• Bug 1295844 - Test suspended videos with webm files
  • done

Team

Product owner:

Eng: Alastor, Daniel, Gerald, JW, Kaku,

Program Management: Blake, Josh

UX: Mark, Morpheus

QA: SoftVision and William

Communications

IRC:

Email:

VidyoRoom: