Firefox/Shutdown Decoders: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
Line 4: Line 4:
The mechanism is that, when a video element is invisible, we replace its original video decoder with '''blank video decoder''' which only produces white frames with right resolution and right time information. The original video decoder is released and the black video decoder is light so that we reduce CPU/GPU & memory usage.
The mechanism is that, when a video element is invisible, we replace its original video decoder with '''blank video decoder''' which only produces white frames with right resolution and right time information. The original video decoder is released and the black video decoder is light so that we reduce CPU/GPU & memory usage.


The drawback is that, while the suspended-video-element is switched back to be visible again, we should resume its original video decoder. The resuming operation must be asynchronous and might be time consuming which depends on the resolution of the video file and whether it contains audio tracks or not.
The drawback is that, while the suspended-video-element is switched back to be visible again, we should resume its original video decoder. The resuming operation must be asynchronous and might be time-consuming which depends on the resolution of the video file and whether it contains audio tracks or not.


In the prototype (Phase 0), we have enabled this feature on the Firefox Nightly channel for any video element. We also add telemetry to collect needed information, especially on the resuming time. Currently (Phase 1), we are going to enable this feature on the Firefox Release channel for videos that is able to be resumed quickly and the criteria is '''1) videos without audio track''' or '''2) videos with both audio and video tracks but with low resolution (480P for now)'''. In the future (Phase 2), our goal is to enable this feature on all videos without observable latency while resuming.
In the prototype (Phase 0), we have enabled this feature on the Firefox Nightly channel for any video element. We also add telemetry to collect needed information, especially on the resuming time. Currently (Phase 1), we are going to enable this feature on the Firefox Release channel for videos that is able to be resumed quickly and the criteria is '''1) videos without audio track''' or '''2) videos with both audio and video tracks but with low resolution (480P for now)'''. In the future (Phase 2), our goal is to enable this feature on all videos without observable latency while resuming.
Line 10: Line 10:
The following is a step-by-step description of suspending decoder working flow.
The following is a step-by-step description of suspending decoder working flow.
# At the very beginning of the decoding framework, the raw media data is sent to demuxer.
# At the very beginning of the decoding framework, the raw media data is sent to demuxer.
# Demuxer helps to separate a combined signal, e.g., a streaming can be separated into audio data and video data by demuxer. After that, audio and video data is sent to audio and video decoder separately.
# Demuxer helps to separate a combined signal, e.g., a streaming can be separated into audio data and video data by demuxer. After that, audio and video data are sent to audio and video decoder separately.
# In data decoding period, audio decoder keeps working as normal because user may be listening to the music. But, Firefox uses a blank video decoder to replace current video decoder if the video element is invisible.
# In data decoding period, audio decoder keeps working as normal because a user may be listening to the music. But, Firefox uses a blank video decoder to replace current video decoder if the video element is invisible.


[[File:Fx_Shutdown_Decoder_Architecture_v2.png|800px|frameless]]
[[File:Fx_Shutdown_Decoder_Architecture_v2.png|800px|frameless]]
Line 35: Line 35:
If you find a bug or have a suggestion, please submit it on Bugzilla:
If you find a bug or have a suggestion, please submit it on Bugzilla:
https://goo.gl/coRinl
https://goo.gl/coRinl
=Known Issues=
===Decoder resuming latency===
While the suspended-video-element is switched back to be visible again, we should resume its original video decoder. The resuming operation '''must be asynchronous''' since we don't want to block the main thread and '''might be time-consuming''' which depends on the resolution of the video file and whether it contains audio tracks or not.
Currently, we have no way to boost the resuming time, however, we have telemetries for collecting the needed time of different resolutions on different platforms.
''Related telemetries'':
* [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-08-28&keys=All!AV%252C241-480!V%252C241-480!AV%252C481-720&max_channel_version=nightly%252F51&measure=VIDEO_SUSPEND_RECOVERY_TIME_MS&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-08-18&table=0&trim=1&use_submission_date=0 VIDEO_SUSPENDED_RECOVERY_TIME_MS]
''Related bugs'':
* [https://bugzil.la/1272919 Bug 1272919 - Video shows the frame that was on screen when I switched from youtube to a different tab for a fraction of a second]
===Video element as content source===
The situation is about the video element itself is used as a content source and its decoder might be suspended at the same time. This issue could be split into two cases:
* Case 1: The decoder is suspended '''AFTER''' the {cpatureStream(), drawImage(), createImageBitmap()} is called.
* Case 2: The decoder is suspended '''BEFORE''' the {cpatureStream(), drawImage(), createImageBitmap()} is called.
'''Case 1''' is relatively easy, we could mark the video element as being a content source while {cpatureStream(), drawImage(), createImageBitmap()} is invoked and then its decoder should never be suspended.
'''Case 2''' is rather complicated. The critical issue is that resuming decoder is an async operation with latency. So, while {cpatureStream(), drawImage(), createImageBitmap()} is invoked on an already-suspended-video, there must be several "blank" frames been leaked, even though we resume the decoder immediately. To completely solve this problem, we must make "resuming decoder" a blocking operation, however, it might block the main thread; otherwise, we leak the blank frames.
''Related bugs'':
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1284389 Bug 1284389 - Don't suspend video elements captured via mozCaptureStream()]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1295921 Bug 1295921 - Don't suspend video decoder for elements as sources to drawImage() and createImageBitmap()]


=Planning=
=Planning=
Line 40: Line 67:
* Help users to reduce device resource usage (CPU & Memory)
* Help users to reduce device resource usage (CPU & Memory)
* Finish tasks across videos
* Finish tasks across videos
* No impact in current user flow
* No impact on current user flow


==Feature Plan==
==Feature Plan==
* Phase 0: Shutdown decoders when video element is invisible
* Phase 0: Shutdown decoders when video element is invisible
* Phase 1: Using a blank video decoder to replace video decoder instead of shutdown decoders directly. In this phase, the mechanism is applied to 1) Low resolution video (480P) and 2) Video film without audio track
* Phase 1: Using a blank video decoder to replace video decoder instead of shutdown decoders directly. In this phase, the mechanism is applied to 1) Low-resolution video (480P) and 2) Video film without audio track
* Phase 2: We're going to enhance the mechanism and make sure it can apply to all video files.
* Phase 2: We're going to enhance the mechanism and make sure it can apply to all video files.


==Success==
==Success==
* Effective cross discipline teams solving problems across platforms
* Effective cross-discipline teams solving problems across platforms
* Validation of key assumptions through telmetrics
* Validation of key assumptions through telemetrics
* video decode suspend in the hands of all our users
* video decode suspend in the hands of all our users


Line 69: Line 96:


== Milestones ==
== Milestones ==
=== Blank Previous Frame ===
* https://bugzil.la/1272919 - Dan
* Reviewed and needs rebasing before landing.
=== Optimizations ===
=== Optimizations ===
* For video elements off screen (using layout notification) - Kaku, http://bugzil.la/1282710
* [https://bugzil.la/1282710 Bug 1282710 - Suspend and resume foreground video decoders according to visibility events]
** Not just for videos in background tabs but when the video element scrolls off the screen
** done.
** mattwoodrow says it's possible to subscribe to visibility events via layout and to ask seth or tnikkel for details.
* [https://bugzil.la/1282012 Bug 1282012 - Seek to nearest keyframe when resuming videos with no audio]
*** tnikkel suggests to start form here: http://searchfox.org/mozilla-central/source/layout/generic/nsIFrame.h#1177
** [https://bugzil.la/1294657 Bug 1294657 - Seek to nearest keyframe when resuming videos with no audio - with audio track but muted]
*** nsVideoFrame has already implemented it: http://searchfox.org/mozilla-central/source/layout/generic/nsVideoFrame.cpp#664
*** pending, excluded from phase 1.
** Need to start video before it scrolls back into viewport
** [https://bugzil.la/1294658 Bug 1294658 - Seek to nearest keyframe when resuming videos with no audio - with audio track but might be silent]
*** k17e: in async pan zoom the viewport is 3.5 times the height of the screen weighted in the direction of movement so we shouldn't need to do anything special here
*** pending, excluded from phase 1.
** k17e: Probably only want to do this for silent video where we can recover really quickly
** [https://bugzil.la/1294656 Bug 1294656 - Seek to nearest keyframe when resuming videos with no audio - no audio track]
* Resume video when mouse pointer starts hovering tab - https://bugzil.la/1274919 - Dan
*** done.
** In review. Needs response to mconley.
* [https://bugzil.la/1274919 Bug 1274919 - Resume suspended video decoders on tab mouse hover.]
** under review.
* [https://bugzil.la/1284389 Bug 1284389 - Don't suspend video elements captured via mozCaptureStream()]
** WIP.
* [https://bugzil.la/1295921 Bug 1295921 - Don't suspend video decoder for elements as sources to drawImage() and createImageBitmap()]
** WIP.
* Resume video when keyboard is used to change tags - Needs Bug
* Resume video when keyboard is used to change tags - Needs Bug
** For example, if it's possible to detect the direction of cycling and videos are within, say, 5 tabs of the current tab, then start decoding again.
** For example, if it's possible to detect the direction of cycling and videos are within, say, 5 tabs of the current tab, then start decoding again.
** Need to hook into the tab navigation/change code and alert media elements, like bug 1274919
** Need to hook into the tab navigation/change code and alert media elements, like bug 1274919
* Seek to nearest keyframe when video has no audio - http://bugzil.la/1282012 - Kaku
** If there's no audio then no A/V sync to key, so just jump to the nearest keyframe.
** This is a win for large background videos such as at:
*** http://www.gyg.com.au
*** https://www.paypal.com/nz/webapps/mpp/home
*** Many more examples - http://www.hongkiat.com/blog/fullsize-video-background-websites/
** Tricky to detect no audio:
*** element is muted
*** video file has no audio track
*** video file has audio track of silence (alwu did work on this for tab audio indicator)
** Don't suspend video decode if piping output through MSG.


=== Telemetry ===
=== Telemetry ===
* What telemetry should we collect about suspending video decode?
* '''Amount of time hidden''' - measure of user value (Bucket results by resolution; i.e. are 720p videos hidden less often?)
** k17e: Amount of time hidden - measure of user value (Bucket results by resolution; i.e. are 720p videos hidden less often?)
** [http://bugzil.la/1285419 Bug 1285419 - Telemetry to support background video decoder suspend: Hidden play time]
*** http://bugzil.la/1285419 "Telemetry to support background video decoder suspend: Hidden play time" -> VIDEO_HIDDEN_PLAY_TIME_MS
** [https://mzl.la/2c0493i VIDEO_HIDDEN_PLAY_TIME_MS]
**** Just something quick, based on existing VIDEO_PLAY_TIME_MS, to get some data soon, no bucketing on this one.
** [http://bugzil.la/1287987 Bug 1287987 - Telemetry to support background video decoder suspend: Percentage hidden/total play time, keyed by audio presence and height ranges]
**** https://telemetry.mozilla.org/new-pipeline/dist.html#max_channel_version=nightly%252F50&measure=VIDEO_HIDDEN_PLAY_TIME_MS
** [https://mzl.la/2c04hQl VIDEO_HIDDEN_PLAY_TIME_PERCENTAGE]
*** http://bugzil.la/1287987 "Percentage hidden/total play time, keyed by audio presence and height ranges" -> VIDEO_HIDDEN_PLAY_TIME_PERCENTAGE
** [http://bugzil.la/1293145 Bug 1293145 - Telemetry to support background video decoder suspend: Percentage video-decode-suspended/total play time]
**** https://telemetry.mozilla.org/new-pipeline/dist.html#max_channel_version=nightly%252F50&measure=VIDEO_HIDDEN_PLAY_TIME_PERCENTAGE
** [https://mzl.la/2c07M9s VIDEO_INFERRED_DECODE_SUSPEND_PERCENTAGE]
** k17e: Recovery time - measure of user cost (separate for noisy vs silent videos) (Bucket results by resolution; i.e. do 720p videos take longer to recover?)
 
** k17e: Key frame spacing - distribution allows better tuning
* '''Recovery time''' - measure of user cost (separate for noisy vs silent videos) (Bucket results by resolution; i.e. do 720p videos take longer to recover?)
*** http://bugzil.la/1289668 "Telemetry to support background video decoder suspend: Inter-keyframe timings" -> measure=VIDEO_INTER_KEYFRAME_AVERAGE_MS
** [https://bugzil.la/1294349 Bug 1294349 - Telemetry to support background video decoder suspend: Recovery time from video-decode-suspended]
**** https://telemetry.mozilla.org/new-pipeline/dist.html#max_channel_version=nightly%252F50&measure=VIDEO_INTER_KEYFRAME_AVERAGE_MS
** [https://mzl.la/2c947Iv VIDEO_SUSPEND_RECOVERY_TIME_MS]
**** https://telemetry.mozilla.org/new-pipeline/dist.html#max_channel_version=nightly%252F50&measure=VIDEO_INTER_KEYFRAME_MAX_MS
 
* '''Key frame spacing''' - distribution allows better tuning
** [http://bugzil.la/1289668 Bug 1289668 - Telemetry to support background video decoder suspend: Inter-keyframe timings]
** [https://mzl.la/2c04Utr VIDEO_INTER_KEYFRAME_AVERAGE_MS]
** [https://mzl.la/2c956bN VIDEO_INTER_KEYFRAME_AVERAGE_PERCENTAGE]


=== Mochitests ===
=== Mochitests ===
Currently no tests, need some to ensure that behaviour doesn't break
* [https://bugzil.la/1284177 Bug 1284177 - Add tests for video suspend in background]
http://bugzil.la/1284177
** done
Add tests for suspending offscreen videos.
* [https://bugzil.la/1294358 Bug 1294358 - Add test for suspended videos still fire 'ended' event]
Test no visible JS events because of suspending decode.
** done
 
* [https://bugzil.la/1295844 Bug 1295844 - Test suspended videos with webm files]
** done
=Team=
=Team=


Revision as of 15:46, 29 August 2016

Overview

Suspending video element's video decoder, when the video element is in background tabs or is invisible even in the foreground tab, is a Firefox feature that reduces CPU/GPU & memory usage.

The mechanism is that, when a video element is invisible, we replace its original video decoder with blank video decoder which only produces white frames with right resolution and right time information. The original video decoder is released and the black video decoder is light so that we reduce CPU/GPU & memory usage.

The drawback is that, while the suspended-video-element is switched back to be visible again, we should resume its original video decoder. The resuming operation must be asynchronous and might be time-consuming which depends on the resolution of the video file and whether it contains audio tracks or not.

In the prototype (Phase 0), we have enabled this feature on the Firefox Nightly channel for any video element. We also add telemetry to collect needed information, especially on the resuming time. Currently (Phase 1), we are going to enable this feature on the Firefox Release channel for videos that is able to be resumed quickly and the criteria is 1) videos without audio track or 2) videos with both audio and video tracks but with low resolution (480P for now). In the future (Phase 2), our goal is to enable this feature on all videos without observable latency while resuming.

The following is a step-by-step description of suspending decoder working flow.

  1. At the very beginning of the decoding framework, the raw media data is sent to demuxer.
  2. Demuxer helps to separate a combined signal, e.g., a streaming can be separated into audio data and video data by demuxer. After that, audio and video data are sent to audio and video decoder separately.
  3. In data decoding period, audio decoder keeps working as normal because a user may be listening to the music. But, Firefox uses a blank video decoder to replace current video decoder if the video element is invisible.

Fx Shutdown Decoder Architecture v2.png


Key Documents

Other Resources

  • Etherpad: Constructing...

Install

Stable Release

Stay tuned.

Developer Release

The developer release is updated each time code is committed to Mozilla-central in HG.

Reporting Issues

If you find a bug or have a suggestion, please submit it on Bugzilla: https://goo.gl/coRinl

Known Issues

Decoder resuming latency

While the suspended-video-element is switched back to be visible again, we should resume its original video decoder. The resuming operation must be asynchronous since we don't want to block the main thread and might be time-consuming which depends on the resolution of the video file and whether it contains audio tracks or not.

Currently, we have no way to boost the resuming time, however, we have telemetries for collecting the needed time of different resolutions on different platforms.


Related telemetries:

Related bugs:

Video element as content source

The situation is about the video element itself is used as a content source and its decoder might be suspended at the same time. This issue could be split into two cases:

  • Case 1: The decoder is suspended AFTER the {cpatureStream(), drawImage(), createImageBitmap()} is called.
  • Case 2: The decoder is suspended BEFORE the {cpatureStream(), drawImage(), createImageBitmap()} is called.

Case 1 is relatively easy, we could mark the video element as being a content source while {cpatureStream(), drawImage(), createImageBitmap()} is invoked and then its decoder should never be suspended.

Case 2 is rather complicated. The critical issue is that resuming decoder is an async operation with latency. So, while {cpatureStream(), drawImage(), createImageBitmap()} is invoked on an already-suspended-video, there must be several "blank" frames been leaked, even though we resume the decoder immediately. To completely solve this problem, we must make "resuming decoder" a blocking operation, however, it might block the main thread; otherwise, we leak the blank frames.


Related bugs:

Planning

Goals

  • Help users to reduce device resource usage (CPU & Memory)
  • Finish tasks across videos
  • No impact on current user flow

Feature Plan

  • Phase 0: Shutdown decoders when video element is invisible
  • Phase 1: Using a blank video decoder to replace video decoder instead of shutdown decoders directly. In this phase, the mechanism is applied to 1) Low-resolution video (480P) and 2) Video film without audio track
  • Phase 2: We're going to enhance the mechanism and make sure it can apply to all video files.

Success

  • Effective cross-discipline teams solving problems across platforms
  • Validation of key assumptions through telemetrics
  • video decode suspend in the hands of all our users

Schedule

Product Milestones

Engineering

Shutdown_Decoders [status: Green]

Current Goals:

  • Ship disabling video decoders for ads
  • Silent, looping videos where user doesn't really care the exact point

Next Milestone:


Milestones

Optimizations

Telemetry

Mochitests

Team

Product owner:

Eng: Alastor, Daniel, Gerald, JW, Kaku,

Program Management: Blake, Josh

UX:

QA: SoftVision and William

Communications

IRC:

Email:

VidyoRoom:

Retrieved from "https://wiki.mozilla.org/index.php?title=Firefox/Shutdown_Decoders&oldid=1146139"