Firefox/Shutdown Decoders: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(→‎Methodology: update the video file link.)
 
(46 intermediate revisions by 4 users not shown)
Line 1: Line 1:
=Overview=
=Overview=
== Background ==
Suspending video element's video decoder, when the video element is in background tabs or is invisible even in the foreground tab, is a Firefox feature that reduces CPU/GPU & memory usage.
Suspending video element's video decoder, when the video element is in background tabs or is invisible even in the foreground tab, is a Firefox feature that reduces CPU/GPU & memory usage.


The mechanism is that, when a video element is invisible, we replace its original video decoder with '''blank video decoder''' which only produces white frames with right resolution and right time information. The original video decoder is released and the black video decoder is light so that we reduce CPU/GPU & memory usage.
The mechanism is that, when a video element is invisible, we replace its original video decoder with '''blank video decoder''' which only produces white frames with right resolution and right time information. The original video decoder is released and the black video decoder is light so that we reduce CPU/GPU & memory usage.


The drawback is that, while the suspended-video-element is switched back to be visible again, we should resume its original video decoder. The resuming operation must be asynchronous and might be time consuming which depends on the resolution of the video file and whether it contains audio tracks or not.
== Trade-off ==
 
The trade-off is that, while the suspended-video-element is switched back to be visible again, we should resume its original video decoder.  
In the prototype (Phase 0), we have enabled this feature on the Firefox Nightly channel for any video element. We also add telemetry to collect needed information, especially on the resuming time. Currently (Phase 1), we are going to enable this feature on the Firefox Release channel for videos that is able to be resumed quickly and the criteria is '''1) videos without audio track''' or '''2) videos with both audio and video tracks but with low resolution (480P for now)'''. In the future (Phase 2), our goal is to enable this feature on all videos without observable latency while resuming.
The resuming operation must be asynchronous and might be time-consuming which depends on the resolution of the video file and whether it contains audio tracks or not.


== Working flow ==
The following is a step-by-step description of suspending decoder working flow.
The following is a step-by-step description of suspending decoder working flow.
# At the very beginning of the decoding framework, the raw media data is sent to demuxer.
# At the very beginning of the decoding framework, the raw media data is sent to demuxer.
# Demuxer helps to separate a combined signal, e.g., a streaming can be separated into audio data and video data by demuxer. After that, audio and video data is sent to audio and video decoder separately.
# Demuxer helps to separate a combined signal, e.g., a streaming can be separated into audio data and video data by demuxer. After that, audio and video data are sent to audio and video decoder separately.
# In data decoding period, audio decoder keeps working as normal because user may be listening to the music. But, Firefox uses a blank video decoder to replace current video decoder if the video element is invisible.
# In data decoding period, audio decoder keeps working as normal because a user may be listening to the music. But, Firefox uses a blank video decoder to replace current video decoder if the video element is invisible.


[[File:Fx_Shutdown_Decoder_Architecture_v2.png|800px|frameless]]
[[File:Fx_Shutdown_Decoder_Architecture_v2.png|800px|frameless]]


=Overall Project Health=


===Key Documents===
<font color="green">'''[GREEN]'''</font>
* Code: https://hg.mozilla.org/integration/mozilla-inbound/file/tip/dom/media
* Issues: Meta Bug - [https://bugzilla.mozilla.org/show_bug.cgi?id=1276556 Bug 1276556] [META] Tracking enable of background tab video decoder suspend
* Product/UX Trello Board: NA
 
===Other Resources===
* Etherpad: Constructing...
 
=Install=
 
===Stable Release===
Stay tuned.
 
===Developer Release===
The developer release is updated each time code is committed to Mozilla-central in HG.
 
=Reporting Issues=
If you find a bug or have a suggestion, please submit it on Bugzilla:
https://goo.gl/coRinl
 
=Planning=
==Goals==
* Help users to reduce device resource usage (CPU & Memory)
* Finish tasks across videos
* No impact in current user flow
 
==Feature Plan==
* Phase 0: Shutdown decoders when video element is invisible
* Phase 1: Using a blank video decoder to replace video decoder instead of shutdown decoders directly. In this phase, the mechanism is applied to 1) Low resolution video (480P) and 2) Video film without audio track
* Phase 2: We're going to enhance the mechanism and make sure it can apply to all video files.
 
==Success==
* Effective cross discipline teams solving problems across platforms
* Validation of key assumptions through telmetrics
* video decode suspend in the hands of all our users
 
==Schedule==
 
===Product Milestones===
 


= Engineering =
=Target Milestone=
Shutdown_Decoders [status: <span style="color:#0f0">Green</span>]
Firefox55


Current Goals:
=Engineer Owner=
* Ship disabling video decoders for ads
Tzuhao Kuo [:kaku]
* Silent, looping videos where user doesn't really care the exact point


Next Milestone:
=QA Contact=
Simona Badau <simona.marcu@softvision.ro><br>
Adrian Florinescu <adrian.florinescu@softvision.ro


= MVP Scope-Bug Tracking =
*{{Bug|1293963}} -[Meta] Suspend-video-decoder: phase-1 shipping
*tracking things only under the scope of phase-1 shipping.
<onlyinclude>
<bugzilla>
{
    "blocks": "1293963",
    "include_fields": "id, priority, summary, status,resolution,assigned_to, last_change_time",
    "order": "bug_id"
}
</bugzilla>
</onlyinclude>


*{{Bug|1352007}}-  [Meta] Suspend-video-decoder
*the highest level meta bug tracking all things related to the suspend-video-decoder feature.
<onlyinclude>
<bugzilla>
{
    "blocks": "1352007",
    "include_fields": "id, priority, summary, status,resolution,assigned_to, last_change_time",
    "order": "bug_id"
}
</bugzilla>
</onlyinclude>


== Milestones ==
*{{Bug|1276556}}-[META] Tracking enable of background tab video decoder suspend
=== Blank Previous Frame ===
*tracking basic functionalities which are all completed
* https://bugzil.la/1272919 - Dan
<onlyinclude>
* Reviewed and needs rebasing before landing.
<bugzilla>
{
    "blocks": "1276556",
    "include_fields": "id, priority, summary, status,resolution,assigned_to, last_change_time",
    "order": "bug_id"
}
</bugzilla>
</onlyinclude>


=== Optimizations ===
=Signoff-Report issued on April 25 by Adrian Florinescu =
* For video elements off screen (using layout notification) - Kaku, http://bugzil.la/1282710
==Feature testing status: COMPLETED==
** Not just for videos in background tabs but when the video element scrolls off the screen
Testing status: COMPLETED (100%).
** mattwoodrow says it's possible to subscribe to visibility events via layout and to ask seth or tnikkel for details.
83 passed (92%), 4 blocked (4%), 3 failed with known bugs (3%), 0 failed with new bugs (0%)
*** tnikkel suggests to start form here: http://searchfox.org/mozilla-central/source/layout/generic/nsIFrame.h#1177
*** nsVideoFrame has already implemented it: http://searchfox.org/mozilla-central/source/layout/generic/nsVideoFrame.cpp#664
** Need to start video before it scrolls back into viewport
*** k17e: in async pan zoom the viewport is 3.5 times the height of the screen weighted in the direction of movement so we shouldn't need to do anything special here
** k17e: Probably only want to do this for silent video where we can recover really quickly
* Resume video when mouse pointer starts hovering tab - https://bugzil.la/1274919 - Dan
** In review. Needs response to mconley.
* Resume video when keyboard is used to change tags - Needs Bug
** For example, if it's possible to detect the direction of cycling and videos are within, say, 5 tabs of the current tab, then start decoding again.
** Need to hook into the tab navigation/change code and alert media elements, like bug 1274919
* Seek to nearest keyframe when video has no audio - http://bugzil.la/1282012 - Kaku
** If there's no audio then no A/V sync to key, so just jump to the nearest keyframe.
** This is a win for large background videos such as at:
*** http://www.gyg.com.au
*** https://www.paypal.com/nz/webapps/mpp/home
*** Many more examples - http://www.hongkiat.com/blog/fullsize-video-background-websites/
** Tricky to detect no audio:
*** element is muted
*** video file has no audio track
*** video file has audio track of silence (alwu did work on this for tab audio indicator)
** Don't suspend video decode if piping output through MSG.


=== Telemetry ===
==Overall feature status after testing: GREEN==
* What telemetry should we collect about suspending video decode?
Reason: 92% of our tests passed.
** k17e: Amount of time hidden - measure of user value (Bucket results by resolution; i.e. are 720p videos hidden less often?)
*** http://bugzil.la/1285419 "Telemetry to support background video decoder suspend: Hidden play time" -> VIDEO_HIDDEN_PLAY_TIME_MS
**** Just something quick, based on existing VIDEO_PLAY_TIME_MS, to get some data soon, no bucketing on this one.
**** https://telemetry.mozilla.org/new-pipeline/dist.html#max_channel_version=nightly%252F50&measure=VIDEO_HIDDEN_PLAY_TIME_MS
*** http://bugzil.la/1287987 "Percentage hidden/total play time, keyed by audio presence and height ranges" -> VIDEO_HIDDEN_PLAY_TIME_PERCENTAGE
**** https://telemetry.mozilla.org/new-pipeline/dist.html#max_channel_version=nightly%252F50&measure=VIDEO_HIDDEN_PLAY_TIME_PERCENTAGE
** k17e: Recovery time - measure of user cost (separate for noisy vs silent videos)  (Bucket results by resolution; i.e. do 720p videos take longer to recover?)
** k17e: Key frame spacing - distribution allows better tuning
*** http://bugzil.la/1289668 "Telemetry to support background video decoder suspend: Inter-keyframe timings" -> measure=VIDEO_INTER_KEYFRAME_AVERAGE_MS
**** https://telemetry.mozilla.org/new-pipeline/dist.html#max_channel_version=nightly%252F50&measure=VIDEO_INTER_KEYFRAME_AVERAGE_MS
**** https://telemetry.mozilla.org/new-pipeline/dist.html#max_channel_version=nightly%252F50&measure=VIDEO_INTER_KEYFRAME_MAX_MS


=== Mochitests ===
==Recommendation from QE: SHIP IT ==
Currently no tests, need some to ensure that behaviour doesn't break
Reason: 1 existing issue was reopened during testing Bug1309494
http://bugzil.la/1284177
Proposed course of action: let this feature ride Fx55.
Add tests for suspending offscreen videos.
Test no visible JS events because of suspending decode.


=Team=
==QA Test Report  ==
*Test Report: [https://wiki.mozilla.org/QA/Shutdown_Decoders Test Report]


Product owner:
==Bugs tracking ==


Eng: Alastor, Daniel, Gerald, JW, Kaku,  
<bugzilla>
    {
        "id":["1309494"],
        "include_fields": "id, priority, summary, status, resolution, assigned_to, last_change_time",
        "order": "bug_id"
    }
</bugzilla>
=UX Spec=


Program Management: Blake, Josh
*UX Spec : [https://mozilla.invisionapp.com/share/K48PCVSEM UX Spec ]


UX:
===Decoder resuming latency===
While the suspended-video-element is switched back to be visible again, we should resume its original video decoder. The resuming operation '''must be asynchronous''' since we don't want to block the main thread and '''might be time-consuming''' which depends on the resolution of the video file and whether it contains audio tracks or not.


QA: SoftVision and William
Currently, we have no way to boost the resuming time, however, we have telemetries for collecting the needed time of different resolutions on different platforms.


=Communications=
<br />


IRC:
=Power consumption (under experiment...)=
[https://docs.google.com/a/mozilla.com/spreadsheets/d/1gvq2BOr0GdZb7hEwDenOBYc25rpCpeHYONkDH6b7IVc/edit?usp=sharing raw data]
==Methodology==
'''Used media file:''' https://drive.google.com/a/mozilla.com/file/d/0Bwk6-CqTcXSDRlYtTjVPZUpJRUU/view?usp=sharing
* mp4, H.264
* video only
* resolution: 1280 x 720
* duration is 2:4:22
'''Test scenario:''' use a local html file to play a local video file and put it into 4 cases. <br />
* case 1: normal playback in foreground tab, full screen. <br />
* case 2: normal playback if foreground tab with video decoder suspended, full screen. <br />
* case 3: normal playback in background tab. <br />
* case 4: normal playback if background tab with video decoder suspended. <br />


Email:  
==Nexus 5==
Normal playback in background tab: 3hr 59min 22sec.<br />
Suspend playback in background tab: 6hr 10min 02sec.<br />
'''Delta: 2hr 10min 40sec, 54%.'''<br />
[[File:Nexus-5.png|1600px|frameless]]


VidyoRoom:
==Nexus 6==
Normal playback in background tab: 5hr 15min 19sec.<br />
Suspend playback in background tab: 8hr 21min 37sec.<br />
'''Delta: 3hr 6min 18sec, 59%.'''<br />
[[File:Nexus-6.png|1600px|frameless]]

Latest revision as of 04:44, 13 July 2017

Overview

Background

Suspending video element's video decoder, when the video element is in background tabs or is invisible even in the foreground tab, is a Firefox feature that reduces CPU/GPU & memory usage.

The mechanism is that, when a video element is invisible, we replace its original video decoder with blank video decoder which only produces white frames with right resolution and right time information. The original video decoder is released and the black video decoder is light so that we reduce CPU/GPU & memory usage.

Trade-off

The trade-off is that, while the suspended-video-element is switched back to be visible again, we should resume its original video decoder. The resuming operation must be asynchronous and might be time-consuming which depends on the resolution of the video file and whether it contains audio tracks or not.

Working flow

The following is a step-by-step description of suspending decoder working flow.

  1. At the very beginning of the decoding framework, the raw media data is sent to demuxer.
  2. Demuxer helps to separate a combined signal, e.g., a streaming can be separated into audio data and video data by demuxer. After that, audio and video data are sent to audio and video decoder separately.
  3. In data decoding period, audio decoder keeps working as normal because a user may be listening to the music. But, Firefox uses a blank video decoder to replace current video decoder if the video element is invisible.

Fx Shutdown Decoder Architecture v2.png

Overall Project Health

[GREEN]

Target Milestone

Firefox55

Engineer Owner

Tzuhao Kuo [:kaku]

QA Contact

Simona Badau <simona.marcu@softvision.ro>
Adrian Florinescu <adrian.florinescu@softvision.ro

MVP Scope-Bug Tracking

  • bug 1293963 -[Meta] Suspend-video-decoder: phase-1 shipping
  • tracking things only under the scope of phase-1 shipping.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


  • bug 1352007- [Meta] Suspend-video-decoder
  • the highest level meta bug tracking all things related to the suspend-video-decoder feature.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


  • bug 1276556-[META] Tracking enable of background tab video decoder suspend
  • tracking basic functionalities which are all completed

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Signoff-Report issued on April 25 by Adrian Florinescu

Feature testing status: COMPLETED

Testing status: COMPLETED (100%). 83 passed (92%), 4 blocked (4%), 3 failed with known bugs (3%), 0 failed with new bugs (0%)

Overall feature status after testing: GREEN

Reason: 92% of our tests passed.

Recommendation from QE: SHIP IT

Reason: 1 existing issue was reopened during testing Bug1309494 Proposed course of action: let this feature ride Fx55.

QA Test Report

Bugs tracking

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1

UX Spec

Decoder resuming latency

While the suspended-video-element is switched back to be visible again, we should resume its original video decoder. The resuming operation must be asynchronous since we don't want to block the main thread and might be time-consuming which depends on the resolution of the video file and whether it contains audio tracks or not.

Currently, we have no way to boost the resuming time, however, we have telemetries for collecting the needed time of different resolutions on different platforms.


Power consumption (under experiment...)

raw data

Methodology

Used media file: https://drive.google.com/a/mozilla.com/file/d/0Bwk6-CqTcXSDRlYtTjVPZUpJRUU/view?usp=sharing

  • mp4, H.264
  • video only
  • resolution: 1280 x 720
  • duration is 2:4:22

Test scenario: use a local html file to play a local video file and put it into 4 cases.

  • case 1: normal playback in foreground tab, full screen.
  • case 2: normal playback if foreground tab with video decoder suspended, full screen.
  • case 3: normal playback in background tab.
  • case 4: normal playback if background tab with video decoder suspended.

Nexus 5

Normal playback in background tab: 3hr 59min 22sec.
Suspend playback in background tab: 6hr 10min 02sec.
Delta: 2hr 10min 40sec, 54%.
Nexus-5.png

Nexus 6

Normal playback in background tab: 5hr 15min 19sec.
Suspend playback in background tab: 8hr 21min 37sec.
Delta: 3hr 6min 18sec, 59%.
Nexus-6.png

Retrieved from "https://wiki.mozilla.org/index.php?title=Firefox/Shutdown_Decoders&oldid=1175640"