Firefox/Shutdown Decoders

From MozillaWiki
Jump to: navigation, search

Overview

Background

Suspending video element's video decoder, when the video element is in background tabs or is invisible even in the foreground tab, is a Firefox feature that reduces CPU/GPU & memory usage.

The mechanism is that, when a video element is invisible, we replace its original video decoder with blank video decoder which only produces white frames with right resolution and right time information. The original video decoder is released and the black video decoder is light so that we reduce CPU/GPU & memory usage.

Trade-off

The trade-off is that, while the suspended-video-element is switched back to be visible again, we should resume its original video decoder. The resuming operation must be asynchronous and might be time-consuming which depends on the resolution of the video file and whether it contains audio tracks or not.

Working flow

The following is a step-by-step description of suspending decoder working flow.

  1. At the very beginning of the decoding framework, the raw media data is sent to demuxer.
  2. Demuxer helps to separate a combined signal, e.g., a streaming can be separated into audio data and video data by demuxer. After that, audio and video data are sent to audio and video decoder separately.
  3. In data decoding period, audio decoder keeps working as normal because a user may be listening to the music. But, Firefox uses a blank video decoder to replace current video decoder if the video element is invisible.

Fx Shutdown Decoder Architecture v2.png

Overall Project Health

[GREEN]

Target Milestone

Firefox55

Engineer Owner

Tzuhao Kuo [:kaku]

QA Contact

Simona Badau <simona.marcu@softvision.ro>
Adrian Florinescu <adrian.florinescu@softvision.ro

MVP Scope-Bug Tracking

  • bug 1293963 -[Meta] Suspend-video-decoder: phase-1 shipping
  • tracking things only under the scope of phase-1 shipping.
Full Query
ID Priority Summary Status Resolution Assigned to Last change time
1276556 P3 [META] Tracking enable of background tab video decoder suspend NEW 2018-03-09T05:15:11Z
1294349 -- Telemetry to support background video decoder suspend: Recovery time from video-decode-suspended RESOLVED FIXED Gerald Squelart [:gerald] (he/him) 2016-08-25T07:57:28Z
1294656 -- Seek to nearest keyframe when resuming videos with no audio - no audio track RESOLVED FIXED Tzuhao Kuo [:kaku] 2016-09-05T03:51:20Z
1305338 P1 [UX] UX Spec for Shutdown Decoder Phase 1 RESOLVED FIXED Mark Liang(:mark_liang) 2017-03-10T06:42:07Z
1309492 P3 If a video element's video is suspended, make it draw black frames. RESOLVED WONTFIX Tzuhao Kuo [:kaku] 2017-03-30T08:23:32Z
1309494 P3 Use spinning loading UI if resuming video element's video decoder is longer than 250ms. RESOLVED FIXED Tzuhao Kuo [:kaku] 2017-05-17T09:08:05Z
1345403 -- Mark a video element as taint and never suspend it again if it is passed into drawImage() RESOLVED FIXED Tzuhao Kuo [:kaku] 2017-03-30T07:46:19Z
1346116 -- Consider a video element is in tree or not to suspend its video decoder RESOLVED FIXED Tzuhao Kuo [:kaku] 2017-03-14T13:27:37Z
1346120 -- If a video element's video is suspended, keep the last decoded video frame on the screen RESOLVED FIXED Tzuhao Kuo [:kaku] 2017-03-30T09:00:23Z
1346235 P1 Fennec has stopped working while playing videos VERIFIED FIXED John Lin [:jhlin][:jolin] 2018-09-24T13:53:46Z
1348432 P1 Missing audio in YouTube videos that were requested within the same tab. Refreshing page fixes audio. VERIFIED FIXED Tzuhao Kuo [:kaku] 2017-04-27T07:31:42Z
1348864 P1 Black frame displayed on video decoder resume RESOLVED DUPLICATE Tzuhao Kuo [:kaku] 2017-04-19T02:14:46Z
1349456 -- Mark a video element as taint and never suspend it again if it is aptured via mozCaptureStream() RESOLVED FIXED Tzuhao Kuo [:kaku] 2017-03-30T07:46:19Z
1349459 -- Mark a video element as taint and never suspend it again if it is passed into createImageBitmap() VERIFIED FIXED Tzuhao Kuo [:kaku] 2017-04-24T08:10:15Z
1365581 P3 [Shutdown Decoder] OGV video elements are not suspended. RESOLVED INVALID Tzuhao Kuo [:kaku] 2017-05-19T02:02:43Z
1365584 P1 [Shutdown Decoder] On WEBM video element resume the player returns a error RESOLVED DUPLICATE Tzuhao Kuo [:kaku] 2017-05-22T10:59:24Z
1369970 -- [Shutdown Decoder] enable the preference. RESOLVED FIXED Tzuhao Kuo [:kaku] 2017-06-08T14:09:25Z

17 Total; 1 Open (5.88%); 13 Resolved (76.47%); 3 Verified (17.65%);


  • bug 1352007- [Meta] Suspend-video-decoder
  • the highest level meta bug tracking all things related to the suspend-video-decoder feature.
Full Query
ID Priority Summary Status Resolution Assigned to Last change time
1274919 P1 Resume suspended video decoders on tab mouse hover. RESOLVED FIXED Alastor Wu [:alwu] 2017-11-16T00:21:17Z
1276557 P3 Disable background video suspend on nightly NEW 2018-03-09T05:15:46Z
1282012 P3 Seek to nearest keyframe when resuming videos with no audio NEW Tzuhao Kuo [:kaku] 2017-04-25T08:21:52Z
1284389 P3 Don't suspend video elements captured via mozCaptureStream() NEW 2018-03-09T05:14:40Z
1291620 P3 Media Tracing via ETW ASSIGNED Dan Glastonbury (:kamidphish) | needinfo me 2017-03-30T09:37:06Z
1293963 P3 [Meta] Suspend-video-decoder: phase-1 shipping NEW Tzuhao Kuo [:kaku] 2017-06-03T08:40:20Z
1295921 P3 Don't suspend video decoder for elements as sources to drawImage() ASSIGNED Tzuhao Kuo [:kaku] 2017-03-30T09:37:06Z
1308184 P1 [Meta] [Phase 1+2] Shutdown video decoders when Firefox is in the background RESOLVED WONTFIX Tzuhao Kuo [:kaku] 2017-03-30T09:32:02Z
1346705 P3 Enable suspend-video-decoder related mochitests on autophone ASSIGNED Tzuhao Kuo [:kaku] 2017-06-05T09:08:46Z
1349461 P3 Don't suspend video decoder for elements as sources to createImageBitmap() ASSIGNED Tzuhao Kuo [:kaku] 2018-11-05T12:59:23Z
1380584 -- [Shield] Pref Flip Study: Shutdown decoder RESOLVED FIXED 2017-11-16T05:00:07Z
1381114 P3 Autoplay is not disabled after restoring session with browser.sessionstore.restore_on_demand false NEW 2017-08-01T03:28:55Z
1381376 -- background video doesn't get resumed when turning off the pref "media.suspend-bkgnd-video.enabled" RESOLVED FIXED Alastor Wu [:alwu] 2017-07-24T02:31:08Z
1381738 P3 Resume video decoding immediately once turning off the pref "media.suspend-bkgnd-video.enabled" NEW 2019-10-26T21:06:26Z
1383610 -- [ShutdownDecoder] Turn off preference on FF55 desktop version. RESOLVED FIXED Tzuhao Kuo [:kaku] 2017-07-28T08:32:43Z
1383653 -- [Shutdown Decoder] video can only be suspended once. RESOLVED FIXED Tzuhao Kuo [:kaku] 2017-07-30T16:49:28Z

16 Total; 10 Open (62.5%); 6 Resolved (37.5%); 0 Verified (0%);


  • bug 1276556-[META] Tracking enable of background tab video decoder suspend
  • tracking basic functionalities which are all completed
Full Query
ID Priority Summary Status Resolution Assigned to Last change time
1224973 P2 Reduced decoder usage for background videos RESOLVED FIXED Dan Glastonbury (:kamidphish) | needinfo me 2017-01-01T11:27:27Z
1272919 P3 Video shows the frame that was on screen when I switched from youtube to a different tab for a fraction of a second VERIFIED FIXED Dan Glastonbury (:kamidphish) | needinfo me 2016-12-27T16:14:40Z
1274626 P3 video fasts forwards when it was sitting in a background tab VERIFIED FIXED Tzuhao Kuo [:kaku] 2016-12-27T16:07:09Z
1275481 P2 Suspend background video decoders on timer RESOLVED FIXED Dan Glastonbury (:kamidphish) | needinfo me 2016-10-03T15:09:44Z
1275538 P1 Crash in mozilla::MediaFormatReader::RequestVideoData RESOLVED FIXED Jean-Yves Avenard [:jya] 2016-05-30T13:37:57Z
1276495 P1 Crash in mozilla::MediaFormatReader::RequestAudioData RESOLVED FIXED Dan Glastonbury (:kamidphish) | needinfo me 2016-07-05T07:56:43Z
1282710 P2 Suspend and resume foreground video decoders according to visibility events. RESOLVED FIXED Tzuhao Kuo [:kaku] 2017-07-14T20:38:56Z
1284177 P3 Add tests for video suspend in background RESOLVED FIXED Dan Glastonbury (:kamidphish) | needinfo me 2017-02-14T08:36:51Z
1285419 P3 Telemetry to support background video decoder suspend: Hidden play time RESOLVED FIXED Gerald Squelart [:gerald] (he/him) 2016-08-30T06:47:21Z
1286875 P3 Firefox Nightly play/pause on Youtube creates syncronization issue between video and audio RESOLVED INVALID 2016-07-27T06:42:46Z
1287987 -- Telemetry to support background video decoder suspend: Percentage hidden/total play time, keyed by audio presence and height ranges RESOLVED FIXED Gerald Squelart [:gerald] (he/him) 2016-08-30T06:47:21Z
1289668 -- Telemetry to support background video decoder suspend: Inter-keyframe timings RESOLVED FIXED Gerald Squelart [:gerald] (he/him) 2016-08-17T01:33:51Z
1293145 -- Telemetry to support background video decoder suspend: Percentage video-decode-suspended/total play time RESOLVED FIXED Gerald Squelart [:gerald] (he/him) 2016-08-30T06:47:21Z
1294345 P3 Remove the TrackSet parameter from MediaDecoderStateMachine::InitiateDecodeRecoverySeek() RESOLVED FIXED JW Wang [:jwwang] [:jw_wang] 2016-08-12T16:49:06Z
1294349 -- Telemetry to support background video decoder suspend: Recovery time from video-decode-suspended RESOLVED FIXED Gerald Squelart [:gerald] (he/him) 2016-08-25T07:57:28Z
1294352 P3 Merge MDSM::InitiateSeek() and MDSM::InitiateDecodeRecoverySeek() RESOLVED FIXED JW Wang [:jwwang] [:jw_wang] 2016-08-12T16:49:09Z
1294358 P3 Add test for suspended videos still fire 'ended' event RESOLVED FIXED Dan Glastonbury (:kamidphish) | needinfo me 2016-08-23T14:04:02Z
1295844 P3 Test suspended videos with webm files. RESOLVED FIXED Dan Glastonbury (:kamidphish) | needinfo me 2016-08-25T23:58:42Z
1299065 -- Telemetry to support background video decoder suspend: invisible elements in the foreground should also be recorded RESOLVED FIXED Tzuhao Kuo [:kaku] 2016-09-12T05:27:30Z
1299718 P3 Telemetry to support background video decoder suspend: the usage of a hidden video element as a content source RESOLVED FIXED Dan Glastonbury (:kamidphish) | needinfo me 2017-02-07T09:12:29Z
1301059 P3 When suspending decode, the BlankDecoder shouldn't be wrapped in a H264Converter RESOLVED FIXED Jean-Yves Avenard [:jya] 2016-09-14T10:11:53Z
1301061 P3 Background video decode will cause issue when use with EME RESOLVED INVALID 2016-10-06T06:46:24Z
1305338 P1 [UX] UX Spec for Shutdown Decoder Phase 1 RESOLVED FIXED Mark Liang(:mark_liang) 2017-03-10T06:42:07Z
1306551 P5 Intermittent dom/media/test/test_background_video_suspend.html | Assertion count 115 is greater than expected range 0-0 assertions from "ASSERTION: Clock should go forwards" RESOLVED FIXED JW Wang [:jwwang] [:jw_wang] 2017-10-30T01:14:26Z
1309492 P3 If a video element's video is suspended, make it draw black frames. RESOLVED WONTFIX Tzuhao Kuo [:kaku] 2017-03-30T08:23:32Z
1309494 P3 Use spinning loading UI if resuming video element's video decoder is longer than 250ms. RESOLVED FIXED Tzuhao Kuo [:kaku] 2017-05-17T09:08:05Z
1310140 P3 Reconcile suspend-video-decoder with the states of MDSM RESOLVED FIXED JW Wang [:jwwang] [:jw_wang] 2016-10-19T15:06:56Z
1313955 P3 Paused videos in background tabs change time position slightly when the tab is selected VERIFIED FIXED Dan Glastonbury (:kamidphish) | needinfo me 2017-02-09T14:26:35Z
1346120 -- If a video element's video is suspended, keep the last decoded video frame on the screen RESOLVED FIXED Tzuhao Kuo [:kaku] 2017-03-30T09:00:23Z

29 Total; 0 Open (0%); 26 Resolved (89.66%); 3 Verified (10.34%);


Signoff-Report issued on April 25 by Adrian Florinescu

Feature testing status: COMPLETED

Testing status: COMPLETED (100%). 83 passed (92%), 4 blocked (4%), 3 failed with known bugs (3%), 0 failed with new bugs (0%)

Overall feature status after testing: GREEN

Reason: 92% of our tests passed.

Recommendation from QE: SHIP IT

Reason: 1 existing issue was reopened during testing Bug1309494 Proposed course of action: let this feature ride Fx55.

QA Test Report

Bugs tracking

Full Query
ID Priority Summary Status Resolution Assigned to Last change time
1309494 P3 Use spinning loading UI if resuming video element's video decoder is longer than 250ms. RESOLVED FIXED Tzuhao Kuo [:kaku] 2017-05-17T09:08:05Z

1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);

UX Spec

Decoder resuming latency

While the suspended-video-element is switched back to be visible again, we should resume its original video decoder. The resuming operation must be asynchronous since we don't want to block the main thread and might be time-consuming which depends on the resolution of the video file and whether it contains audio tracks or not.

Currently, we have no way to boost the resuming time, however, we have telemetries for collecting the needed time of different resolutions on different platforms.


Power consumption (under experiment...)

raw data

Methodology

Used media file: https://drive.google.com/a/mozilla.com/file/d/0Bwk6-CqTcXSDRlYtTjVPZUpJRUU/view?usp=sharing

  • mp4, H.264
  • video only
  • resolution: 1280 x 720
  • duration is 2:4:22

Test scenario: use a local html file to play a local video file and put it into 4 cases.

  • case 1: normal playback in foreground tab, full screen.
  • case 2: normal playback if foreground tab with video decoder suspended, full screen.
  • case 3: normal playback in background tab.
  • case 4: normal playback if background tab with video decoder suspended.

Nexus 5

Normal playback in background tab: 3hr 59min 22sec.
Suspend playback in background tab: 6hr 10min 02sec.
Delta: 2hr 10min 40sec, 54%.
Nexus-5.png

Nexus 6

Normal playback in background tab: 5hr 15min 19sec.
Suspend playback in background tab: 8hr 21min 37sec.
Delta: 3hr 6min 18sec, 59%.
Nexus-6.png