Confirmed users
425
edits
Networking/Archive/DASH/Implementation: Difference between revisions
Networking/Archive/DASH/Implementation (view source)
Revision as of 14:11, 24 April 2023
, 24 April 2023Manuel Bucher moved page Networking/DASH/Implementation to Networking/Archive/DASH/Implementation: move to necko archive
(Add info on VP8/Video and Vorbis/Audio decoders' ability to adapt) |
m (Manuel Bucher moved page Networking/DASH/Implementation to Networking/Archive/DASH/Implementation: move to necko archive) |
||
| (9 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
{{Template:Archived}} | |||
''Work in progress'' | ''Work in progress'' | ||
| Line 6: | Line 8: | ||
With current native video, Necko buffers the data until it is safe to start playing. nsMediaChannelStream downloads data via HTTP and puts that data in an nsMediaCache. nsMediaCache in turn makes this data available to decoder threads via read() and seek() APIs. The decoders then read the data and enqueue it in A/V queues. | With current native video, Necko buffers the data until it is safe to start playing. nsMediaChannelStream downloads data via HTTP and puts that data in an nsMediaCache. nsMediaCache in turn makes this data available to decoder threads via read() and seek() APIs. The decoders then read the data and enqueue it in A/V queues. | ||
Chris Pearce has a blog post [[http://blog.pearce.org.nz/2011/02/firefox-4-video-decoder-architecture.html here]] that describes the current architecture. | |||
[[image:Dash-impl-current-video-arch.png]] | |||
[[:image:Dash-impl-current-video-arch.svg|SVG version]] | |||
Figure 1 Current Gecko Video Architecture (taken from http://pearce.org.nz/uploaded_images/video-architecture.svg with additions for Necko code). | |||
== High Level Approaches - Segment Request and Delivery to MediaStream/MediaCoder == | == High Level Approaches - Segment Request and Delivery to MediaStream/MediaCoder == | ||
| Line 12: | Line 20: | ||
# One nsMediaDASHStream class is created which manages the monitoring of local capabilities/load and adapts download by switching streams as necessary. Then one nsMediaCache provides data to a single nsWebMDecoder. | # One nsMediaDASHStream class is created which manages the monitoring of local capabilities/load and adapts download by switching streams as necessary. Then one nsMediaCache provides data to a single nsWebMDecoder. | ||
[[image:Dash-impl-high-level-opt-1.png]] | |||
Figure 2 - High level view of option 1: Single Audio/Video decoder. | |||
'''Description needed''' | '''Description needed''' | ||
| Line 18: | Line 28: | ||
# One nsMediaDASHStream as before, but with multiple nsWebMDecoders, one for each encoded stream available on the server. Only one nsWebMDecoder would be used at a time. | # One nsMediaDASHStream as before, but with multiple nsWebMDecoders, one for each encoded stream available on the server. Only one nsWebMDecoder would be used at a time. | ||
[[image:dash-impl-high-level-opt-2.png]] | |||
Figure 3 - High level view of option 1: Multiple audio/video decoders. | |||
'''Description needed''' | '''Description needed''' | ||
| Line 35: | Line 47: | ||
: To avoid this, we could require that each segment include extra packets to allow correct decoding. Or, we can just support no audio adaptation to start with (similar to Apple HLS). Rob O'Callahan is working on a Media Streams API which includes cross-fading - this may be a longer term solution, after starting with non-adaptive audio. | : To avoid this, we could require that each segment include extra packets to allow correct decoding. Or, we can just support no audio adaptation to start with (similar to Apple HLS). Rob O'Callahan is working on a Media Streams API which includes cross-fading - this may be a longer term solution, after starting with non-adaptive audio. | ||
== MPD == | == Use of libdash from ITEC == | ||
Code from libdash has been released under MPL2.0 by Christian Timmerer and Christopher Mueller of ITEC, Austria. The plan is to fork the codebase and integrate it with Gecko code. At a high level this involves switching the DOMParser/libxml and HTTP code in libdash for nsDOMParser and Necko's HTTP code. As of Feb 16, 2012, this is under way. | |||
This adds extra components in the above high level diagram, namely DASHManager, MPDParser and the MPD classes, as well as the Adaptation Logic. The following sections will reflect the base design used in libdash including any changes. | |||
== DASHManager == | |||
Manages all DASH activity in Gecko, similar to nsHttpConnectionMgr. MPD and MPDParser objects are obtained here. | |||
== MPDParser == | |||
Will use nsDOMParser to read XML from the nsHTTPChannel and populate the MPD classes. | |||
'''Diagram needed''' | '''Diagram needed''' | ||
'''Description needed''' | '''Detailed Description needed''' | ||
== MPD Classes and Objects == | |||
As per libdash | |||
'''Diagram needed''' | '''Diagram needed''' | ||
'''Description needed''' | '''Description needed''' | ||
=== MPD | == Plan == | ||
A staged approach will be taken for implementation, with early stages as follows: | |||
* Parse an MPD file off the network and choose a single stream for media playback. No adaptation for this first phase. | |||
* Add a basic adaptation logic. | |||
* (Long term) Expose JS APIs as follows: | |||
*# Buffering/request handling | |||
*# MPD handling | |||
*# Adaptation logic | |||
*# Logging, monitoring. | |||
Please note that (4) would correspond to the DASH metrics defined in the Annex of the DASH specification. It would allow application developers to generate reports back to the content provider in order to improve Quality of Service/Experience. | |||
Also note that JS APIs may be exposed earlier; the primary goal, however, is to get adaptive video working in Gecko, and JS APIs (while very desirable) should not be blocking work in these first stages. | |||
'''Diagram needed''' | '''Diagram needed''' | ||
'''Description needed''' | '''Description needed''' | ||
== Capability/Load Monitoring == | == Capability/Load Monitoring == | ||
| Line 56: | Line 90: | ||
== Media Segments == | == Media Segments == | ||
Processed as per [http://sourceforge.net/apps/trac/matroska/wiki/DASH_Profile DASH/Matroska/WebM] | |||
'''Diagram needed''' | '''Diagram needed''' | ||
'''Description needed''' | '''Description needed''' | ||
== | == Related Bugs == | ||
Overall DASH Tracking {{bug|702122}} add support for DASH (WebM) | |||
Other bugs to be created: | |||
* Add libdash code to mozilla-central, integrated with nsDOMParser and nsHttpChannel | |||
** Exclude code from general build for now. | |||
** Code should read in MPD, be able to produce debug output for MPD, and populate MPD classes | |||
* Update codebase with bugfixes from Christopher Mueller. | |||
* Choose stream from MPD and playback, non-adaptively | |||
* Integrate adaptation logic | |||
== Related Pages == | == Related Pages == | ||
*[[Networking/DASH]] | *[[Networking/DASH]] | ||
*[[Networking/DASH/Testing]] | *[[Networking/DASH/Testing]] | ||