Networking/Archive/RTSP

Background

RTSP (Real-Time Streaming Protocol) provides initiation and VCR-like control of streaming presentations. RTSP was standardized in 1998. RTSP builds on existing standards: It closely resembles HTTP in operation, and it can use SDP for session description and RTP for media transport.
For more detail about RTSP, please refer to http://en.wikipedia.org/wiki/Real_Time_Streaming_Protocol and RFC 2326.

Protocol Stack

The complete multimedia protocol stack is illustrated in this figure, which shows the relationship between the RTSP/RTP framework and the supporting setup and control protocols.

Call Setup and Control

Various call setup, control, and advertisement protocols can be used to start an RTP session, depending on the application scenario:

• For the purpose of starting interactive session, be it a voice telephony call or a video conference, there are two standards. The original standard in this area was ITU recommendation H.323, and more recently the IETF has defined the Session Initiation Protocol (SIP).
• For the purpose of starting a noninteractive session--for example, video on demand--the main standard is the RTSP.
• The original use of RTP was with IP multicast and the lightweight sessions model of conferencing. This design used the Session Announcement Protocol (SAP) and IP multicast to announce ongoing sessions, such as seminars and TV broadcasts, that were open to the public.

Session Description

Session Description Protocol (SDP) is a means of describing the session, which conveys information such as transport addresses on the media flows, the format of the media, the RTP payload formats and profile that are to be used, the times when the session is active, and the purpose of the session.
SDP is in text file format, so it is human-readable and can be easily parsed.

Terminology and Reference

• RTSP: Real Time Streaming Protocol. An application-level protocol for control over the delivery of data with real-time properties. Defined in RFC 2326.
• RTP: Real-time Transport Protocol. Defines a standardized packet format for delivering audio and video over IP networks. Defined in RFC 3550.
• RTCP: RTP Control Protocol. Provides out-of-band statistics and control information for an RTP session. Defined in RFC 3550.
• SDP: Session Description Protocol. A format for describing streaming media initialization parameters. Defined in RFC 4566.
• RTP audio video profile - http://en.wikipedia.org/wiki/RTP_audio_video_profile

History (Important Bugs)

• bug 831645 - Support rtsp streaming framework
  • This is the first bug of RTSP streaming feature on B2G, which implemented most of the RTSP framework in Firefox OS.

• bug 992568 - [RTSP] Refactor RtspChannel to support HTTP->RTSP redirection and rendering inside the browser
  • Although the basic framework was ready, before FFOS v1.4, RTSP on B2G is rendered in the video app and supports only <a> and <video> tags. We had made a decision (in https://bugzilla.mozilla.org/show_bug.cgi?id=963565#c13) to render RTSP directly inside the B2G browser app. This bug serves as the collective implementation of several features, including:
    • Support RTSP in URL bar and <iframe>.
    • Support HTTP->RTSP redirection.
    • Render RTSP inside the browser.

• bug 957937 - (b2g-RTSP-2.0) [META] RTSP for 2.0
  
  • This is the meta bug of RTSP features in v2.0. We made a huge improvement in quality and reliability of RTSP feature in this version.
    

Supported RTP Payload Formats (Codecs)

RTP payload formats define how particular media types are transported within RTP. Payload formats are referenced by RTP profiles, and they may also define certain properties of the RTP data transfer protocol.
For a complete list of RTP payload formats, refer to http://en.wikipedia.org/wiki/RTP_audio_video_profile.

This tables lists the payload formats (codecs) supported by our RTSP client right now.

Payload Type (PT)	Name	Type	Description	Comment
dynamic	H264	video	H.264 video (MPEG-4 Part 10)
dynamic	MP4A-LATM	audio	MPEG-4 Audio
dynamic	H263-1998	video	H.263 video, second version (1998)
dynamic	H263-2000	video	H.263 video, third version (2000)
dynamic (or profile)	AMR	audio	Adaptive Multi-Rate audio
dynamic (or profile)	AMR-WB	audio	Adaptive Multi-Rate Wideband audio (ITU-T G.722.2)
dynamic (or profile)	MP4V-ES	video	MPEG-4 Visual
dynamic (or profile)	mpeg4-generic	audio/video	MPEG-4 Elementary Streams

For unsupported payload formats, such as MP3, an error message will be displayed on the browser.

Implementation

Framework Overview

This class diagram illustrates the most important components in our current RTSP framework. We can divide the framework into four major sub-modules:

• HTML Media Element
• RTSP Channel
• RTSP Controller
• Stagefright Library

HTML Media Element

Code path:

m-c/content/html/content/public/HTMLMediaElement.h
m-c/content/html/content/src/HTMLMediaElement.cpp
m-c/content/media/MediaDecoder.h/cpp
m-c/content/media/MediaResource.h/cpp
m-c/content/media/RtspMediaResource.h/cpp

RtspMediaResource provides an interface to deliver and control RTSP media data to RtspOmxDecoder.
Note that the design and behavior of RTSP streaming are quite different from HTTP streaming. Please refer to the comments in RtspMediaResource.h for more detail.

RTSP Channel

Code path:

m-c/netwerk/protocol/RtspHandler.h/cpp
m-c/netwerk/protocol/RtspChannelChild.h/cpp
m-c/netwerk/protocol/RtspChannelParent.h/cpp

Unlike other channels in Necko, RtspChannelChild is a dummy channel used to aid MediaResource creation in HTMLMediaElement. Network control and data flows are managed by RtspController objects, which is also responsible for IPC with the parent process.
For now, the existence of RtspChannelParent is only to support HTTP->RTSP redirection.

RTSP Controller

Code path:

m-c/netwerk/base/public/nsIStreamingProtocolController.idl
m-c/netwerk/protocol/rtsp/controller/RtspController.h/cpp
m-c/netwerk/protocol/rtsp/controller/RtspControllerChild.h/cpp
m-c/netwerk/protocol/rtsp/controller/RtspControllerParent.h/cpp

Both RtspControllerChild and RtspController inherit from and implement the interfaces nsIStreamingProtocolListener and nsIStreamingProtocolController. As its name indicates, nsIStreamingProtocolController interface is used to control and retrieve meta data from a media stream.

Stagefright Library

Code path:

m-c/netwerk/protocol/rtsp/rtsp/RTSPSource.h/cpp
m-c/netwerk/protocol/rtsp/rtsp/RTSPConnectionHandler.h
m-c/netwerk/protocol/rtsp/rtsp/*

We reused the stagefright library from AOSP. Source files for RTSP client were mostly ported to the folder "netwerk/protocol/rtsp/rtsp".
However, please note that currently we still depend on the utilities from Android foundation, such as ABuffer, ALooper and AMessage, etc.
The class RTSPSource is used directly by RtspController to perform operations, such as establishing RTSP connection and sending RTSP requests.

TCP-Interleaved RTP