TPEPlatform/WebRTC B2G

Summary

This document puts focus on B2G only. Cross platform issues are not involved.

bug 750011 - [meta] Support B2G for WebRTC
- A meta issue to trace all WebRTC_B2G blocking issues.
WebRTC workweek minutes(3rd Jun ~ 7th Jun)
Audio issues

Network

On B2G, we put focus on

E10S: chrome/ content process module on FirefoxOS
E10S: packet filter in chrome process.
Network interface enumeration and prioritization: WLAN & 3G connection transition
SDP
- SDP parsing(encode/ decode)
- Request video frame parameter, such as video resolution, depend on HW capability.
ICE: error reporting.

Bug No	Description	Statue	Target	Assigned
bug 825708	We should use interface properties to determine ICE priorities	Open	Gecko 26	Patrick
bug 869869	e10s for UDP socket	Open	Gecko 26	SC
bug 870660	Packet filter for UDP e10s	Open	Gecko 26	Patrick
bug 881761	NSS for WebRTC in content process	Open	Gecko 26	???
bug 881935	SDP: Support negotiation of video resolution	Open	Gecko 26	Adam Roach
bug 881982	ICE: handle dynamic network interface change	Open	Gecko 26	Patrick
bug 884196	ICE: report error on network interface change	Open	Gecko 26	Patrick
bug 881935	Support negotiation of video resolution	Open	Gecko 26	SWU/ Adam

Performance

Performance-wise, we need to do more optimization on B2G because of weaker hardware. FirefoxOS run on ARM, instead of x86. We need to leverage BSP/GPU to boost up whole webrtc performance. Here list options for optimization in B2G.

H.264/AAC codec support: create a prototype first to eveluate performance gain by using HW codec.
Pipeline optimization. To make sure no useless or redundant opertion in encode/ decode pipeline.
- Example: bug 873003 - duplicate video frames be process in encode thread.
Use reasonable timer interval in Process Thread on B2G. Less statistic data collection, no NACK.
Test OPUS with lower complexity and decide whether B2G uses OPUS or G711 as default audio codec

Bug No	Description	Statue	Target	Assigned
bug 884365	Audio realtime input clock mismatch	Open	Gecko 26	Randell
bug 861050	WebRTC performance issue on B2G	Open	Gecko 26	Steven

TODO: H.264 encoder/ decoder - john.

Permission

bug 853356:

Media Resource Management

The media resources on B2G, H/W codec, camera and mic, are limited and will be accessed by multiple processes. We need a centralized manager to handle how to dispatch these resources. We also need to define the media behavior when the process holds a media resource and switch to background.

H/W codec management
camera resource management
microphone resource management
user stories of media under multiplrocesses

Threads in WebRTC

WebRTC is composed by capture module, coding module and streamming protocol module. To address performance bottleneck, we need to be familiar with webrtc threading module, which include the role of each thread and relationship between threads.

Here are the threads in WebRTC(signaling threads are excluded)

(MediaStreamGraph) Media stream graph run thread: audio/video coding.(MediaStreamGraphImpl::RunThread in MediaStreamGraph.cpp)
(Network) Socket transport service: send/receive packets. (Entry point of user space callback function??)
(Capture) Camera capture thread (from camera api): recieve image frame from camera.(Entry point of user space callback function??)
(Capture) Audio capture thread: recieve audio frame from microphone. (Entry point of user space callback function??)
(Process) Process thread (worker threads in GIPS): doing many other tasks. Process thread has a task queue for client to inject tasks into.

In a nut shell, we can divide these threads into three categories.

encode path

Encode path start from capture(getUserMedia).
MediaPipelineListner listen update notification(NotifyQueueTrackChanges) from MSG Run Thread and
- Encode audio chunks in MSG Run Thread.
- Encode video chunks in another thread(ViECapter Thread).
  - Put Encoded media data into Transport Service Thread to network

Camera capture
Mic capture

Decode path

Steven, please update whole story from network/ jitter buffer to renderer.

Audio decode
Video decode
RTP receive

Process dispatcher threads

Process thread is a dispatcher thread. A user registers a Module into a ProcessThread, ProcessThread will callback to Module::Process in specific duration(>= 100 ms).
Implementation of ProcessThread is located in process_thread_impl.cc
http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/utility/source/process_thread_impl.cc
ModuleRtpRtcpImpl::Process() is an example enjected module.(Main goal of this function is to do RTCP jobs)
http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/rtp_rtcp/source/rtp_rtcp_impl.cc

RTCP - NACK/ Statistic

Process threads

Reference