MozillaWiki

TPEPlatform/WebRTC B2G: Difference between revisions

Page Discussion

TPEPlatform/WebRTC B2G (view source)

Revision as of 23:44, 1 April 2014

4,273 bytes added ,  1 April 2014
→‎Performance
 
(36 intermediate revisions by 4 users not shown)
Line 1: Line 1:
==Summary==
==Summary==
This document puts focus on B2G only. Cross platform issues are not involved.
This document puts focus on B2G only. Cross platform issues are not involved.
*{{bug|750011}} - [meta] Support B2G for WebRTC
#WebRTC B2G META issue
** A meta issue to trace all WebRTC_B2G blocking issues.
#*{{bug|750011}} - [meta] Support B2G for WebRTC
*[[TPEPlatform/WebRTC_B2G/Workweek_note|WebRTC workweek minutes(3rd Jun ~ 7th Jun)]]
#[[TPEPlatform/WebRTC_B2G/Workweek_note|WebRTC workweek minutes(3rd Jun ~ 7th Jun)]]
*[[Media/WebRTC_Audio_Issues| Audio issues]]
#Relative WIKIs
#*[[Media/WebRTC_Audio_Issues| Audio issues]]
 
In bug tables, item description with read color means blocker issues


==Network==
==Network==
On B2G, we put focus on  
On B2G, we put focus on  
#E10S: chrome/ content process module on FirefoxOS
#E10S
#*Sandboxed. Move UDP socket into Chrome process on FirefoxOS
#*Packet filter in Chrome process.
#Network interface enumeration and prioritization: WLAN & 3G connection transition  
#Network interface enumeration and prioritization: WLAN & 3G connection transition  
#SDP
#*SDP parsing for video/ audio frame parameter. For example, maximum frame rate, or maximum frame size.
#*Request video/audio frame parameter base on HW capability.
#ICE
#*Error reporting: application is able to get ICE error/ failed callback.


{| class="fullwidth-table sortable" style="width: 100%;"
{| class="fullwidth-table sortable" style="width: 100%;"
Line 21: Line 31:
| {{bug|825708}}
| {{bug|825708}}
| We should use interface properties to determine ICE priorities
| We should use interface properties to determine ICE priorities
| Open
| FIXED
| Gecko 26
| Gecko 26
| Patrick
| Patrick
|-
|-
| {{bug|869869}}
| {{bug|869869}}
| e10s for UDP socket
| <font color="red">e10s for UDP socket</font>
| Open
| Open
| Gecko 26
| Gecko 26
Line 32: Line 42:
|-
|-
| {{bug|870660}}
| {{bug|870660}}
| Packet filter for UDP e10s
| <font color="red">Packet filter for UDP e10s</font>
| Open
| Open
| Gecko 26
| Gecko 26
Line 38: Line 48:
|-
|-
| {{bug|881761}}
| {{bug|881761}}
| NSS for WebRTC in content process
| <font color="red">NSS for WebRTC in content process</font>
| Open
| Open
| Gecko 26
| Gecko 26
| ???
| Patrick
|-
| {{bug|881935}}
| SDP: Support negotiation of video resolution
| Open
| Gecko 26
| Adam Roach
|-
|-
| {{bug|881982}}
| {{bug|881982}}
Line 59: Line 63:
| Open
| Open
| Gecko 26
| Gecko 26
| Patrick
| Shian-Yow
|-
|-
| {{bug|881935}}
| {{bug|881935}}
| Support negotiation of video resolution
| Support negotiation of video resolution
| Open
| FIXED
| Gecko 26
| Gecko 26
| SWU/ Adam
| Shian-Yow
|}
|}


Line 77: Line 81:
#Use reasonable timer interval in Process Thread on B2G. Less statistic data collection, no NACK.
#Use reasonable timer interval in Process Thread on B2G. Less statistic data collection, no NACK.
#Test OPUS with lower complexity and decide whether B2G uses OPUS or G711 as default audio codec
#Test OPUS with lower complexity and decide whether B2G uses OPUS or G711 as default audio codec
#H.264 coding module
#*H.264 encoder/ decoder with HW codec
#*H.264 RTP transport.


{| class="fullwidth-table sortable" style="width: 100%;"
{| class="fullwidth-table sortable" style="width: 100%;"
Line 88: Line 96:
| {{bug|884365}}
| {{bug|884365}}
| Audio realtime input clock mismatch  
| Audio realtime input clock mismatch  
| Open
| FIXED
| Gecko 26
| Gecko 26
| Randell
| Randell
Line 97: Line 105:
| Gecko 26
| Gecko 26
| Steven
| Steven
|-
| {{bug|896391}}
| memcpy from camera preview's GraphicBuffer is slow
| Open
|
| Steven
|-
| {{bug|877954}}
| Adapt video encode resolution & framerate according to available bandwidth and CPU use
| FIXED
| Gecko 28
| gpascutto
|}
|}


TODO: H.264 encoder/ decoder - john.
==gUM==
 
{| class="fullwidth-table sortable" style="width: 100%;"
==Permission==
|-
 
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239); width="100px" | '''Bug No'''
{{bug|853356}}:
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Description'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Statue'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Target'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Assigned'''
|-
| {{bug|853356}}
| <font color="red">Display camera/ microphone permission acquisition prompt by ContentPermmissionReques</font>
| FIXED
| Gecko 26
| Alfredo
|-
| {{bug|898949}}
| [B2G getUserMedia] Display front/back camera list on permission prompt
| FIXED
| Gecko 26
| S.C
|-
| {{bug|913896}}
| Display audio (microphone) permission in permission acquisition prompt
| FIXED
| Gecko 26
| Fred Lin
|}


==Media Resource Management==
==Media Resource Management==
Line 112: Line 154:
#user stories of media under multiplrocesses
#user stories of media under multiplrocesses


==Threads in WebRTC==
==WebRTC Threading Modal==
WebRTC is composed by capture module, coding module and streamming protocol module. To address performance bottleneck, we need to be familiar with webrtc threading module, which include role of each thread and relationship between two thread.
WebRTC is composed by capture module, coding module and streamming protocol module. To address performance bottleneck, we need to be familiar with webrtc threading module, which include the role of each thread and relationship between threads.


Here are the threads in WebRTC(signaling threads are excluded)
Here are the threads in WebRTC(signaling threads are excluded)
# Media stream graph run thread: audio/video synchronization and dispatching
#(MediaStreamGraph) Media stream graph run thread: audio/video coding.(MediaStreamGraphImpl::RunThread in MediaStreamGraph.cpp)
# socket transport service: send/receive packets
#(Network) Socket transport service: send/receive packets. (Entry point of user space callback function??)
# camera capture thread (from camera api): get images from camera
#(Capture) Camera capture thread: On FFOS, video frames are callback through MediaEngineWebRTCVideoSource::OnNewFrame and the source is camera api. For other platforms, the images are from MediaEngineWebRTCVideoSource::DeliverFrame, the callback interface of GIPS, and the source is implemented in GIPS. Then MSG thread keeps pulling the latest frames by MediaEngineWebRTCVideoSource::NotifyPull.
# audio capture thread: get audio streaming from mic
#(Capture) Audio capture thread: recieve audio frame from microphone. All audio streams are input through MediaEngineWebRTCAudioSource::Process. In "Process" function, the audio is saving to media track. The mechanism may change since it has clock drift problem(bug 884365).
# process thread (worker thread in GIPS): doing many other tasks. Process thread has a task queue for client to inject tasks into.  
#(Process) Process thread (worker threads in GIPS): doing many other tasks. Process thread has a task queue for client to inject tasks into.  
Basically, we may divide these threads into three categories.  
 
*'''encode path'''
In a nut shell, we can divide these threads into three categories.  
**Encode path start from capture(getUserMedia).
===encode path===
**MediaPipelineListner listen update notification(NotifyQueueTrackChanges) from MSG Run Thread and
*Encode path start from capture(getUserMedia).
***Encode audio chunks in MSG Run Thread.
*MediaPipelineListner listen update notification(NotifyQueueTrackChanges) from MSG Run Thread and
***Encode video chunks in another thread(ViECapter Thread).
**Encode audio chunks in MSG Run Thread.
****Put Encoded media data into Transport Service Thread to network
**Encode video chunks in another thread(ViECapter Thread).
***Put Encoded media data into Transport Service Thread to network
<gallery>
<gallery>
FIle:Camera_cap_.jpg|Camera capture
FIle:Camera_cap_.jpg|Camera capture
File:MicCapture_.jpg|Mic capture
File:MicCapture_.jpg|Mic capture
</gallery>
</gallery>
*'''decode path'''
===Decode path===
**''Steven, please update whole story from network/ jitter buffer to renderer.''
*''Steven, please update whole story from network/ jitter buffer to renderer.''
<gallery>
<gallery>
File:AudioDecode_.jpg|Audio decode
File:AudioDecode_.jpg|Audio decode
Line 139: Line 182:
File:ReceiveRTPPakcets_.jpg|RTP receive  
File:ReceiveRTPPakcets_.jpg|RTP receive  
</gallery>
</gallery>
*'''Process dispatcher threads'''
===Process dispatcher threads===
**RTCP - NACK/ Statistic  
Process thread is a dispatcher thread. A client can register a Module into a ProcessThread. ProcessThread will callback to Module::Process in specific duration(>= 100 ms).<br>
Implementation of ProcessThread is located in [http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/utility/source/process_thread_impl.cc process_thread_impl.cc]<br>
Here are modules that implement the Process function:<br>
[http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/video_engine/call_stats.cc call_stats.cc], [http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/video_engine/vie_remb.cc vie_remb.cc], [http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/video_engine/vie_sync_module.cc vie_sync_module.cc], [http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/voice_engine/monitor_module.cc monitor_module.cc], [http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/audio_device/audio_device_impl.cc audio_device_impl.cc], [http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/pacing/paced_sender.cc paced_sender.cc], [http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/video_capture/video_capture_impl.cc video_capture_impl.cc], [http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/audio_conference_mixer/source/audio_conference_mixer_impl.cc audio_conference_mixer_impl.cc], [http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/rtp_rtcp/source/rtp_rtcp_impl.cc rtp_rtcp_impl.cc], [http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/audio_coding/main/source/audio_coding_module_impl.cc audio_coding_module_impl.cc], [http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/video_coding/main/source/video_coding_impl.cc video_coding_impl.cc]
 
*RTCP - NACK/ Statistic  
<gallery>
<gallery>
File:ProcessThread_.jpg|Process threads
File:ProcessThread_.jpg|Process threads
</gallery>
</gallery>
==Next Step==
#Integrate WebRTC with MediaRecord.
==FAQ==
* How to know the current resolution captured by camera?
Use gdb and break at MediaEngineWebRTCVideoSource::OnNewFrame()


==Reference==
==Reference==
*Real-time communication with WebRTC: Google I/O 2013. http://www.youtube.com/watch?feature=player_embedded&v=p2HzZkd2A40
*Real-time communication with WebRTC: Google I/O 2013. http://www.youtube.com/watch?feature=player_embedded&v=p2HzZkd2A40
Pchangbin
5

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/671960...959854"