TPEPlatform/MediaRecorder: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(45 intermediate revisions by 2 users not shown)
Line 7: Line 7:
MediaRecorder takes media stream, which may come from web audio or media source or user media, as input. User may process all these media streams before recording. In mozCamera, there is nothing that user can do between capturerer and recorder, except canned video effect.
MediaRecorder takes media stream, which may come from web audio or media source or user media, as input. User may process all these media streams before recording. In mozCamera, there is nothing that user can do between capturerer and recorder, except canned video effect.


The following table reveals encode and associate container format we are going to support:
To totally replace mozCamera, we need to support these three WebAPI
#MediaRecorder: replace video recording function in mozCameraControl
#ImageCapture: replace takePhoto function in mozCameraControl
#CameraControl: except the first two function, we still need an API to control camera for 3A/ Zoom/ etc..
 
 
On B2G, there are two strategies
#B2G 1.3 on JerryBean: Base on JB::MediaCodec, we will support OmxVideoTrackEncoder and OmxAudioTrackEncoder to leverage HW encoder.
#B2G 1.3 on ICS: in this case, we still use VP8 and WebM.
 
==Tracking Issues==
META bugs for Encoding framework:
{| class="fullwidth-table sortable" style="width: 100%;"
{| class="fullwidth-table sortable" style="width: 100%;"
|-
|-
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" width="50px" | '''Type'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" width="100px" | '''Bug No'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Encoder'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Description'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Detailed'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Container'''
|-
|-
| rowspan=4 valign="top" | Audio
| {{bug|803414}}
| Opus
| MediaRecorder WebAPI
| mono/ stereo
| Ogg
|-
|-
| Vobis
| {{bug|879688}}
| mono/ stereo
| MediaEncoder pipeline
| Ogg
|-
|-
| AAC
| {{bug|888177}}
| mono/ stereo/ 5.0/ 5.1
| ImageCapture WebAPI
| MPEG-4
|-
| PCM
| 16 bit LPCM
| Wave
|-
| rowspan=2 valign="top" | Video
| H.264
| Baseline Profile
| MPEG-4
|-
| VP8
|  
| WebM
|}
|}


==Encoder framework==
Main tracking issues
META bugs for Encoding framework:
*{{bug|879688}}: Implement MediaEncoder
*{{bug|842243}}: Implement Audio encoding path
*{{bug|879669}}: Implement Video encoding path
*{{bug|888177}}: Implement ImageCapture API
 
Dependency among modules
*GetUserMedia
**Recoderging Pipeline
***Audio Encoding - Wav/ Opus/ Vobis/ AAC/ HEAAC
***Video Encoding - VP8/ VP9/ H.264/ HEVC
***Container multiplexer - MP4/ OGG/ WebM
**ImageCapture
***Image Encoding - JPG?
===Class Diagram===
'''The class diagram of encoder framework'''
[[File:Class_Diagram-Encoder.jpg|1000px]]
===Sequential Diagram===
'''The sequence diagram of receiving input source from MediaStreamGraph, and outputting final container data from MediaEncoder to MediaRecorder'''
[[File:Sequence_Diagram-Encoder.jpg|1000px]]
 
==VP9 or H.264==
List pros and cons for each audio/ video codec. We need more clear view to prioritize codec support sequence.
 
Requirements for a video codec:
#Image quality - comparative data was sought, but without defining a baseline
#Performance - what resolutions / frame rates can be achieved in software on some common systems
#Power consumption of hardware and/or software implementations
#Hardware support
#IPR status
 
==Codec/ Container support==
Schedule is not well define yet. All the information list bellow is not confirmed.<br>
===B2G===
{| class="fullwidth-table sortable" style="width: 100%;"
{| class="fullwidth-table sortable" style="width: 100%;"
|-
|-
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" align="center" width="50px" | '''Type'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Bug No'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Milestone'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Description'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Target '''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Statue'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" align="center" width="100px"| '''Issue'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Assigned'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Assignee'''
|-
|-
|-
|rowspan=3|Audio
| Opus/ OGG
| Gecko 24
| {{bug|842243}}
| {{bug|842243}}
| Audio encoding path
| FIXED
| Shelly
| Shelly
|-
|-
| LPCM/ Wav
| {{bug|919905}}
| TBD
| Refactor encoder architecture to support the video encoder module
|  
| FIXED
|  
| Randy
|-
| <font color="red">BLOCKER: </font>{{bug|879669}}
| Video encoding path
| Open
| Randy
|-
| {{bug|891705}}
| WebM 1.0 container writer
| Open
| Randy
|-
| {{bug|883749}}
| Vorbis encoding
| Open
| Benjamin
|-
|-
| <strike>MP3</strike>
| {{bug|881840}}
| <strike>Not defined</strike>
| VP8 encoding
| <strike>{{bug|884162}}</strike>
| Open
| Randy
|-
|-
|rowspan=3|Video
| H.264/ AAC/ MP4
| Gecko 28
| {{bug|879668}}
| {{bug|879668}}
|
| VideoOmxEncoder encoding
| Open
| Shelly, John
|-
|-
| VP8/ Vobis/ WebM
| {{bug|920934}}
| Gecko 27
| AudioOmxEncoder encoding
| {{bug|881840}}
| Open
|
| Shelly, John
|-
|-
| VP9/ Opus/ WebM
| {{bug|891704}}
| ??/??
| MP4Writer container writer
| TBD
| Open
|
| Alfredo
|}
|}


===Android===
==Testing Plan==
#Jason Smith: https://etherpad.mozilla.org/media-recording-testing
 
{| class="fullwidth-table sortable" style="width: 100%;"
{| class="fullwidth-table sortable" style="width: 100%;"
|-
|-
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Milestone'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Bug No'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Target '''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Description'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Issue'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Statue'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Assignee'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Target'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Assigned'''
|-
| {{bug|889772}}
| Audio Recording - Marionette Test Case
| Open
| Gecko 25
| Randy Lin
|-
|-
| Opus/ OGG
| TBD
| 6/14
| Video Recording - Marionette Test Case
| {{bug|842243}}
| Open
| Shelly
| Gecko 26
| Randy Lin
|}
|}


===Linux===
==Schedule==
 
{| class="fullwidth-table sortable" style="width: 100%;"
{| class="fullwidth-table sortable" style="width: 100%;"
|-
|-
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" align="center" width="50px" | '''Type'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Milestone'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Milestone'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Target '''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Target '''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Issue '''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" align="center" | '''Platform'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Assignee'''
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Assignee'''
|-
|-
| Opus/ OGG
|rowspan=5|Audio
| 6/14
| Opus
| {{bug|842243}}
| Gecko 24
| All
| Shelly
| Shelly
|}
===Windows===
{| class="fullwidth-table sortable" style="width: 100%;"
|-
|-
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Milestone'''
| Vorbis
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Target Date'''
| Gecko 28
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Issue No'''
| All
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Assignee'''
| Benjamin
|-
|-
| Opus/ OGG
| OmxAudio
| 6/14
| B2G 1.3
| {{bug|842243}}
| B2G 1.3 with Android Jerry Bean
| Shelly
| Shelly
|}
===MacOS===
{| class="fullwidth-table sortable" style="width: 100%;"
|-
|-
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Milestone'''
| <strike>LPCM/ Wav</strike>
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Target Date'''
| <strike>  </strike>
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Issue No'''
| <strike>  </strike>
| style="background: none repeat scroll 0% 0% rgb(239, 239, 239);" | '''Assignee'''
| <strike>  </strike>
|-
| <strike>MP3</strike>
| <strike>  </strike>
| <strike>  </strike>
| <strike>  </strike>
|-
|rowspan=3|Video
| OmxVideo
| B2G 1.3
| B2G 1.3 with Android Jerry Bean
| Shelly Lin, John Lin
|-
| VP8
| Gecko 28
| All
| Randy Lin, Benjamin Chen
|-
|-
| Opus/ OGG
| VP9
| 6/14
| TBD
| {{bug|842243}}
| TBD
| Shelly
|  
|}
|}


Line 176: Line 171:
**https://groups.google.com/forum/#!topic/mozilla.dev.media/y-KoshuUR44
**https://groups.google.com/forum/#!topic/mozilla.dev.media/y-KoshuUR44
*[http://dev.w3.org/2011/webrtc/editor/getusermedia.html GetUserMedia Spec]
*[http://dev.w3.org/2011/webrtc/editor/getusermedia.html GetUserMedia Spec]
*[http://www.w3.org/TR/2013/WD-mediastream-recording-20130205/ MediaStream Recording Spec]
*[https://dvcs.w3.org/hg/dap/raw-file/default/media-stream-capture/MediaRecorder.html MediaStream Recording Spec]
*[https://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-capture/ImageCapture.html Image Capture Spec]
*[https://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-capture/ImageCapture.html Image Capture Spec]
*[http://www.w3.org/TR/html-media-capture/ HTML media capture (only supports simple capture functionalities)], associated bug {{bug|741393}}
*[http://www.w3.org/TR/html-media-capture/ HTML media capture (only supports simple capture functionalities)], associated bug {{bug|741393}}

Latest revision as of 06:40, 28 October 2013

Summary

We already have mozCamera for take photo or video recording. mozCamera itself is not part of any standard web spec, which means

  1. It works on Firefox platform only.
  2. Totally isolate in media world. There is no way to connect data stream from mozCamera with media processing module, such as WebAudio, or recording module, such as MediaRecorder.

MediaRecorder takes media stream, which may come from web audio or media source or user media, as input. User may process all these media streams before recording. In mozCamera, there is nothing that user can do between capturerer and recorder, except canned video effect.

To totally replace mozCamera, we need to support these three WebAPI

  1. MediaRecorder: replace video recording function in mozCameraControl
  2. ImageCapture: replace takePhoto function in mozCameraControl
  3. CameraControl: except the first two function, we still need an API to control camera for 3A/ Zoom/ etc..


On B2G, there are two strategies

  1. B2G 1.3 on JerryBean: Base on JB::MediaCodec, we will support OmxVideoTrackEncoder and OmxAudioTrackEncoder to leverage HW encoder.
  2. B2G 1.3 on ICS: in this case, we still use VP8 and WebM.

Tracking Issues

META bugs for Encoding framework:

Bug No Description
bug 803414 MediaRecorder WebAPI
bug 879688 MediaEncoder pipeline
bug 888177 ImageCapture WebAPI

Main tracking issues

Bug No Description Statue Assigned
bug 842243 Audio encoding path FIXED Shelly
bug 919905 Refactor encoder architecture to support the video encoder module FIXED Randy
BLOCKER: bug 879669 Video encoding path Open Randy
bug 891705 WebM 1.0 container writer Open Randy
bug 883749 Vorbis encoding Open Benjamin
bug 881840 VP8 encoding Open Randy
bug 879668 VideoOmxEncoder encoding Open Shelly, John
bug 920934 AudioOmxEncoder encoding Open Shelly, John
bug 891704 MP4Writer container writer Open Alfredo

Testing Plan

  1. Jason Smith: https://etherpad.mozilla.org/media-recording-testing
Bug No Description Statue Target Assigned
bug 889772 Audio Recording - Marionette Test Case Open Gecko 25 Randy Lin
TBD Video Recording - Marionette Test Case Open Gecko 26 Randy Lin

Schedule

Type Milestone Target Platform Assignee
Audio Opus Gecko 24 All Shelly
Vorbis Gecko 28 All Benjamin
OmxAudio B2G 1.3 B2G 1.3 with Android Jerry Bean Shelly
LPCM/ Wav
MP3
Video OmxVideo B2G 1.3 B2G 1.3 with Android Jerry Bean Shelly Lin, John Lin
VP8 Gecko 28 All Randy Lin, Benjamin Chen
VP9 TBD TBD

Reference

Retrieved from "https://wiki.mozilla.org/index.php?title=TPEPlatform/MediaRecorder&oldid=738907"