TPEPlatform/MediaRecorder

Summary

We already have mozCamera for take photo or video recording. mozCamera itself is not part of any standard web spec, which means

1. It works on Firefox platform only.
2. Totally isolate in media world. There is no way to connect data stream from mozCamera with media processing module, such as WebAudio, or recording module, such as MediaRecorder.

MediaRecorder takes media stream, which may come from web audio or media source or user media, as input. User may process all these media streams before recording. In mozCamera, there is nothing that user can do between capturerer and recorder, except canned video effect.

The following table reveals encode and associate container format we are going to support:

Type	Encoder	Detailed	Container
Audio	Opus	mono/ stereo	Ogg
	Vobis	mono/ stereo	Ogg
	AAC	mono/ stereo/ 5.0/ 5.1	MPEG-4
	PCM	16 bit LPCM	Wave
Video	H.264	Baseline Profile	MPEG-4
	VP8		WebM

Encoder framework

META bugs for Encoding framework:

• bug 879688: Implement MediaEncoder
• bug 842243: Implement Audio encoding path
• bug 879669: Implement Video encoding path
• bug 888177: Implement ImageCapture API

Dependency among modules

• GetUserMedia
  • Recoderging Pipeline
    • Audio Encoding - Wav/ Opus/ Vobis/ AAC/ HEAAC
    • Video Encoding - VP8/ VP9/ H.264/ HEVC
    • Container multiplexer - MP4/ OGG/ WebM
  • ImageCapture
    • Image Encoding - JPG?

Class Diagram

The class diagram of encoder framework

Sequential Diagram

The sequence diagram of receiving input source from MediaStreamGraph, and outputting final container data from MediaEncoder to MediaRecorder

VP9 or H.264

List pros and cons for each audio/ video codec. We need more clear view to prioritize codec support sequence.

Requirements for a video codec:

1. Image quality - comparative data was sought, but without defining a baseline
2. Performance - what resolutions / frame rates can be achieved in software on some common systems
3. Power consumption of hardware and/or software implementations
4. Hardware support
5. Licensing

Codec/ Container support

Schedule is not well define yet. All the information list bellow is not confirmed.

B2G

Type	Milestone	Target	Issue	Assignee
Audio	Opus/ OGG	Gecko 24	bug 842243	Shelly
	LPCM/ Wav	TBD
	MP3	Not defined	bug 884162
Video	H.264/ AAC/ MP4	Gecko 28	bug 879668
	VP8/ Vobis/ WebM	Gecko 27	bug 881840
	VP9/ Opus/ WebM	??/??	TBD

Android

Milestone	Target	Issue	Assignee
Opus/ OGG	6/14	bug 842243	Shelly

Linux

Milestone	Target	Issue	Assignee
Opus/ OGG	6/14	bug 842243	Shelly

Windows

Milestone	Target Date	Issue No	Assignee
Opus/ OGG	6/14	bug 842243	Shelly

MacOS

Milestone	Target Date	Issue No	Assignee
Opus/ OGG	6/14	bug 842243	Shelly

Reference

• Meeting Minutes
• Mail thread
  • https://groups.google.com/forum/#!topic/mozilla.dev.media/y-KoshuUR44
• GetUserMedia Spec
• MediaStream Recording Spec
• Image Capture Spec
• HTML media capture (only supports simple capture functionalities), associated bug bug 741393
• Android JB encoder matrix
• iOS encorder matrix
• Firefox playback format
• VP9 Goolge I/O event
• Why VP8 should be that codec? (rtcweb-vp8-01)
• H.264 as Mandatory to Implement Video Codec for WebRTC
• H.264/AVC as Mandatory-to-Implement Video Codec for RTCweb