Media/multichannel

Overview

Current Overall Status

1. Multiple channel support for cubeb on Windows is implemented
2. Mixing module for cubeb is implemented and mixing test is added
3. Have a WIP patch to make multiple channel works in Firefox Windows
4. Surveying a more robust way to test the multiple channel support

Roadmap

There are three phases for the implementation.

1. Basic support
   1. Implement basic support for each backends
      1. Implement multiple channel support in cubeb on Windows
      2. Support multiple channel for Firefox Windows based on previous cubeb's work
      3. Repeat the work for OSX, Linux, ... and other platforms.
   2. Implement mixing module beyond backends
      1. Implement the defined mixing mechanisms(e.g., 3f2 downmix)
      2. Design alternative mechanisms for the undefined conversions
   3. Testing: Design a reliable test for multiple channel support
      1. Fake a virtual audio device on Windows
      2. Intercept the audio output data to verify via the faked device.
      3. Repeat the work for OSX, Linux, ... and other platforms.
2. Integration with device-switching support(this has not implemented yet)
3. Fancy feature support
   1. stereo-to-5.1 simulation

Integration cubeb with gecko

The Gecko's AudioConverter downmix the audio data which has more than two channels into stereo or mono before passing it to cubeb. The reason to do that is because cubeb has no multi-channel support. Once cubeb implements multi-channel support and mixing mechanism, there is no need to downmix first in gecko.

We should move the downmix mechanism from gecko into cubeb one by one on each platform. If we implement the multi-channel support on Windows, then we should disallow the downmix mechanism in gecko on Windows and let cubeb to do that job. The downmix in gecko will be removed until the multi-channel support is implemented on each backend.

Timeline

Evaluate dates here.

Status tracking

Cubeb

Full Query

ID	Summary	Product	Component	Resolution	Assigned to	Depends on	Blocks	Whiteboard	Target milestone
1619726	Use the new Rust audio mixer in cubeb windows	Core	Audio/Video: cubeb		C.M.Chang[:chunmin]		1073786, 1368938		---
1300024	Support audio 5.1 on Android	Core	Audio/Video: cubeb		C.M.Chang[:chunmin]		1073786, 1286101		---
1368938	Support audio 5.1 with non-SMPTE layout devices	Core	Audio/Video: cubeb		Paul Adenot (:padenot)	1619726	1073786, 1325023		---
1627827	Audio goes wrong channel with Motu 828 mkii	Core	Audio/Video: cubeb		C.M.Chang[:chunmin]	1628132	1073786	[media-audio]	---

4 Total; 4 Open (100%); 0 Resolved (0%); 0 Verified (0%);

Playback

Insert bug list for playback here

Multiple Channel Support

Supported channels

To support more channels beyond Stereo(who has left and right channels), we need to define what channels we use:

Code	Channel Name
Ｍ	Mono
L	Left(Front Left)
R	Right(Front Right)
C	Center(Front Center)
LS	Left Surround(Side Left)
RS	Right Surround(Side Right)
RLS	Rear Left Surround(Back Left)
RRC	Rear Center(Back Center)
RRS	Rear Right Surround(Bake Right)
LFE	Low Frequency Effects

Channel Layout

Channel layout specifies the order of input/output channel data in audio buffer. For example, if the layout is stereo, then we have two channel data. The first data is for left channel, the second one is right channel.

Channel layout has various definitions, but SMPTE's format is most common:

Name	Channels
DUAL-MONO	L	R
DUAL-MONO-LFE	L	R	LFE
MONO	M
MONO-LFE	M	LFE
STEREO	L	R
STEREO-LFE	L	R	LFE
3F	L	R	C
3F-LFE	L	R	C	LFE
2F1	L	R	RC
2F1-LFE	L	R	LFE	RC
3F1	L	R	C	RC
3F1-LFE	L	R	C	LFE	RC
2F2	L	R	LS	RS
2F2-LFE	L	R	LFE	LS	RS
3F2	L	R	C	LS	RS
3F2-LFE	L	R	C	LFE	LS	RS
3F3R-LFE	L	R	C	LFE	RC	LS	RS
3F4-LFE	L	R	C	LFE	RLS	RRS	LS	RS

Mixing

When the number of audio input channels is different from the number of audio output channels, we need to convert the audio input data to fit the audio output's configuration.

Downmix

When input channels is larger than output channels, we need to compress the audio input data. The conversion is called downmix(downward mixing). The table 2 in ITU-R BS.775-3 defines equations to convert audio from 3F2 to 1F(mono), 2F(stereo), 3F, 2F1, 3F1 and 2F2. We can simply add a LFE value to expand the downmix matrix. It will allow it to convert audio from 3F2-LFE(5.1 surround sound) to 1F(mono), 2F(stereo), 3F, 2F1, 3F1 and 2F2 and their LFEs. Here is the code example.

Upmix

When input channels is smaller than output channels, we need to expand the audio input data.

Bypass

When input channels is equal to output channels but they have different layout, we need to bypass the audio input data.

References

• Multi-channel support discussion on Github
• [WIP] Multiple channels support for cubeb on Windows