Plugins:PepperAudioAPI

Status

Under consideration.

Problem Summary

Low level audio component of the Pepper API.

Low Level Audio in PEPPER

• Last modified: December 7, 2009
• Author: Nicholas Fullagar (Google)
• Contributors:

Objective:

• First pass implementation is to provide 16bit, stereo, CD (44.1kHz) or DAT (48kHz) quality audio.
• Deliver low latency audio on systems that have good audio drivers. (<=20ms)
• Be flexible enough to scale additional features in future versions
  • additional sample formats
  • additional speaker configurations
  • audio input
• It will be the application's responsibility to provide higher level audio functionality.
  • If an audio device is available, this API will guarantee
  • stereo output (regardless of physical speaker configuration)
  • the availability of both 44.1kHz and 48kHz output.
• the availability of int16 as a sample format.
• The basic audio interface is a pure "C" API.
• Only linear PCM formats will be supported.
• Straight forward to secure.
• Implemented in Pepper's device API

The browser might be responsible for applying additional user volume or muting control,
and may upsample/downsample the audio data before mixing with other browser audio
sources.

All future versions of PEPPER will support version 1 (this proposal) as a bare minimum.

Trusted PEPPER modules will be able to open more than one audio context.
Untrusted PEPPER modules might be restricted to a single audio context.
In either environment, it is assumed multi-threading is available.

This document contains two models - one based on callbacks, and one based on a blocking
push model. While both models are fundamentally the same, they do carry some subtle
differences:

• Blocking Push API provides more control, and assumes threading will be explicitly handled by the application. When writing trusted PEPPER modules without a platform independent threading library, this may imply the application writer to #if WIN32, #if PTHREADS, etc. around the application's audio thread setup code.
• Callback API does not require application to setup/teardown audio thread, and the API entry points are all on the NPAPI thread -- only the callback occurs on an implicit, dedicated audio thread.
• There might be some subtle implementation details when it comes to how each model needs to signal between the audio device and the PEPPER audio interface. These signals will need to be as fast and low latency as possible, so minimizing how many signals are required will be important.
  

Background:

Native Client plug-ns have up until now used an audio API very similar to a blocking
flushContext() call, usually done on a thread dedicated to audio. The API call was
implemented as a syscall, and at the service runtime (kernel) level used SDL to deliver low
latency audio across multiple platforms. As Native Client moves into a more secure
sandbox environment, untrusted code will no longer be able to communicate directly with
audio devices in the same process. In the new audio model, trusted code in another
process will be attached to the audio device, and the sandboxed Native Client untrusted
application will use shared memory and sockets to communicate its audio needs outside the
sandbox. The new API for this model will be part of PEPPER, a variant of NPAPI that is
designed to be platform neutral.

Please refer to Appendix A for a list of links pertaining to audio.
Please refer to Appendix B for PEPPER design wiki -- where this document will migrate once
approved.

Both models will need additional PEPPER events. Please refer to Appendix C for audio
related events.

Both models support multi-channel configurations. Please refer to Appendix E for multi-
channel output scenarios.
Please refer to Appendix F for input scenarios.
Please refer to Appendix G for contributers.

Changes to Pepper's Device API

Previously, Pepper's device API consisted of the methods initializeContext, flushContext, and destroyContext. Audio extends this API to queryCapability, queryConfig, initializeContext, flushContext, getState, setState, and destroyContext.

• queryCapability - ask a device about a specific capability
• queryConfig - ask a device about a configuration (a set of capabilities)
• initializeContext - given a configuration, initialize a device context
• flushContext - flush context to device, either blocking or with a completion callback
• getStateContext - get state information from device context
• setStateContext - set device context state
• destroyContext - shutdown and destroy device context

NPAPI Thead, Plug-ins, and Plug-in Instances

The following device methods must be called from the NPAPI thread (Tp). They should not
be called from any other thread, unless it is via NPN_ThreadAsyncCall.

• queryCapability, queryConfig, initializeContext, getStateContext, setStateContext,destroyContext
  

The only device method that can be called from another thread is flushContext when using
the push model. When using the callback model, it is expected that the user supplied
callback will be invoked on a thread dedicated to audio. This dedicated audio thead is not
an NPAPI thread.

When creating an audio context, usually this is done as part of a plug-in instance. In
untrusted Pepper modules, there will be one instance per sandboxed process. However,
this could change in the future, and an untrusted Pepper plug-in could host multiple
instances in the same sandbox. In this case, it may be desirable to have one audio device,
with it's own application supplied mixer, to be shared amongst mutliple untrusted plug-in
instances. This audio context should be created at Plug-in initialization, using NULL as the
npp instance argument.