11
edits
Plugins:PepperAudioAPI: Difference between revisions
Jump to navigation
Jump to search
→Problem Summary
No edit summary |
|||
| Line 33: | Line 33: | ||
*Implemented in Pepper's device API | *Implemented in Pepper's device API | ||
The browser might be responsible for applying additional user volume or muting control, | The browser might be responsible for applying additional user volume or muting control,and may upsample/downsample the audio data before mixing with other browser audiosources. | ||
All future versions of PEPPER will support version 1 (this proposal) as a bare minimum. | All future versions of PEPPER will support version 1 (this proposal) as a bare minimum. | ||
Trusted PEPPER modules will be able to open more than one audio context. | Trusted PEPPER modules will be able to open more than one audio context.Untrusted PEPPER modules might be restricted to a single audio context.In either environment, it is assumed multi-threading is available. | ||
This document contains two models - one based on callbacks, and one based on a | This document contains two models - one based on callbacks, and one based on a blockingpush model. While both models are fundamentally the same, they do carry some subtledifferences: | ||
*Blocking Push API provides more control, and assumes threading will be explicitly handled by the application. When writing trusted PEPPER modules without a platform independent threading library, this may imply the application writer to #if WIN32, #if PTHREADS, etc. around the application's audio thread setup code. | *Blocking Push API provides more control, and assumes threading will be explicitly handled by the application. When writing trusted PEPPER modules without a platform independent threading library, this may imply the application writer to #if WIN32, #if PTHREADS, etc. around the application's audio thread setup code. | ||
*Callback API does not require application to setup/teardown audio thread, and the API entry points are all on the NPAPI thread -- only the callback occurs on an implicit, dedicated audio thread. | *Callback API does not require application to setup/teardown audio thread, and the API entry points are all on the NPAPI thread -- only the callback occurs on an implicit, dedicated audio thread. | ||
*There might be some subtle implementation details when it comes to how each model needs to signal between the audio device and the PEPPER audio interface. These signals will need to be as fast and low latency as possible, so minimizing how many signals are required will be important. | *There might be some subtle implementation details when it comes to how each model needs to signal between the audio device and the PEPPER audio interface. These signals will need to be as fast and low latency as possible, so minimizing how many signals are required will be important. | ||
=====Background:===== | =====Background:===== | ||
Native Client plug-ns have up until now used an audio API very similar to a | Native Client plug-ns have up until now used an audio API very similar to a blockingflushContext() call, usually done on a thread dedicated to audio. The API call wasimplemented as a syscall, and at the service runtime (kernel) level used SDL to deliver lowlatency audio across multiple platforms. As Native Client moves into a more securesandbox environment, untrusted code will no longer be able to communicate directly withaudio devices in the same process. In the new audio model, trusted code in anotherprocess will be attached to the audio device, and the sandboxed Native Client untrustedapplication will use shared memory and sockets to communicate its audio needs outside thesandbox. The new API for this model will be part of PEPPER, a variant of NPAPI that isdesigned to be platform neutral. | ||
Please refer to Appendix A for a list of links pertaining to audio. | Please refer to Appendix A for a list of links pertaining to audio.Please refer to Appendix B for PEPPER design wiki -- where this document will migrate onceapproved. | ||
Both models will need additional PEPPER events. Please refer to Appendix C for | Both models will need additional PEPPER events. Please refer to Appendix C for audiorelated events. | ||
Both models support multi-channel configurations. Please refer to Appendix E for multi- | Both models support multi-channel configurations. Please refer to Appendix E for multi-channel output scenarios.Please refer to Appendix F for input scenarios.Please refer to Appendix G for contributers. | ||
===Changes to Pepper's Device API=== | ===Changes to Pepper's Device API=== | ||
Previously, Pepper's device API consisted of the methods initializeContext, flushContext, and destroyContext. Audio extends this API to queryCapability, queryConfig, initializeContext, flushContext, getState, setState, and destroyContext. | Previously, Pepper's device API consisted of the methods initializeContext, flushContext, and destroyContext. Audio extends this API to queryCapability, queryConfig, initializeContext, flushContext, getState, setState, and destroyContext. | ||
:queryCapability - ask a device about a specific capability | |||
:queryConfig - ask a device about a configuration (a set of capabilities) | |||
:initializeContext - given a configuration, initialize a device context | |||
:flushContext - flush context to device, either blocking or with a completion callback | |||
:getStateContext - get state information from device context | |||
:setStateContext - set device context state | |||
:destroyContext - shutdown and destroy device context | |||
===NPAPI Thead, Plug-ins, and Plug-in Instances=== | ===NPAPI Thead, Plug-ins, and Plug-in Instances=== | ||
The following device methods must be called from the NPAPI thread (Tp). They should | The following device methods must be called from the NPAPI thread (Tp). They should notbe called from any other thread, unless it is via NPN_ThreadAsyncCall. | ||
:queryCapability, queryConfig, initializeContext, getStateContext, setStateContext,destroyContext | |||
The only device method that can be called from another thread is flushContext when | The only device method that can be called from another thread is flushContext when usingthe push model. When using the callback model, it is expected that the user suppliedcallback will be invoked on a thread dedicated to audio. This dedicated audio thead is notan NPAPI thread. | ||
When creating an audio context, usually this is done as part of a plug-in instance. In | When creating an audio context, usually this is done as part of a plug-in instance. In untrusted Pepper modules, there will be one instance per sandboxed process. However, this could change in the future, and an untrusted Pepper plug-in could host multiple instances in the same sandbox. In this case, it may be desirable to have one audio device, with it's own application supplied mixer, to be shared amongst mutliple untrusted plug-in instances. This audio context should be created at Plug-in initialization, using NULL as thenpp instance argument. | ||
===Obtaining Pepper Extensions and the Pepper Audio Interface=== | |||
Regardless of callback vs. push model, first the Pepper audio interface must be acquired. | |||
<tt> | |||
NPPepperExtensions *pepper;<br> | |||
NPAudio *audio = NULL;<br> | |||
/* Get PEPPER extensions */<br> | |||
NPERR ret = NPN_GetValue(instance, NPNVPepperExtensions, &pepper);<br> | |||
if (NPERR_NO_ERROR == ret) {<br> | |||
/* successfully obtained Pepper extensions, now acquire audio... */<br> | |||
audio = pepper->acquireDevice(NPAudioDeviceID);<br> | |||
} else {<br> | |||
/* Pepper extensions are not available */<br> | |||
}<br> | |||
/* check for acquisition of audio device interface */<br> | |||
if (NULL != audio) {<br> | |||
/* Pepper audio device interface is available */<br> | |||
... | |||
</tt> | |||
Once an interface to the audio device is acquired, an application can query capabilities of the device, and from there, create a configuration (a set of capabilities.) In this simple example, the application picks 44.1kHz, stereo output using int16 sample types. It uses query Capability to retrieve a recommended sample frame count for the output buffer. | |||
<tt> | |||
...<br> | |||
/* fill out request, use sample buffer size from configuration */<br> | |||
NPAudioConfig request;<br> | |||
NPAudioConfig config;<br> | |||
int32 ret;<br> | |||
request.sampleRate = NPAudioSampleRate44100Hz;<br> | |||
request.sampleType = NPAudioSampleTypeInt16;<br> | |||
ret = audio->queryCapability(npp,<br> | |||
NPAudioQuerySampleFrameCount44100Hz, &request.sampleFrameCount);<br> | |||
request.outputChannelMap = NPAudioChannelStereo;<br> | |||
request.inputChannelMap = NPAudioChannelNone;<br> | |||
request.flags = 0;<br> | |||
request.callback = NULL;<br> | |||
request.userData = NULL;<br> | |||
ret = audio->queryConfig(npp, &request, &config);<br> | |||
if (NPERR_NO_ERROR == ret) {<br> | |||
/* config struct now contains a valid audio device config */<br> | |||
... | |||
</tt> | |||
This example uses a very standard audio device configuration. When requesting less common configurations, an application should be prepared to work with the configuration returned by queryConfig(), which may end up being different than the requested configuration. For example, a device might return that it is capable of emitting audio at 96kHz. It could also return that it is capable of 5.1 channel output. However, when the application requests a configuration of 5.1 channel output at 96kHz, query Config may return back that in this case, the device is only capable of 48kHz playback over 5.1 channels. The application then has to decide, take the 5.1 output at 48kHz, or try another configuration, | |||
perhaps stereo at 96kHz, depending on which is more important - number of output | |||
channels or sample rate. | |||
To make the most common cases easy, Pepper audio guarantees availability of 44.1kHz, 48kHz, int16 sample types, and stereo output. Pepper audio will automatically resample if needed, and do stereo->mono mixdown if needed. | |||
If an application desires, it can query Pepper audio for the system's internal sample rate. Using this sample rate will avoid resampling (which may occur at either the device driver level or a software mixer.) | |||
===Mode One: Callback Model=== | |||
The callback model has some restrictions -- all functions except the callback itself will occur on the NPAPI thread. The callback will occur on a high priority thread dedicated to serving audio. This thread is created implicitly during initialization, and cleaned up automatically at shutdown. | |||