MozillaWiki

Plugins:PepperAudioAPI: Difference between revisions

Page Discussion

Plugins:PepperAudioAPI (view source)

Revision as of 16:02, 26 April 2010

15,232 bytes added ,  26 April 2010
→‎Problem Summary
 
(39 intermediate revisions by 3 users not shown)
Line 6: Line 6:


Low level audio component of the Pepper API.
Low level audio component of the Pepper API.
For more on Pepper, please see [[NPAPI:Pepper|Pepper Overview]]


=== Specification  ===
=== Specification  ===
Line 12: Line 14:


*Last modified: December 7, 2009  
*Last modified: December 7, 2009  
*Author: Nicholas Fullagar (Google)
*Contributors: [https://wiki.mozilla.org/Plugins:PepperAudioAPI#Appendix_G_-_Contributers Appendix G]
*Contributors:


=====Objective:=====
=====Objective:=====


*First pass implementation is to provide 16bit, stereo, CD (44.1kHz) or DAT (48kHz) quality audio.  
*First pass implementation is to provide 16bit, stereo, CD (44.1kHz) or DAT (48kHz) quality audio.  
*Deliver low latency audio on systems that have good audio drivers. (<=20ms)  
*Deliver low latency audio on systems that have good audio drivers. (<=20ms)  
*Be flexible enough to scale additional features in future versions  
*Be flexible enough to scale additional features in future versions  
Line 31: Line 32:
*Only linear PCM formats will be supported.  
*Only linear PCM formats will be supported.  
*Straight forward to secure.  
*Straight forward to secure.  
*Implemented in Pepper's device API  
*Implemented in Pepper's NPDevice API  


The browser might be responsible for applying additional user volume or muting control,and may upsample/downsample the audio data before mixing with other browser audiosources.  
The browser might be responsible for applying additional user volume or muting control,and may upsample/downsample the audio data before mixing with other browser audio sources.  


All future versions of PEPPER will support version 1 (this proposal) as a bare minimum.  
All future versions of PEPPER will support version 1 (this proposal) as a bare minimum.


Trusted PEPPER modules will be able to open more than one audio context.Untrusted PEPPER modules might be restricted to a single audio context.In either environment, it is assumed multi-threading is available.  
Trusted PEPPER modules will be able to open more than one audio context. Untrusted PEPPER modules might be restricted to a single audio context. In either environment, it is assumed multi-threading is available.


This document contains two models - one based on callbacks, and one based on a blockingpush model. While both models are fundamentally the same, they do carry some subtledifferences:  
This document contains two models - one based on callbacks, and one based on a blocking push model. While both models are fundamentally the same, they do carry some subtle differences:  


*Blocking Push API provides more control, and assumes threading will be explicitly handled by the application. When writing trusted PEPPER modules without a platform independent threading library, this may imply the application writer to #if WIN32, #if PTHREADS, etc. around the application's audio thread setup code.  
*Blocking Push API provides more control, and assumes threading will be explicitly handled by the application. When writing trusted PEPPER modules without a platform independent threading library, this may imply the application writer to #if WIN32, #if PTHREADS, etc. around the application's audio thread setup code.  
*Callback API does not require application to setup/teardown audio thread, and the API entry points are all on the NPAPI thread -- only the callback occurs on an implicit, dedicated audio thread.  
*Callback API does not require application to setup/teardown audio thread, and the API entry points are all on the NPAPI thread -- only the callback occurs on an implicit, dedicated audio thread.  
*There might be some subtle implementation details when it comes to how each model needs to signal between the audio device and the PEPPER audio interface. These signals will need to be as fast and low latency as possible, so minimizing how many signals are required will be important.
*There might be some subtle implementation details when it comes to how each model needs to signal between the audio device and the PEPPER audio interface. These signals will need to be as fast and low latency as possible, so minimizing how many signals are required will be important.


=====Background:=====
=====Background:=====
Native Client plug-ns have up until now used an audio API very similar to a blockingflushContext() call, usually done on a thread dedicated to audio. The API call wasimplemented as a syscall, and at the service runtime (kernel) level used SDL to deliver lowlatency audio across multiple platforms. As Native Client moves into a more securesandbox environment, untrusted code will no longer be able to communicate directly withaudio devices in the same process. In the new audio model, trusted code in anotherprocess will be attached to the audio device, and the sandboxed Native Client untrustedapplication will use shared memory and sockets to communicate its audio needs outside thesandbox. The new API for this model will be part of PEPPER, a variant of NPAPI that isdesigned to be platform neutral.


Please refer to Appendix A for a list of links pertaining to audio.Please refer to Appendix B for PEPPER design wiki -- where this document will migrate onceapproved.  
NPAPI plug-ins have traditionally done audio output either via third party audio libraries or directly with platform specific audio APIs.  This document describes a unified audio API for PEPPER, an extension of NPAPI for platform independent developement.


Both models will need additional PEPPER events. Please refer to Appendix C for audiorelated events.  
Please refer to Appendix A for a list of links pertaining to audio. Please refer to Appendix B for PEPPER design wiki -- where this document will migrate once approved.  


Both models support multi-channel configurations. Please refer to Appendix E for multi-channel output scenarios.Please refer to Appendix F for input scenarios.Please refer to Appendix G for contributers.  
Both models will need additional PEPPER events.  Please refer to Appendix C for audio related events.
 
Both models support multi-channel configurations. Please refer to Appendix E for multi-channel output scenarios. Please refer to Appendix F for input scenarios. Please refer to Appendix G for contributors.


===Changes to Pepper's Device API===
===Changes to Pepper's Device API===
Previously, Pepper's device API consisted of the methods initializeContext, flushContext, and destroyContext. Audio extends this API to queryCapability, queryConfig, initializeContext, flushContext, getState, setState, and destroyContext.   
 
Previously, Pepper's device API consisted of the methods initializeContext, flushContext, and destroyContext. Audio extends this API to queryCapability, queryConfig, initializeContext, flushContext, getState, setState, and destroyContext.   


:queryCapability - ask a device about a specific capability  
:queryCapability - ask a device about a specific capability  
Line 69: Line 72:
The following device methods must be called from the NPAPI thread (Tp). They should notbe called from any other thread, unless it is via NPN_ThreadAsyncCall.  
The following device methods must be called from the NPAPI thread (Tp). They should notbe called from any other thread, unless it is via NPN_ThreadAsyncCall.  


:queryCapability, queryConfig, initializeContext, getStateContext, setStateContext,destroyContext
:queryCapability, queryConfig, initializeContext, getStateContext, setStateContext, destroyContext


The only device method that can be called from another thread is flushContext when usingthe push model. When using the callback model, it is expected that the user suppliedcallback will be invoked on a thread dedicated to audio. This dedicated audio thead is notan NPAPI thread.  
The only device method that can be called from another thread is flushContext when using the push model. When using the callback model, it is expected that the user supplied callback will be invoked on a thread dedicated to audio. This dedicated audio thread is not an NPAPI thread.  


When creating an audio context, usually this is done as part of a plug-in instance. In untrusted Pepper modules, there will be one instance per sandboxed process. However, this could change in the future, and an untrusted Pepper plug-in could host multiple instances in the same sandbox. In this case, it may be desirable to have one audio device, with it's own application supplied mixer, to be shared amongst mutliple untrusted plug-in instances. This audio context should be created at Plug-in initialization, using NULL as thenpp instance argument.
When creating an audio context, usually this is done as part of a plug-in instance. In untrusted Pepper modules, there will be one instance per sandboxed process. However, this could change in the future, and an untrusted Pepper plug-in could host multiple instances in the same sandbox. In this case, it may be desirable to have one audio device, with it's own application supplied mixer, to be shared amongst multiple untrusted plug-in instances. This audio context should be created at plug-in initialization, using NULL as the npp instance argument.


===Obtaining Pepper Extensions and the Pepper Audio Interface===
===Obtaining Pepper Extensions and the Pepper Audio Interface===
Line 81: Line 84:
<pre>
<pre>
NPPepperExtensions *pepper;
NPPepperExtensions *pepper;
NPAudio *audio = NULL;
NPDevice *audio = NULL;
/* Get PEPPER extensions */
/* Get PEPPER extensions */
NPERR ret = NPN_GetValue(instance, NPNVPepperExtensions, &pepper);
NPERR ret = NPN_GetValue(instance, NPNVPepperExtensions, &pepper);
if (NPERR_NO_ERROR == ret) {
if (NPERR_NO_ERROR == ret) {
/* successfully obtained Pepper extensions, now acquire audio... */
  /* successfully obtained Pepper extensions, now acquire audio... */
audio = pepper->acquireDevice(NPAudioDeviceID);
  audio = pepper->acquireDevice(NPPepperAudioDevice);
} else {
} else {
/* Pepper extensions are not available */
  /* Pepper extensions are not available */
}
}
/* check for acquisition of audio device interface */
/* check for acquisition of audio device interface */
if (NULL != audio) {
if (NULL != audio) {
/* Pepper audio device interface is available */
  /* Pepper audio device interface is available */
...
  ...
</pre>
</pre>


Once an interface to the audio device is acquired, an application can query capabilities of the device, and from there, create a configuration (a set of capabilities.) In this simple example, the application picks 44.1kHz, stereo output using int16 sample types. It uses query Capability to retrieve a recommended sample frame count for the output buffer.
Once an interface to the audio device is acquired, an application can query capabilities of the device, and from there, create a configuration (a set of capabilities.) In this simple example, the application picks 44.1kHz, stereo output using int16 sample types. It uses queryCapability() to retrieve a recommended sample frame count for the output buffer.


<pre>
<pre>
...
...
/* fill out request, use sample buffer size from configuration */
/* fill out request, use sample buffer size from configuration */
NPAudioConfig request;
NPDeviceContextAudioConfig request;
NPAudioConfig config;
NPDeviceContextAudioConfig config;
int32 ret;
int32 ret;
request.sampleRate = NPAudioSampleRate44100Hz;
request.sampleRate = NPAudioSampleRate44100Hz;
Line 115: Line 118:
ret = audio->queryConfig(npp, &request, &config);
ret = audio->queryConfig(npp, &request, &config);
if (NPERR_NO_ERROR == ret) {
if (NPERR_NO_ERROR == ret) {
/* config struct now contains a valid audio device config */
  /* config struct now contains a valid audio device config */
...
  ...
</pre>
</pre>


Line 124: Line 127:
channels or sample rate.  
channels or sample rate.  


To make the most common cases easy, Pepper audio guarantees availability of 44.1kHz, 48kHz, int16 sample types, and stereo output. Pepper audio will automatically resample if needed, and do stereo->mono mixdown if needed.
To make the most common cases easy, Pepper audio guarantees availability of 44.1kHz, 48kHz, int16 sample types, and stereo output. Pepper audio will automatically re-sample if needed, and do stereo->mono mixdown if needed.


If an application desires, it can query Pepper audio for the system's internal sample rate. Using this sample rate will avoid resampling (which may occur at either the device driver level or a software mixer.)
If an application desires, it can query Pepper audio for the system's internal sample rate. Using this sample rate will avoid re-sampling (which may occur at either the device driver level or a software mixer.)


===Mode One: Callback Model===
===Mode One: Callback Model===
Line 142: Line 145:


/* fill out request, use sample buffer size from configuration */
/* fill out request, use sample buffer size from configuration */
NPAudioConfig request;
NPDeviceContextAudioConfig request;
NPAudioConfig config;
NPDeviceContextAudioConfig config;
NPAudioContext context;
NPDeviceContextAudio context;
int32 ret;
int32 ret;
request.sampleRate = NPAudioSampleRate44100Hz;
request.sampleRate = NPAudioSampleRate44100Hz;
Line 183: Line 186:


<pre>
<pre>
void appCallback(const NPAudioContext *context) {
void appCallback(const NPDeviceContextAudio *context) {
   /* assume using int16 format */
   /* assume using int16 format */
   int16 *outBuffer16 = (int16 *)context->outBuffer;
   int16 *outBuffer16 = (int16 *)context->outBuffer;
Line 199: Line 202:
satisfy scheduling demands. This model assumes the application programmer will setup the dedicated audio thread (and has access to a thread API such as pthreads.)
satisfy scheduling demands. This model assumes the application programmer will setup the dedicated audio thread (and has access to a thread API such as pthreads.)


=====Example Usage=====
====Example Usage====


<pre>
<pre>
/* From the application's NPAPI thread... */
/* From the application's NPAPI thread... */
/* setup request structure, use sample_buffer_size from configuration */
/* setup request structure, use sample_buffer_size from configuration */
NPAudioConfiguration request;
NPDeviceContextAudioConfig request;
NPAudioConfiguration config;
NPDeviceContextAudioConfig config;
pthread_t appAudioThreadID = 0;
pthread_t appAudioThreadID = 0;
volatile appAudioDone = false;
volatile appAudioDone = false;
Line 240: Line 243:
/* application's thread dedicated to audio */
/* application's thread dedicated to audio */
void* appAudioThread(void *userData) {
void* appAudioThread(void *userData) {
   /* context is sent via pthread's userData function arg */
   /* context & interface is sent via pthread's userData function arg */
   NPAudioContext *context = (NPAudioContext *)userData;
   NPDeviceContextAudio *context = (NPDeviceContextAudio *)userData;
   /* get pepper audio extensions (so we can call flushContext) */
   /* get pepper audio extensions (so we can call flushContext) */
   NPAudio *audio = (NPAudio *)context->userData;
   NPDevice *audio = (NPDevice *)context->userData;
   /* simple audio loop for this example, poll global variable */
   /* simple audio loop for this example, poll global variable */
   while (!appAudioDone) {
   while (!appAudioDone) {
Line 266: Line 269:
</pre>
</pre>


===Switching to Low Power (Silence)===


=====Switching to Low Power (Silence)=====
Callback Mode: While an application is performing audio output, it will receive callbacks at a regular interval. If an application needs to enter a state where audio output isn't required, it may wish to suppress the callback to save power and/or CPU cycles. The application can suppress the audio callback in two ways: 1) It can shutdown the audio context, and at a later time, when the application needs to resume audio output, it can re-initialize a new audio context. This approach may have long latency. 2) If setStateContext(npp, NPAudioContextStateCallback, NPAudioCallbackStop) is invoked on the NPAPI thread, callbacks will be suspended and the implicit audio thread will go into a sleep state. To resume audio playback and callbacks, the NPAPI thread should invoke setStateContext(npp,NPAudioContextStateCallback, NPAudioCallbackStart), which will resume callbacks in a low latency time frame.
 
Callback Mode: While an application is performing audio output, it will receive callbacks at a regular interval. If an application needs to enter a state where audio output isn't required, it may wish to suppress the callback to save power and/or CPU cycles. The application can suppress the audio callback in two ways: 1) It can shutdown the audio context, and at a later time, when the application needs to resume audio output, it can re-initialize a new audio context. This approach may have long latency. 2) If setStateContext(npp, NPAudioContextStateCallback, NPAudioCallbackStop) is invoked on the NPAPI thread, callbacks will be suspended and the implicit audio thread will go into a sleep state. To resume audio playback and callbacks, the NPAPI thread should invoke setStateContext(npp,NPAudioContextStateCallback, NPAudioCallbackStop), which will resume callbacks in a low latency time frame.


Push Mode: In the blocking push model, flushContext() will block (sleep) until the audio context is ready to consume another chunk of sample data. If the audio thread does not call flushContext() within a short period of time (the duration of which depends on the sample frame count and sample frequency), the audio context will automatically emit silience. Normally, an audio thread will continuously make calls to the blocking flushContext() call to emit long continuous periods of audio output. If the application enters a state where no audio output is needed for an extended duration and it wishes to reduce CPU load, the application has one of two options. 1) Shutdown the audio context and exit the audio thread. When audio playback needs to resume, it can re-spawn the audio thread and re-initialize an audio context. This approach may have a long latency. 2) The application can sleep the audio thread during long periods of silence by waiting on a pthread cond_var. When the main thread is ready to resume audio playback, it can signal the condition variable to wake up the sleeping audio thread. This approach is expected to resume audio output in a low latency time frame.
Push Mode: In the blocking push model, flushContext() will block (sleep) until the audio context is ready to consume another chunk of sample data. If the audio thread does not call flushContext() within a short period of time (the duration of which depends on the sample frame count and sample frequency), the audio context will automatically emit silence. Normally, an audio thread will continuously make calls to the blocking flushContext() call to emit long continuous periods of audio output. If the application enters a state where no audio output is needed for an extended duration and it wishes to reduce CPU load, the application has one of two options. 1) Shutdown the audio context and exit the audio thread. When audio playback needs to resume, it can re-spawn the audio thread and re-initialize an audio context. This approach may have a long latency. 2) The application can sleep the audio thread during long periods of silence by waiting on a pthread cond_var. When the main thread is ready to resume audio playback, it can signal the condition variable to wake up the sleeping audio thread. This approach is expected to resume audio output in a low latency time frame.


=====Basic API=====
===Basic API===
The basic Pepper API consists of: queryCapability, queryConfig, initializeContext, setStateContext, getStateContext, flushContext, and destroyContext.
The basic Pepper API consists of: queryCapability, queryConfig, initializeContext, setStateContext, getStateContext, flushContext, and destroyContext.


Line 320: Line 322:


/* forward decls */
/* forward decls */
typedef struct NPAudioContext NPAudioContext;
typedef struct NPDeviceContextAudio NPDeviceContextAudio;
typedef struct NPAudioConfig NPAudioConfig;
typedef struct NPDeviceContextAudioConfig NPDeviceContextAudioConfig;


/* user supplied callback function */
/* user supplied callback function */
typedef void (*NPAudioCallback)(const NPAudioContext *context);
typedef void (*NPAudioCallback)(const NPDeviceContextAudio *context);


/* Audio config structure */
/* Audio config structure */
struct NPAudioConfig {
struct NPDeviceContextAudioConfig {
int32 sampleRate;
int32 sampleRate;
int32 sampleType;
int32 sampleType;
Line 339: Line 341:


/* Audio context structure */
/* Audio context structure */
struct NPAudioContext {
struct NPDeviceContextAudio {
NPP npp;
NPP npp;
NPAudioConfig config;
NPDeviceContextAudioConfig config;
void *outBuffer;
void *outBuffer;
void *inBuffer;
void *inBuffer;
void *private;
void *privatePtr;
};
};
</pre>
</pre>
Line 352: Line 354:
inputs:  
inputs:  
  NPP npp
  NPP npp
Plug-in instance npp
Plug-in instance npp
NPDeviceCapability query
  NPDeviceCapability query
NPAudioCapabilitySampleRate
NPAudioCapabilitySampleRate
NPAudioCapabilitySampleType
NPAudioCapabilitySampleType
NPAudioCapabilitySampleFrameCount
NPAudioCapabilitySampleFrameCount
NPAudioCapabilitySampleFrameCount44100Hz
NPAudioCapabilitySampleFrameCount44100Hz
NPAudioCapabilitySampleFrameCount48000Hz
NPAudioCapabilitySampleFrameCount48000Hz
NPAudioCapabilitySampleFrameCount96000Hz
NPAudioCapabilitySampleFrameCount96000Hz
NPAudioCapabilityOutputChannelMap
NPAudioCapabilityOutputChannelMap
NPAudioCapabilityInputChannelMap
NPAudioCapabilityInputChannelMap
outputs:
outputs:
*value
*value
Line 371: Line 373:
</pre>
</pre>
'''NPError queryConfig(NPP npp, NPAudioConfig *request, NPAudioConfig *obtain)'''
'''NPError queryConfig(NPP npp, NPDeviceContextAudioConfig *request, NPDeviceContextAudioConfig *obtain)'''
<pre>
<pre>
   inputs:
   inputs:
NPP npp
NPP npp
Plug-in instance npp
Plug-in instance npp
NPAudioConfig *request
NPDeviceContextAudioConfig *request
A requested configuration, which is a set of capabilities
A requested configuration, which is a set of capabilities
outputs:
outputs:
NPAudioConfig *obtain
NPDeviceContextAudioConfig *obtain
The set of capabilities obtained, which may or may not match
The set of capabilities obtained, which may or may not match
request input.
request input.
Line 388: Line 390:
Okay if request & obtain pointers are the same.
Okay if request & obtain pointers are the same.
</pre>
</pre>
'''NPError initializeContext(NPP npp, const NPAudioConfig *config,
'''NPError initializeContext(NPP npp, const NPDeviceContextAudioConfig *config,
NPAudioContext *context)'''
NPDeviceContextAudio *context)'''
<pre>
<pre>
inputs:
inputs:
NPP npp
NPP npp
Plug-in instance npp
Plug-in instance npp
NPAudioConfig *config - a structure with which to configure the
NPDeviceContextAudioConfig *config - a structure with which to configure the
audio device
audio device
int32 sampleRate
int32 sampleRate
Line 431: Line 433:
pointer to user data for callback. Can be NULL.
pointer to user data for callback. Can be NULL.
outputs:
outputs:
NPAudioContext *context
NPDeviceContextAudio *context
Filled in, used to identify audio context.
Filled in, used to identify audio context.
returns:
returns:
Line 452: Line 454:
</pre>
</pre>
'''NPError setStateContext(NPP npp, NPAudioContext *context, int32 state, int32 value)'''
'''NPError setStateContext(NPP npp, NPDeviceContextAudio *context, int32 state, int32 value)'''
<pre>
<pre>
         inputs:
         inputs:
NPP npp
NPP npp
Plug-in instance npp
Plug-in instance npp
NPAudioContext *context
NPDeviceContextAudio *context
audio context to apply state
audio context to apply state
int32 state
int32 state
Line 479: Line 481:
</pre>
</pre>
'''NPError getStateContext(NPP npp, NPAudioContext *context, int32 state, int32 *value)'''
'''NPError getStateContext(NPP npp, NPDeviceContextAudio *context, int32 state, int32 *value)'''
<pre>
<pre>
inputs:
inputs:
NPP npp
NPP npp
Plug-in instance npp
Plug-in instance npp
NPAudioContext *context
NPDeviceContextAudio *context
audio context to apply state
audio context to apply state
int32 state
int32 state
Line 502: Line 504:
</pre>
</pre>
'''NPError flushContext(NPP npp, NPAudioContext *context)'''
'''NPError flushContext(NPP npp, NPDeviceContextAudio *context)'''
<pre>
<pre>
         inputs:
         inputs:
Line 508: Line 510:
For audio, this function is invoked from a non-NPAPI thread,
For audio, this function is invoked from a non-NPAPI thread,
so npp is ignored.
so npp is ignored.
NPAudioContext *context
NPDeviceContextAudio *context
context->outBuffer
context->outBuffer
pointer to output data. If this field is NULL, no output
pointer to output data. If this field is NULL, no output
Line 543: Line 545:
</pre>
</pre>
'''NPError destroyContext(NPP npp, NPAudioContext *context)'''
'''NPError destroyContext(NPP npp, NPDeviceContextAudio *context)'''
<pre>
<pre>
         inputs:
         inputs:
NPP npp
NPP npp
Plug-in instance npp
Plug-in instance npp
NPAudioContext context
NPDeviceContextAudio *context
audio context to shutdown
audio context to shutdown
returns:
returns:
Line 569: Line 571:
</pre>
</pre>
'''void (NPAudioCallback *)(NPAudioContext *context);'''
'''void (NPAudioCallback *)(NPDeviceContextAudio *context);'''
<pre>
<pre>
         inputs:
         inputs:
NPAudioContext context
NPDeviceContextAudio context
Audio context that generated this callback. Fields within the
Audio context that generated this callback. Fields within the
context
context
Line 615: Line 617:
isochronously on the same callback.
isochronously on the same callback.
</pre>
</pre>
===queryCapability() Details===
This section goes into more detail on what queryCapability() returns based on the input
enum.
<pre>
notes:
queryCapability() can be called before initializeContext to probe
capabilities of the audio device. queryCapability() should only be
called from the NPAPI thread. If the output parameter is NULL,
queryCapability will return NPERR_INVALID_PARAM.
queryCapability(npp, NPAudioCapabilitySampleRate, &value)
output value:
Current 'native' sample rate of the underlying audio system. If an
application can use this rate, less upsampling/downsampling is
likely to occur at the driver level.
NPAudioSampleRate44100Hz
NPAudioSampleRate48000Hz
NPAudioSampleRate96000Hz
notes:
Both NPAudioSampleRate44100Hz and NPAudioSampleRate48000Hz are
always available, and will be upsampled/downsampled as needed. By
querying for the native sample rate, an application can avoid
implicit sample rate conversion.
queryCapability(npp, NPAudioCapabilitySampleType, &value)
output value:
Native sample format of the underlying audio system
NPAudioSampleTypeInt16
NPAudioSampleTypeFloat32
notes:
NPAudioSampleTypeInt16 will always be available. By querying for
the sample format, an application may be able to discover higher
quality output formats.
queryCapability(npp, NPAudioCapabilitySampleFrameCount, &value)
Query the audio system for the sample buffer frame count recommended by
NPAudioQuerySampleRate.
output value:
Native sample buffer size of the underlying audio system ranging
from NPAudioMinSampleFrameCount...NPAudioMaxSampleFrameCount.
This sample buffer frame count relates to the sample rate returned
by queryCapability(npp, NPAudioCapabilitySampleRate, &sampleRate)
notes:
Upon successful initialization of the audio system, an application
will always be able to request and obtain the native sample buffer
frame count when specifying the sample frequency returned by
NPAudioQuerySampleRate.
queryCapability(npp, NPAudioCapabilitySampleFrameCount44100Hz, &value)
Query the audio system for the sample buffer frame count to use for
44.1kHz output.
output value:
Recommended sample buffer frame count ranging from
NPAudioMinSampleFrameCount...NPAudioMaxSampleFrameCount.
notes:
Upon successful initialization of the audio system at 44.1kHz, an
application will always be able to request and obtain this sample
buffer frame count.
queryCapability(npp, NPAudioQuerySampleFrameCount48000Hz, &value)
Query the audio system for the sample buffer frame count to use for
48kHz output.
output value:
Recommended sample buffer frame count ranging from
NPAudioMinSampleFrameCount...NPAudioMaxSampleFrameCount.
notes:
Upon successful initialization of the audio system at 48kHz, an
application will always be able to request and obtain this buffer
frame count.
queryCapability(npp, NPAudioCapabilitySampleFrameCount96000Hz, &value)
Query the audio system for the sample buffer frame count to use for
96kHz output.
output value:
Recommended sample buffer frame count ranging from
NPAudioMinSampleFrameCount...NPAudioMaxSampleFrameCount.
notes:
Upon successful initialization of the audio system at 96kHz, an
application will always be able to request and obtain this buffer
frame count.
queryCapability(npp, NPAudioCapabilityOutputChannelMap, &value)
Query the audio system for the output/speaker arrangement.
output value:
NPAudioChannelNone
NPAudioChannelMono
NPAudioChannelStereo
NPAudioChannelThree
NPAudioChannelFour
NPAudioChannelFive
NPAudioChannelFiveLFE
NPAudioChannelSeven
NPAudioChannelSevenLFE
notes:
Upon successful initialization of the audio system, an application
will always be able to request and obtain this channel setup.
Additionally, NPAudioChannelMono and NPAudioChannelStereo will
always be available, regardless of physical speaker configuration:
mono output -> mono speaker, or
mono output -> center speaker, or
mono output -> left & right speakers
stereo output -> left & right speakers, or
stereo output -> mono speaker
queryCapability(npp, NPAudioCapabilityInputChannelMap, &value)
Query the audio system for the input/microphone arrangement.
output value:
NPAudioChannelNone
NPAudioChannelMono
NPAudioChannelStereo
NPAudioChannelThree
NPAudioChannelFour
NPAudioChannelFive
NPAudioChannelSeven
notes:
Upon successful initialization of the audio system, an application
will always be able to request and obtain this channel setup. If
NPAudioChannelNone is returned, either the device has no input, or
the implementation doesn't support audio input.
</pre>
=== Appendix A - Links to audio related info  ===
[http://www.kaourantin.net/2008/05/adobe-is-making-some-noise-part-1.html http://www.kaourantin.net/2008/05/adobe-is-making-some-noise-part-1.html]
[http://osdl.sourceforge.net/main/documentation/rendering/SDL-audio.html http://osdl.sourceforge.net/main/documentation/rendering/SDL-audio.html]
[http://stackoverflow.com/questions/1595190/sounds-effects-in-iphone-game http://stackoverflow.com/questions/1595190/sounds-effects-in-iphone-game]
[http://insanecoding.blogspot.com/2009/06/state-of-sound-in-linux-not-so-sorry.html http://insanecoding.blogspot.com/2009/06/state-of-sound-in-linux-not-so-sorry.html]
[http://developer.apple.com/Mac/library/documentation/MusicAudio/Conceptual/AudioUnitProgrammingGuide/AudioUnitDevelopmentFundamentals/AudioUnitDevelopmentFundamentals.html#//apple_ref/doc/uid/TP40003278-CH7-SW12 http://developer.apple.com/Mac/library/documentation/MusicAudio/Conceptual/AudioUnitProgrammingGuide/AudioUnitDevelopmentFundamentals/AudioUnitDevelopmentFundamentals.html#//apple_ref/doc/uid/TP40003278-CH7-SW12]
=== Appendix B - PEPPER wiki  ===
[https://wiki.mozilla.org/Plugins:PlatformIndependentNPAPI https://wiki.mozilla.org/Plugins:PlatformIndependentNPAPI]
===Appendix C - PEPPER Audio Events===
Adds:
<pre>
NPEventType_Audio
struct NPAudioEvent {
  int32 subtype;
  int32 value;
};
</pre>
where subtype can be:
<tt>NPEventAudioDeviceChanged</tt> - audio can continue to play on the current device, but an application may gain additional capability by shutting down, and re-initializing. An example of this would be a laptop currently emitting audio to headphones (stereo). The user unplugs the headphones, and plugs in an optical spdif that feeds into a home theater system. In this case, the application may wish to re-initialize to gain multi- channel output. Another example - a user plugs in a microphone and/or enables microphone permissions.
<tt>NPEventAudioMute</tt> - the browser has muted/unmuted audio on this plug-in instance. To save battery life, the application may wish to suspend streaming of audio while muted.
:value is 0 - entering mute state.
:value is 1 - leaving mute state.
===Appendix D - Configuration Precedence===
In cases where either the device or the implementation can't support all audio features, queryConfig will need to gracefully scale back. When scaling back features, there should be a precedence to what features should be supported vs what features should be dropped. In all cases, 44.1kHz, 48kHz and stereo output should be fully supported.
*When the application asks for simultaneous audio input and output, but the underlying system can only support one or the other, queryConfig will favor audio output over audio input.
*When the application asks for multi-channel output and a sample rate >48kHz (likely 96kHz), queryConfig will prefer lower sample rates (48kHz) over reducing the number of output channels.
*If the implementation doesn't suport a "dot one" low frequency channel, queryConfig will select the equivalent multi-channel format without the LFE channel. Applications should assume the left and right front channels are full range.
===Appendix E - Multi-Channel Output Configurations===
Interleave orders are fixed. Which configurations are supported depends on both the physical configuration and the underlying implementation. There is one exception:
NPAudioChannelStereo will always be supported.
<pre>
      +-----+
front | ch0 |
      +-----+
  +----------+
  | listener |
  +----------+
</pre>
If the physical speaker arrangement is multi-channel, 1 channel output will pick the front center speaker if one is present. If no front center speaker is available, 1 channel output will occur on both the front left and front right speakers. For audio output, NPAudioChannelStereo will always be available to the application, regardless of physical speaker configuration.
Sample buffer format:
<pre>
+-----+-----+-----+
| ch0 | ch0 | ch0 | ...
+-----+-----+-----+
<-----> single sample frame
</pre>
'''Format NPAudioChannelStereo'''
<pre>
front +-----+    +-----+ front
left  | ch0 |    | ch1 | right
(L)  +-----+    +-----+ (R)
          +----------+
          | listener |
          +----------+
</pre>
If the physical speaker arrangement is mono, ch0 and ch1 will be combined and output on the mono speaker. For audio output, NPAudioChannelStereo will always be availabe to the application, regardless of physical speaker configuration.
Sample buffer interleave order is L, R:
<pre>
+------+------+------+------+
| ch0  | ch1  | ch0  | ch1  | ...
+------+------+------+------+
    sample        sample
<-  frame 0 -> <- frame 1 ->
</pre>
'''Format NPAudioChannelThree'''
<pre>
                front
                center (C)
front +-----+  +-----+    +-----+ front
left  | ch0 |  | ch2 |    | ch1 | right
(L)  +-----+  +-----+    +-----+ (R)
              +----------+
              | listener |
              +----------+
</pre>
Sample buffer interleave order is L, R, C:
<pre>
+------+------+------+------+------+------+
| ch0  | ch1  | ch2  | ch0  | ch1  | ch2  | ...
+------+------+------+------+------+------+
<- sample frame 0  -> <- sample frame 1 ->
</pre>
'''Format NPAudioChannelFour'''
<pre>
front +-----+              +-----+ front
left  | ch0 |              | ch1 | right
(L)  +-----+              +-----+ (R)
+----------+
| listener |
+----------+
      +-----+              +-----+
      | ch2 |              | ch3 |
left  +-----+              +-----+ right
back (Lb)                          back (Rb)
</pre>
Sample buffer interleave order is L, R, Lb, Rb:
<pre>
+------+------+------+------+------+------+------+------+
| ch0  | ch1  | ch2  | ch3  | ch4  | ch0  | ch1  | ch2  | ...
+------+------+------+------+------+------+------+------+
<---  sample frame 0          --->  <---  sample frame 1...
</pre>
'''Format NPAudioChannelFive'''
             
<pre>
              front
              center (C)
front +-----+  +-----+  +-----+ front
left  | ch0 |  | ch2 |  | ch1 | right
(L)  +-----+  +-----+  +-----+ (R)
            +----------+
            | listener |
    +-----+ +----------+ +-----+
    | ch3 |              | ch4 |
left +-----+              +-----+ right
surround (Ls)            surround (Rs)
</pre>
Sample buffer interleave order is L, R, C, Ls, Rs:
<pre>
+------+------+------+------+------+------+------+------+
| ch0  | ch1  | ch2  | ch3  | ch4  | ch0  | ch1  | ch2  | ...
+------+------+------+------+------+------+------+------+
<---  sample frame 0  ---> <--- sample frame 1...
</pre>
'''Format NPAudioChannelFiveLFE'''
   
<pre>
                front
                center (C)
front +-----+  +-----+  +-----+ front  +-----+
left  | ch0 |  | ch2 |  | ch1 | right  | ch3 | subwoofer (LFE)
(L)  +-----+  +-----+  +-----+ (R)    +-----+
                +----------+
                | listener |
      +-----+  +----------+  +-----+
      | ch4 |                | ch5 |
left  +-----+                +-----+ right
surround (Ls)                surround (Rs)
</pre>
Sample buffer interleave order is L, R, C, LFE, Ls, Rs:
<pre>
+------+------+------+------+------+------+------+------+
| ch0  | ch1  | ch2  | ch3  | ch4  | ch5  | ch0  | ch1  | ...
+------+------+------+------+------+------+------+------+
<---  sample frame 0                  ---> <---  sample frame 1...
</pre>
'''Format NPAudioChannelSeven'''
<pre>
                front
                center (C)
front +-----+  +-----+  +-----+ front 
left  | ch0 |  | ch2 |  | ch1 | right 
(L)  +-----+  +-----+  +-----+ (R)   
                +----------+
                | listener |
      +-----+  +----------+  +-----+
      | ch5 |                | ch6 |
left  +-----+                +-----+ right
surround                              surround
(Ls)      +------+      +------+    (Rs)
    left  | ch3  |      | ch4  | right
    back  +------+      +------+ back
    (Lb)                        (Rb)
</pre>
Sample buffer interleave order is L, R, C, Lb, Rb, Ls, Rs:
<pre>
+------+------+------+------+------+------+------+------+
| ch0  | ch1  | ch2  | ch3  | ch4  | ch5  | ch6  | ch0  | ...
+------+------+------+------+------+------+------+------+
<---  sample frame 0                        ---> <---  sample frame 1...
</pre>
'''Format NPAudioChannelSevenLFE'''
   
<pre>
                front
                center (C)
front +-----+  +-----+  +-----+ front  +-----+
left  | ch0 |  | ch2 |  | ch1 | right  | ch3 | subwoofer (LFE)
(L)  +-----+  +-----+  +-----+ (R)    +-----+
                +----------+
                | listener |
      +-----+  +----------+  +-----+
      | ch6 |                | ch7 |
left  +-----+                +-----+ right
surround                              surround
(Ls)      +------+      +------+    (Rs)
    left  | ch4  |      | ch5  | right
    back  +------+      +------+ back
    (Lb)                        (Rb)
</pre>
Sample buffer interleave order is L, R, C, LFE, Lb, Rb, Ls, Rs:
<pre>
+------+------+------+------+------+------+------+------+
| ch0  | ch1  | ch2  | ch3  | ch4  | ch5  | ch6  | ch7  | ...
+------+------+------+------+------+------+------+------+
<---              sample frame 0                  ---> <---  sample frame 1...
</pre>
===Appendix F - Multi-Channel Input Formats===
:Interleave orders are fixed.
:Format NPAudioChannelMono - input is mono microphone input
:Format NPAudioChannelStereo - input is stereo line in. Interleave order same as Appendix E.
:Format NPAudioChannelFive - input is SPDIF. Interleave order same as Appendix E.
:Format NPAudioChannelFiveOne - input is SPDIF. Interleave order same as Appendix E.
:Format NPAudioChannelSeven - input is SPDIF. Interleave order same as Appendix E.
:Format NPAudioChannelSevenOne - input is SPDIF. Interleave order same as Appendix E.
===Appendix G - Contributors===
<table>
<tr><td>Andrew Scherkus</td><td>scherkus@google.com</td></tr>
<tr><td>Brad Chen</td><td>bradchen@google.com</td></tr>
<tr><td>Carlos Pizano</td><td>cpu@google.com</td></tr>
<tr><td>Chris Rogers</td><td>crogers@google.com</td></tr>
<tr><td>Darin Fisher</td><td>darin@google.com</td></tr>
<tr><td>David Sehr</td><td>sehr@google.com</td></tr>
<tr><td>Dominic Mazzoni</td><td>dmazzoni@google.com</td></tr>
<tr><td>Ian Lewis</td><td>ilewis@google.com</td></tr>
<tr><td>Nicholas Fullagar</td><td>nfullagar@google.com</td></tr>
<tr><td>Seth Alt</td><td>seth@sweepingdesign.com</td></tr>
<tr><td>Stewart Miles</td><td>smiles@google.com</td></tr>
</table>
Josh
Confirmed users, Bureaucrats and Sysops emeriti
1,680

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/188181...218382"