Confirmed users
656
edits
User:Corban/AudioAPI: Difference between revisions
Jump to navigation
Jump to search
no edit summary
mNo edit summary |
No edit summary |
||
| Line 10: | Line 10: | ||
* Corban Brook | * Corban Brook | ||
== Current | == Current API == | ||
We have developed a proof of concept, experimental build of Firefox which extends the HTMLMediaElement (e.g., affecting <video> and <audio>) and implements the following basic API for reading and writing raw audio data: | |||
===== Reading Audio ===== | ===== Reading Audio ===== | ||
| Line 59: | Line 59: | ||
audioOutput.mozAudioWrite(samples.length, samples); | audioOutput.mozAudioWrite(samples.length, samples); | ||
</pre> | </pre> | ||
== DOM Implementation == | |||
===== nsIDOMAudioData ===== | |||
Audio data (raw and spectrum) is currently returned in a pseudo-array named '''nsIDOMAudioData'''. In future this will be changed to use the much faster native WebGL Array. | |||
<pre> | |||
interface nsIDOMAudioData : nsISupports | |||
{ | |||
readonly attribute unsigned long length; | |||
float item(in unsigned long index); | |||
}; | |||
</pre> | |||
The '''length''' attribute indicates the number of elements of data returned. | |||
The '''item()''' method provides a getter for audio elements. | |||
===== nsIDOMNotifyAudioWrittenEvent ===== | |||
Audio data is made available via the following event: | |||
* '''Event''': AudioWrittenEvent | |||
* '''Event handler''': onaudiowritten | |||
The '''AudioWrittenEvent''' is defined as follows: | |||
<pre> | |||
interface nsIDOMNotifyAudioWrittenEvent : nsIDOMEvent | |||
{ | |||
readonly attribute nsIDOMAudioData mozFrameBuffer; | |||
readonly attribute nsIDOMAudioData mozSpectrum; | |||
}; | |||
</pre> | |||
The '''mozFrameBuffer''' attribute contains the raw audio data (float values) obtained from decoding a single frame of audio. This is of the form <nowiki>[left, right, left, right, ...]</nowiki>. All audio frames are normalized to a length of 4096 or greater, where shorter frames are padded with 0 (zero). | |||
The '''mozSpectrum''' attribute contains a pre-calculated FFT for this frame of audio data. It is calculated on using the first 4096 float values in the current audio frame only, which may include zeros used to pad the buffer. It is always 2048 elements in length. | |||
===== nsIDOMHTMLMediaElement additions ===== | |||
<pre> | |||
void mozWriteAudio(in PRUint32 count, [array, size_is(count)] in float valueArray); | |||
void mozSetup(in PRUint32 channels, in PRUint32 rate, in float volume); | |||
</pre> | |||
The '''mozSetup()''' method allows an <audio> or <video> element to be setup for writing from script. This method '''must''' be called before '''mozWriteAudio''' can be called, since an audio stream has to be created for the media element. It takes three arguments: | |||
# '''channels''' - the number of audio channels (e.g., 2) | |||
# '''rate''' - the audio's sample rate (e.g., 44100 samples per second) | |||
# '''volume''' - the initial volume to use (e.g., 1.0) | |||
The choices made for '''channel''' and '''rate''' are significant, because they determine the frame size you must use when passing data to '''mozWriteAudio()'''. | |||
The '''mozWriteAudio()''' method can be called after '''mozSetup()'''. It allows a frame of audio (or multiple frames, but whole frames) to be written directly from script. It takes two arguments: | |||
# '''count''' - the number of elements in this frame (e.g., 4096) | |||
# '''valueArray''' - an array of floats, which represent a complete frame of audio (or multiple frames, but whole frames). | |||
Both '''mozWriteAudio()''' and '''mozSetup()''' will throw exceptions if called out of order, or if audio frame sizes do not match. | |||