Confirmed users
656
edits
Audio Data API Review Version: Difference between revisions
Jump to navigation
Jump to search
→Writing Audio
| Line 330: | Line 330: | ||
Audio data written using the '''mozWriteAudio()''' method needs to be written at a regular interval in equal portions, in order to keep a little ahead of the current sample offset (current sample offset of hardware can be obtained with '''mozCurrentSampleOffset()'''), where a little means something on the order of 500ms of samples. For example, if working with 2 channels at 44100 samples per second, a writing interval of 100ms, and a pre-buffer equal to 500ms, one would write an array of (2 * 44100 / 10) = 8820 samples, and a total of (currentSampleOffset + 2 * 44100 / 2). | Audio data written using the '''mozWriteAudio()''' method needs to be written at a regular interval in equal portions, in order to keep a little ahead of the current sample offset (current sample offset of hardware can be obtained with '''mozCurrentSampleOffset()'''), where a little means something on the order of 500ms of samples. For example, if working with 2 channels at 44100 samples per second, a writing interval of 100ms, and a pre-buffer equal to 500ms, one would write an array of (2 * 44100 / 10) = 8820 samples, and a total of (currentSampleOffset + 2 * 44100 / 2). | ||
===== Other Approachs to Writing Audio ===== | |||
1) Connect Decode Thread and Worker | |||
'''Idea:''' | |||
Pass audio data from the decoding thread to a worker for processing. | |||
Create a worker thread, and pass it to the HTMLMediaElement. The decode thread gets the worker via the HTMLMediaElement, and passes messages to the worker as audio is decoded, bypassing the main thread. The worker processes the audio, and returns, allowing audio to be modified before it is played. | |||
'''TODO:''' | |||
* Modify HTMLMediaElement so you can pass it a Worker as the destination for audio data. | |||
* Allow the decoder to get a thread-safe reference to this worker, so it can be used by the decoder thread. The worker would be handed off to the audio decoding thread as an nsIWorker interface pointer | |||
* Have the decode thread pass a message to the worker (ideally synchronously) so that the worker can return modified audio data, which then gets played as normal. Perhaps pass some kind of event object to the worker. When the worker is done modifying the data, it can call a method on the event object to say "hey, put this data back in the audio decoder thread's buffers now and let the audio decoder thread proceed." | |||
'''Issues:''' | |||
* The worker code was written with the assumption that the code is either running on the main thread or on the worker thread. Teaching it about a third thread would be necessary. | |||
* May need to add a C++ PostMessage call to the nsIWorker interface, as the current one assumes it's being called through XPConnect. | |||
* How do we get the data back from the worker to the decode thread? | |||
* Keep a reference to the worker in the HTMLMediaElement use cycle collection to handle lifetimes | |||
'''Notes:''' | |||
* DecodeAudioData function in the webm backend is what you need to decode vorbis | |||
* nsOggReader::DecodeAudioData pushes audio decoded data onto the audio queue. Instead of this, push to worker. Maybe shift that out of the nsOggReader (the adding to the audio queue) and into the nsBuiltin code. | |||
2) Introduce a Callback to Decode Thread | |||
'''Idea:''' | |||
Allow js to push data to the Audio Queue, whether modified data from the decoder, or generated data. Have a callback that gets the data as soon as the decode thread is done with it, and the callback modifies it (synch) before it goes into the queue to be played. This callback could then postMessage to a worker, which processes it on another thread, and then puts it back into the audio queue (audio thread). The audio thread no longer queues things directly. | |||
'''Notes:''' | |||
* This could replace the current mozWriteAudio() method. | |||
===== Complete Example: Creating a Web Based Tone Generator ===== | ===== Complete Example: Creating a Web Based Tone Generator ===== | ||