User:Roc/AudioBufferProposal

Overview

This proposal aims to solve the data-race problem with AudioBuffer's mutable Float32Arrays, in a way that provides a high degree of compatibility with existing API usage, but avoids requiring memory copies in almost all cases (even for code using existing APIs).

Specification Changes

Text in italics is non-normative.

AudioBuffer is extended with one new method:

partial interface AudioBuffer {
  void copyChannelDataTo(long channelNumber, unsigned long start, unsigned long length, Float32Array destination);
}

The copyChannelDataTo method copies a range of samples from the specified channel of the AudioBuffer to the destination array. If start is less than zero, start plus length is greater than the AudioBuffer's length, or length is greater than the destination array's length, an INDEX_SIZE_ERR exception must be thrown.

Note: This method can be used to fill part of an array by passing in a Float32Array that's a view onto the larger array. Note: When reading data from an AudioBuffer's channels, and the data can be processed in chunks, copyChannelDataTo should be preferred to calling getChannelData() and accessing the resulting array, because it may avoid unnecessary memory allocation and copying.

An internal operation "acquire the contents of an AudioBuffer" is invoked when the contents of an AudioBuffer are needed by some API implementation. This operation returns immutable channel data to the invoker. When an "acquire the contents" operation occurs, run the following steps:

1. If any of the AudioBuffer's ArrayBuffers have been neutered, abort these steps and return zero-length channel data buffers to the invoker.
2. Neuter all ArrayBuffers for arrays previously returned by getChannelData() on this AudioBuffer.
3. The underlying data buffers for those ArrayBuffers are retained by the AudioBuffer and returned to the invoker.
4. New ArrayBuffers containing copies of the data are attached to the AudioBuffer to be returned by the next call to getChannelData().

Note: Acquiring the contents of an AudioBuffer can be implemented without copying channel data. In particular, the last step can be performed lazily at the next getChannelData() call (if there is one; there often won't be). A sequence of consecutive "acquire the contents" operations with no intervening getChannelData() (e.g. multiple AudioBufferSourceNodes playing the same AudioBuffer) can be implemented with no allocations or copying. Note: Implementations can perform an additional optimization: if getChannelData() is called on an AudioBuffer, fresh ArrayBuffers have not yet been allocated, but all invokers of previous "acquire the contents" operations on an AudioBuffer have stopped using the AudioBuffer's data, the raw data buffers can be recycled for use with new AudioBuffers, avoiding any reallocation or copying of the channel data.

The "acquire the contents of an AudioBuffer" operation is invoked in the following cases:

• When AudioBufferSourceNode.start() is called, it acquires the contents of the node's buffer. If the operation fails, nothing is played.
• When a ConvolutionNode's 'buffer' is set to an AudioBuffer while the node is connected to an output node, or a ConvolutionNode is connected to an output node while the ConvolutionNode's 'buffer' is set to an AudioBuffer, it acquires the contents of the AudioBuffer. * When dispatch of an AudioProcessingEvent completes, it acquires the contents of its outputBuffer.

Note: For AudioBufferSourceNode and ConvolutionNode, a node "acquires the contents of an AudioBuffer" at exactly the moment Jer's proposal would make the buffer immutable by virtue of being associated with that node.

Additional Notes

A good implementation following the advice above will not allocate or copy buffers of channel data any more than an implementation of the "freely share memory" proposal, except when an application calls getChannelData() on an AudioBuffer that is "live". If such an application writes to the returned array, that is deprecated behavior under any proposal, but this proposal defines more predictable results than the "freely share memory" proposal. If such an application only reads the returned array, it can probably be modified to use copyChannelDataTo instead, which will reduce the memory overhead to a negligible level.

This proposal relies heavily on ArrayBuffer neutering. Some people want to avoid the use of neutering, but the TAG declined to endorse that position when explicitly asked.

Webkit and Blink developers have indicated that they'll keep around the webkit-prefixed AudioContext API for a long time. This proposal provides substantial compatibility with that API, which has value for authors and browser implementers, especially given the amount of content already written to that API.

I argue that this proposal is no more complicated than other proposals. Compared to Jer's proposal, this proposal has less API surface, and less complexity for Web developers who don't need to read or write channel data after an AudioBuffer has been used (since they don't have to think about node liveness), or who are not very concerned about performance. For developers who are very concerned about performance, and who want to modify channel data between uses of an AudioBuffer, the proposals are very similar; they'll have to think about node liveness. (Jer's proposal throws an exception when writing data to an in-use AudioBuffer, which is informative for Web authors who want to avoid copies; but with my proposal we can provide a similar alert through Web developer tools.) My proposal may be more complicated for implementers --- or not, since Jer's proposal requires implementations to track precisely when an AudioBuffer is associated with a live AudioNode, and this proposal does not.

Compared to the "freely share memory" proposal, assuming that proposal is fully fleshed out to define what authors and implementations are allowed to do, this proposal is almost the same for Web developers. The only extra complexity for Web developers is that they should call copyChannelDataTo to read channel contents instead of reading from getChannelData() arrays, if an AudioBuffer could be in use. (Web developers writing to getChannelData() arrays should ensure that the AudioBuffer is not in use, under both proposals.) For implementers, the "freely share memory" proposal is probably less complex, although the "ArrayBuffer is neutered while we're playing it" problem must be solved trickily while this proposal solves it easily. (Of course, implementers who don't wish to rely on undefined C++ behavior, or who wish to use non-shared-memory hardware, will have a hard time with the "freely share memory" proposal (or be forced to make copies at inopportune times).)