Labs/Jetpack/Reboot/JEP/120

JEP 120 - Audio JEP

• Champion: Jim Garrison - jim@garrison.cc
• Status: Under Review
• Bug Ticket:
• Type: API
• Difficulty: 4

Proposal

Implementation of the audio JEP will allow the developer to record audio from the user's microphone and store it in an Ogg/Vorbis file.

Alternatively, the developer can capture the audio stream to memory by providing a callback that receives each block of audio and knows how to process or store it. And then the audio byte array can be encoded to an Ogg/Vorbis file.

Key Issues

• How do we implement audio capture in a cross-platform way? portaudio seems to be the best way at the moment. (libsydneyaudio, already used for playback of HTML5 <audio> and <video> elements, has an API for audio capture but it is unimplemented.)
• Should the streaming and encoding APIs always use 44.1 kHz audio, or should we increase API complexity to support other sample rates as well?

API Methods

The Audio module will be available under jetpack.audio.

To begin recording to a file, call: jetpack.audio.recordToFile.

To stop the recording, call: jetpack.audio.stopRecording. This function will return the path of the Ogg/Vorbis file that the audio was recorded to.

To check if a recording is in progress (simultaneous recording of audio is not allowed), access the jetpack.audio.isRecording property.

(An exception will be raised if you attempt to record while another recording is already in progress.)

To record a stream to memory, call: jetpack.audio.recordToPipe. The callback receives a byte array and the number of samples contained in that array. The array will consist of PCM float stereo (interleaved) samples at a frequency of 44100Hz.

We're working on adding support for re-encoding this byte stream (before or after manipulation) back into an Ogg/Vorbis file. Check out http://hg.mozilla.org/labs/jetpack/file/tip/components/experimental/audio/AudioEncoder.cpp if you want to help!

Use Cases

• Integration with a web service that does speech recognition
• Client for an audio twitter-like social network (e.g. audioboo, twaud.io)
• TwitterFox or a similar extension could add support for attaching audio to a tweet
• Live visualization of audio implemented in HTML/CSS/JS/Canvas
• Voice memos (https://jetpack.mozillalabs.com/demos.html)
• Several possible ways to interface with regular web sites
  • An extension could provide a reference implementation of audio capture for the HTML5 <device> element as that specification develops (see https://labs.ericsson.com/blog/beyond-html5-audio-capture-web-browsers and http://www.whatwg.org/specs/web-apps/current-work/multipage/commands.html#devices)
  • Somebody could write a jetpack extension that exposes a non-standard Javascript API that web sites can access to record audio.
  • A web site may want to provide a site-specific extension that allows users to record and upload snippets of audio.
  • Somebody could make an extension that allows the user to record audio semlessly to any file upload form element, thus being able to easily upload audio to any web site.
• Web sites would benefit:
  • Users could seamlessly attach audio to an email on gmail or any webmail site
  • A language instruction wiki (e.g. Wikiotics) could allow users to easily upload spoken phrases for language lessons
  • A greeting card site could seamlessly allow users to upload personalized audio messages
  • A DJ/playlist site could allow users to record quick introductions between songs

Examples

Recording to a file

This Jetpack shows a 'Record' button on the status bar, which when clicked will start recording audio to a file. On clicking it again, the recording is stopped, a notification with the file path is displayed and the audio is played back.

var path = '';

jetpack.statusBar.append({
  html: 'Record<i>!</i>',
  width: 55,
  onReady: function(w) {
    $(w).click(function() {
      if (jetpack.audio.isRecording) {
        path = jetpack.audio.stopRecording();
        jetpack.notifications.show("Saved to " + path + ", now playing!");
        $(jetpack.tabs.focused.contentDocument).find('body').
          append('<audio src="file://'+path+'" autoplay="true"></audio>');
      } else {
        jetpack.notifications.show("Recording...");
        jetpack.audio.recordToFile();
      }
    });
  }
});

Recording to memory

The Jetpack audio module allows access to the raw audio stream. Here's an example:

jetpack.future.import("audio");
var start = 0;
var samples = 0;

function myCallback(data, num) {
  samples += num;
}

jetpack.statusBar.append({
  html: 'Stream<i>!</i>',
  width: 45,
  onReady: function(doc) {
    $(doc).click(function() {
      if (jetpack.audio.isRecording) {
        jetpack.audio.stopRecording();
        jetpack.notifications.show("Received " + samples +
          " in " + (Date.now() - start) + " seconds.");
      } else {
        start = Date.now();
        jetpack.audio.recordToPipe(myCallback);
        jetpack.notifications.show("Started recording to pipe");
      }
    });
  }
});

References

• Labs/Jetpack/JEP/18
• https://mozillalabs.com/blog/2009/07/jetpack-0-4-audio-recording-page-mods/