Labs/Jetpack/Reboot/JEP/120: Difference between revisions

From MozillaWiki
< Labs‎ | Jetpack‎ | Reboot‎ | JEP
Jump to navigation Jump to search
Line 85: Line 85:


<pre class="brush:js;toolbar:false">
<pre class="brush:js;toolbar:false">
jetpack.future.import("audio");
var start = 0;
var start = 0;
var samples = 0;
var samples = 0;

Revision as of 19:42, 23 April 2010

JEP 120 - Audio JEP

  • Champion: Jim Garrison - jim@garrison.cc
  • Status: Under Review
  • Bug Ticket:
  • Type: API
  • Difficulty: 4

Proposal

Implementation of the audio JEP will allow the developer to record audio from the user's microphone and store it in an Ogg/Vorbis file.

Alternatively, the developer can capture the audio stream to memory by providing a callback that receives each block of audio and knows how to process or store it. And then the audio byte array can be encoded to an Ogg/Vorbis file.

Key Issues

  • How do we implement audio capture in a cross-platform way? portaudio seems to be the best way at the moment. (libsydneyaudio, already used for playback of HTML5 <audio> and <video> elements, has an API for audio capture but it is unimplemented.)
  • Should the streaming and encoding APIs always use 44.1 kHz audio, or should we increase API complexity to support other sample rates as well?

API Methods

The Audio module will be available under jetpack.audio.

To begin recording to a file, call: jetpack.audio.recordToFile.

To stop the recording, call: jetpack.audio.stopRecording. This function will return the path of the Ogg/Vorbis file that the audio was recorded to.

To check if a recording is in progress (simultaneous recording of audio is not allowed), access the jetpack.audio.isRecording property.

(An exception will be raised if you attempt to record while another recording is already in progress.)

To record a stream to memory, call: jetpack.audio.recordToPipe. The callback receives a byte array and the number of samples contained in that array. The array will consist of PCM float stereo (interleaved) samples at a frequency of 44100Hz.

We're working on adding support for re-encoding this byte stream (before or after manipulation) back into an Ogg/Vorbis file. Check out http://hg.mozilla.org/labs/jetpack/file/tip/components/experimental/audio/AudioEncoder.cpp if you want to help!

Use Cases

  • Integration with a web service that does speech recognition
  • Client for an audio twitter-like social network (e.g. audioboo, twaud.io)
  • TwitterFox or a similar extension could add support for attaching audio to a tweet
  • Live visualization of audio implemented in HTML/CSS/JS/Canvas
  • Voice memos (https://jetpack.mozillalabs.com/demos.html)
  • Several possible ways to interface with regular web sites
  • Web sites would benefit:
    • Users could seamlessly attach audio to an email on gmail or any webmail site
    • A language instruction wiki (e.g. Wikiotics) could allow users to easily upload spoken phrases for language lessons
    • A greeting card site could seamlessly allow users to upload personalized audio messages
    • A DJ/playlist site could allow users to record quick introductions between songs

Examples

Recording to a file

This Jetpack shows a 'Record' button on the status bar, which when clicked will start recording audio to a file. On clicking it again, the recording is stopped, a notification with the file path is displayed and the audio is played back. 

var path = '';

jetpack.statusBar.append({
  html: 'Record<i>!</i>',
  width: 55,
  onReady: function(w) {
    $(w).click(function() {
      if (jetpack.audio.isRecording) {
        path = jetpack.audio.stopRecording();
        jetpack.notifications.show("Saved to " + path + ", now playing!");
        $(jetpack.tabs.focused.contentDocument).find('body').
          append('<audio src="file://'+path+'" autoplay="true"></audio>');
      } else {
        jetpack.notifications.show("Recording...");
        jetpack.audio.recordToFile();
      }
    });
  }
});

Recording to memory

The Jetpack audio module allows access to the raw audio stream. Here's an example: 

var start = 0;
var samples = 0;

function myCallback(data, num) {
  samples += num;
}

jetpack.statusBar.append({
  html: 'Stream<i>!</i>',
  width: 45,
  onReady: function(doc) {
    $(doc).click(function() {
      if (jetpack.audio.isRecording) {
        jetpack.audio.stopRecording();
        jetpack.notifications.show("Received " + samples +
          " in " + (Date.now() - start) + " seconds.");
      } else {
        start = Date.now();
        jetpack.audio.recordToPipe(myCallback);
        jetpack.notifications.show("Started recording to pipe");
      }
    });
  }
});

References

Retrieved from "https://wiki.mozilla.org/index.php?title=Labs/Jetpack/Reboot/JEP/120&oldid=218101"