Labs/Jetpack/Reboot/JEP/120: Difference between revisions

From MozillaWiki
< Labs‎ | Jetpack‎ | Reboot‎ | JEP
Jump to navigation Jump to search
(initial audio jep)
(proposal withdrawn)
 
(5 intermediate revisions by the same user not shown)
Line 2: Line 2:


* Champion: Jim Garrison - jim@garrison.cc
* Champion: Jim Garrison - jim@garrison.cc
* Status: Under Review
* Status: Withdrawn
* Bug Ticket:
* Type: API
* Difficulty: 4


=== Proposal ===
=== Proposal ===


Implementation of the audio JEP will allow the developer to record audio from the user's microphone and store it in an Ogg/Vorbis file.
This proposal has been withdrawn (for now, at least) in light of the developments at http://lists.w3.org/Archives/Public/public-device-apis/2010Jul/0096.html
 
Alternatively, the developer can capture the audio stream to memory by providing a callback that receives each block of audio and knows how to process or store it.  And then the audio byte array can be encoded to an Ogg/Vorbis file.
 
=== Key Issues ===
 
* How do we implement audio capture in a cross-platform way?  portaudio seems to be the best way at the moment.  (libsydneyaudio, already used for playback of HTML5 <audio> and <video> elements, has an API for audio capture but it is unimplemented.)
* Should the streaming and encoding APIs always use 44.1 kHz audio, or should we increase API complexity to support other sample rates as well?
 
=== Dependencies & Requirements ===
 
=== Capabilities Required (if applicable) ===
 
=== API Methods ===
 
The Audio module will be available under <code>jetpack.audio</code>.
 
To begin recording to a file, call: <code>jetpack.audio.recordToFile</code>.
 
To stop the recording, call: <code>jetpack.audio.stopRecording</code>. This function will return the path of the Ogg/Vorbis file that the audio was recorded to.
 
To check if a recording is in progress (simultaneous recording of audio is not allowed), access the <code>jetpack.audio.isRecording</code> property.
 
(An exception will be raised if you attempt to record while another recording is already in progress.)
 
To record a stream to memory, call: <code>jetpack.audio.recordToPipe</code>.  The callback receives a byte array and the number of samples contained in that array. The array will consist of PCM float stereo (interleaved) samples at a frequency of 44100Hz.
 
We're working on adding support for re-encoding this byte stream (before or after manipulation) back into an Ogg/Vorbis file. Check out http://hg.mozilla.org/labs/jetpack/file/tip/components/experimental/audio/AudioEncoder.cpp if you want to help!
 
=== Use Cases ===
 
* Integration with a web service that does speech recognition
* Client for an audio twitter-like social network (e.g. audioboo, twaud.io)
* TwitterFox or a similar extension could add support for attaching audio to a tweet
* Live visualization of audio implemented in HTML/CSS/JS/Canvas
* Voice memos (https://jetpack.mozillalabs.com/demos.html)
* Several possible ways to interface with regular web sites
** An extension could provide a reference implementation of audio capture for the HTML5 &lt;device> element as that specification develops (see https://labs.ericsson.com/blog/beyond-html5-audio-capture-web-browsers and http://www.whatwg.org/specs/web-apps/current-work/multipage/commands.html#devices)
** Somebody could write a jetpack extension that exposes a non-standard Javascript API that web sites can access to record audio.
** A web site may want to provide a site-specific extension that allows users to record and upload snippets of audio.
** Somebody could make an extension that allows the user to record audio semlessly to any file upload form element, thus being able to easily upload audio to any web site.
* Web sites would benefit:
** Users could seamlessly attach audio to an email on gmail or any webmail site
** A language instruction wiki (e.g. Wikiotics) could allow users to easily upload spoken phrases for language lessons
** A greeting card site could seamlessly allow users to upload personalized audio messages
** A DJ/playlist site could allow users to record quick introductions between songs
 
=== Examples ===
 
==== Recording to a file ====
 
This Jetpack shows a 'Record' button on the status bar, which when clicked will start recording audio to a file. On clicking it again, the recording is stopped, a notification with the file path is displayed and the audio is played back.
 
<pre class="brush:js;toolbar:false;">
var path = '';
 
jetpack.statusBar.append({
  html: 'Record<i>!</i>',
  width: 55,
  onReady: function(w) {
    $(w).click(function() {
      if (jetpack.audio.isRecording) {
        path = jetpack.audio.stopRecording();
        jetpack.notifications.show("Saved to " + path + ", now playing!");
        $(jetpack.tabs.focused.contentDocument).find('body').
          append('<audio src="file://'+path+'" autoplay="true"></audio>');
      } else {
        jetpack.notifications.show("Recording...");
        jetpack.audio.recordToFile();
      }
    });
  }
});
</pre>
 
==== Recording to memory ====
 
The Jetpack audio module allows access to the raw audio stream. Here's an example:
 
<pre class="brush:js;toolbar:false">
jetpack.future.import("audio");
var start = 0;
var samples = 0;
 
function myCallback(data, num) {
  samples += num;
}
 
jetpack.statusBar.append({
  html: 'Stream<i>!</i>',
  width: 45,
  onReady: function(doc) {
    $(doc).click(function() {
      if (jetpack.audio.isRecording) {
        jetpack.audio.stopRecording();
        jetpack.notifications.show("Received " + samples +
          " in " + (Date.now() - start) + " seconds.");
      } else {
        start = Date.now();
        jetpack.audio.recordToPipe(myCallback);
        jetpack.notifications.show("Started recording to pipe");
      }
    });
  }
});
</pre>
 
=== References ===
 
* [[Labs/Jetpack/JEP/18]]
* https://mozillalabs.com/blog/2009/07/jetpack-0-4-audio-recording-page-mods/

Latest revision as of 04:17, 17 July 2010

JEP 120 - Audio JEP

  • Champion: Jim Garrison - jim@garrison.cc
  • Status: Withdrawn

Proposal

This proposal has been withdrawn (for now, at least) in light of the developments at http://lists.w3.org/Archives/Public/public-device-apis/2010Jul/0096.html

Retrieved from "https://wiki.mozilla.org/index.php?title=Labs/Jetpack/Reboot/JEP/120&oldid=238996"