[[File:Meemoo.png|thumb|300px|right|Meemoo is "the Arduino for Web"]]

== Meemoo (Hackable Web Apps) ==

[http://meemoo.org Meemoo] is a framework for hackable web apps by Forrest Oliphant and an incredible tool for web making. My main goal in the project is to extend Meemoo to make modules creation easier. Based on a collaborative [http://github.com/automata/water live code editor] I want people creating new modules for hackable web apps inside their own browsers, without external tools.

Give Meemoo a try! Go http://meemoo.org/iframework

Fork Meemoo! Go http://github.com/meemoo

== Weekly Report ==

A daily log of my activities around Meemoo GSoC 2012 project is at: http://etherpad.mozilla.org/meemoo2012. Here is a resume of those activities week by week.

=== Before GSoC period ===

My first contact with Meemoo was by Alex McLean, who presented Meemoo as an interesting livecoding environment on Web. This was around middle of March and after some days I knew Meemoo was part of Moz GSoC.

At March 29 through April 23 I was in contact with Forrest. First of all I created a hello-world module (now we are calling this external modules by "NodeBox-Iframe") to understand how modules communicates inside Meemoo. I was surprised how easy was that! This first contact with Meemoo generated this tutorial: https://github.com/meemoo/iframework/wiki/Creating-a-Module.

I was amazed with Meemoo and started to write my proposal to Moz GSoC. I read about Moz focus on "creating webmakers" by Mark Surman, Dave Humphrey and others mozillians, so I wanted to join the initiative, and Meemoo seems to be an amazing tool around that. In this meantime I studied Backbone.js, window.postMessage and all the Meemoo's architecture: iframework (the environment) and meemoo.js (the embeddable module lib).

In this period I reported some issues on iframework and created a bunch of simple modules and apps:

* http://meemoo.org/iframework/#gist/2248496
* http://meemoo.org/iframework/#gist/2248582
* http://meemoo.org/iframework/#gist/2306646

I also fixed an issue around actions history:

* http://github.com/meemoo/iframework/issues/24

And at April 23 I was crossing my fingers :-)

=== Community bonding period ===

==== Week 1 (Apr 23 - Apr 30) ====

For this period I originally proposed:

April 23 - May 21 (4 weeks): discussing with my mentor about the exact features and
means of implementation around Meemoo, syncing with my mentor works and doing
some prototypes.

I started thinking about the live code editor (how to implement that? as a module? how to evaluate the generated code?) and decided to go with an iframe editor using Ace: [http://i.imgur.com/VPy35.png] and [http://i.imgur.com/Dllre.png].

==== Week 2 (May 1 - May 7) ====

I discussed some ideas around the live code editor with Forrest and at the end of week Forrest pointed me to Meteor. I studied Meteor planning to use it to make the live code editor collaborative. And with Forrest we planned to use Meteor as a way to make the whole Meemoo (iframework) collaborative! I created a test app using Meteor and feel surprised. I just didn't liked all the magic needed to generate DOM, but we can use something in the middle to make the "view source" work.

==== Week 3 (May 8 - May 14) ====

Renato Fabbri invited me to use Meemoo on "Cúpula dos Povos" at "Rio+20" conference, here in Brazil. I planned to create some modules to do photo sequencing and 3D forms synthesis based on parameters extracted from audio tags. I started creating a supershape.js module: http://meemoo.org/iframework/#/gist/2644999

Forrest pointed me to toolness' friendlycode (now the editor behind Moz Thimble) and I got some ideas for the live code editor (mainly the tips thing).

I started to plan how to use the audio APIs (Web Audio, Audio Data, MediaStream) in a compatible manner inside Meemoo. I decided to create a Web Audio API module.

We discovered RecursiveDrawing.com and some ideas exploded :-)

Jussi pointed me to his mike.js (https://github.com/jussi-kalliokoski/mike.js) to get raw data from mic while the W3C Audio spec isn't defined.

==== Week 4 (May 15 - May 21) ====

We were getting more and more ideas for apps and modules so we decided to store everything here:

* https://github.com/meemoo/iframework/wiki/Module-ideas
* https://github.com/meemoo/iframework/wiki/Share-your-apps

Forrest was working at Hackable Clock app to Moz Summer of Code Party meeting (https://etherpad.mozilla.org/May22 ) and I forked the app with some audio thing: http://meemoo.org/iframework/#gist/2765356 ;-)

I also started to study the Web Audio API (https://gist.github.com/2728285)

Forrest started to design the Native Nodes for iframework and it make my ideas to change a bit. I decided to make the live code editor and the audio nodes as native nodes.

=== Coding period ===

==== Week 5 (May 22 - May 28) ====

Talked with Joe, the creator of Audiolet, about Web Audio API integration and he gave me some good ideas. But I still need to understand how to use the native Web Audio nodes instead of JavaScriptAudioNode.

Forrest presented Meemoo as a demo day at his University and meet Sebastian, the creator of an awesome WebPd fork!

For this week I originally proposed for GSoC:

May 22 - May 28 (1 week): fixing the remaining bugs on iframework and meemoo.js,
getting them ready to work on the live code editor.

But Forrest was working on a refactoring, so I delayed the work on iframework and focused on the live code editor and audio nodes. I thought a lot about them and I decided:

* Having the live code editor as a friendlycode fork (livecode branch) and as a native node invoked by "view source" and "create new module" events inside iframework
* Having the audio nodes using Web Audio API or Audiolet (selected by browser type, for now) and as native nodes. As we got more resolution from W3C Audio spec we can remove the specific code

==== Week 6 (May 29 - Jun 4) ====

For the next two weeks I proposed:

May 29 - June 11 (2 weeks): extending iframework to create the code editor.
I already have a simple editor implemented using Ace [1] and another using
CodeMirror [2]. I will integrate one of those editors inside iframework.

I implemented some changes on live code editor (now a fork of friendlycode, the core of Moz Thimble).

I created some use cases (see http://etherpad.mozilla.org/meemoo2012, beginning of pad) and tested those with Web Audio API and everything worked well but just in a Chrome canary build. We need to have a node inside Audiolet to get raw data from audio and video tags sources. I have to discuss more with Joe about that.

Google launched Blockly and I planned to use that inside the live code editor, as an optional "mode" (like Emacs modes).

Forrest created a new branch for native nodes and I fetched. Things started to get a form!

==== Week 7 (Jun 5 - Jun 11) ====

I worked in the audio nodes and the live code editor. They are not native nodes of Meemoo yet, but standalone pages. I'm planning to integrate them with iframework in 2 or 3 days.

==== Week 8: (Jun 12 - Jun 18) ====

Originally proposed:

June 12 - June 18 (1 week): integrating the editor with share.js or etherpad-lite,
to make possible collaborative coding.

I have to discuss with Forrest because we planned to make the whole iframework collaborative. But for now, maybe we could have just the live code editor collaborative, based on Meteor.

I'm planning to have the editor and audio nodes as native nodes this week too. UPDATE: the audio nodes is dependant of native nodes and we have lots of work before touching the media nodes

I discussed with Forrest and we are concentrating on native nodes for now. I have started a Meteor app to use as an external live code editor while we don't have native nodes. I'm testing how to update the whole iframe every time we change the Meteor editor. When we got native nodes, we are planning to use Meteor to make the whole iframework collaborative, but it is not possible now.

Forrest: I think we should experiment with just modules for now, because I'm
guessing it will take a bigger rewrite to make iframework work with it.
I like the idea of different apps on different systems feeding
images/sounds/data into the same meteor module.

Vilson: OK! So a version of the editor in meteor...

Forrest integrated Grunt, so now we can start doing tests and the deploying is much easier.

We also started to think more seriously about an integration with Google Blockly, like a module to do logic operations. I'm planning a Blockly mode for the live code editor to edit JavaScript.

==== Week 9: (Jun 19 - Jun 25) ====

Originally proposed:

June 19 - June 25 (1 week): creating a Web server based on Django or node.js
to store modules created by everyone. Every module is refered by its URL in
Meemoo, which makes the server design simple.

Forrest worked on local storage and we are using Gist to store public apps. So, for now we don't need a server. Maybe this will come with the Meteor "backend". We had a big push to iframework repos with all those features. Meemoo is rocking! :-D

In this way I got this week to continue the work on the editor and created a module for mr.doob's Harmony: http://meemoo.org/iframework/#gist/2959806 and we are having fun :-)

http://i.imgur.com/So1ik.gif
http://a4.sphotos.ak.fbcdn.net/hphotos-ak-snc7/c9.0.403.403/p403x403/601267_323131427772915_595524936_n.jpg
http://a4.sphotos.ak.fbcdn.net/hphotos-ak-prn1/c0.0.337.337/p403x403/550432_323079607778097_1216176047_n.jpg
http://a7.sphotos.ak.fbcdn.net/hphotos-ak-ash3/c9.0.403.403/p403x403/552783_323134351105956_963782996_n.jpg

Some ideas for the live editor UI from http://codepen.io:

Vilson: Lots of good ideas from codepen. I think we can use the live editor
for lots of other projects too. Thinking on a editor to use "creative coding"
libs like three.js, paper.js or audiolet.

We are using the Facebook page of Meemoo to post some updates, together with Twitter: http://facebook.com/meemooapp

==== Week 10: (Jun 26 - Jul 2) ====

Originally proposed:

June 26 - July 2 (1 week): beginning creating new modules inside the editor:
a processing.js edit-and-run module, an event sequencer and an audio sampler
(based on Audiolet's BufferPlayer, using Mozilla's Audio Data API).

The whole media nodes, including audio, are dependant of native nodes. So we are delaying this. I'm creating another modules like meemoo-supershape.js as iframe nodes.

Forrest created math evaluation nodes! And now we can have cool animations like these:

* http://meemoo.org/iframework/#gist/3002307
* http://meemoo.org/iframework/#gist/3018598

And we continue having lots of fun :-D

http://a8.sphotos.ak.fbcdn.net/hphotos-ak-prn1/c0.0.403.403/p403x403/548607_326966247389433_427146130_n.jpg
http://i.imgur.com/jYDuh.gif
http://i.imgur.com/B67r2.gif
http://i.imgur.com/sDIz0.gif
http://i.imgur.com/TcO6N.png
http://i.imgur.com/Ch1PD.png
http://i.imgur.com/JR5Zl.png
http://i.imgur.com/hWdVl.png

I changed my priorities for:

# live code editor as meteor app
# native nodes
# audio nodes using web audio api
# app view as "show only this node"
# fullscreen view as "show only this node in fs"

Discussed with Forrest about app view and for now we are thinking on a button to hide the whole app but the current module.

Taking some inspiration from https://github.com/sullerandras/collaborative-html-editor-with-meteor to the live code editor refresh iframe thing.

Forrest wrote about his progress on June: http://meemoo.org/blog/2012-06-27-nearing-beta/

http://i.imgur.com/nLFKo.gif
http://i.imgur.com/y2GKs.gif

I'll post mine soon.

==== Week 11: (Jul 3 - Jul 9) ====

Originally proposed:

July 3 - July 9 (1 week): testing to ensure that the editor perfectly works
before the mid-term evaluations, asking for feedback from my mentor and the
community.

I have the editor almost done! I'll deploy on meteor.com soon and start testing this weekend!

Thinking together with Forrest about ideas for more Mozilla Webmakers projects like https://wiki.mozilla.org/Webmakers/Projects/MeemooClock-DIY

* How to create your own GIF?
* How to create crazy 3D shapes?
* How to create your meme and publish on Facebook?

==== Week 12: (Jul 10 - Jul 16) ====

Originally proposed:

July 9 - July 13 (4 days): submitting the project for mid-term evaluation,
talking with my mentor about the current results and discussing what
should be changed.

I wrote the mid-term evaluation and passed!

This week I started getting in trouble with Meteor. URL routing wasn't implemented yet and I had to do it by hand using Backbone Routing. It is working now.

I deployed the app at http://liveeditor.meteor.com and the source at http://github.com/automata/meemoo-liveeditor

We decided to have two views for each module: "edit" and "module". User could then change between the two views. I planned to use http://liveeditor.meteor.com/foo/edit and http://liveeditor.meteor.com/foo/module to have both modes for the "foo" module.

I continued working on three fronts: ''the editor'', ''the audio nodes'' and ''hacking new modules''.

We noticed Ward Cunningham "Federated Wiki" was similar with Meemoo approach but focusing on "distributed wikis" instead of "hackable apps". Maybe we can work together soon!

==== Week 13: (Jul 17 - Jul 23) ====

Originally proposed:

July 14 - July 23 (more than a week): continue creating new modules: an
audio synthesizer (like [3]), a video mixer (inspired by Forrest thesis
on Meemoo), a tone matrix clone [4] and a simple din clone [5]

My choice for Meteor was based on the easy to develop a WebSocket based collaborative editor that could update the generated HTML inside a module automatically. Unfortunately Meteor reveals not a good option to "non-standard" Webby things like I'm doing: an Etherpad-like editor. I decided to go with another approach: implementing the whole WebSocket thing by hand. The frontend is almost done, so it is not a big problem but I'll need to study things like patching and WebSockets.

In the modules front we got in touch with Japanese community of audio synth g200kg. The guys created amazing synths and I created two new modules to have them inside Meemoo:

* http://github.com/automata/meemoo-webmodular
* http://github.com/automata/meemoo-webbeeper

Demos apps are here:

* http://meemoo.org/iframework/#gist/3168087
* http://meemoo.org/iframework/#gist/3168177

We even have Beethoven's Moonlight Sonata in Meemoo :-D

* http://meemoo.org/iframework/#gist/3208230

Both modules communicates using MIDI-like messages. Soon I'll implement a better MIDI type for Meemoo, respecting the draft spec from Jussi and Chris from W3C Audio WG. For now we are using MIDI messages similar of g200kg's WebMIDILink: http://www.g200kg.com/en/docs/webmidilink/spec.html.

I was not totally OK with the original proposal. I have to write some other modules but I'm in touch with Forrest to hack priority ones first.

==== Week 14: (Jul 24 - Jul 30) ====

Originally Proposed:

July 24 - August 6 (two weeks): working on iframework to improve its UX,
creating a theme system and zoom/panning functionalities based on ThreeNode.js.

This will have to wait. I passed this week studying Meteor, share.js, Node.js and WebSockets and comparing them. I still tried to implement the editor in Meteor but failed.

I contacted Carols from Google for support to go to FISL but unfortunately I could not go to Porto Alegre :-(

==== Week 15: (Jul 31 - Aug 6) ====

Starting a countdown to ''pencils down date''!

This week we are working on a Meteor module to share state between multiple connected clients. A first version is online at http://meemoo.org/iframework/#gist/3209721. Just open that in various tabs and change the slider! The module: http://meemoo-tunnel.meteor.com and its source: http://github.com/automata/meemoo-tunnel. The point is simple but opens lots of possibilities: someone using the meemoo-tunnel can send data for other remote iframeworks using meemoo-tunnel as well. We can create, for example, a 20-channel audio synth (one channel per computer) controlled by all the participants, or a visual projection controlled remotely by someone's mobile device!

In the editor front I'm implementing the editor in Node using ShareJS instead of Meteor. ShareJS is more interesting for my approach because I only need to share text changes done in a CodeMirror editor. I don't want the entire application state being shared. Maybe we can go for Etherpad Lite in a near future, but I'm happy with ShareJS for now.

About the audio nodes, we decided to use Web Audio API in Chrome and Audiolet in Firefox. I talked with Robert O'Callahan and he created a [https://bugzilla.mozilla.org/show_bug.cgi?id=779297 watching bug] making possible to follow Moz progress in implementing Web Audio API in Firefox. Hoping to help the media team as much as possible in the following months. Matt Diamond created an [https://github.com/h5bp/lazyweb-requests/issues/82 awesome thread] about a possible standard to interchange Web Audio API based apps. A nice find to us because we are thinking about ways to standardize those apps inside Meemoo and [http://www.g200kg.com MIDI] [http://github.com/automata/meemoo-webmodular support too].

And thanks to the incredible work of Forrest the [https://github.com/meemoo/iframework/tree/master/src/nodes native nodes works]! YEY! I'm studying them and learning a lot!

At the end of week Pomax and Atul (THANKS!) fixed [https://github.com/mozilla/webpagemaker/issues/485 #485] in webpagemaker (Moz Thimble). Now we can use pages created in Thimble inside iframes and then inside Meemoo. I started talking with Atul about the possibility of JS editing inside Thimble.

I also created another module, the [http://github.com/automata/meemoo-intro meemoo-intro]. The app is here: http://meemoo.org/iframework/#gist/3282702 (UPDATE: now Forrest made it the default Meemoo app, yey!). It is an interactive guide to Meemoo newcomers. We have to work a lot on that to explains Meemoo as much as possible. Oh, yes, I used popcorn.js for that, it is awesome!

==== Week 16: (Aug 7 - Aug 13) ====

Originally proposed:

August 7 - August 13 (1 week): refactoring and fixing the code of
iframework, meemoo.js, meemoo-modules and Meemoo modules server,
finishing and styling the documentation. Creating example apps using the
new modules and commiting the last updates into Meemoo repositories.
Final review with my mentor and the community before the "pencils-down"
date.

I have lots of works yet! I'm pushing code reviewing for the next week. I don't see GSoC as 12-weeks projects but as a start to more work inside Meemoo and Mozilla projects like Thimble/WebMaker. In this way my priorities are in hacking as much as possible the modules I created, the code editor and iframework, and them report my entire progress and do code review in the next week.

I participated in the WebMakers call and Forrest presented the meemoo-intro app! Thanks Fo! http://meemoo.org/iframework/#gist/3282702

This week some guys (@janesconference and @thealphanerd -> he is also a GSoCer! Small world!) mailed about [http://github.com/automata/osc-web osc-web] and we talked about a way to make it a standard and use it inside Meemoo. For now, we use OSC inside Meemoo running a local nodejs server as a 'bridge' between OSC and HTTP. Forrest suggested to use a Java applet (like midibridge does) and I'm thinking about that.

About the editor, after some fixes in the CodeMirror integration with ShareJS, it is online at http://li7e.org. I sent the fix to ShareJS too. The code of "li7e" (liVe with a rotated V ;-)) is on http://github.com/automata/li7e. Planning to test some things on li7e and them suggest them to Atul to have that in Thimble!

So, now we can create a new Meemoo module inside our own browsers! The steps are something like:
* copy some code and paste in the editor: http://li7e.org/paperjs/edit
* hack the code to include Meemoo and get something to send (or receive)
* copy the url and paste in the 'add module' menu: http://li7e.org/paperjs
* share your app: http://meemoo.org/iframework/#gist/3345317
* and fork http://meemoo.org/iframework/#gist/3345422 :-)

I started the audio nodes based in Chris Wilson's [https://github.com/cwilso/WebAudio/blob/master/js/main.js Web Audio API playground]. Created some simple nodes in my branch 'audio': https://github.com/automata/iframework/blob/audio/src/nodes/audio.js.

Created the apps gallery in meemoo.org: http://meemoo.org/hack-our-apps/

==== Week 17: (Aug 14 - Aug 20) ====

Originally proposed:

August 14 - August 19 (some days): last days to do minor fixes and
stay in contact as much as possible with my mentor to polish the project.

Attended to WebMakers call today and was presented to TowTruck from Moz Labs: http://vimeo.com/36754286. It is awesome! It is so close of the editor (LI7E) we are creating for collaborative coding! I'll try to talk more with Atul about that.

For this week: Web Audio API native nodes, better meemoo-intro, documentation and code review of all the repos created/pushed.

== More information ==

The official Meemoo page is: http://meemoo.org

The official project details is in: http://www.google-melange.com/gsoc/project/google/gsoc2012/automata/14001

And the accepted proposal is publicly available at: http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/automata/1

== About the authors ==

Forrest Oliphant (mentor): http://forresto.com ___ [http://twitter.com/forresto @forresto]

Vilson Vieira (student): http://automata.cc ___ [http://twitter.com/aut0mata @aut0mata]

SummerOfCode/2012/Meemoo

SummerOfCode/2012/Meemoo

2012-04-25T12:15:30Z

Vilsonvieira:

SummerOfCode/2012/Meemoo

2012-04-25T12:09:11Z

Vilsonvieira: Created page with "Meemoo is a framework for hackable web apps by Forrest Oliphant and an incredible tool for web making. My main goal in the project is to extend Meemoo to make modules creation ea..."

Meemoo is a framework for hackable web apps by Forrest Oliphant and an incredible tool for web making. My main goal in the project is to extend Meemoo to make modules creation easier. Based on a collaborative [http://github.com/automata/water live code editor] I want people creating new modules for hackable web apps inside their own browsers, without external tools.

== More information ==

The weekly report is being writting at Meemoo's blog: http://meemoo.org/blog

A daily log of my activities around Meemoo GSoC 2012 project is at: http://etherpad.mozilla.org/meemoo2012

The oficial project details in: http://www.google-melange.com/gsoc/project/google/gsoc2012/automata/14001

== About the authors ==

Forrest Oliphant (mentor): http://forresto.com ___ [http://twitter.com/forresto @forresto]

Vilson Vieira (student): http://automata.cc ___ [http://twitter.com/automata @automata]

User:Vilsonvieira

2012-04-19T15:59:11Z

Vilsonvieira: Created page with "I create algorithms that create. ~ http://automata.cc"

I create algorithms that create.

~ http://automata.cc

Audio Data API

2011-06-01T20:26:20Z

Vilsonvieira:

== Defining an Enhanced API for Audio (Draft Recommendation) ==

===== Abstract =====

The HTML5 specification introduces the <audio> and <video> media elements, and with them the opportunity to dramatically change the way we integrate media on the web. The current HTML5 media API provides ways to play and get limited information about audio and video, but gives no way to programatically access or create such media. We present a new Mozilla extension to this API, which allows web developers to read and write raw audio data.

===== Authors =====

* David Humphrey ([http://twitter.com/humphd @humphd])
* Corban Brook ([http://twitter.com/corban @corban])
* Al MacDonald ([http://twitter.com/f1lt3r @F1LT3R])
* Yury Delendik
* Ricard Marxer ([http://twitter.com/ricardmp @ricardmp])
* Charles Cliffe ([http://twitter.com/ccliffe @ccliffe])

===== Other Contributors =====

* Thomas Saunders
* Ted Mielczarek

== API Tutorial ==

This API extends the HTMLMediaElement and HTMLAudioElement (e.g., affecting <video> and <audio>), and implements the following basic API for reading and writing raw audio data:

===== Reading Audio =====

Audio data is made available via an event-based API. As the audio is played, and therefore decoded, sample data is passed to content scripts in a framebuffer for processing after becoming available to the audio layer--hence the name, '''MozAudioAvailable'''. These samples may or may not have been played yet at the time of the event. The audio samples returned in the event are raw, and have not been adjusted for mute/volume settings on the media element. Playing, pausing, and seeking the audio also affect the streaming of this raw audio data.

Users of this API can register two callbacks on the <audio> or <video> element in order to consume this data:

<pre>
<audio id="audio" src="song.ogg"></audio>
<script>
var audio = document.getElementById("audio");
audio.addEventListener('MozAudioAvailable', audioAvailableFunction, false);
audio.addEventListener('loadedmetadata', loadedMetadataFunction, false);
</script>
</pre>

The '''loadedmetadata''' event is a standard part of HTML5. It now indicates that a media element (audio or video) has useful metadata loaded, which can be accessed using three new attributes:

* mozChannels
* mozSampleRate
* mozFrameBufferLength

Prior to the '''loadedmetadata''' event, accessing these attributes will cause an exception to be thrown, indicating that they are not known, or there is no audio. These attributes indicate the '''number of channels''', audio '''sample rate per second''', and the '''default size of the framebuffer''' that will be used in '''MozAudioAvailable''' events. This event is fired once as the media resource is first loaded, and is useful for interpreting or writing the audio data.

The '''MozAudioAvailable''' event provides two pieces of data. The first is a framebuffer (i.e., an array) containing decoded audio sample data (i.e., floats). The second is the time for these samples measured from the start in seconds. Web developers consume this event by registering an event listener in script like so:

<pre>
<audio id="audio" src="song.ogg"></audio>
<script>
var audio = document.getElementById("audio");
audio.addEventListener('MozAudioAvailable', someFunction, false);
</script>
</pre>

An audio or video element can also be created with script outside the DOM:

<pre>
var audio = new Audio();
audio.src = "song.ogg";
audio.addEventListener('MozAudioAvailable', someFunction, false);
audio.play();
</pre>

The following is an example of how both events might be used:

<pre>
var channels,
rate,
frameBufferLength,
samples;

function audioInfo() {
var audio = document.getElementById('audio');

// After loadedmetadata event, following media element attributes are known:
channels = audio.mozChannels;
rate = audio.mozSampleRate;
frameBufferLength = audio.mozFrameBufferLength;
}

function audioAvailable(event) {
var samples = event.frameBuffer;
var time = event.time;

for (var i = 0; i < frameBufferLength; i++) {
// Do something with the audio data as it is played.
processSample(samples[i], channels, rate);
}
}
</pre>

===== Complete Example: Visualizing Audio Spectrum =====

This example calculates and displays FFT spectrum data for the playing audio:

[[File:fft.png]]

<pre>
<!DOCTYPE html>
<html>
<head>
<title>JavaScript Spectrum Example</title>
</head>
<body>
<audio id="audio-element"
src="song.ogg"
controls="true"
style="width: 512px;">
</audio>
<div><canvas id="fft" width="512" height="200"></canvas></div>

<script>
var canvas = document.getElementById('fft'),
ctx = canvas.getContext('2d'),
channels,
rate,
frameBufferLength,
fft;

function loadedMetadata() {
channels = audio.mozChannels;
rate = audio.mozSampleRate;
frameBufferLength = audio.mozFrameBufferLength;

fft = new FFT(frameBufferLength / channels, rate);
}

function audioAvailable(event) {
var fb = event.frameBuffer,
t = event.time, /* unused, but it's there */
signal = new Float32Array(fb.length / channels),
magnitude;

for (var i = 0, fbl = frameBufferLength / 2; i < fbl; i++ ) {
// Assuming interlaced stereo channels,
// need to split and merge into a stero-mix mono signal
signal[i] = (fb[2*i] + fb[2*i+1]) / 2;
}

fft.forward(signal);

// Clear the canvas before drawing spectrum
ctx.clearRect(0,0, canvas.width, canvas.height);

for (var i = 0; i < fft.spectrum.length; i++ ) {
// multiply spectrum by a zoom value
magnitude = fft.spectrum[i] * 4000;

// Draw rectangle bars for each frequency bin
ctx.fillRect(i * 4, canvas.height, 3, -magnitude);
}
}

var audio = document.getElementById('audio-element');
audio.addEventListener('MozAudioAvailable', audioAvailable, false);
audio.addEventListener('loadedmetadata', loadedMetadata, false);

// FFT from dsp.js, see below
var FFT = function(bufferSize, sampleRate) {
this.bufferSize = bufferSize;
this.sampleRate = sampleRate;
this.spectrum = new Float32Array(bufferSize/2);
this.real = new Float32Array(bufferSize);
this.imag = new Float32Array(bufferSize);
this.reverseTable = new Uint32Array(bufferSize);
this.sinTable = new Float32Array(bufferSize);
this.cosTable = new Float32Array(bufferSize);

var limit = 1,
bit = bufferSize >> 1;

while ( limit < bufferSize ) {
for ( var i = 0; i < limit; i++ ) {
this.reverseTable[i + limit] = this.reverseTable[i] + bit;
}

limit = limit << 1;
bit = bit >> 1;
}

for ( var i = 0; i < bufferSize; i++ ) {
this.sinTable[i] = Math.sin(-Math.PI/i);
this.cosTable[i] = Math.cos(-Math.PI/i);
}
};

FFT.prototype.forward = function(buffer) {
var bufferSize = this.bufferSize,
cosTable = this.cosTable,
sinTable = this.sinTable,
reverseTable = this.reverseTable,
real = this.real,
imag = this.imag,
spectrum = this.spectrum;

if ( bufferSize !== buffer.length ) {
throw "Supplied buffer is not the same size as defined FFT. FFT Size: " + bufferSize + " Buffer Size: " + buffer.length;
}

for ( var i = 0; i < bufferSize; i++ ) {
real[i] = buffer[reverseTable[i]];
imag[i] = 0;
}

var halfSize = 1,
phaseShiftStepReal,
phaseShiftStepImag,
currentPhaseShiftReal,
currentPhaseShiftImag,
off,
tr,
ti,
tmpReal,
i;

while ( halfSize < bufferSize ) {
phaseShiftStepReal = cosTable[halfSize];
phaseShiftStepImag = sinTable[halfSize];
currentPhaseShiftReal = 1.0;
currentPhaseShiftImag = 0.0;

for ( var fftStep = 0; fftStep < halfSize; fftStep++ ) {
i = fftStep;

while ( i < bufferSize ) {
off = i + halfSize;
tr = (currentPhaseShiftReal * real[off]) - (currentPhaseShiftImag * imag[off]);
ti = (currentPhaseShiftReal * imag[off]) + (currentPhaseShiftImag * real[off]);

real[off] = real[i] - tr;
imag[off] = imag[i] - ti;
real[i] += tr;
imag[i] += ti;

i += halfSize << 1;
}

tmpReal = currentPhaseShiftReal;
currentPhaseShiftReal = (tmpReal * phaseShiftStepReal) - (currentPhaseShiftImag * phaseShiftStepImag);
currentPhaseShiftImag = (tmpReal * phaseShiftStepImag) + (currentPhaseShiftImag * phaseShiftStepReal);
}

halfSize = halfSize << 1;
}

i = bufferSize/2;
while(i--) {
spectrum[i] = 2 * Math.sqrt(real[i] * real[i] + imag[i] * imag[i]) / bufferSize;
}
};
</script>
</body>
</html>
</pre>

===== Writing Audio =====

It is also possible to setup an <audio> element for raw writing from script (i.e., without a ''src'' attribute). Content scripts can specify the audio stream's characteristics, then write audio samples using the following methods:

<code>mozSetup(channels, sampleRate)</code>

<pre>
// Create a new audio element
var audioOutput = new Audio();
// Set up audio element with 2 channel, 44.1KHz audio stream.
audioOutput.mozSetup(2, 44100);
</pre>

<code>mozWriteAudio(buffer)</code>

<pre>
// Write samples using a JS Array
var samples = [0.242, 0.127, 0.0, -0.058, -0.242, ...];
var numberSamplesWritten = audioOutput.mozWriteAudio(samples);

// Write samples using a Typed Array
var samples = new Float32Array([0.242, 0.127, 0.0, -0.058, -0.242, ...]);
var numberSamplesWritten = audioOutput.mozWriteAudio(samples);
</pre>

<code>mozCurrentSampleOffset()</code>

<pre>
// Get current audible position of the underlying audio stream, measured in samples.
var currentSampleOffset = audioOutput.mozCurrentSampleOffset();
</pre>

Since the '''MozAudioAvailable''' event and the '''mozWriteAudio()''' method both use '''Float32Array''', it is possible to take the output of one audio stream and pass it directly (or process first and then pass) to a second:

<pre>
<audio id="a1"
src="song.ogg"
controls>
</audio>
<script>
var a1 = document.getElementById('a1'),
a2 = new Audio(),
buffers = [];

function loadedMetadata() {
// Mute a1 audio.
a1.volume = 0;
// Setup a2 to be identical to a1, and play through there.
a2.mozSetup(a1.mozChannels, a1.mozSampleRate);
}

function audioAvailable(event) {
// Write the current framebuffer
var frameBuffer = event.frameBuffer; // frameBuffer is Float32Array
writeAudio(frameBuffer);
}

a1.addEventListener('MozAudioAvailable', audioAvailable, false);
a1.addEventListener('loadedmetadata', loadedMetadata, false);

function writeAudio(audioBuffer) {
// audioBuffer is Float32Array
buffers.push({buffer: audioBuffer, position: 0});

// If there's buffered data, write that
while(buffers.length > 0) {
var buffer = buffers[0].buffer;
var position = buffers[0].position;
var written = a2.mozWriteAudio(buffer.subarray(position));
// // If all data wasn't written, keep it in the buffers:
if(position + written < buffer.length) {
buffers[0].position = position + written;
break;
}
buffers.shift();
}
}
</script>
</pre>

Audio data written using the '''mozWriteAudio()''' method needs to be written at a regular interval in equal portions, in order to keep a little ahead of the current sample offset (the sample offset that is currently being played by the hardware can be obtained with '''mozCurrentSampleOffset()'''), where a little means something on the order of 500ms of samples. For example, if working with 2 channels at 44100 samples per second, a writing interval of 100ms, and a pre-buffer equal to 500ms, one would write an array of (2 * 44100 / 10) = 8820 samples, and a total of (currentSampleOffset + 2 * 44100 / 2).

It's also possible to auto detect the minimal duration of the pre-buffer, such that the sound is played without interruptions, and lag between writing and playback is minimal. To do this start writing the data in small portions and wait for the value returned by '''mozCurrentSampleOffset()''' to be more than 0.

<pre>
var prebufferSize = sampleRate * 0.020; // Initial buffer is 20 ms
var autoLatency = true, started = new Date().valueOf();
...
// Auto latency detection
if (autoLatency) {
prebufferSize = Math.floor(sampleRate * (new Date().valueOf() - started) / 1000);
if (audio.mozCurrentSampleOffset()) { // Play position moved?
autoLatency = false;
}
}
</pre>

===== Complete Example: Creating a Web Based Tone Generator =====

This example creates a simple tone generator, and plays the resulting tone.

<pre>
<!DOCTYPE html>
<html>
<head>
<title>JavaScript Audio Write Example</title>
</head>
<body>
<input type="text" size="4" id="freq" value="440"><label for="hz">Hz</label>
<button onclick="start()">play</button>
<button onclick="stop()">stop</button>

<script type="text/javascript">
function AudioDataDestination(sampleRate, readFn) {
// Initialize the audio output.
var audio = new Audio();
audio.mozSetup(1, sampleRate);

var currentWritePosition = 0;
var prebufferSize = sampleRate / 2; // buffer 500ms
var tail = null, tailPosition;

// The function called with regular interval to populate
// the audio output buffer.
setInterval(function() {
var written;
// Check if some data was not written in previous attempts.
if(tail) {
written = audio.mozWriteAudio(tail.subarray(tailPosition));
currentWritePosition += written;
tailPosition += written;
if(tailPosition < tail.length) {
// Not all the data was written, saving the tail...
return; // ... and exit the function.
}
tail = null;
}

// Check if we need add some data to the audio output.
var currentPosition = audio.mozCurrentSampleOffset();
var available = currentPosition + prebufferSize - currentWritePosition;
if(available > 0) {
// Request some sound data from the callback function.
var soundData = new Float32Array(available);
readFn(soundData);

// Writting the data.
written = audio.mozWriteAudio(soundData);
if(written < soundData.length) {
// Not all the data was written, saving the tail.
tail = soundData;
tailPosition = written;
}
currentWritePosition += written;
}
}, 100);
}

// Control and generate the sound.

var frequency = 0, currentSoundSample;
var sampleRate = 44100;

function requestSoundData(soundData) {
if (!frequency) {
return; // no sound selected
}

var k = 2* Math.PI * frequency / sampleRate;
for (var i=0, size=soundData.length; i<size; i++) {
soundData[i] = Math.sin(k * currentSoundSample++);
}
}

var audioDestination = new AudioDataDestination(sampleRate, requestSoundData);

function start() {
currentSoundSample = 0;
frequency = parseFloat(document.getElementById("freq").value);
}

function stop() {
frequency = 0;
}
</script>
</body>
</html>
</pre>

== DOM Implementation ==

===== nsIDOMNotifyAudioAvailableEvent =====

Audio data is made available via the following event:

* '''Event''': AudioAvailableEvent
* '''Event handler''': onmozaudioavailable

The '''AudioAvailableEvent''' is defined as follows:

<pre>
interface nsIDOMNotifyAudioAvailableEvent : nsIDOMEvent
{
// frameBuffer is really a Float32Array
readonly attribute jsval frameBuffer;
readonly attribute float time;
};
</pre>

The '''frameBuffer''' attribute contains a typed array ('''Float32Array''') with the raw audio data (32-bit float values) obtained from decoding the audio (e.g., the raw data being sent to the audio hardware vs. encoded audio). This is of the form <nowiki>[channel1, channel2, ..., channelN, channel1, channel2, ..., channelN, ...]</nowiki>. All audio frames are normalized to a length of channels * 1024 by default, but could be any length between 512 and 16384 if the user has set a different length using the '''mozFrameBufferLength''' attribute.

The '''time''' attribute contains a float representing the time in seconds of the first sample in the '''frameBuffer''' array since the start of the audio track.

===== nsIDOMHTMLMediaElement additions =====

Audio metadata is made available via three new attributes on the HTMLMediaElement. By default these attributes throw if accessed before the '''loadedmetadata''' event occurs. Users who need this info before the audio starts playing should not use '''autoplay''', since the audio might start before a loadmetadata handler has run.

The three new attributes are defined as follows:

<pre>
readonly attribute unsigned long mozChannels;
readonly attribute unsigned long mozSampleRate;
attribute unsigned long mozFrameBufferLength;
</pre>

The '''mozChannels''' attribute contains the number of channels in the audio resource (e.g., 2). The '''mozSampleRate''' attribute contains the number of samples per second that will be played, for example 44100. Both are read-only.

The '''mozFrameBufferLength''' attribute indicates the number of samples that will be returned in the framebuffer of each '''MozAudioAvailable''' event. This number is a total for all channels, and by default is set to be the number of channels * 1024 (e.g., 2 channels * 1024 samples = 2048 total).

The '''mozFrameBufferLength''' attribute can also be set to a new value, if users want lower latency, or larger amounts of data, etc. The size given '''must''' be a number between 512 and 16384. Using any other size will result in an exception being thrown. The best time to set a new length is after the '''loadedmetadata''' event fires, when the audio info is known, but before the audio has started or '''MozAudioAvailable''' events begun firing.

===== nsIDOMHTMLAudioElement additions =====

The HTMLAudioElement has also been extended to allow write access. Audio writing is achieved by adding three new methods:

<pre>
void mozSetup(in long channels, in long rate);
unsigned long mozWriteAudio(array); // array is Array() or Float32Array()
unsigned long long mozCurrentSampleOffset();
</pre>

The '''mozSetup()''' method allows an <audio> element to be setup for writing from script. This method '''must''' be called before '''mozWriteAudio''' or '''mozCurrentSampleOffset''' can be called, since an audio stream has to be created for the media element. It takes two arguments:

# '''channels''' - the number of audio channels (e.g., 2)
# '''rate''' - the audio's sample rate (e.g., 44100 samples per second)

The choices made for '''channel''' and '''rate''' are significant, because they determine the amount of data you must pass to '''mozWriteAudio()'''. That is, you must pass an array with enough data for each channel specified in '''mozSetup()'''.

The '''mozSetup()''' method, if called more than once, will recreate a new audio stream (destroying an existing one if present) with each call. Thus it is safe to call this more than once, but unnecessary.

The '''mozWriteAudio()''' method can be called after '''mozSetup()'''. It allows audio data to be written directly from script. It takes one argument, '''array'''. This is a JS Array (i.e., new Array()) or a typed float array (i.e., new Float32Array()) containing the audio data (floats) you wish to write. It must be 0 or N elements in length, where N % channels == 0, otherwise an exception is thrown.

The '''mozWriteAudio()''' method returns the number of samples that were just written, which may or may not be the same as the number in '''array'''. Only the number of samples that can be written without blocking the audio hardware will be written. It is the responsibility of the caller to deal with any samples that don't get written in the first pass (e.g., buffer and write in the next call).

The '''mozCurrentSampleOffset()''' method can be called after '''mozSetup()'''. It returns the current position (measured in samples) of the audio stream. This is useful when determining how much data to write with '''mozWriteAudio()'''.

All of '''mozWriteAudio()''', '''mozCurrentSampleOffset()''', and '''mozSetup()''' will throw exceptions if called out of order. '''mozSetup()''' will also throw if a ''src'' attribute has previously been set on the audio element (i.e., you can't do both at the same time).

===== Security =====

Similar to the <canvas> element and its '''getImageData''' method, the '''MozAudioAvailable''' event's '''frameBuffer''' attribute protects against information leakage between origins.

The '''MozAudioAvailable''' event's '''frameBuffer''' attribute will throw if the origin of audio resource does not match the document's origin. NOTE: this will affect users who have the security.fileuri.strict_origin_policy set, and are working locally with file:/// URIs.

===== Compatibility with Audio Backends =====

The current MozAudioAvailable implementation integrates with Mozilla's decoder abstract base classes, and therefore, any audio decoder which uses these base classes automatically dispatches MozAudioAvailable events. At the time of writing, this includes the Ogg and WebM decoders but '''not''' the Wave decoder.

== Additional Resources ==

A series of blog posts document the evolution and implementation of this API: http://vocamus.net/dave/?cat=25. Another overview by Al MacDonald is available [http://weblog.bocoup.com/web-audio-all-aboard here].

Al has also written 2 very good tutorials and video demos of [http://weblog.bocoup.com/read-html5-audio-data-with-firefox-4 reading] and [http://weblog.bocoup.com/generate-sound-with-javascript-in-firefox-4 writing] audio with the API.

The BBC Research and Development Blog has also done an excellent overview of the API [http://www.bbc.co.uk/blogs/researchanddevelopment/2010/11/mozilla-audio-data-api.shtml here].

=== Bug ===

The work on this API is available in Mozilla [https://bugzilla.mozilla.org/show_bug.cgi?id=490705 bug 490705].

=== Obtaining Builds ===

[http://nightly.mozilla.org/ Firefox trunk nightlies] include the Audio Data API (starting with 2010-08-26 builds).

=== JavaScript Audio Libraries ===

* We have started work on a JavaScript library to make building audio web apps easier. Details are [[Audio Data API JS Library|here]] and http://github.com/corbanbrook/dsp.js.
* [http://github.com/corbanbrook/audionode.js audionode.js] acts as a javascript bridge between the Web Audio API and the Audio Data API allowing us to run the examples http://weare.buildingsky.net/processing/audionode.js/examples/index.html.
* [http://github.com/notmasteryet/audiodata/ Audio Data API Objects] - An high level abstraction (and an usage example) of the Audio Data API.
* [http://github.com/bfirsh/dynamicaudio.js dynamicaudio.js] - An interface for writing audio with a Flash fall back for older browsers.
* [https://beatdetektor.svn.sourceforge.net/svnroot/beatdetektor/trunk/core/js/beatdetektor.js Beat Detektor] by Charles Cliffe, uses dsp.js to add beat detection.
* [https://github.com/jussi-kalliokoski/audiolib.js/ audiolib.js] by Jussi Kalliokoski, a powerful audio tools library for JavaScript, compatible with the Audio Data API and Chrome's Audio API.
* [https://github.com/oampo/Audiolet Audiolet] - A JavaScript library for real-time audio synthesis and composition from within the browser
*[https://github.com/grantgalitz/XAudioJS XAudioJS] - A JavaScript library that provides a raw audio sample writing access to the mozilla audio data and web audio APIs. Provides a basic write and callback system so the developer can be assured to have gapless audio for these two APIs. Also provides a fallback WAV PCM data URI generator that is not guaranteed to be gapless.
* [http://www.gregjopa.com/2011/05/calculate-note-frequencies-in-javascript-with-music-js/ Music.js, library containing functions and data sets to generate notes, intervals, chords, and scales]
* [https://github.com/BillyWM/jsmodplayer Javascript .MOD and .XM music player]

=== Working Audio Data Demos ===

'''NOTE: we recently took down two servers that were hosting many of these demos. We are working to find a new home for them.'''

A number of working demos have been created, including:

* [http://videos.mozilla.org/serv/blizzard/audio-slideshow Overview slideshow demo of various features] ([http://www.youtube.com/watch?v=oJ1UsLoPX3E video here])

* Audio Visualizations
** http://www.nihilogic.dk/labs/pocket_full_of_html5/ (Demo by Jacob Seidelin)
** http://weare.buildingsky.net/processing/dsp.js/examples/fft.html
** http://www.storiesinflight.com/jsfft/visualizer/index.html (Demo by Thomas Sturm)

* Applying Realtime Audio Effects
** Volume, pitch, etc. UI for audio - https://developer.mozilla.org/en-US/demos/detail/voron (homepage: http://kievII.net)
** JS IIR Filter http://weare.buildingsky.net/processing/dsp.js/examples/filter.html (video [http://vimeo.com/11335434 here])
** Vocodes a formant with a carrier wave http://weare.buildingsky.net/processing/dsp.js/examples/vocoder.html
** Biquad Filter example http://weare.buildingsky.net/processing/dsp.js/examples/biquad.html
** Graphic EQ example http://weare.buildingsky.net/processing/dsp.js/examples/grapheq.html
** Delay effect http://code.almeros.com/code-examples/delay-firefox-audio-api/ (video of older version [http://vimeo.com/11780707 here])
** Reverb effect http://code.almeros.com/code-examples/reverb-firefox-audio-api/ (video [http://vimeo.com/13386796 here])

* Generating and Playing Audio
** [http://www.oampo.co.uk/2011/05/technocrat/ Ambient techno machine]
** [http://www.gregjopa.com/2011/05/calculate-note-frequencies-in-javascript-with-music-js/ Music.js, library containing functions and data sets to generate notes, intervals, chords, and scales]
** [https://hacks.mozilla.org/2011/01/html5guitar/ HTML5 Guitar Tab Player]
** [http://automata.cc/src/vivace/experiments/matrix.html Tone matrix using Audiolet.js]
** [http://www.oampo.co.uk/labs/audiolet-demo/ Generating music in JS via audiolet.js], [http://www.oampo.co.uk/labs/breakbeat/ breakbeat demo]
** [http://audioscene.org/scene-files/humph/sfxr/ sfxr.js] - sound effect generator/editor for video games.
** [http://jonbro.tk/blog/2010/09/19/html_5_chip_tracker.html JavaScript Chip Tracker app] (demo by Jonathan Brodsky)
** JavaScript Audio Sampler http://weare.buildingsky.net/processing/dsp.js/examples/sampler.html
** SamplePlayer, SampleLoader, Sequencer and Keyboard http://code.almeros.com/code-examples/sampler-firefox-audio-api/ (video [http://vimeo.com/13805893 here])
** Square Wave Generation http://weare.buildingsky.net/processing/dsp.js/examples/squarewave.html
** Random Noise Generation http://weare.buildingsky.net/processing/dsp.js/examples/nowave.html
** JS Multi-Oscillator Synthesizer http://weare.buildingsky.net/processing/dsp.js/examples/synthesizer.html (video [http://vimeo.com/11411533 here])
** Bloop http://async5.org/audiodata/examples/bloop-ea/bloop-audiodata.html
** JavaScript Text to Speech engine http://async5.org/audiodata/tts/index.html
** Toy Piano http://async5.org/audiodata/examples/piano.html (and the sample-based piano http://async5.org/audiodata/examples/piano-s/piano2.html)
** Csound Shaker Instrument http://async5.org/audiodata/csound/shaker.htm and Bar Instrument http://async5.org/audiodata/csound/bar.htm
** Canon Theremin Piano http://mtg.upf.edu/static/media/canon-theremin-piano.html (by Zacharias Vamvakousis zackbam@gmail.com).
** Manipulate music example using mouse and accelerometer http://blog.dt.in.th/stuff/audiodata/ (Thai Pangsakulyanont)
** Tuning exploration, Wicki keyboard and Karplus-Strong synthesizer http://www.toverlamp.org/static/wickisynth/wickisynth.html (Piers Titus van der Torren)
** Modular Synthesizer with MIDI, control and audio ports. http://www.niiden.com/jstmodular/ (Jussi Kalliokoski)
** Dual-axis Theremin controlling pitch and volume with cursor position. http://stu.ie/?p=2599 (Stuart Gilbert)
** JavaScript "Image to Sound" generator http://zhangjw.bai-hua.org/audio_test6.html (ZhangJW)
** XAudioJS library test page http://www.grantgalitz.org/sound_test/

* Beat Detection (also showing use of WebGL for 3D visualizations)
** http://people.mozilla.com/~prouget/demos/boomboom/index.html
** http://cubicvr.org/CubicVR.js/bd3/BeatDetektor1HD.html (video [http://vimeo.com/11345262 here])
** http://cubicvr.org/CubicVR.js/bd3/BeatDetektor2HD.html (video of older version [http://vimeo.com/11345685 here])
** http://cubicvr.org/CubicVR.js/bd3/BeatDetektor3HD.html (video [http://www.youtube.com/watch?v=OxoFcyKYwr0&fmt=22 here])
** http://cubicvr.org/CubicVR.js/bd3/BeatDetektor3HDFX.html (same, but with more effects)
** http://cubicvr.org/CubicVR.js/bd3/BeatDetektor4HD.html (video [http://www.youtube.com/watch?v=dym4DqpJuDk&fmt=22 here])
** http://cubicvr.org/CubicVR.js/bd_fluid_sim/BD_GPUFluid.html

* Writing Audio from JavaScript, Digital Signal Processing
** API Example: [http://audioscene.org/?p=171 Inverted Waveform Cancellation]
** API Example: [http://audioscene.org/?p=255 Stereo Splitting and Panning]
** API Example: [http://audioscene.org/?p=267/ Mid-Side Microphone Decoder]
** API Example: [http://audioscene.org/?p=279 Ambient Extraction Mixer]
** API Example: [http://audioscene.org/?p=302 Worker Thread Audio Processing]

* Audio Games
** http://www.oampo.co.uk/labs/fireflies/

=== Demos Needing to be Updated to New API ===

* FFT visualization (calculated with js)
** Experimental JavaScript port Pure Data http://mccormick.cx/dev/webpd/ with demo http://mccormick.cx/dev/webpd/demos/processingjs-basic-example-with-audio/index.html

** http://ondras.zarovi.cz/demos/audio/
** http://code.bocoup.com/processing-js/3d-fft/viz.xhtml

* Visualizing sound using the video element
** http://bocoup.com/core/code/firefox-audio/whale-fft2/whale-fft.html (video [http://vimeo.com/8872808 here])



=== Third Party Discussions ===

A number of people have written about our work, including:

* http://ajaxian.com/archives/amazing-audio-sampling-in-javascript-with-firefox
* http://createdigitalmusic.com/2010/05/03/real-sound-synthesis-now-an-open-standard-in-the-browser/
* http://www.webmonkey.com/2010/05/new-html5-tools-make-your-browser-sing-and-dance/
* http://www.wired.co.uk/news/archive/2010-05/04/new-html5-tools-give-adobe-flash-the-finger
* http://hacks.mozilla.org/2010/04/beyond-html5-experiments-with-interactive-audio/
* http://schepers.cc/?p=212
* http://createdigitalmusic.com/2010/05/27/browser-madness-3d-music-mountainscapes-web-based-pd-patching/
* http://news.slashdot.org/story/10/05/26/1936224/Breakthroughs-In-HTML-Audio-Via-Manipulation-With-JavaScript
* http://ajaxian.com/archives/amazing-audio-api-javascript-demos
* http://www.webmonkey.com/2010/08/sampleplayer-makes-your-browser-sing-sans-flash/