User:David.humphrey/www2010

WWW2010 Audio Data Talk

This is an outline for our talk.

Outline

• Introductions
  • Al, Dave
  • Audio Data
• Brief outline of talk, demos

• History of the work
  • Thomas Saunders and Al propose the idea
  • Dave patches Firefox (Minefield) to expose raw audio data
  • Corban Brook and Al work on methods for calculating FFT
  • Ted Mielczarek came-up with an idea for writing audio from JavaScript, and Dave implemented it.
  • Since then many demos, libraries, etc. have been written

• Current status of work
  • API details and links to working builds in Mozilla Wiki
  • Implementation details in Mozilla Bugzilla bug 490705

• Goals for today and this talk
  • Convince you that the web is ready for audio data
  • Show you that JavaScript is fast enough for real time digital sound processing
  • Explore some initial possibilities for audio data in the browser
  • Examine what an audio data API for the web might look like
  • Look forward to the future and what audio data might enable on the web

• API Design
  • audiowritten event
  • event.mozFrameBuffer
  • event.mozSpectrum
  • mozSetup(channels, sampleRate, volume)
  • mozWriteAudio(samples.length, samples [])

• Demos
  • Accessing raw audio data in JavaScript (real-time audiowritten event, mozFrameBuffer, frame padding with 0s)
  • Javascript FFT Spectrum(speed of JavaScript to handle this)
  • Spectrum data natively (mozSpectrum)
  • Visualizing spectrum (2D canvas, JavaScript)
  • Beat Detection (audio synchronization and visual cues, WebGL)
  • Audio data from video element
  • Building a Square wave (constructs a square wave out of sine wave components, plays audio as wave is constructed, mozWriteAudio)
  • JavaScript Synthesizer (creating, manipulating sound with JavaScript oscillators, envelopes and filters)
  • Audio filter (audiowritten event, applies lowpass filter on live signal, audio write)
  • TODO-F1LT3R... Bloop (audio write, sound generation from JavaScript, interactive multi-touch, etc.)

• Conclusions
  • The web is very excited about audio data. People are building Firefox with our patch, downloading our builds, and making demos already.