MediaStreamAPI - Revision history

Tschneidereit at 14:47, 3 June 2012

2012-06-03T14:47:37Z

← Older revision		Revision as of 14:47, 3 June 2012
Line 1:		Line 1:
	Moved!		Moved!

	The latest version of the draft is now at [~~http~~://hg.~~mozilla~~.org/~~users/rocallahan_mozilla.com~~/~~specs~~/raw-file/tip/~~StreamProcessing~~/StreamProcessing.html].		The latest version of the draft is now at [https://dvcs.w3.org/hg/audio/raw-file/tip/streams/StreamProcessing.html].

Roc at 05:25, 16 June 2011

2011-06-16T05:25:53Z

← Older revision		Revision as of 05:25, 16 June 2011
Line 1:		Line 1:
	Moved!		Moved!

	The latest version of the draft is now at [[http://hg.mozilla.org/users/rocallahan_mozilla.com/specs/raw-file/tip/StreamProcessing/StreamProcessing.html]].		The latest version of the draft is now at [http://hg.mozilla.org/users/rocallahan_mozilla.com/specs/raw-file/tip/StreamProcessing/StreamProcessing.html].

Roc: Replaced content with "Moved! The latest version of the draft is now at http://hg.mozilla.org/users/rocallahan_mozilla.com/specs/raw-file/tip/StreamProcessing/StreamProcessing.html."

2011-06-16T05:25:19Z

Replaced content with "Moved! The latest version of the draft is now at http://hg.mozilla.org/users/rocallahan_mozilla.com/specs/raw-file/tip/StreamProcessing/StreamProcessing.html."

Show changes

Roc: /* Scenarios */

2011-05-23T21:36:04Z

Scenarios

Roc: /* Scenarios */

2011-05-23T11:43:16Z

Scenarios

← Older revision		Revision as of 11:43, 23 May 2011
Line 29:		Line 29:
	10) Trigger a sound sample to be played through the effects graph ASAP but without causing any blocking		10) Trigger a sound sample to be played through the effects graph ASAP but without causing any blocking

	11) ~~Synchronized MIDI + Audio capture~~		11) Trigger a sound sample to be played through the effects graph at a given time

	12) ~~Synchronized MIDI + Audio playback~~ (~~Would that just work if streams could contain MIDI data?~~)		12) Capture video from a camera and analyze it (e.g. face recognition)

	13) Capture video ~~from~~ a ~~camera~~ and ~~analyze it~~ (e.g. ~~face recognition~~)		13) Capture video, record it to a file and upload the file (e.g. Youtube)

	14) Capture video, record it ~~to a file~~ and upload ~~the file~~ (e.g. ~~Youtube~~)		14) Capture video from a canvas element, record it and upload (e.g. Screencast/"Webcast" or composite multiple video sources with effects into a single canvas then record)

	15) ~~Capture video from a canvas element, record it and upload~~ (~~e.g. Screencast/"Webcast" or composite multiple video sources with effects into a single canvas then record~~)		15) Synchronized MIDI + Audio capture

			16) Synchronized MIDI + Audio playback (Would that just work if streams could contain MIDI data?)

	=== Straw-man Proposal ===		=== Straw-man Proposal ===

Roc: /* Examples */

2011-05-23T11:42:26Z

Examples

← Older revision		Revision as of 11:42, 23 May 2011
Line 348:		Line 348:
	</script>		</script>

	13) Capture video from a camera and analyze it (e.g. face recognition)		11) Trigger a sound sample to be played through the effects graph in five seconds

			<script>
			var effectsMixer = ...;
			var audio = new Audio(...);
			function triggerSound() {
			var audio = new Audio(...);
			var stream = audio.captureStream();
			stream.waitForUse = true;
			audio.play();
			effectsMixer.addStream(stream, window.currentTime + 5);
			stream.onended = function() { effectsMixer.removeStream(stream); }
			}
			</script>

			12) Capture video from a camera and analyze it (e.g. face recognition)

	<script>		<script>
Line 357:		Line 372:
	</script>		</script>

	14) Capture video, record it to a file and upload the file (e.g. Youtube)		13) Capture video, record it to a file and upload the file (e.g. Youtube)

	<script>		<script>
Line 375:		Line 390:
	</script>		</script>

	15) Capture video from a canvas, record it to a file then upload		14) Capture video from a canvas, record it to a file then upload

	<canvas width="640" height="480" id="c"></canvas>		<canvas width="640" height="480" id="c"></canvas>
	<script>		<script>
	var canvas = document.getElementById("c");		var canvas = document.getElementById("c");
	var streamRecorder = canvas.~~createStream()~~.record();		var streamRecorder = canvas.stream.record();
	function stopRecording() {		function stopRecording() {
	streamRecorder.getRecordedData(gotData);		streamRecorder.getRecordedData(gotData);

Roc: /* Canvas Recording */

2011-05-23T11:37:32Z

Canvas Recording

← Older revision		Revision as of 11:37, 23 May 2011
Line 190:		Line 190:

	interface HTMLCanvasElement {		interface HTMLCanvasElement {
	Stream ~~createStream()~~;		readonly attribute Stream stream;
	};		};

	'~~createStream~~' ~~produces~~ a stream containing the "live" contents of the canvas as video frames, and no audio.		'stream' is a stream containing the "live" contents of the canvas as video frames, and no audio.

	==== Examples ====		==== Examples ====

Roc: /* Media elements */

2011-05-23T11:35:04Z

Media elements

← Older revision		Revision as of 11:35, 23 May 2011
Line 61:		Line 61:

	interface HTMLMediaElement {		interface HTMLMediaElement {
	~~// Returns the stream of "what the element is playing" ---~~
	~~// whatever the element is currently playing, after its~~
	~~// volume and playbackrate are taken into account.~~
	~~// While the element is not playing (e.g. because it's paused,~~
	~~// seeking, or buffering), the stream is blocked. When the element~~
	~~// is in the ended state, the stream is in the ended state.~~
	~~// When something else causes this stream to be blocked,~~
	~~// we block the output of the media element.~~
	readonly attribute Stream stream;		readonly attribute Stream stream;

Line 74:		Line 66:
	Stream captureStream();		Stream captureStream();

	~~// When set, do not produce direct audio output. Audio output~~
	~~// is still sent to 'stream'.~~
	// This attribute is NOT reflected into the DOM. It's initially false.		// This attribute is NOT reflected into the DOM. It's initially false.
	attribute boolean captureAudio;		attribute boolean captureAudio;

	~~// Can be set to a Stream. Blocked streams play silence and show the last video frame.~~
	attribute any src;		attribute any src;
	};		};

			'stream' returns the stream of "what the element is playing" --- whatever the element is currently playing, after its volume and playbackrate are taken into account. While the element is not playing (e.g. because it's paused, seeking, or buffering), the stream is blocked. When the element is in the ended state, the stream is in the ended state. When something else causes this stream to be blocked, we block the output of the media element.

			When 'captureAudio' is set, the element does not produce direct audio output. Audio output is still sent to 'stream'.

			'src' can be set to a Stream. Blocked streams play silence and show the last video frame.

	==== Stream extensions ====		==== Stream extensions ====

Roc: /* Stream mixing and processing */

2011-05-23T11:33:19Z

Stream mixing and processing

← Older revision		Revision as of 11:33, 23 May 2011
Line 120:		Line 120:
	};		};

	This object combines multiple streams with synchronization to create a new stream. While any input stream is blocked and not live, the StreamProcessor is blocked. While the StreamProcessor is blocked, all its input streams are forced to be blocked. (Note that this can cause other StreamProcessors using the same input stream(s) to block, etc.)		This object combines multiple streams with synchronization to create a new stream. While any input stream is blocked and not live, the StreamProcessor is blocked. While the StreamProcessor is blocked, all its input streams are forced to be blocked. (Note that this can cause other StreamProcessors using the same input stream(s) to block, etc.) A StreamProcessor is ended if all its inputs are ended (including if there are no inputs).

	'inputs' returns the current set of input streams. A stream can be used as multiple inputs to the same StreamProcessor, so 'inputs' can contain multiple references to the same stream.		'inputs' returns the current set of input streams. A stream can be used as multiple inputs to the same StreamProcessor, so 'inputs' can contain multiple references to the same stream.

Roc: /* Media elements */

2011-05-23T11:31:44Z

Media elements

← Older revision		Revision as of 11:31, 23 May 2011
Line 69:		Line 69:
	// When something else causes this stream to be blocked,		// When something else causes this stream to be blocked,
	// we block the output of the media element.		// we block the output of the media element.
	readonly attribute stream;		readonly attribute Stream stream;

	// Returns the same stream as 'stream', but also sets the captureAudio attribute.		// Returns the same stream as 'stream', but also sets the captureAudio attribute.

← Older revision		Revision as of 21:36, 23 May 2011
Line 9:		Line 9:
	These are higher-level than use-cases.		These are higher-level than use-cases.

	1) Play video with processing effect applied to the audio track		1) Play video with processing effect applied to the audio track (e.g. high-pass filter)

	2) Play video with processing effects mixing in out-of-band audio tracks (in sync)		2) Play video with processing effects mixing in out-of-band audio tracks (in sync) (e.g. mixing in an audio commentary with audio ducking)

	3) Capture microphone input and stream it out to a peer with a processing effect applied to the audio		3) Capture microphone input and stream it out to a peer with a processing effect applied to the audio (e.g. XBox 360 chat with voice distortion)

	4) Capture microphone input and visualize it as it is being streamed out to a peer and recorded		4) Capture microphone input and visualize it as it is being streamed out to a peer and recorded (e.g. Internet radio broadcast)

	5) Capture microphone input, visualize it, mix in another audio track and stream the result to a peer and record		5) Capture microphone input, visualize it, mix in another audio track and stream the result to a peer and record (e.g. Internet radio broadcast)

	6) Receive audio streams from peers, mix them with spatialization effects, and play		6) Receive audio streams from peers, mix them with spatialization effects, and play (e.g. live chat with spatial feature)

	7) Seamlessly chain from the end of one input stream to another		7) Seamlessly chain from the end of one input stream to another (e.g. playlists, audio/video editing)

	8) Seamlessly switch from one input stream to another, e.g. ~~to implement~~ adaptive streaming		8) Seamlessly switch from one input stream to another (e.g. adaptive streaming)

	9) Synthesize samples from JS data		9) Synthesize samples from JS data (e.g. game emulators or MIDI synthesizer)

	10) Trigger a sound sample to be played through the effects graph ASAP but without causing any blocking		10) Trigger a sound sample to be played through the effects graph ASAP but without causing any blocking (e.g. game sound effects)

	11) Trigger a sound sample to be played through the effects graph at a given time		11) Trigger a sound sample to be played through the effects graph at a given time (e.g. game sound effects)

	12) Capture video from a camera and analyze it (e.g. face recognition)		12) Capture video from a camera and analyze it (e.g. face recognition)

	13) Capture video, record it to a file and upload the file (e.g. Youtube)		13) Capture video and audio, record it to a file and upload the file (e.g. Youtube upload)

	14) Capture video from a canvas element, record it and upload (e.g. Screencast/"Webcast" or composite multiple video sources with effects into a single canvas then record)		14) Capture video from a canvas element, record it and upload (e.g. Screencast/"Webcast", or composite multiple video sources with effects into a single canvas then record)

	15) Synchronized MIDI + Audio capture		15) Synchronized MIDI + Audio capture

	16) Synchronized MIDI + Audio playback ~~(Would that just work if streams could contain MIDI data?)~~		16) Synchronized MIDI + Audio playback

	=== Straw-man Proposal ===		=== Straw-man Proposal ===