Media/WebAudio - Revision history

Mreavy: update summary

2015-09-23T15:10:14Z

update summary

← Older revision		Revision as of 15:10, 23 September 2015
Line 1:		Line 1:


	'''Web Audio Perf meta bug:''' https://bugzilla.mozilla.org/show_bug.cgi?id=~~webaudioperf~~ <br/>		'''Web Audio Perf Parity meta bug:''' https://bugzilla.mozilla.org/show_bug.cgi?id=webaudioperf_parity <br/>
	'''All open Web Audio bugs:''' http://mzl.la/1HQZkmU <br/>		'''All open Web Audio bugs:''' http://mzl.la/1HQZkmU <br/>
	'''NOTE:''' If a bug doesn't have a priority (P1 to P5), we are not planning to fix it by the end of Q3. If anyone thinks it needs to be fixed, please ping padenot and/or mreavy.<br/>		'''NOTE:''' If a bug doesn't have a priority (P1 to P5), we are not planning to fix it by the end of Q3. If anyone thinks it needs to be fixed, please ping padenot and/or mreavy.<br/>
	'''WebRTC and Web Audio plans coming out of Whistler:''' https://docs.google.com/document/d/1eJLEenV4T5R5uiattNXU4PIy9w7hUvysj3lvkZPJVIM/edit#heading=h.hit4o9naa62o -- This wiki will be updated to reflect the new info in this doc		'''WebRTC and Web Audio plans coming out of Whistler:''' https://docs.google.com/document/d/1eJLEenV4T5R5uiattNXU4PIy9w7hUvysj3lvkZPJVIM/edit#heading=h.hit4o9naa62o -- This wiki will be updated to reflect the new info in this doc
			----------------------
			== Web Audio API performance improvement, phase 2 (Q3) ==
			* Firefox 44 is our target release for "improved web audio performance" (as good or better than our competition)
			** Web Audio perf parity bug, everything higher than Rank=14, is the target: {{Bug\|1189514}}
			** Benchmarks: http://ouija.allizom.org/grafana/index.html#/dashboard/file/webaudio.json
			*** (Mac and Android benchmarks are having trouble running in automation. Investigating why with dminor.)


	----------------------		----------------------
	== Web Audio API performance improvement, phase 1 (Q2) ==		== Web Audio API performance improvement, phase 1 (Q2) ==
Line 30:		Line 38:
	Some comments about the expected difficulty and whether it's blocked is at the end of each point in [brackets].		Some comments about the expected difficulty and whether it's blocked is at the end of each point in [brackets].

	* Web Audio API suspend/resume
	** Gaia is hacking around the lack of this API for now, but we need it, there has been around 3-4 high-profile battery consumption regression because of that.<br />
	** The spec is now finished, so we are good to implement.
	** [Landed in Fx 40]
	* Possible audio output device switch fixes for Windows
	** We need to add some code to handle audio output device switching in windows/WASAPI. This is driver/windows version dependent, so some testing is needed beforehand.
	** [Landed in Fx 38]
	* Web Audio API Audio Worker		* Web Audio API Audio Worker
	** This is _very_ important to implement quickly and right when the spec is done.		** This is _very_ important to implement quickly and right when the spec is done.
Line 80:		Line 81:
	** We have an issue where on some platforms, the latency is too high (more than 1000/60ms between callbacks), and we are not able to paint the all the frame when they are due (we seem to have hacked around it for now). iirc roc has ideas on how to do that. It should also reduce the video latency.		** We have an issue where on some platforms, the latency is too high (more than 1000/60ms between callbacks), and we are not able to paint the all the frame when they are due (we seem to have hacked around it for now). iirc roc has ideas on how to do that. It should also reduce the video latency.
	** [no idea of difficulty yet -- thoughts welcome]		** [no idea of difficulty yet -- thoughts welcome]
	* MSG optimizations
	** When randomly profiling Firefox during WebRTC calls/gUM applications/Web Audio API applications, I see that the MSG is using too much CPU compared to what it should use. Since the MSG is pretty central to our overall real-time media story, it should logically be something that we optimize. For example, I think that investing around week in optimizing the MSG would be as interesting in CPU usage wins as adding support for an hardware AEC. I have notes somewhere on things we could do (somewhat low-hanging fruits), and the CPU usage gain (of course during a specific scenario, since different scenarios stress the MSG code in different ways).<br />
	** [not too hard]
	* Sandbox hardening		* Sandbox hardening
	** Audio input and output stream are somewhat syscall heavy, and do scary things with buffers all the time. Depending on the technique the security people are going to use/are using, we might need to change some code on our side. Because we have non-traditional requirements on latency, the classic solutions ("just use ipdl") might not work.		** Audio input and output stream are somewhat syscall heavy, and do scary things with buffers all the time. Depending on the technique the security people are going to use/are using, we might need to change some code on our side. Because we have non-traditional requirements on latency, the classic solutions ("just use ipdl") might not work.
	** We should look into mutexes and other synchronization primitives		** We should look into mutexes and other synchronization primitives
	** The page we used as a reference, but not sure if it's up to date: <[[Sandbox]]>		** The page we used as a reference, but not sure if it's up to date: <[[Sandbox]]>
			* MSG optimizations
			** Remove blocking from MSG
			** [Landed in Fx 43].<br />
			* Web Audio API suspend/resume
			** Gaia is hacking around the lack of this API for now, but we need it, there has been around 3-4 high-profile battery consumption regression because of that.<br />
			** The spec is now finished, so we are good to implement.
			** [Landed in Fx 40]
			* Possible audio output device switch fixes for Windows
			** We need to add some code to handle audio output device switching in windows/WASAPI. This is driver/windows version dependent, so some testing is needed beforehand.
			** [Landed in Fx 38]

Mreavy: quick update to bug status -- bigger changes to contents coming soon

2015-07-30T19:34:29Z

quick update to bug status -- bigger changes to contents coming soon

← Older revision		Revision as of 19:34, 30 July 2015
Line 18:		Line 18:
	* {{bug\|1169321}} - We need the benchmarks to run on android, there is a bug preventing us to see the results		* {{bug\|1169321}} - We need the benchmarks to run on android, there is a bug preventing us to see the results
	* {{bug\|1157768}} - This is the same as 926838, but for x86, optimizing the most expensive node of the Web Audio API.		* {{bug\|1157768}} - This is the same as 926838, but for x86, optimizing the most expensive node of the Web Audio API.

	~~'''Ongoing''':~~
	* {{bug\|1140450}} - This is still not closed, the team is exploring alternative ways to be even more efficient. (This is a P2; so it's not a "must-have" for our first phase improvements, but we still expect to land it in Fx 42.)		* {{bug\|1140450}} - This is still not closed, the team is exploring alternative ways to be even more efficient. (This is a P2; so it's not a "must-have" for our first phase improvements, but we still expect to land it in Fx 42.)

Mreavy: /* Web Audio API performance improvement, phase 1 (Q2) */ updated bug 1157768

2015-07-16T00:51:12Z

Web Audio API performance improvement, phase 1 (Q2): updated bug 1157768

← Older revision		Revision as of 00:51, 16 July 2015
Line 17:		Line 17:
	* {{bug\|1157137}} - ScriptProcessorNode would sometimes have a bug where internal latency would build up		* {{bug\|1157137}} - ScriptProcessorNode would sometimes have a bug where internal latency would build up
	* {{bug\|1169321}} - We need the benchmarks to run on android, there is a bug preventing us to see the results		* {{bug\|1169321}} - We need the benchmarks to run on android, there is a bug preventing us to see the results
			* {{bug\|1157768}} - This is the same as 926838, but for x86, optimizing the most expensive node of the Web Audio API.

	'''Ongoing''':		'''Ongoing''':
	* {{bug\|1157768}} - This is the same as 926838, but for x86, optimizing the most expensive node of the Web Audio API. (Close to done -- will land in Fx 42.)
	* {{bug\|1140450}} - This is still not closed, the team is exploring alternative ways to be even more efficient. (This is a P2; so it's not a "must-have" for our first phase improvements, but we still expect to land it in Fx 42.)		* {{bug\|1140450}} - This is still not closed, the team is exploring alternative ways to be even more efficient. (This is a P2; so it's not a "must-have" for our first phase improvements, but we still expect to land it in Fx 42.)

Mreavy: description of doc

2015-07-02T08:43:57Z

description of doc

← Older revision		Revision as of 08:43, 2 July 2015
Line 4:		Line 4:
	'''All open Web Audio bugs:''' http://mzl.la/1HQZkmU <br/>		'''All open Web Audio bugs:''' http://mzl.la/1HQZkmU <br/>
	'''NOTE:''' If a bug doesn't have a priority (P1 to P5), we are not planning to fix it by the end of Q3. If anyone thinks it needs to be fixed, please ping padenot and/or mreavy.<br/>		'''NOTE:''' If a bug doesn't have a priority (P1 to P5), we are not planning to fix it by the end of Q3. If anyone thinks it needs to be fixed, please ping padenot and/or mreavy.<br/>
	'''WebRTC and Web Audio plans coming out of Whistler:''' https://docs.google.com/document/d/1eJLEenV4T5R5uiattNXU4PIy9w7hUvysj3lvkZPJVIM/edit#heading=h.hit4o9naa62o		'''WebRTC and Web Audio plans coming out of Whistler:''' https://docs.google.com/document/d/1eJLEenV4T5R5uiattNXU4PIy9w7hUvysj3lvkZPJVIM/edit#heading=h.hit4o9naa62o -- This wiki will be updated to reflect the new info in this doc
	----------------------		----------------------
	== Web Audio API performance improvement, phase 1 (Q2) ==		== Web Audio API performance improvement, phase 1 (Q2) ==

Mreavy: added google doc link

2015-07-02T08:42:57Z

added google doc link

← Older revision		Revision as of 08:42, 2 July 2015
Line 4:		Line 4:
	'''All open Web Audio bugs:''' http://mzl.la/1HQZkmU <br/>		'''All open Web Audio bugs:''' http://mzl.la/1HQZkmU <br/>
	'''NOTE:''' If a bug doesn't have a priority (P1 to P5), we are not planning to fix it by the end of Q3. If anyone thinks it needs to be fixed, please ping padenot and/or mreavy.<br/>		'''NOTE:''' If a bug doesn't have a priority (P1 to P5), we are not planning to fix it by the end of Q3. If anyone thinks it needs to be fixed, please ping padenot and/or mreavy.<br/>
			'''WebRTC and Web Audio plans coming out of Whistler:''' https://docs.google.com/document/d/1eJLEenV4T5R5uiattNXU4PIy9w7hUvysj3lvkZPJVIM/edit#heading=h.hit4o9naa62o
	----------------------		----------------------
	== Web Audio API performance improvement, phase 1 (Q2) ==		== Web Audio API performance improvement, phase 1 (Q2) ==

Mreavy: /* Web Audio API performance improvement, Q2 */ update description

2015-07-02T08:39:43Z

Web Audio API performance improvement, Q2: update description

← Older revision		Revision as of 08:39, 2 July 2015
Line 5:		Line 5:
	'''NOTE:''' If a bug doesn't have a priority (P1 to P5), we are not planning to fix it by the end of Q3. If anyone thinks it needs to be fixed, please ping padenot and/or mreavy.<br/>		'''NOTE:''' If a bug doesn't have a priority (P1 to P5), we are not planning to fix it by the end of Q3. If anyone thinks it needs to be fixed, please ping padenot and/or mreavy.<br/>
	----------------------		----------------------
	== Web Audio API performance improvement, Q2 ==		== Web Audio API performance improvement, phase 1 (Q2) ==

	'''Done''':		'''Done''':
Line 19:		Line 19:
	'''Ongoing''':		'''Ongoing''':
	* {{bug\|1157768}} - This is the same as 926838, but for x86, optimizing the most expensive node of the Web Audio API. (Close to done -- will land in Fx 42.)		* {{bug\|1157768}} - This is the same as 926838, but for x86, optimizing the most expensive node of the Web Audio API. (Close to done -- will land in Fx 42.)
	* {{bug\|1140450}} - This is still not closed, the team is exploring alternative ways to be even more efficient. (This is a P2; so it's not a "must-have" for Q2, but we still expect to land it in Fx 42.)		* {{bug\|1140450}} - This is still not closed, the team is exploring alternative ways to be even more efficient. (This is a P2; so it's not a "must-have" for our first phase improvements, but we still expect to land it in Fx 42.)

Mreavy: /* Web Audio API performance improvement, Q2 */ update status of open bugs

2015-07-02T08:38:31Z

Web Audio API performance improvement, Q2: update status of open bugs

← Older revision		Revision as of 08:38, 2 July 2015
Line 18:		Line 18:

	'''Ongoing''':		'''Ongoing''':
	* {{bug\|1157768}} - This is the same as 926838, but for x86, optimizing the most expensive node of the Web Audio API. (Close to done -- ~~we needed to do extra build work to deal with licensing issues~~.)		* {{bug\|1157768}} - This is the same as 926838, but for x86, optimizing the most expensive node of the Web Audio API. (Close to done -- will land in Fx 42.)
	* {{bug\|1140450}} - This is still not closed, the team is exploring alternative ways to be even more efficient. (This is a P2; so it's not a "must-have" for Q2, but we still expect to land it in ~~time~~.)		* {{bug\|1140450}} - This is still not closed, the team is exploring alternative ways to be even more efficient. (This is a P2; so it's not a "must-have" for Q2, but we still expect to land it in Fx 42.)

Mreavy: /* Web Audio API performance improvement, Q2 */

2015-06-17T19:27:15Z

Web Audio API performance improvement, Q2

← Older revision		Revision as of 19:27, 17 June 2015
Line 15:		Line 15:
	* {{bug\|1127188}} - when a document goes away (for example when the a page is reloaded), AudioContext are now aggressively prevented from doing any further processing, improving the experience for developers that often reload their page, and saving resources when navigating away from pages with Web Audio API code		* {{bug\|1127188}} - when a document goes away (for example when the a page is reloaded), AudioContext are now aggressively prevented from doing any further processing, improving the experience for developers that often reload their page, and saving resources when navigating away from pages with Web Audio API code
	* {{bug\|1157137}} - ScriptProcessorNode would sometimes have a bug where internal latency would build up		* {{bug\|1157137}} - ScriptProcessorNode would sometimes have a bug where internal latency would build up
			* {{bug\|1169321}} - We need the benchmarks to run on android, there is a bug preventing us to see the results

	'''Ongoing''':		'''Ongoing''':
	* {{bug\|1157768}} - This is the same as 926838, but for x86, optimizing the most expensive node of the Web Audio API.		* {{bug\|1157768}} - This is the same as 926838, but for x86, optimizing the most expensive node of the Web Audio API. (Close to done -- we needed to do extra build work to deal with licensing issues.)
	* {{bug\|1140450}} - This is still not closed, the team is exploring alternative ways to be even more efficient.		* {{bug\|1140450}} - This is still not closed, the team is exploring alternative ways to be even more efficient. (This is a P2; so it's not a "must-have" for Q2, but we still expect to land it in time.)
	* {{bug\|1169321}} - ~~We need the benchmarks to run on android~~, ~~there is a bug preventing us~~ to ~~see the results~~


Line 25:		Line 26:

	----------------------		----------------------

	== 2015 Web Audio ({output,input},MSG) Development Plan ==		== 2015 Web Audio ({output,input},MSG) Development Plan ==

Padenot: linkify bugs

2015-05-28T16:30:47Z

linkify bugs

← Older revision		Revision as of 16:30, 28 May 2015
Line 8:		Line 8:

	'''Done''':		'''Done''':
			* {{bug\|1050645}} - We now have automatic benchmarks running on CI, on windows, mac and linux, where we benchmark our performance against Chrome. Unfortunately, this was put in place after some optimization landed, so we don't see those improvements. We need to have it running on mobile, it should work but does not.
	* bug 1050645 - We now have automatic benchmarks running on CI, on windows, mac and linux, where we benchmark our performance against Chrome. Unfortunately, this was put in place after some optimization landed, so we don't see those improvements. We need to have it running on mobile, it should work but does not.		* {{bug\|1140448}} - AudioParam is now very fast. This is very apparent in benchmarks
	* bug 1140448 - AudioParam is now very fast. This is very apparent in benchmarks		* {{bug\|926838}} - The FFT code is now at least twice as fast on ARM. Same, obvious in benchmark, or simply running any app that uses a ConvolverNode
	* bug 926838 - The FFT code is now at least twice as fast on ARM. Same, obvious in benchmark, or simply running any app that uses a ConvolverNode		* {{bug\|1140450}} - Resampling complexity has been lowered. This is a 50% decrease in complexity, and leads to wins accross the a number of benchmarks. Very obvious performance increase on the benchmarks. The students working on this are currently exploring ways of making it even faster. This is optimizing one of the most popular AudioNode of the Web Audio API.
	* bug 1140450 - Resampling complexity has been lowered. This is a 50% decrease in complexity, and leads to wins accross the a number of benchmarks. Very obvious performance increase on the benchmarks. The students working on this are currently exploring ways of making it even faster. This is optimizing one of the most popular AudioNode of the Web Audio API.
	* Various other optimizations of the internal MediaStreamGraph code, leading to less visible optimizations		* Various other optimizations of the internal MediaStreamGraph code, leading to less visible optimizations
	* bug 1127188 - when a document goes away (for example when the a page is reloaded), AudioContext are now aggressively prevented from doing any further processing, improving the experience for developers that often reload their page, and saving resources when navigating away from pages with Web Audio API code		* {{bug\|1127188}} - when a document goes away (for example when the a page is reloaded), AudioContext are now aggressively prevented from doing any further processing, improving the experience for developers that often reload their page, and saving resources when navigating away from pages with Web Audio API code
	* bug 1157137 - ScriptProcessorNode would sometimes have a bug where internal latency would build up		* {{bug\|1157137}} - ScriptProcessorNode would sometimes have a bug where internal latency would build up

	'''Ongoing''':		'''Ongoing''':
	* bug 1157768 - This is the same as 926838, but for x86, optimizing the most expensive node of the Web Audio API.		* {{bug\|1157768}} - This is the same as 926838, but for x86, optimizing the most expensive node of the Web Audio API.
	* bug 1140450 - This is still not closed, the team is exploring alternative ways to be even more efficient.		* {{bug\|1140450}} - This is still not closed, the team is exploring alternative ways to be even more efficient.
	* bug 1169321 - We need the benchmarks to run on android, there is a bug preventing us to see the results		* {{bug\|1169321}} - We need the benchmarks to run on android, there is a bug preventing us to see the results

Padenot: list down perf work for q2

2015-05-28T16:28:02Z

list down perf work for q2

@@ Line 1: / Line 1: @@
 '''Web Audio Perf meta bug:''' https://bugzilla.mozilla.org/show_bug.cgi?id=webaudioperf <br/>
 '''All open Web Audio bugs:''' http://mzl.la/1HQZkmU <br/>
 '''NOTE:''' If a bug doesn't have a priority (P1 to P5), we are not planning to fix it by the end of Q3.  If anyone thinks it needs to be fixed, please ping padenot and/or mreavy.<br/>
 ----------------------
-'''2015 Web Audio ({output,input},MSG) Development Plan''' - Some comments about the expected difficulty and whether it's blocked is at the end of each point in [brackets].
+== Web Audio API performance improvement, Q2 ==
 *  Web Audio API suspend/resume
 ** Gaia is hacking around the lack of this API for now, but we need it, there has been around 3-4 high-profile battery consumption regression because of that.<br />

← Older revision		Revision as of 19:34, 30 July 2015
Line 18:		Line 18:
	* {{bug\|1169321}} - We need the benchmarks to run on android, there is a bug preventing us to see the results		* {{bug\|1169321}} - We need the benchmarks to run on android, there is a bug preventing us to see the results
	* {{bug\|1157768}} - This is the same as 926838, but for x86, optimizing the most expensive node of the Web Audio API.		* {{bug\|1157768}} - This is the same as 926838, but for x86, optimizing the most expensive node of the Web Audio API.

	~~'''Ongoing''':~~
	* {{bug\|1140450}} - This is still not closed, the team is exploring alternative ways to be even more efficient. (This is a P2; so it's not a "must-have" for our first phase improvements, but we still expect to land it in Fx 42.)		* {{bug\|1140450}} - This is still not closed, the team is exploring alternative ways to be even more efficient. (This is a P2; so it's not a "must-have" for our first phase improvements, but we still expect to land it in Fx 42.)

← Older revision		Revision as of 00:51, 16 July 2015
Line 17:		Line 17:
	* {{bug\|1157137}} - ScriptProcessorNode would sometimes have a bug where internal latency would build up		* {{bug\|1157137}} - ScriptProcessorNode would sometimes have a bug where internal latency would build up
	* {{bug\|1169321}} - We need the benchmarks to run on android, there is a bug preventing us to see the results		* {{bug\|1169321}} - We need the benchmarks to run on android, there is a bug preventing us to see the results
			* {{bug\|1157768}} - This is the same as 926838, but for x86, optimizing the most expensive node of the Web Audio API.

	'''Ongoing''':		'''Ongoing''':
	* {{bug\|1157768}} - This is the same as 926838, but for x86, optimizing the most expensive node of the Web Audio API. (Close to done -- will land in Fx 42.)
	* {{bug\|1140450}} - This is still not closed, the team is exploring alternative ways to be even more efficient. (This is a P2; so it's not a "must-have" for our first phase improvements, but we still expect to land it in Fx 42.)		* {{bug\|1140450}} - This is still not closed, the team is exploring alternative ways to be even more efficient. (This is a P2; so it's not a "must-have" for our first phase improvements, but we still expect to land it in Fx 42.)