Paladin/Misc/Android Audio Notes

Here are some links from an IRC conversation about Android audio, and log of much of the conversation itself.

Log from channel # starting 2011-11-15 12:04:26 -0800

Link to a time in the conversation by visiting a URL like https://wiki.mozilla.org/Paladin/Misc/Android_Audio_Notes#12:00

	» automata joined the chat room.	12:20
	» cpearce joined the chat room.	12:34
	» automata left the chat room.	12:34
	» oneman left the chat room.	13:32
	» oneman joined the chat room.	13:49
	» sicking left the chat room.	15:08
	» sicking joined the chat room.	15:14
	» sicking left the chat room.	15:35
	» rillian_lime left the chat room.	15:35
	» rillian_lime joined the chat room.	15:35
	» sicking joined the chat room.	15:45
	» anant left the chat room.	16:49
	» sicking left the chat room.	16:55
jesup	derf: I'll probably be on anyways, but is there anything you need from me at the codec meeting?	17:05
dmose	jesup: is now a good time to chat about audio stuff?	17:11
jesup	dmose: not horrible; the 3-year-old is watching videos on the tablet. Dangerous things, those tablets	17:13
dmose	chuckles	17:14
dmose	we're trying to figure out how soon to give ours access to such things	17:14
jesup	Much easier for her to browse youtube. than a computer with a mouse. And with youtubve you have to watch	17:14
dmose	true	17:14
jesup	Good idea for a website: curated videos from youtube for kids	17:15
jesup	the 11-month-old is "helping" unload the dishwasher	17:16
jesup	Someone is making a ruggedized 7" (?) android tablet preloaded with stuff for kids - not out yet I think	17:16
jesup	Flash on sesamestreet.org doesn't always work well with touch....	17:17
jesup	so, audio?	17:17
dmose	good point on the sesamestreet	17:18
dmose	on the upside, nice to see adobe acknowledging that flash is on its way out	17:18
dmose	yeah, so you had mentioned last week after roc had filled me in on the current state of his audio work	17:19
jesup	Yeah, in around 5 years, or more	17:19
dmose	that you wanted to chat more	17:19
dmose	so i was just responding to that	17:19
dmose	i assumed/hoped you had some wisdom to impart :-)	17:20
jesup	Right. I wanted to touch base with you on webrtc audio and how it might interact with some of the things you're doing	17:20
dmose	ok	17:20
jesup	We talked a little about it at AllHands	17:20
dmose	yeah	17:20
dmose	so, in general, i think the first person shooter use case is a pretty obvious one	17:20
jesup	To lay the groundwork, what's the problem space you're dealing with/want to address?	17:21
jesup	(I think I know, but better to make sure)	17:21
dmose	primarily 3d gaming on the web	17:21
dmose	https://wiki.mozilla.org/Paladin may be helpful	17:21
dmose	and if you follow the (more…) link, that may be helpful too	17:22
jesup	ok. (I'd looked a little before)	17:23
jesup	So for audio, you have the typical in-game audio: background, music, and event-driven audio	17:24
dmose	that sounds right to me	17:24
jesup	And then possibly streaming or inter-player audio	17:24
dmose	yeah	17:24
jesup	I assume there's an assumption of multi-player, at least as an option	17:24
dmose	eventually, yes	17:25
dmose	that's not something we're thinking about much right now	17:25
dmose	but it's clearly something the market wants for these games	17:25
jesup	is the target "serious" heavy-duty 3d FPS type games, or casual-gamer (maybe 3d) games, or across the spectrum?	17:26
dmose	long-term, across the spectrum	17:27
dmose	but crawl, walk, run	17:27
dmose	and we're also trying to care significantly about mobile	17:27
jesup	gotcha.	17:27
jesup	Ah. That will be.... fun	17:27
dmose	so we're going to tend to pick our targets based on some combination of "what's most practical" * " most useful"	17:27
jesup	Did we talk about the sound issues on Android?	17:27
dmose	no we didn't	17:28
dmose	i'd love to know more	17:28
jesup	Well... it's bad. 1 sec	17:29
dmose	chuckles	17:29
dmose	good times!	17:29
	» lloyd\|mbp is now known as lloyd.	17:29
	» lloyd left the chat room.	17:29
jesup	http://mindtherobot.com/blog/555/android-audio-problems-hidden-limitations-and-opensl-es/	17:31
jesup	Google Android Audio Latency and you'll get a bunch of hits.	17:31
jesup	Or look at Andoird bug 3434	17:31
dmose	ah, that's super helpful	17:31
dmose	t	17:31
dmose	hanks	17:31
jesup	Basically, latency sucks	17:31
jesup	there are tricks to minimize it, but they don't always work, depending on many htings	17:32
kinetik	i hope they've improved things in ICS.	17:32
dmose	fun times	17:32
jesup	http://www.kvraudio.com/forum/printview.php?t=331703&start=0	17:33
jesup	"The only thing I recall is an article discussing ICS and the fact that there hadn't been any significant headway made. "	17:33
jesup	:-(	17:33
dmose	that's unfortunate	17:33
jesup	So, that's there. It's probably ok for games, thoiugh even there it requires work	17:36
jesup	Music apps it sucks for	17:36
dmose	good to know; i'm gonna go slap this on wiki page somewhere	17:36
jesup	Interactive stuff like webrtc - not good, though we can probably work, with trouble	17:36
dmose	that adds excitement	17:37
dmose	presumably chrome will have the same issues though	17:37
dmose	so perhaps that'll help motivate lower layer fixes	17:37
jesup	There are tricks to improve it, especially if you can pre-load and prepare your sounds (games often do)	17:37
dmose	yeah, we're going to need an asset-loading story	17:37
dmose	since games actually real-time apps	17:37
jesup	chrome may not, since it's the kernel drivers and flingers that are the issue	17:38
dmose	can you unpack that a bit?	17:38
jesup	chrome's kernel and androids are different; and I think the audio "flinger" layer is also. Some of the problems devolve from the interfaces for Java/Dalvik I understand.	17:39
jesup	I'm no expert on this, but I've taken note iof it since my old company had been planning to build Android-based standalone videophones	17:39
dmose	i don't mean chromeos	17:39
dmose	i mean chrome-the-browser	17:39
dmose	which is supposed to replace "browser" on android at some point	17:40
jesup	Aha. Again, it's the kernel/flinger, not the browser that's the problem	17:40
kinetik	(some stuff on android latency here: http://www.youtube.com/watch?v=gfiYUL2exT8#t=40m32s)	17:40
dmose	right, but my point being that if google cares about webrtc in their browser on android	17:40
jesup	Some of the hacks used by app developers for low-latency audio may be tough for a browser to do	17:40
dmose	that may motivate them to prioritize fixing the kernel	17:40
dmose	then again, maybe not :-)	17:41
jesup	Yes. But they're in no rush for ebrtc on android last I knew	17:41
dmose	kinetik: ah, thanks!	17:41
jesup	kinetik: thanks!	17:41
jesup	We spent a while discussing it at AllHands	17:41
dmose	ah well	17:41
dmose	too bad	17:41
jesup	kinetik: any revelations since then?	17:41
jesup	dmose: kinetik is another good person to talk to about this	17:42
jesup	(audio in general, low-latency)	17:42
dmose	good to know! i just started following the "areweplayingyet" tracking bug	17:42
kinetik	jesup: not so far, i'm planning to start working on libcubeb backend for android shortly	17:43
kinetik	and waiting for ICS to see what might've changed, latency wise	17:43
jesup	As for Webrtc and gaming: one thing that may help you a bunch is the low-latency congestion control we're doing, at least for media streams.	17:43
jesup	We're going to try to work it into the data streams too	17:43
dmose	are data streams == PeerConnection?	17:44
rillian_lime	they're a thing hanging off peerconnection objects	17:44
jesup	Multiplayer can be enhanced with audio chat betwen players/teams/"break-rooms", etc	17:44
jesup	yes	17:44
jesup	direct p2p data channels (mostly, sometimes has to bounce off a relay)	17:45
dmose	that'd be sweet	17:45
jesup	We hope to support reliable streams and datagrams, unreliable datagrams, and possibly partly-reliable datagrams (limited retry)	17:45
dmose	wow	17:45
dmose	sounds like a lot of work	17:45
dmose	but awesome!	17:45
jesup	encrypted, congestion-controlled, with the app having input into who gets the bits	17:46
jesup	(prioritized)	17:46
jesup	We're leveraging SCTP -over-UDP (really over DTLS-over-UDP)	17:46
jesup	(I assume, IETF consensus call on that tomorrow)	17:47
jesup	My proposal	17:47
dmose	break a leg :-)	17:47
jesup	Most of that comes pretty much for free with SCTP	17:47
jesup	The main things we're adding/improving is congestion cotnrol	17:47
dmose	does the code we have already include SCTP?	17:48
jesup	That's somewhat of my baby	17:48
dmose	nice	17:48
jesup	No, but we're working with the SCTP authors of the BSD implementation; they've split it out into user-space (actually updated the split)	17:48
dmose	very cool!	17:48
dmose	just waits for the day when firefox ships with a loadable kernel module	17:49
jesup	Even ignoreing media, there's a lot you can leverage there	17:49
jesup	b2g - inthe kernel	17:49
dmose	fair point	17:49
jesup	:-)	17:49
jesup	Webrtc and media - enables chat, group chat, locality-based chat with fading, messages, push-to-talk, etc	17:50
dmose	yeah, there's a lot to like	17:50
dmose	the interesting thing about gaming is that pretty much wants to inhale and use every possible capability of a given platform	17:51
dmose	so we definitely have to pick our priorities :-)	17:51
jesup	A useful demo/framework for some devs might be a 3-d meeting/chat/presentation with local audio between players "near" each other and a PA system.	17:51
jesup	It's not actually a game, but shows how you can use the tech to make games, with owrking examples	17:52
dmose	yeah, intriguing	17:52
jesup	Maybe a "slideshow" as well, or enterpise bridge main display :-)	17:53
jesup	and it removes the need to make the "game" portion, plus no one dings you on the game not being good as a game	17:53
dmose	very intriguing	17:54
dmose	does ircloggr have a way to show stuff in past->present order?	17:54
dmose	i want to AIM some of the other paladin folks at the logs of this discussion	17:55
jesup	I have to run in a sec, but another thing it will probably be able to do is to leave invites ("OFFERS") open with the server, and let it "fork" the OFFER to a play who has moved close enough so your browser and theirs can iopen direct audio channels	17:55
jesup	It's logged - see title	17:55
jesup	A bit painful, though	17:55
dmose	yeah, i'm looking at those logs, and they're kinda hard to read because they're in backwards order	17:55
jesup	If you're in chatzilla you can save the channel view	17:56
dmose	yeah, i'll just do that	17:56
dmose	that forking invites thing is a little brain-warping	17:56
jesup	We wouldn't mind a good demo/example of what can be doen with it either	17:57
dmose	duly noted	17:57
jesup	Normally forking is when you call someone, and they are logged in via N different devices. The offer is forked to all of them	17:57
dmose	ah, that's helpful context	17:57
jesup	they all ring, and when one picks up the others are cancelled - but they don't have to be (think in a house wiht N phones)	17:58
jesup	Or an offer to a meta-address for the workgroup, leading to a "mesh" conference (no central mixer). Trickier, but doable	17:58
dmose	cool	17:59
jesup	If they can remain open, the game server can leverage it.	17:59
jesup	Gotta run	17:59
jesup	hoep that helped	17:59
dmose	ok, thanks for the brain dump	18:00
jesup	ta	18:00
dmose	that's super helpful	18:00

Paladin/Misc/Android Audio Notes

Navigation menu

Search