Changes

Speech recognition on any modern handsets is almost a standard feature, and the target of SpeechRTC is bring it to Firefox OS and other Mozilla products by creating a scalable and flexible services platform with focus on delivering great experience to users and empowerement of developers finally offering full support to Web Speech API and other tools.

SpeechRTC is already used on two published Firefox OS apps[1], and proved to run both online as offline at any device with 1.3, even unagi. So the fast track is first integrate it with FxOS as some OS level integrated apps, to build the foundations and then release the Web Speech API to developers on sequence.

On online mode, Audio is captured encoded on Opus through MediaRecorder and streamed through websockets to a nodejs application at the server that handle the connection with the decoder. It also have methods to change language model when necessary.

On offline mode, audio is captured as pcm from gUm and streamed to a web worker "thread", who treats it, and handle its processing with the decoder api ported to js by emscripten. As online, aslo has language model switch support. Despite this proved to work, the ideal approach is run the decode on a separate cpp process and communicate with it through IPC, to make it run even on phones with constrained cpu.

On online mode, the nodejs application who receive audio and grammars from peers is responsible to handle the connection with the voice server, who then decode opus to pcm and pass it to the decoder when dealing with recognition, or swicth the language model when requested to. Some argued with me on the past about also running the decode on node, but I decided that would be better to the project decouple it, since we may need to use different decoders and voice servers that may not run on javascript.

Third-party licensing is extremely costly (usual unit is millions) and lead to an unwanted dependency. Write a decoder from scratch is tough, and requires highly specialized and difficult to find engineers.

For LVSCR we can also consider Julius and benchmark it since he has great proved results.

We should also build scripts to automatically adapt the acoustic model per user with his own voice, to constantly auto-improve the service individually for him but also for the service as overall.

Some argued with me about privacy on online services. At the ideal screnario, actually online recognition is required only for LVSCR, while FSG can be handled offline if architected correctly. I think letting users to choose or not to let us use his voice to improve models is how other OSes handle this issue.

The same speech server can be designed to run both online as offline, letting the responsibility to handle transmission to the middleware that handle the connections with the front.

After we build boths online as offline backends on scalable way, we connect it with the already ready Web Speech API on Gecko, and release the api to developers and automatically starts to support every web app already developer with Web Speech API support that currently only runs on Chrome.