View source for User:Tedd/Speech Recognition

= Architecture =

== API limitation ==
Acess to the speech recognition is limited to certified apps (at the time of writing), using the 'Func' parameter inside the WebIDL constructor:
<pre>
 Func="SpeechRecognition::IsAuthorized"
</pre>

'''IsAuthorized''' is implemented inside the SpeechRecognition class:
<pre>
bool
SpeechRecognition::IsAuthorized(JSContext* aCx, JSObject* aGlobal)
{
  bool inCertifiedApp = IsInCertifiedApp(aCx, aGlobal);
  bool enableTests = Preferences::GetBool(TEST_PREFERENCE_ENABLE);
  bool enableRecognitionEnable = Preferences::GetBool(TEST_PREFERENCE_RECOGNITION_ENABLE);
  bool enableRecognitionForceEnable = Preferences::GetBool(TEST_PREFERENCE_RECOGNITION_FORCE_ENABLE);
  return (inCertifiedApp || enableRecognitionForceEnable || enableTests) && enableRecognitionEnable;
}
</pre>

== Relevant files in the Gecko source tree ==
pocketsphinx library code:
<pre>
./media/pocketsphinx
</pre>

Speech recognition WebIDL:
<pre>
./dom/webidl/SpeechRecognitionResultList.webidl
./dom/webidl/SpeechRecognitionResult.webidl
./dom/webidl/SpeechRecognitionAlternative.webidl
./dom/webidl/SpeechRecognition.webidl
./dom/webidl/SpeechRecognitionEvent.webidl
./dom/webidl/SpeechRecognitionError.webidl
</pre>

WebIDL implementation (C++):
<pre>
gecko/dom/media/webspeech/recognition/SpeechRecognitionResult.h
gecko/dom/media/webspeech/recognition/SpeechRecognition.h
gecko/dom/media/webspeech/recognition/SpeechRecognitionResultList.h
gecko/dom/media/webspeech/recognition/SpeechRecognitionResult.cpp
gecko/dom/media/webspeech/recognition/SpeechRecognitionAlternative.cpp
gecko/dom/media/webspeech/recognition/SpeechRecognitionResultList.cpp
gecko/dom/media/webspeech/recognition/SpeechRecognition.cpp
gecko/dom/media/webspeech/recognition/SpeechRecognitionAlternative.h
</pre>

Recognition service IDL:
<pre>
./dom/media/webspeech/recognition/nsISpeechRecognitionService.idl
</pre>

Implementation of the IDL interface (C++):
<pre>
./dom/media/webspeech/recognition/PocketSphinxSpeechRecognitionService.cpp
./dom/media/webspeech/recognition/PocketSphinxSpeechRecognitionService.h
</pre>

Events Implementation (C++):
<pre>
./dom/events/SpeechRecognitionError.h
./dom/events/SpeechRecognitionError.cpp
</pre>

== Association between components for speech recognition ==
Speech recognition functionality is available in JavaScript through WebIDL which is bound to a C++ class, which in return uses the nsISpeechRecognitionSerivce interface to communicate with the actual recognition service.
This section should illustrate how each component is associated with one another.

In JavaScript (given the right permissions) a 'SpeechRecognition' object can be created:
<pre>
var speech = new SpeechRecognition();
speech.start(stream);
</pre>

The invoked function is defined inside a WebIDL file ([https://dxr.mozilla.org/mozilla-central/rev/1f4cf75c894862cf3634d6014d8de9c807a054a7/dom/webidl/SpeechRecognition.webidl SpeechRecognition.webidl]):
<pre>
interface SpeechRecognition : EventTarget {
    ...
    void start(optional MediaStream stream);
    ...
}
</pre>

The SpeechRecognition interface and the start method, are itself implemented in a C++ class ([https://dxr.mozilla.org/mozilla-central/rev/1f4cf75c894862cf3634d6014d8de9c807a054a7/dom/media/webspeech/recognition/SpeechRecognition.cpp#714 SpeechRecognition::Start]):
<pre>
void
SpeechRecognition::Start(const Optional<NonNull<DOMMediaStream>>& aStream, ErrorResult& aRv)
{
  ...
  nsresult rv;
  rv = mRecognitionService->Initialize(this);
  ...
}
</pre>

'''mRecognitionService''' is an instance of the class that implements the [https://dxr.mozilla.org/mozilla-central/rev/1f4cf75c894862cf3634d6014d8de9c807a054a7/dom/media/webspeech/recognition/nsISpeechRecognitionService.idl nsISpeechRecognitionService] interface.
<pre>
interface nsISpeechRecognitionService : nsISupports {
    void initialize(in SpeechRecognitionWeakPtr aSpeechRecognition);
    ...
}
</pre>

In case of pocketpshinx, this class is defined in [https://dxr.mozilla.org/mozilla-central/rev/1f4cf75c894862cf3634d6014d8de9c807a054a7/dom/media/webspeech/recognition/PocketSphinxSpeechRecognitionService.h PocketSphinxSpeechRecognitionService.h] which implements the [https://dxr.mozilla.org/mozilla-central/rev/1f4cf75c894862cf3634d6014d8de9c807a054a7/dom/media/webspeech/recognition/PocketSphinxSpeechRecognitionService.cpp#197 Initialize] function as well:
<pre>
NS_IMETHODIMP
PocketSphinxSpeechRecognitionService::Initialize(
    WeakPtr<SpeechRecognition> aSpeechRecognition)
{
...
}
</pre>

This class uses the [https://dxr.mozilla.org/mozilla-central/source/media/pocketsphinx pocketsphinx] library for the speech recognition, an example of the library use is shown [https://dxr.mozilla.org/mozilla-central/rev/1f4cf75c894862cf3634d6014d8de9c807a054a7/dom/media/webspeech/recognition/PocketSphinxSpeechRecognitionService.cpp#108 here]:
<pre>
rv = ps_process_raw(mPs, &mAudiovector[0], mAudiovector.Length(), FALSE,
                    FALSE);

rv = ps_end_utt(mPs);
confidence = 0;
</pre>

== Library functions used in pocketsphinx service class ==
The following exported functions from the library are used inside the SpeechRecognition (pocketsphinx) service 
<pre>
int ps_start_utt(ps_decoder_t *ps);
int ps_process_raw(ps_decoder_t *ps, int16 const *data, size_t n_samples, int no_search, int full_utt);
int ps_end_utt(ps_decoder_t *ps);
char const *ps_get_hyp_final(ps_decoder_t *ps, int32 *out_is_final);
int32 ps_get_prob(ps_decoder_t *ps);
logmath_t *ps_get_logmath(ps_decoder_t *ps);
arg_t const *ps_args(void);
ps_decoder_t *ps_init(cmd_ln_t *config);
int ps_set_jsgf_string(ps_decoder_t *ps, const char *name, const char *jsgf_string);
int ps_set_search(ps_decoder_t *ps, const char *name);
</pre>