Accessibility/Mobile/ScreenReader

Introduction

Firefox OS allows blind users to operate a device with speech assistance. This function is called a "screen reader", it aids the user by allowing them to explore the screen with speech synthesis and activate items via special gestures. Also see: Firefox OS Screen Reader Presentation

Activation

Quick Toggle

A blind user who has a Firefox OS device does not require any sighted assistance for activating the screen reader. Press volume up and then volume down 3 times in sequence and you will hear the first audible feedback telling you to do the same volume buttons sequence one more time to activate the screen reader. You will hear "screen reader started", and you can now continue using your device with a screen reader. To toggle off the screen reader the same volume sequence applies.

A video demonstrating quick toggle

Toggle via Settings

Another way to start and stop the screen reader is via the Settings app, in the Accessibility section.

A video demonstrating activating screen reader via settings

What to do if you have 1.3 (1.4 flatfish) device

Usage

The screen reader allows you to explore and touch anything on the screen. Anything that is under your finger will be spoken but not activated automatically. This is a good way to get an idea of the user interface.

To read all the items that are on the screen in sequential order swipe to the right to go forward and to the left to go backward. The active item is highlighted by a rectangle around it, while it is spoken.

Once you have reached the item that you want to activate, double tap anywhere on the screen. If you reached a range widget, you can swipe up or down to change its value.

A video demonstrating screen reader navigation

To speed up using the keyboard when entering text you do not need to double tap on the chosen key. Lifting a finger once you have reached the key by touch will input it. This also works for entering numbers.

A video demonstrating typing with the screen reader

When viewing a large document or application like a Home screen, you can scroll down or up to the next or previous screen by swiping up or down with two fingers.

A video demonstrating scrolling the home screen

To bring down the utility tray, that has past notifications and shortcut buttons for the most common settings, swipe down with two fingers starting from the top edge of the screen.

A video demonstrating how to get to the utility tray with the screen reader

To quickly switch between applications and web pages simply swipe left or right with two fingers starting from the edge of the screen.

A video demonstrating edge gestures with the screen reader

Implementation

Architecture

There are two parts of ScreenReader, the following figure shows the architecture:

The first part is AccessFu.jsm which is imported by shell.html, it is controller part of the ScreenReader system. It will listen to mozBroser-opened event from browser elements and inject content-script.js to them. Also, it listens to gestures and part of keyboard inputs then passes the action commands to target element. In the other hand, it deal with presenting the output results like drawing visual cursor and triggering the TTS(text-to-speech) engine.
The other part is the browser controller part, it contains modules like content-script.js, ContentControl.jsm and EventManager.jsm. Moreover, every browser elements have their own accessibility tree which is a subtree of DOM tree. Also there is a pivot with nsIAccessiblePivot type to indicate the currently focused element in the tree. The main task of browser controller part is to manage the element traverse in accessibility tree.

Move the visual cursor

There are two main steps to move the visual cursor. When receiving an action command, a browser element will traverse its own elements (intra-browser) or pass the movement action to other browser elements (inter-browse).

The following is the flow of screen reader execute an action command:

When browser receives an action command, it will check if the current pointing element is a browser element, if so, passes the comment to it via frame message manager to the child, so that it can reach the deepest browser elements. After the browser element checking, the browser element will try to move visual cursor inside itself; If it do can move, the browser element will move the visual cursor and do the browser element check mentioned above again, if can not move, it will pass the command back to its parent to seek the opportunity to move.

How does visual cursor move in the browser element?

As previously mentioned, there is an accessibility tree for every browser element. When browser element receives the action command, The ContentControl.jsm injected to the browser element will get access to the virtual cursor via Utils.jsm, and make the virtual cursor to traverse accessibility tree according the action command. If virtual cursor can move, it will send an AccEvent with element position information through EventManager.jsm to AccessFu.jsm to notice the focused element changed. For more details about accessibility, check the following link [Accessibility](https://developer.mozilla.org/en-US/docs/Web/Accessibility).

Core Modules and Components

AccessFu.jsm

AccessFu.jsm is imported by settings.js in shell.html. It is the mainly controller of ScreenReader. When a browser element is loaded to the system, AccessFu.jsm will inject it with a script "content-script.js ", which will initialize ContentControl.jsm and EventManager.jsm. Also, AccessFu.jsm captures user actions including gestures, key inputs and transfers them to action commands then sends it to target browser element for visual cursor moving. Moreover, it listens to visual cursor updating message from content. After receiving the cursor-moved event, it will rerender the visual cursor and send the chrome message to system.html for the voice reading.

ContentControl.jsm

ContentControl.jsm handles the action commands which will change nsIAccessiblePivot's target element, gets the virtual pivot interface nsIAccessiblePivot via Utils.jsm and makes it execute corresponding action.

Utils.jsm

Utils.jsm supports ScreenReader Javascript modules the entry points to the components like Accessible elements, Message Managers and Window elements.

nsIAccessiblePivot

Every nsIDocAccessible contains a nsIAccessiblePivot pointing to an Accessible element which indicates targeted element in this Accessibility tree. nsIAccessiblePivot can traverse Accessibility tree by its member functions(e.g, moveNext/movePrev/moveToPoint). After its traversal functions, it will fire an AccEvent for announcing status updating.

EventManager.jsm

EventManager.jsm listens to any event that will update the visual cursor's position like AccEvent for nsIAccessiblePivot position updating or the veiwport change event like scroll, wheel and resize. It will send the position visual cursor should updates to AccessFu.jsm.

Presentation.jsm

Presentation.jsm is an interface for all presenter classes. A presenter could be, for example, a speech output module, or a visual cursor indicator.