Accessibility/Video a11y Study08: Difference between revisions

The three months are over during which the Mozilla Foundation provided me with a grant towards analysing the situation of accessibility for the HTML5 <video> and <audio> elements, particularly with a view towards Ogg support. This post provides a summary of my findings and recommendations on how to progress video a11y in Firefox, as well as a list of actual progress made already. One of the biggest achievements is that there is now a mailing list at Xiph for video a11y and that this community will continue working on these issues, if slowly.

The study took a broad view on what constitutes "accessibility" for audio and video including and beyond means of providing access to people with disabilities. The study analysed means of generally attaching textual information to audio and video, and enabling search engines with better access to these text representations. The requirements document is available [https://wiki.mozilla.org/Accessibility/Video_a11y_requirements in the Mozilla wiki].

== Results & Recommendations of the Video Accessibility study ==

First of all one has to recognize that some accessibility data is in text format (e.g. closed captions), and others is actually supplementary media data that accompanies the core video or audio content. Examples for such non-text accessibility data are open captions (i.e. captions that are burnt into the video data), bitmap captions (i.e. graphics files that are blended on top of the video data), or audio descriptions (i.e. descriptive spoken annotations that are aligned with pauses in the original audio track). Most non-text accessibility data actually has a textual representation: closed captions for example come as text that can easily be turned on or off by the video player. Also, textual audio descriptions can be rendered by a screen reader or through a braille device.

It needs to be understood that sign language is for deaf people often their first language, while transcribed spoken speech is often their first foreign language. Similarly, listening to natural speech in audio descriptions is a lot more relaxing than listening to screen readers or reading braille for blind people.  Also, there is already a large amount of audio and video accessibility data available that is not in textual format, e.g. the bitmaps used for captions on DVDs. It would be a shame to exclude such data from being used on the Web.

Considering these circumstances, it is really important to enable the association of non-text captions, sign language and audio annotations with video.

Existing sign language video or audio descriptions usually come as part of the video - either directly part of the given audio or video track by being burnt-in (e.g. picture-in-picture sign language video, or open captions), or as a separate track. QuickTime, mpeg2 or mpeg4 are container formats that are typically used to encapsulate such extra tracks with the original audio or video file. Ogg is capable of the same multi-track encapsulation and provides synchronisation between the tracks. The Ogg skeleton headers can further provide a clear indication of the available tracks inside an Ogg file, which can be used to enable media players to offer audio and video track selection to a user.

Also please note that we recommend development of a server-side dynamic content adaptation scheme that allows the browser to request - on behalf of his user - adequate accessibility tracks together with the content. This is described in more detail in section 6 below.

* CC: closed captions (for the deaf)

* TAD: textual audio descriptions (for the blind; to be used as braille or through TTS)

The ability to dynamically associate a time-aligned text file with an audio or video file at the moment of playback is very powerful. It has allowed the creation of a whole community of subtitling fans, the fansubbers, which provides accessibility to almost all movies and feature files.

   <itext category="SUB" lang="fr" src="translation_webservice/fr/caption.srt" style="valign: top;"></itext>

How this will actually work is as yet unclear. One approach is to render the out-of-band text files as HTML straight into the DOM of the current Web page. This raises security issues. Another approach is to render it into a kind of iframe. Also, SVG has a "text" element that serves a similar purpose, so the specification could be aligned with that.

Of all the formats that were analysed, DFXP is the most flexible format and is capable of producing multiple categories. It still needs to be checked, whether it would be possible for all of the given categories to be supported by DFXP, since DFXP is developed as an exchange format for subtitles and captions in particular. In any case, DFXP is not optimal for a Web-based time-aligned text format, since it redefines a lot of HTML, SMIL and CSS constructs, rather than re-using existing HTML, javascript and CSS. The re-definition was necessary because DFXP was developed as a generic format for time-aligned text, that needs to work in any situation, including outside the Web. However, for purposes of the Web (and for many Web-capable media players), reuse of HTML, CSS and javascript would lead to a format for which it would be much easier to provide implementations.

One concern that has been raised with TDHT is that HTML may be too comprehensive a format for the needs of time-aligned text. Time-aligned text is predominantly text that should be stylable by the Website that is using it. HTML however is pretty bad at separating data from styling, unless CSS is used exclusively. It may be better to create a format that is more simlar to RSS than to HTML in its simplicity.

TDHT and DFXP are current solutions for out-of-band time-aligned text. Other options for such comprehensive time-aligned text solutions need to be analysed and experimented with.

Ogg, QuickTime, Flash, mpeg4, and 3gpp are containers that are typically used to encapsulate such extra tracks with the original audio-visual file. All of these are capable of encapsulating 3GPP TimedText, which is a subpart of DFXP.

Just like out-of-band time-aligned text come in multiple formats, in-line text codecs do, too. This study motivated the Xiph community to define a framework for mapping any type of text codec into Ogg through the so-called [http://wiki.xiph.org/index.php/OggText OggText mapping]. A first implementation of this format exists for encapsulating srt files into Ogg.

In a Web framework (such as Firefox) it doesn't sense to support all possible in-line text codecs. Instead, it is useful to only support one comprehensive format, or at most a simple format (like srt) and a comprehensive format (like TDHT or Kate). Then, an existing format can be transcoded to one of these two formats for in-line encoding and Web delivery. For example, DFXP could be mapped to TDHT before encapsulation into Ogg.

When looking at what in-line time-aligned text codecs to support in Ogg, one should also look at what existing media players (outside the Web) are able to decode. There is actually support for decoding of Kate and CMML Ogg tracks in typical open source media players like mplayer or vlc. However, not much content has been produced other than test content that uses these formats to provide subtitles, captions, or other time-aligned text. The most uptake that either format has achieved is as an export format for Metavid: http://www.metavid.org/. Mostly, the solutions for using subtitles with Ogg has been to use srt files and have the media players synchronise them.

To make this available, the Web browser needs to have the ability to gain a full picture of the tracks available for a video or audio resource before being able to choose the ones requested through the user's preferences. The text tracks may come from the video or audio resource itself, or through "text" tracks inside the "video" tag. To gain information about the text tracks available inside an Ogg file, a browser can start downloading the first few bytes of the Ogg file, which contain the header data and therefore the description of the tracks available inside it. For Ogg, the skeleton track knows which text tracks are available. However, we need to add the category description into skeleton to allow per-category content selection.

During the study, some progress was already made towards the four areas of work, in particular towards the wecond and third.

The aim of the study was to "deliver a well thought-out recommendation for how to support the different types of accessibility needs for audio and video, including a specification of the actual file format(s) to use. At minimum, a Ogg solution for captions and subtitles is expected, and a means towards including sign language video tracks, audio annotations, transcripts, scripts, story boards, karaoke, metadata, and semantic annotations is proposed."

* for how to include subtitles into Web pages with a &lt;video> element,

* for how to encapsulate time-aligned text into Ogg, and