Video:Subtitles:Brainstorming

From MozillaWiki
Jump to: navigation, search

discussions are currently at: https://lists.mozilla.org/listinfo/community-web-standards

Vision

  • The potential: community-driven subtitling could make video global
  • The problem: fragmented & closed tools create fragmented communities
  • The solution: build open translation communities around the viewing experience
  • Outcomes:
    • more participation
    • more transcription,
    • more languages served,
    • better availability,
    • more user control over the content they want to view
  • Broad impact: the end of language barriers for popular video
  • more at https://wiki.mozilla.org/File:Subtitles_Proposal.pdf

Overview

These are some notes from a brainstorming session between Felipe Sanches, Dean Janson, Chris Hofmann, Rodrigo Silva and Aurelio Heckert at the 2009 FISL Conference in Porto Alegre, Brazil.

We got together to map out some ideas around Video Subtitles and their future on the web, with the emphasis on finding ways that will:

  • engage more contribution toward localization of video though subtitles
  • make the creation of subtitles easier
  • streamline and preserve openness in the content management of subtitles
  • allow easy discovery of subtitles where and when they are available,
  • and keep users in control of subtitle sources

There is plenty to do to understand and connect all these issues and we want to engage many others to help us learn existing systems and search for future improvements. The basic plan might look something like this:

  • look at prototypes and the landscape of how people and sites are doing subtitles.
  • figure out best practices and key pieces that are needed to help preserve open participation in the the creation and use of subtitles
  • figure out what a reference implementation for some client side stuff that would help users stay in control of where the subtitles come from if available.
  • figure out ways to turn this into specs for server and client and develop standards where needed to ensure interoperability

Some Areas and Issues to Look At

There are a lot of complex issues to look at. First step is to find a way to simplify and set priorities on the pieces that are most important to standardizes, but with any eye on a longer term plan that works toward addressing all the issues.

  • what is the landscape for current open subtitle systems?
  • how do they work?
  • could they become a reference for server and client components of an open specification?
  • what kind of centralized/distributed architectures are out there, and what kind are needed?
  • how could closed systems be back-fitted to provide more open solutions?
  • What meta data should be associated with each subtitle file
    • "timing data" to allow syncing of the same subtitle file across many transcoding speeds
    • <definition of canonical 'base' version>
    • for each transcoding speed
      • initial time shift
      • stretch factor
    • pointer to subtitle files available on the server(s) where the video is hosted
    • pointer to other subtitle hosts where the video provider want offer them
    • each video can have more than one subtitle in the same language (i.e. subs for alternative audio tracks, comments, closed-captioning, etc). For each of these there should be a list of available translations to other languages.
    • copyright/remixing authorization to provide other subtitles
    • how collections of video files (and subtitle files) might be related on a given topic - http://www.opencastproject.org/project/metadata


  • How could index's or directories be set up to identify videos that need translation contribution for individual videos across many languages? The goal here being to help localization contributors to find and prioritize contribution to subtitle work.
    • what tools are needed to enable more and better creation and sharing of subtitle files?
  • How could a distributed model be set up for hosting subtitle files and meta data to avoid lock-in and control issues over the content?
  • How does a server offer/direct/control the client to subtitle files it knows about for each of the videos it hosts? a website could display the subtitles, similar to subyo, except that it could work w/ subtitles from any open subs database (and do reverse browsing/lookup to find videos that are already subbed). The last idea is a wordpress plugin that enables open subtitle search/display on all videos embedded on a given blog (so if you're watching a video a subtitle button could sit below it). And of course this would all be in addition to Miro and other apps that supported the open subtitling system.


  • How does the client match up and use available information from the server and services to provide the most simple UI and selection of subtitle files given particular language preference(s)? - offer a FF plugin, so it searches across open databases for any compatible videos.
  • how does the client offer then 'gracefully degrade' when first, second, third language preferences for subtitles can't be honoured..
    • how might we support degradation paths like es-mx, es-ar, es-es, es-*
  • how could clients be provided with lists of subtitle services/subtitle files ala anti-malware blocklists, to provide greater user choice and control over selection of subtitles for any given video?
  • how could client components be set up to work across firefox, miro, and other apps that support an open subtitling system.
  • How could reputation systems/karma systems be established for:
    • repo hosting services,
    • translation contributor quality,
    • or best available translation for a given video?
  • should we implement a web-crawler to improve video/subtiles metadata and to feed the transcription/subtitling community of volunteers with content that requires effort priority?

Discussion

Follow up discussion and comment are encouraged on:

community-web-standards@lists.mozilla.org ( subscription info: https://lists.mozilla.org/listinfo/community-web-standards )

aka

http://groups.google.com/group/mozilla.community.web-standards/topics?lnk

Reading and References

Existing PCF Notes

Subtitle Strategy

We want to create a healthy ecosystem of subtitling communities users can connect to with Miro, with a browser plugin, and working towards an open standard that is widely implemented.


Open Subtitle Lookup Standard (OSL)

the goal is to create a way that clients can ask a website if it has subtitles for a particular URL or file. this could be a desktop client, browser plugin, web service, etc.

getmiro.com/opensub or opensub.org (website graphic - a submarine convertible watching a drive in movie underwater with subtitles)


WHAT WOULD BE IN AN OSL STANDARD? HERE'S SOME THOUGHTS:

client sends any / as many available of the following: 

   a. hash

   b. url

   c. title

   d. preferred language

   e. duration

   f. file type

server returns any / all of the following: 

   a. 'exact file match' for hash or URL

           1. title of match

           2. top-level url of the match, as a unique identifier

           3. list of urls to actual subtitle files for all available languages, specifying language, creator, file type for each

           4. hash of file

   b. list of potential non-exact results

           1. title of each result

           2. source of each result

           3. file type of each result

           4. url of each result

           5. whether result is a subtitle file, or a link to a video with hardcoded subs (tons of subtitling happens using hardcoded subs)


NOTES: 

       a. we can create an rss extension to specify the canonical source for subtitles

       b. should we create any report-back mechanism to quality control?

               1. each subtitle result has a URL    

               2. OSL could have an option for a client to report back if supposed match was positive or negative

               3. more elaborate feedback requires visiting URL

        c. the expectation is that clients could query multiple subtitle servers and have some heirarchy for which ones to trust more.

       d. it will be up to the servers to determine whether a match is exact or not and all the logic of matching, search, and lookup.


MIRO SUBTITLE FEATURES

PART 1

If Miro has subtitles for a video in the directory with the video, there should be a dropdown called 'Subtitles' that appears in the hover / underbar and in a menu. It should say 'Subtitles Off' as the first and default choice and have the subtitle files available under that.

[ Subtitles Off v ] [ Track 1 ] [ Track 2 ]

Miro should remember what the user selected and keep that selected, even if the user stops playing the video.

Do subtitle files have names for themselves? Or at least language names? Can we say something better than 'Track 1' and 'Track 2'?


PART 2 also see -- http://https://launchpad.net/subdownloader http://trac.opensubtitles.org/projects/opensubtitles/wiki/XMLRPC


Every video should have a 'get subtitles' button on the hover, on the 'more details', on the black bar. Clicking the button should do an automatic lookup on opensubtitles.org and other subtitle services for the file. Exact matches should download automatically into the folder for that video file. With each release of Miro, we include more subtitle lookup sites.

Boxee basically does this for opensubtitles.org. They give you a list of matches to pick from. http://www.xbmczone.com/plugin_details.asp?id=49

PART 3

There should be a preference panel that lets you specify which language to show subtitles in first or to only download certain languages.

There should be a way to find inexact matches.

There should be a way to add / remove / prioritize subtitle services.


Wordpress Plugin

USAGE: blogger installs WP Miro Subtitle extension and whenever an embedded video has subtitles available, the viewer can select them. Also, every video has a button that says "translate this video."

TRANSLATIONS: translations would be done via an interface in the Miro Guide. subtitles for videos would be exported to an open subtitle database. videos that had been translated could be shared with a Miro Guide permalink, so people without a firefox extension could easily watch/share it.

DISPLAYING SUBTITLES: We could do the same thing subyo does (eg. http://subyo.com/main/watch/bkEvy-9yVyQ/3.htm ). I believe they do it all with javascript: http://subyo.com/j/j.js

firefox extension implementation

PREFS / SETUP 

   a. set a list of lookups in order of preference

   b. set preferred language order

WATCHING SUBTITLES 

   c. when on a video page, lookup happens and top result in your language starts displaying with the video -- (how hard is it to sync up subtitle overlay with random flash players?  what about specific flash players?)  (what about something really hacky where you press play on the subtitles at the same time you play the video?)  (can we copy JS from here -- http://subyo.com/main/watch/bkEvy-9yVyQ/3.htm  )  (if you make it work for youtube that's 70%? of everything right there)

   d. drop down appears in subtitle area that allows you to pick from available results

   e. 'find subtitles' button lets you explore results more fully and choose?

   f. in Firefox and in Miro, make it easy for users to rate subtitles (like "perfect, imperfect sync, imperfect translation, not a match").  That way, we grow the list of exact matches, and flag subs for improvement.

CREATING SUBTITLES 

   a. there will be a button on the extension that says 'translate this video' and takes you to a miro guide page, with the video embedded.  the user can create a translation and submit it to opensubtitles.org

   b. creation interface will look like dotsub.  see screenshot:



_____________________________


What We Need to Build

I'm trying to simplify the concepts here into exactly what we need to build. (Holmes, does this capture what you are thinking and the key stuff from your proposal?)

A. Subtitle support in Miro. If subtitle files are available in the same directory as the video, Miro should recognize this and show a drop down in the UI to let people pick. There should also be a preference to automatically show subtitles in a certain language, when available. (#11495 ) SEE DETAILS BELOW.

B. Subtitle lookup API standard -- Open Subtitle Lookup (OSL) Like CDDB, we need a way that Miro (and other software) can look up a video in an online database based on hash or URL and then download matching subtitle files. (SEE NOTES BELOW)

C. Subtitle repository system. We should create web software that can serve as a subtitle repository and can implement the server side of point B. This will be both a demonstration implementation and something that communities can use to create repositories.

Dean: I did some research today on http://opensubtitles.org/ and here's what I've got. Advantages: they've developed a protocol for doing hash and title based searching for video subtitles — it's not totally clear how much traction they have, but they claim to have over 1/2 million subs. They're the closest thing (I can find) that does what we want to do. The protocol seems well documented and decently thought out — need developer opinion to validate this. Disadvantages: there is no open implementation of the server. They are the central core for the service. You must register your application (useragent) before their API will work w/ your app. They use the term open wantonly, and don't seem particularly interested in the social side of this stuff. Suggestion: we may want to consider creating a protocol that supersedes theirs. We could add all the things that we thing are important (decentralization, more openness), but make sure that we can still interoperate with their database/API. Here is their devFAQ: http://trac.opensubtitles.org/projects/opensubtitles/wiki/DevReadFirst

Dean: also, possibly of interest: http://uriplay.org/about/faq/

D. Repository list in Miro. Miro should be able to subscribe to subtitle repositories and communicate with them, as in B.

E. (OPTIONAL) We should make a Firefox plugin or an aggregator website that let's people watch an streaming flash videos while looking up and displaying subtitles, the same way that Miro will. This could start with something that just works on YouTube.

F. (OPTIONAL) We should use the same type of repository communication system to enable community search engines around links to particular content. Users can publish all the video links they know about to these sites and can send search requests to these sites when they are looking for video. (NPR: does that sound right? Are we insane for trying to create some kind of search engine system? HW: it's less a search engine system and more a linksharing system with search)

G. (OPTIONAL) A simple subtitle builder — so you can play a web video and subtitle it. Maybe something decent/open already exists? Maybe dotSub can serve this role (they do allow srt exporting)?

H. ("olmes"... OPTIONAL) The ability to attach other kinds of metadata to video files. Like a donation link, links to other sizes, formats of the video, links to other instances of the video online (in case it's unfairly taken down, for example).



__________________________

HOLMES' OSI PROPOSAL


Subtitles on everything for everyone Enabling community subtitling and translation for every video on the net.


1. Overview:

The potential: community-driven subtitling could make video global Inside a community of interest, collaborative subtitling is a proven model. An individual or small group uses available tools to create a time coded transcript in the original language, which others can then translate. The first step can be quite time consuming, but the second can happen quickly. Viewers can search sites that serve their community for subtitle files in a language they speak, and use a compatible video player (like VLC) to display subtitles alongside video. The tasks, roles, and tools are well understood, and the process is applicable to anything from films to daily programming to short video clips. Video is already much better than audio or text at crossing global divides, because many popular videos are language-neutral. Community subtitling can cover the rest, making online video a truly global medium.

The problem: fragmented tools create fragmented communities The problem with community subtitling is that the experience for viewers and contributors is highly fragmented; only the most motivated users overcome these barriers. The viewer's video player does not tell her anything about the availability of subtitles. Viewers must search the web, already knowing what they are looking for. When the viewer finds matching subtitles, she needs to pair them the video using a compatible video player. If the video is stored on the web in a proprietary player like Flash, this can be difficult or impossible.

Similar barriers face volunteer translators: when viewing a video, it is never clear where to find time-coded transcripts, or where to submit a new translation or correction. Different subtitle sites exist for different types of content and different languages. The end result of these hurdles is that subtitle communities only tackle the most sought-after videos, and even then, only serve the most enthusiastic and tech-capable viewers. New subtitling services are cropping up, but each exists as its own separate world: separate from the primary viewing context (e.g. Youtube) and separate from each other.

The solution: build open translation communities around the viewing experience By connecting subtitle search and submission directly to the viewing experience, we can create a subtitling community around nearly every video online. By maintaining an open database of videos and matching subtitles, we can maximize the benefit of users' efforts. When the viewer watches a video, an indicator will tell her when subtitles are available, and in what languages. Once selected, the subtitles will display immediately, just like on a DVD. While watching a video, the user is one click away from offering feedback and corrections or submitting a new translation. Even better, we can create this experience around the vast majority of videos online. Every video podcast, every downloaded video, and every video on major video sharing sites would be covered, first in the open-source aggregator Miro (phase 1) and then in the Firefox browser using a browser plugin (phase 2). Each piece can be open source and the pieces can communicate across open APIs; the system itself will be decentralized and open to collaboration.

Outcomes: more transcription, more languages served, better availability We're confident that by connecting collaborative subtitle creation with the viewing experience we can create orders of magnitude improvements on every relevant metric. First, any subtitle that is created will become far more available once viewers can link those subtitles to a video with a single click. Second, translation is much easier than transcription, it can be divided among multiple translators or even wiki'd, and once a translation exists in a common second language like English, the pool of translators widens. If we make subtitles available to more viewers and give translators good ways to collaborate, the pool of translators will increase proportionally. Finally, the increased exposure and demand for subtitles will encourage more transcription, both by communities and by video creators themselves. Creating time-coded transcriptions is the most specialized and time-consuming part of the process. However, tools and collaboration methods are improving. We believe an increase in audience will bring more energy into the process, stimulate further improvement of collaborative tools for transcribing, and encourage creators of the original video to make time coded transcripts available. Massive amounts of energy goes into the creation, criticism, and promotion of online video. The tools we propose will bring that energy into the collaborative subtitling process and make collaborative subtitles a ubiquitous feature of the online video space.

Broad impact: the end of language barriers for popular video The system we're proposing is the first step in an open, non-hierarchical structure that has the power to make any video available in any language where it has a motivated audience. Any video with a sufficiently large or particularly passionate audience will find volunteers ready to create a time-coded transcript and offer translations. This popularity threshold is unknown and will vary, but we do know that as translation communities grow and tools improve, the threshold will become progressively lower. Online video already exists in highly international, language-neutral spaces (because much popular content is language neutral). By giving viewers easy ways to find subtitles and participate in their creation, we can make language barriers porous.








2. How it works

The system consists of two parts: a subtitle fetcher built into a video player and a web-service. The web-service is a website where users maintain a database of videos and corresponding subtitles. The subtitle fetcher is a plugin (or feature) in a video player that checks this web service for available subtitles as the user watches video.

Subtitle fetcher in Miro (mockup). An indication appears below the video area when subtitles are available. When subtitles are available, the user can choose from a list of languages. Once selected, subtitles are downloaded and displayed in the video window. The user can visit the subtitle repository to search for subtitles and submit one, even if no subtitles are available. The "Find more" and "Submit new" links open a browser window and take the user to a subtitle repository website. Users can adjust how subtitles and notifications are displayed, and can choose to load subtitles automatically for preferred languages. The system will allow users to add multiple subtitle repositories, enabling a competitive ecosystem for subtitles with no centralized point of control (these repositories can, in turn, elect to share data with each other).

Subtitle fetcher Firefox addon on Youtube (mockup): The user installs a Firefox addon which uses a modified, subtitle-capable Youtube player to display each video. The Firefox addon will support Youtube first, followed by other popular video sharing sites. Sites can also choose to offer collaborative subtitling support directly to their users without requiring a Firefox addon (the addon is simply for ensuring compatibility with large sites like Youtube that might not offer such a service). The interface for choosing subtitles and participating is the same as in the Miro subtitle fetcher.



Repository Site: The subtitle fetcher searches a subtitle repository website for subtitles matching a given video. The fetcher searches based on a range of available information (including hash, URL, title, and duration). Sometimes, as in the case of a matching hash or URL, the search will find an exact match (that is, we will be almost 100% sure that the subtitles indeed match the video). In this case, the fetcher will automatically acquire all available subtitles from the site and display these subtitles in the language selector shown in the mockups above.

However, even if a search by hash or URL comes up negative, it's possible that matching subtitles still exist (the video may have been re-encoded or reposted to another site, for example). In this case, users can click "Find more" in the subtitle fetcher, and be taken to the repository site to see a list of possible matches. If the user finds a match, they can create a new correspondence between the identifiers in their search (e.g. hash or URL) and the existing subtitle file. When they return to watch the video, the subtitle fetcher will load these new subtitles. In this way, the community will build a list of links between subtitles for a given video and instances of that video across various sites and formats as the services progresses.

Clicking "submit new" in the subtitle fetcher will take users to a submission page on the repository site. If no subtitles for the video exist already in the system, the subtitle fetcher will automatically submit all available information on the video (title, hash, url, duration, etc) and users will be able to submit a new subtitle file. The submission page will feature links to tutorials on how to create time-coded transcripts using free tools. If subtitles exist already, the user will be able to submit a new translation. We will start with some simple capacity for moderating and improving translations (such as a wiki) and add more functionality for collaboration as we go.

3. Execution

(The order of this work might change depending on interest from the open source community or collaborating organizations / projects)

Phase 1: Subtitle fetcher in Miro, open APIs, open source alpha of repository site.

In the first phase we will create a working implementation of the system for all Miro viewers. Development of the subtitle fetcher in Miro is more straightforward (as compared to developing a Flash player / Firefox addon) so it makes sense as a first step. We will define the Open APIs through which the fetcher talks to the repository, and we will create and host a working repository site that includes all the functionality described above. Phase 1 will conclude with a working system for finding, viewing, and submitting subtitles in the hands of hundreds of thousands of Miro users.

Phase 2: Subtitle fetcher in Firefox and on the repository site. Beta of repository site.

In the second phase we will extend the viewing experience to the Firefox browser (for Youtube and non-flash video) and build the alpha repository site into a full-featured site for collaboration. We will develop a Firefox addon that will bring our interface for finding, viewing, and submitting subtitles to every video on Youtube, and to a wide range of non-Flash web video (including any video hosted using the <video> tag). At this point, any subtitled video will also be viewable on the repository site, though only Firefox users will be able to view subtitles while browsing other sites. We will add a set of features to the repository site to make it a full-featured tool for collaboration: users will be able to enter their abilities (familiarity with transcription tools, languages spoken) and interests (genres, etc) in their profiles, and request help from other users in creating subtitles and completing translations. The site will help users to divide tasks and to improve existing translations in response to feedback.

Phase 3: Support more sites / browsers / embedding methods / types of annotation. Build subtitle creation interface.

In the third phase we will extend the user experience to every popular way people watch video, and build-in a simple interface for subtitle creation. We will adapt our Firefox addon to more websites and embedding methods (it needs to be customized to the specific way each site stores or plays video). We will implement the subtitle fetcher in Internet Explorer and other video players. We can incorporate a simple system for adding subtitles into the viewing experience, both within Miro and in the browser. The specific goals for this phase are best defined as Phase 2 concludes, based on knowledge we gain in the process, feedback from users, and the nature of competitors and collaborators in the space.

4. Other uses and possibilities

This system for subtitling and translation would work just as well for storing and sharing any kind of video metadata, so it has many other potential uses. For example, the repository site could store comments, information about the creator, even information on how to donate to support the creator. All of this could be maintained both by the creator or by users using the same systems that we use for maintaining correspondences between subtitles and videos. As videos propagate to other sites and services, users can help link those new occurrences back to a single page for each piece of content. The repository site beta would be a great starting point for any kind of open voluntary donation system for internet works.

There is already work happening in Miro's open-source community to support time-specific comments and annotations. Two volunteers (from Sao Paolo, Brazil) are implementing support for Youtube-style annotations, as well as an open system (separate from Youtube) for storing annotations. They already have Youtube annotation support working for Miro Linux in testing versions. As we proceed with creating the subtitling infrastructure, we can pursue these other possibilities depending on interest from users, outside organizations, or volunteer developers.

5. Similar projects in this space

dotSUB - dotSUB (dotsub.com) was one of the first web 2.0 subtitling services. Unfortunately, in order to use the service you need to upload your video to the service. In simple practical terms, videos hosted on major sharing sites like Youtube, this presents a challenge for most users; it will be much less likely for subtitles to sprout up organically out of the needs and skills of the audience. In broad structural terms, the dependence on a closed service is politicall/socially problematic and unsustainable.

Opensubtitles.org - OpenSubtitles.org is an open catalog of user-submitted .srt subtitle files. The service is primarily oriented around mainstream film and cinema (the copyright implications of this are unclear) and depends on a complicated combination of tools. Some architectural decisions (for example, indexing by IMDB#) and its user interface make it non-ideal for the vast majority for user-created video.

Subdownloader - Subdownloader (www.subdownloader.net) is an open-source project that grew out of OpenSubtitles.org. It scans the user's computer for videos, and searches a database (currently Opensubtitles.org) for videos with matching hashes. When it finds a match, it automatically downloads subtitle files according to the user's preferred language. We have been in contact with the SubDownloader team, and there may be opportunities to work together on hash-based searching and on building the repository.

Youtube - Youtube now lets video publishers include captions/subtitles with their programming. However, there is no system for viewers submitting subtitles (except by downloading and re-uploading the same video) or for collaborative translation. This step by Youtube will be important for accessibility and useful for organizations that do subtitling and translation in-house. However, it does not create an open system for subtitling and translating videos (not even on Youtube, not to mention the internet at large). If many publishers begin using captions in Youtube, we can incorporate these captions into our system. And even if Youtube expands their service someday to allow for audience-submitted subtitles, an open system for all non-Youtube video will still be crucial.

Overstream.net / Subtitles.in / Babelbear.com - There is a growing set of services that let people create and view user-submitted subtitles on top of Youtube videos for playback within each site. None of these take a particularly open approach: subtitles are locked within the site. Also, without some kind of browser plugin, there is no way for a user watching a video on Youtube to know that it has been subtitled on one of these sites. We will work with each site to try to get them to open up their platform. Either way, we can make our browser plugin search and display videos from these sites as well (just by searching each site, and loading their own subtitle players).

Related ideas discussion

  • I think it needs to offer some easy shifting to the subtitles subtitle events ( items in time ). It is common to see movies that get started at different points in time. I wonder if a format can/should address these issues?
  • Support to remote content association. I think it is interesting concept to be something that any perosn can understand and write. However it is also important for a format to be flexible and to interoperate with other data formats in a distributed fashion, thus allowing developers to experiment advanced applications. One scenario to consider is the fact this is about acessibility. Consider a video with english subtitle and also acessible captioning mode ( the sort of extreme caption ). These are interesting cases because, in the future, one could associate some mechanisms in the scene ( like vibration mechanisms etc ) to provide additional function - aside from the subtitles to the eyes. Notice that if we go this direction it could offer something that has the today's need ( open subtitles ) but also would keep an eye on tomorrow's needs ( acessible understanding and richer time-based events ). It would eventually allow others to keep agregating information to videos so that the video frames eventually are not as important. I like to think o fthe SMIL side of subtitles, the cross-reference side of it and where richer remote objects can also be associated to the scene.
Retrieved from "https://wiki.mozilla.org/index.php?title=Video:Subtitles:Brainstorming&oldid=233250"