Firefox OS/1UA Multi-Screen

Introduction

Goal

Our mission is to support "1 UA multi-screen" rendering on FxOS. Try to imagine the following scenario, you have a computer set up with dual monitors, launch a Firefox browser on your primary display, launch another Firefox browser(or open a link in new window), and drag that new window to the secondary monitor. Our concept is similar but slightly different, where the baseline of this project is to allow a FxOS device to connect and render contents to any non-FxOS display, or device that doesn't support Firefox browser.

The picture shows a FxOS device(at the bottom) has connected to a dummy display(the Acer monitor) with HDMI cord, and to a tablet(at the lower right) with Wifi Display in the same time. Both Marketplace and Dialer are launched from the FxOS device and displayed to the external displays.

Requirements

• Connect with external displays
  • For wireless connection, we have implemented the Wifi Display protocol. That is, any device capable of being a Miracast Sink, can be connected with a FxOS device.
  • For wired connection, we have implemented the HDMI output. This requires the support of hotplug detection in Hardware Composer.
• Open multiple top-level windows on FxOS
  • Gecko supports opening multiple top-level windows on browser naturally, while the FxOS supports single top-level window only, thus we are going to bring this ability back for FxOS.

Advantages

• Having multiple top-level windows means that we will have multiple DOM trees and layer trees exist in the system. The secondary window should have no difference regarding to functionality then the original one, which means that we should be able to launch WebApps, web pages, and multimedia on this secondary window.
• With the case of "1 UA multi-screen" rendering, contents of remote screen are rendered by primary device, but in the resolution of its display.
• Open arbitrary numbers of top-level windows and associate each of them to a display device is doable theoretically, but it depends on the support of hardware capability.

Terminology

Top-level window

A Window object without a parent, currently in FxOS, document loaded into the one and only top-level window is shell.html.

UA

In short of User Agent. A user agent is any software that retrieves, renders and facilitates end user interaction with Web content. -- From User Agent Accessibility Guidelines (UAAG) 2.0

Display device/Screen

Refer to devices for presentation in visual, e.g. monitor, TV, projector, tablet, ...etc.

Primary Display

Refer to the display of FxOS mobile device.

Secondary/Remote/External/Virtual Display

Refer to displays connected with primary device.

To be more specific, HWC has categorized the display types into follows: Primary the default built-in display, External the display connected through HDMI cord, and Virtual as displays which make composited output available within the system. In our case, the display connected via Wifi Display is a kind of virtual display.

Wifi Display, Miracast

User Stories

Architecture Designs

Overview

An overview of multi-screen on FxOS

This is an overview structure design from top to down.

Gecko

(needs update) DisplayDevice is the new introduced object, it simply represents information of a display, such as display's size, dpi, surface format, and more important, some native data such as FrameBufferSurface and ANativeWindow. The default built-in device could be a DisplayDevice with type of primary, and it would be the one and only primary DisplayDevice. Other connected/disconnected DisplayDevice can be added into/removed from our system through interfaces of AddDevice/RemoveDevice in GonkDisplay. Hardware Composer has defined displays connected via HDMI as "external", and displays which make composited output available within the system as "virtual", thus we share the same here.

Although it might be possible to connect multiple displays of the same type in the same time, we have limited the mapping of display type and display device to one to one at this point, any second DisplayDevice with type that already exist in the list asking for connection would result in an error. We'd like to keep the design simple at the early stage and it doesn't sound like a difficult extension anyway. As you can see from the graph above, GonkDisplay manages a list of DisplayDevices. Any component who requests data of a DisplayDevice from GonkDisplay should provide a display type.

Gaia

Since we are in the proof-of-concept stage, we need to cut down some user stories in order to merge them back to master first. The flowchart below is a temporary scenario that we intend to implement for now.

The event structures are defined as follow:

{
  type: 'mozContentEvent',
  detail: {
    type: 'get-display-list'
  }
}

{
  type: 'mozChromeEvent',
  detail: {
    type: 'get-display-list-success',
    display: [
      {
        id: <the unique id which was passed to the associated system-remote in the initializing stage>,
        name: <the name we intend to use in the dialog>
      },
      {
        id: <id>,
        name: <name>
      },
      ...
    ]
  }
}

{
  type: 'mozChromeEvent',
  detail: {
    type: 'get-display-list-error',
    error: <reason>
  }
}

Tracking Bugs

Work in progress

Top blockers of bug 1116089 - [Meta] Support "1 UA multi-screen" on FxOS

14 Total; 0 Open (0%); 14 Resolved (100%); 0 Verified (0%);

Blockers of bug 1138287 - Refactor GonkDisplay/nsWindow/hwc to support multiple displays

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);

Blockers of bug 1138269 - [Meta] Launch a remote desktop to the connected external display

3 Total; 0 Open (0%); 3 Resolved (100%); 0 Verified (0%);

Blockers of bug 1138271 - [Meta] Launch any apps or web pages to the connected external display

5 Total; 0 Open (0%); 5 Resolved (100%); 0 Verified (0%);

Future Plans