Firefox OS/1UA Multi-Screen

From MozillaWiki
Jump to: navigation, search

Introduction

Goal

Our mission is to support "1 UA multi-screen" rendering on FxOS. Try to imagine the following scenario, you have a computer set up with dual monitors, launch a Firefox browser on your primary display, launch another Firefox browser(or open a link in new window), and drag that new window to the secondary monitor. Our concept is similar but slightly different, where the baseline of this project is to allow a FxOS device to connect and render contents to any non-FxOS display, or device that doesn't support Firefox browser.

The picture shows a FxOS device(at the bottom) has connected to a dummy display(the Acer monitor) with HDMI cord, and to a tablet(at the lower right) with Wifi Display in the same time. Both Marketplace and Dialer are launched from the FxOS device and displayed to the external displays.

Requirements

  • Connect with external displays
    • For wireless connection, we have implemented the Wifi Display protocol. That is, any device capable of being a Miracast Sink, can be connected with a FxOS device.
    • For wired connection, we have implemented the HDMI output. This requires the support of hotplug detection in Hardware Composer.
  • Open multiple top-level windows on FxOS
    • Gecko supports opening multiple top-level windows on browser naturally, while the FxOS supports single top-level window only, thus we are going to bring this ability back for FxOS.

Advantages

  • Having multiple top-level windows means that we will have multiple DOM trees and layer trees exist in the system. The secondary window should have no difference regarding to functionality then the original one, which means that we should be able to launch WebApps, web pages, and multimedia on this secondary window.
  • With the case of "1 UA multi-screen" rendering, contents of remote screen are rendered by primary device, but in the resolution of its display.
  • Open arbitrary numbers of top-level windows and associate each of them to a display device is doable theoretically, but it depends on the support of hardware capability.

Terminology

Top-level window
A Window object without a parent, currently in FxOS, document loaded into the one and only top-level window is shell.html.
UA
In short of User Agent. A user agent is any software that retrieves, renders and facilitates end user interaction with Web content. -- From User Agent Accessibility Guidelines (UAAG) 2.0
Display device/Screen
Refer to devices for presentation in visual, e.g. monitor, TV, projector, tablet, ...etc.
Primary Display
Refer to the display of FxOS mobile device.
Secondary/Remote/External/Virtual Display
Refer to displays connected with primary device.
To be more specific, HWC has categorized the display types into follows: Primary the default built-in display, External the display connected through HDMI cord, and Virtual as displays which make composited output available within the system. In our case, the display connected via Wifi Display is a kind of virtual display.
Wifi Display, Miracast

User Stories

Architecture Designs

Overview

An overview of multi-screen on FxOS

This is an overview structure design from top to down.

Gecko

Multi screen of gecko-1.png
Multi screen of gecko-2.png
Multi screen of gecko-3.png

(needs update) DisplayDevice is the new introduced object, it simply represents information of a display, such as display's size, dpi, surface format, and more important, some native data such as FrameBufferSurface and ANativeWindow. The default built-in device could be a DisplayDevice with type of primary, and it would be the one and only primary DisplayDevice. Other connected/disconnected DisplayDevice can be added into/removed from our system through interfaces of AddDevice/RemoveDevice in GonkDisplay. Hardware Composer has defined displays connected via HDMI as "external", and displays which make composited output available within the system as "virtual", thus we share the same here.

Although it might be possible to connect multiple displays of the same type in the same time, we have limited the mapping of display type and display device to one to one at this point, any second DisplayDevice with type that already exist in the list asking for connection would result in an error. We'd like to keep the design simple at the early stage and it doesn't sound like a difficult extension anyway. As you can see from the graph above, GonkDisplay manages a list of DisplayDevices. Any component who requests data of a DisplayDevice from GonkDisplay should provide a display type.

Gaia

Since we are in the proof-of-concept stage, we need to cut down some user stories in order to merge them back to master first. The flowchart below is a temporary scenario that we intend to implement for now.

Multiscreen-gaia.png

The event structures are defined as follow:

{
  type: 'mozContentEvent',
  detail: {
    type: 'get-display-list'
  }
}

{
  type: 'mozChromeEvent',
  detail: {
    type: 'get-display-list-success',
    display: [
      {
        id: <the unique id which was passed to the associated system-remote in the initializing stage>,
        name: <the name we intend to use in the dialog>
      },
      {
        id: <id>,
        name: <name>
      },
      ...
    ]
  }
}

{
  type: 'mozChromeEvent',
  detail: {
    type: 'get-display-list-error',
    error: <reason>
  }
}

Tracking Bugs

Work in progress

Top blockers of bug 1116089 - [Meta] Support "1 UA multi-screen" on FxOS

Full Query
ID Summary Priority Status
1138269 [Meta] Launch a remote desktop to the connected external display -- RESOLVED
1138271 [Meta] Launch any apps or web pages to the connected external display -- RESOLVED
1138287 Refactor GonkDisplay/nsWindow/hwc to support multiple displays -- RESOLVED
1151936 GLContextEGL's EGLSurface override handling -- RESOLVED
1152135 Split EGLSurface buffer swap and HWC buffer swap -- RESOLVED
1152370 Use DisplaySurface as abstract display surface instead of FramebufferSurface -- RESOLVED
1156981 Split CompositorParent's scheduling of composition to CompositorScheduler -- RESOLVED
1157441 Remove hwc buffer swap from LayerManagerComposite::Render() -- RESOLVED
1157874 Remove screen related global variables from nsWindow.cpp -- RESOLVED
1161874 Add an entry point to trigger wifi display on B2G -- RESOLVED
1169176 Investigate the value of external display's dpi returned by hwc P3 NEW
1173258 Crash when terminating an app which is playbacking a video at an external display -- RESOLVED
1186000 Support screen mirroring to HDMI display on gonk -- RESOLVED
1186800 Hardware composer cannot get the correct display size of an external display. -- RESOLVED

14 Total; 1 Open (7.14%); 13 Resolved (92.86%); 0 Verified (0%);

Blockers of bug 1138287 - Refactor GonkDisplay/nsWindow/hwc to support multiple displays

Full Query
ID Summary Priority Status
1138288 Extend nsScreenManagerGonk to support multiple screens -- RESOLVED
1168701 Move nsScreenGonk/nsScreenManagerGonk to a separate file. -- RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);

Blockers of bug 1138269 - [Meta] Launch a remote desktop to the connected external display

Full Query
ID Summary Priority Status
1138290 Opens a new window at the connected external display -- RESOLVED
1142391 [Meta] Create shell-remote and system-remote for external displays -- RESOLVED
1156647 Add multi-screen item to the developer menu in settings -- RESOLVED

3 Total; 0 Open (0%); 3 Resolved (100%); 0 Verified (0%);

Blockers of bug 1138271 - [Meta] Launch any apps or web pages to the connected external display

Full Query
ID Summary Priority Status
1138269 [Meta] Launch a remote desktop to the connected external display -- RESOLVED
1156650 Show display dialog when launching app from homescreen -- RESOLVED
1157079 Add "open in external display" option to browser context menu -- RESOLVED
1157085 Redirect all "webapps-launch" events to system-remote when there's an external display -- RESOLVED
1163914 Expose connected external display list to gaia system app -- RESOLVED

5 Total; 0 Open (0%); 5 Resolved (100%); 0 Verified (0%);


Future Plans