Platform/GFX/OffMainThreadCompositing: Difference between revisions

From MozillaWiki
< Platform‎ | GFX
Jump to navigation Jump to search
No edit summary
No edit summary
Line 1: Line 1:
=Current status=
=Current status=


Running on Android and Mac, currently fixing Android bugs.
Shipped on Android and FirefoxOS.
Usable on Linux and Mac (buggy and off by default).
Work in progress on Windows.


=Goals=
=Goals=


The main goal is to improve responsiveness. In the longer term, goals include:
The main goal is to improve responsiveness. this architecture has the following advantages over our on-the-main-thread approach:
* Reduce main thread contention
* Reduce main thread contention
* Asynchronous scrolling
* Asynchronous scrolling
Line 14: Line 16:
=Design=
=Design=


[[File:gfx-layers-architecture.png|thumb|650px|Gfx layers class diagram]]
[[File:LayersRefactoring.png|thumb|650px|Gfx layers class diagram]]


A new thread will handle composition. This thread will receive updates from the content thread, much as the chrome process receives updates from the content process in e10s. Indeed, as much as possible, the shadow layers machinery developed for e10s will be adapted for off-main-thread compositing. The design may need to diverge in some places from the e10s approach: for example, on Android, updates from the content thread to the compositing thread need to be asynchronous.  ('''note''': With the current code ... this might not be the case with the new frontend, and even in the current code, the cases that require async PLayers could be fixed at the widget/android level too.  We'll need to make a tradeoff.)
A new thread will handle composition. This thread will receive updates from the content thread, much as the chrome process receives updates from the content process in e10s. Indeed, as much as possible, the shadow layers machinery developed for e10s will be adapted for off-main-thread compositing. The design may need to diverge in some places from the e10s approach: for example, on Android, updates from the content thread to the compositing thread need to be asynchronous.  ('''note''': With the current code ... this might not be the case with the new frontend, and even in the current code, the cases that require async PLayers could be fixed at the widget/android level too.  We'll need to make a tradeoff.)
Line 20: Line 22:
=Plan=
=Plan=


Our highest priority platform for this project has been Android, but we want to extend this to all supported platforms: Mac, Windows (D3D9 + D3D10), Linux, B2G (Gonk) and software-only (BasicLayers; possibly, there is ongoing discussion as to whether this is the best use of resources to improve the experience for 'low-end' users).
Our highest priority platform for this project has been mobile platforms, but we want to extend this to all supported platforms: Mac, Windows (D3D9 + D3D10), Linux, and software-only (BasicLayers; possibly, there is ongoing discussion as to whether this is the best use of resources to improve the experience for 'low-end' users).


==Current status==
==Current status==
Line 26: Line 28:
* Only scrolling is asynchronous/off main thread. All other features require further work.
* Only scrolling is asynchronous/off main thread. All other features require further work.
=== Operating systems ===
=== Operating systems ===
* Android: Released. We continue to iron out OMTC bugs as the Android test base expands, but we are already quite high-quality.
* Android and B2G: Released. We continue to iron out OMTC bugs as the test base expands, but we are already quite high-quality.
* OS X: {{bug|756601}} Alpha quality. We began development of OMTC on this platform, but we haven't worked on it much since then. It pretty much works but is disabled by default.
* OS X: {{bug|756601}} Alpha quality. We began development of OMTC on this platform, but we haven't worked on it much since then. It pretty much works but is disabled by default.
* Linux: {{bug|722012}} In initial implementation stages. We're still sorting out basic issues with X and GL, as we don't support OMTC with BasicLayers yet.
* Linux: {{bug|722012}} In initial implementation stages. We're still sorting out basic issues with X and GL, as we don't support OMTC with BasicLayers yet.
* B2G (Gonk): {{bug|741837}} Alpha quality. The B2G team has been working on getting it enabled and working through the implementation.
* BasicLayers (software only): {{bug|703484}} In initial implementation stages. Marco Castelluccio, a volunteer contributor, is working through implementing off-main-thread BasicLayers, which also involves making our Cairo library thread-safe.
* BasicLayers (software only): {{bug|703484}} In initial implementation stages. Marco Castelluccio, a volunteer contributor, is working through implementing off-main-thread BasicLayers, which also involves making our Cairo library thread-safe.
* Windows:
* Windows:
** Direct3D 10: {{bug|756606}} Not started yet; has some fundamental problems with textures which require the layers refactoring.
** Direct3D 10: {{bug|756606}} Work in progress; has some fundamental problems with textures which require the layers refactoring.
** Direct3D 9: {{bug|756608}} Not started yet; but, in theory, should be fairly straightforward (there are already shadow layers implemented, but I don't think they are used anywhere).
** Direct3D 9: {{bug|756608}} Work in progress; but, in theory, should be fairly straightforward (there are already shadow layers implemented, but I don't think they are used anywhere).


==Future Work==
==Future Work==
===WebGL===
===WebGL===
Currently, we read WebGL content back from the graphics card, and then send it to the compositor. This is terrible for performance.
The WebGL streaming buffer project helped improve our WebGL compositing performances.
* Sharing WebGL backing textures between threads {{bug|728524}} will let us avoid reading back.
Even though there is stil some work in progress, most of the work has landed already.
* Double-buffering WebGL {{bug|716859}} will let us hand off a texture to the compositor, then go back to drawing to the WebGL context.
* Sharing WebGL backing textures between threads {{bug|728524}} lets us avoid reading back.
* Double-buffering WebGL {{bug|716859}} lets us hand off a texture to the compositor, then go back to drawing to the WebGL context.
* Not using glFinish() to synchronize between threads {{bug|697831}} (most likely using the ARB_sync extension) will let us hand those textures off faster.
* Not using glFinish() to synchronize between threads {{bug|697831}} (most likely using the ARB_sync extension) will let us hand those textures off faster.


'''Assignee''': Cody Brocious (Daeken)<br>
'''Assignee''': Jeff Giblert, Cody Brocious (Daeken)<br>
'''Status''': Being worked
'''Status''': Mostly done


===Video===
===Video===
Video decoding is already off-main-thread, but it needs a synchronization point between the Gecko thread and the video thread to publish decoded frames. We need to be able to publish frames directly to the compositor thread, never involving the main thread.
The video pipeline can now decode and composite video frames without touching the content thread (async-video). To improve performances and audio/video synchronization, there is some work in adding synchronization logic on the compositor side.  
* Asynchronous video composition is {{bug|706172}}
* Asynchronous video composition (already landed) is {{bug|706172}}


'''Assignee''': Nicolas Silva (nical)<br>
'''Assignee''': Nicolas Silva (nical)<br>
'''Status''': Blocked on Linux implementation of OMTC
'''Status''': async-video done, improvements planned.


===CSS Animations===
===CSS Animations===
Line 72: Line 74:
* {{bug|717872}}
* {{bug|717872}}


'''Assignee''': UNASSIGNED<br>
'''Assignee''': Joe Drew <br>
'''Status''': No work done
'''Status''': Work in progress


===Off-main-thread layers implementations===
===Off-main-thread layers implementations===
Line 79: Line 81:


==== Layers refactoring ====
==== Layers refactoring ====
The layers system has been heavily refactored to more easily facilitate OMTC on all platforms.


We are currently refactoring the layers system to more easily facilitate OMTC. The biggest changes are to the compositing side, and to how textures/buffers/handles are passed between the drawing and compositing components. The aim is for most OMTC code to be backend-independent. Compositing will be done by a Compositor class (rather than the ShadowLayerManager, which will still have responsibility for managing the shadow layer tree). There will be backend-specific compositor sub-classes, but the shadow layer tree and it's manager should be backend-independent. Buffers/textures will be handled by a TextureClient/TextureHost pair which will have backend-specific subclasses and will handle all inter-thread communication. The layer classes can then be agnostic about backend and IPC mechanism.
The biggest changes are to the compositing side, and to how textures/buffers/handles are passed between the drawing and compositing components. The aim is for most OMTC code to be backend-independent. Compositing is done by a Compositor class (rather than the ShadowLayerManager, which still have responsibility for managing the shadow layer tree). There are backend-specific compositor sub-classes, but the shadow layer tree and it's manager are backend-independent. Buffers/textures are handled by TextureClient/TextureHost pairs which have backend-specific subclasses and handle all inter-thread communication. The layer classes can then be agnostic about backend and IPC mechanism.
 
The [https://wiki.mozilla.org/Gecko:Overview#Graphics Gecko overview] wiki page contains good documentation about the current (post refactoring) layers system.


==== Android ====
==== Android ====


'''Assignee''': Benoit Girard/Ali Juma<br>
'''Assignee''': Benoit Girard/Ali Juma<br>
'''Status''': Beta quality
'''Status''': Shipped


==== Mac OS X ====
==== Mac OS X ====


'''Assignee''': UNASSIGNED<br>
'''Assignee''': UNASSIGNED<br>
'''Status''': Alpha quality
'''Status''': Beta quality


==== Linux ====
==== Linux ====


'''Assignee''': Nicolas Silva<br>
'''Assignee''': Nicolas Silva<br>
'''Status''': In development
'''Status''': In development, testable but buggy not optimized.


==== Direct3D 9 ====
==== Direct3D 9 ====


'''Assignee''': UNASSIGNED<br>
'''Assignee''': Nick Cameron<br>
'''Status''': No work done
'''Status''': In development


==== Direct3D 10 ====
==== Direct3D 10 ====


'''Assignee''': UNASSIGNED<br>
'''Assignee''': Bas Schouten<br>
'''Status''': No work done
'''Status''': Alpha quality


==== Basic (Software-only) ====
==== Basic (Software-only) ====
Line 120: Line 125:
=Development=
=Development=


Development is happening on mozilla-central and the graphics branch (layers refactoring).
Development is happening on mozilla-central.


=Test cases=
=Test cases=
http://people.mozilla.com/~ajuma/omtc/
http://people.mozilla.com/~ajuma/omtc/

Revision as of 15:05, 22 June 2013

Current status

Shipped on Android and FirefoxOS. Usable on Linux and Mac (buggy and off by default). Work in progress on Windows.

Goals

The main goal is to improve responsiveness. this architecture has the following advantages over our on-the-main-thread approach:

  • Reduce main thread contention
  • Asynchronous scrolling
  • Preventing tearing
  • Asynchronous video
  • Asynchronous CSS animation

Design

Gfx layers class diagram

A new thread will handle composition. This thread will receive updates from the content thread, much as the chrome process receives updates from the content process in e10s. Indeed, as much as possible, the shadow layers machinery developed for e10s will be adapted for off-main-thread compositing. The design may need to diverge in some places from the e10s approach: for example, on Android, updates from the content thread to the compositing thread need to be asynchronous. (note: With the current code ... this might not be the case with the new frontend, and even in the current code, the cases that require async PLayers could be fixed at the widget/android level too. We'll need to make a tradeoff.)

Plan

Our highest priority platform for this project has been mobile platforms, but we want to extend this to all supported platforms: Mac, Windows (D3D9 + D3D10), Linux, and software-only (BasicLayers; possibly, there is ongoing discussion as to whether this is the best use of resources to improve the experience for 'low-end' users).

Current status

Features

  • Only scrolling is asynchronous/off main thread. All other features require further work.

Operating systems

  • Android and B2G: Released. We continue to iron out OMTC bugs as the test base expands, but we are already quite high-quality.
  • OS X: bug 756601 Alpha quality. We began development of OMTC on this platform, but we haven't worked on it much since then. It pretty much works but is disabled by default.
  • Linux: bug 722012 In initial implementation stages. We're still sorting out basic issues with X and GL, as we don't support OMTC with BasicLayers yet.
  • BasicLayers (software only): bug 703484 In initial implementation stages. Marco Castelluccio, a volunteer contributor, is working through implementing off-main-thread BasicLayers, which also involves making our Cairo library thread-safe.
  • Windows:
    • Direct3D 10: bug 756606 Work in progress; has some fundamental problems with textures which require the layers refactoring.
    • Direct3D 9: bug 756608 Work in progress; but, in theory, should be fairly straightforward (there are already shadow layers implemented, but I don't think they are used anywhere).

Future Work

WebGL

The WebGL streaming buffer project helped improve our WebGL compositing performances. Even though there is stil some work in progress, most of the work has landed already.

  • Sharing WebGL backing textures between threads bug 728524 lets us avoid reading back.
  • Double-buffering WebGL bug 716859 lets us hand off a texture to the compositor, then go back to drawing to the WebGL context.
  • Not using glFinish() to synchronize between threads bug 697831 (most likely using the ARB_sync extension) will let us hand those textures off faster.

Assignee: Jeff Giblert, Cody Brocious (Daeken)
Status: Mostly done

Video

The video pipeline can now decode and composite video frames without touching the content thread (async-video). To improve performances and audio/video synchronization, there is some work in adding synchronization logic on the compositor side.

  • Asynchronous video composition (already landed) is bug 706172

Assignee: Nicolas Silva (nical)
Status: async-video done, improvements planned.

CSS Animations

Currently CSS animations are driven by the main thread. It'd be better if the style and layout code could assign attributes to relevant layers that would then be interpolated using specified functions on the compositor thread.

Assignee: David Zbarsky (dzbarsky) Status: Being worked

Plugins

Plugins are already drawn asynchronously (at least on OS X), similarly to how HTML5 video works. Publishing those plugin "frames" to the compositor is directly analagous.

Assignee: UNASSIGNED
Status: No work done

Animated images

Animated images are drawn using the main thread, but at least conceivably we could publish all the frames to the compositor and have the compositor know how to animate the images.

This could help us to animate throbbers, etc, but we don't currently have any layers support at all for animated images, so it will be more work for potentially less gain.

Assignee: Joe Drew
Status: Work in progress

Off-main-thread layers implementations

Features listed above won't work off the main thread unless we have a layers implementation that knows how to draw off the main thread.

Layers refactoring

The layers system has been heavily refactored to more easily facilitate OMTC on all platforms.

The biggest changes are to the compositing side, and to how textures/buffers/handles are passed between the drawing and compositing components. The aim is for most OMTC code to be backend-independent. Compositing is done by a Compositor class (rather than the ShadowLayerManager, which still have responsibility for managing the shadow layer tree). There are backend-specific compositor sub-classes, but the shadow layer tree and it's manager are backend-independent. Buffers/textures are handled by TextureClient/TextureHost pairs which have backend-specific subclasses and handle all inter-thread communication. The layer classes can then be agnostic about backend and IPC mechanism.

The Gecko overview wiki page contains good documentation about the current (post refactoring) layers system.

Android

Assignee: Benoit Girard/Ali Juma
Status: Shipped

Mac OS X

Assignee: UNASSIGNED
Status: Beta quality

Linux

Assignee: Nicolas Silva
Status: In development, testable but buggy not optimized.

Direct3D 9

Assignee: Nick Cameron
Status: In development

Direct3D 10

Assignee: Bas Schouten
Status: Alpha quality

Basic (Software-only)

This layers implementation is used when we have no hardware-accelerated layers backend; for example, when we're on a system with too-old drivers.

Assignee: Marco Castelluccio
Status: In development

Tracking bug

The tracking bug is bug 598873. Tree view: [1]

Development

Development is happening on mozilla-central.

Test cases

http://people.mozilla.com/~ajuma/omtc/

Retrieved from "https://wiki.mozilla.org/index.php?title=Platform/GFX/OffMainThreadCompositing&oldid=669307"