Platform/GFX/OffMainThreadCompositing: Difference between revisions

From MozillaWiki
< Platform‎ | GFX
Jump to navigation Jump to search
Line 30: Line 30:
=Plan=
=Plan=


Our highest priority platform for this project is Android. Since we expect that a significant portion of the desktop and mobile implementations will overlap, and since desktop development still proceeds more quickly than mobile development, we will actually develop the OS X implementation first, porting to Android as we go along.
Our highest priority platform for this project has been Android, but we want to extend this to all supported platforms: Mac, Windows (D3D9 + D3D10), Linux, B2G (Gonk) and software-only (BasicLayers).


Overview of our current plan:
==Current status==
=== Features ===
* Only scrolling is asynchronous/off main thread. All other features require further work.
=== Operating systems ===
* Android: Beta quality. We continue to iron out OMTC bugs as the Android test base expands, but we are already quite high-quality.
* OS X: {{bug|756601}} Alpha quality. We began development of OMTC on this platform, but we haven't worked on it much since then.
* Linux: {{bug|722012}} In initial implementation stages. We're still sorting out basic issues with X and GL, as we don't support OMTC with BasicLayers yet.
* B2G (Gonk): {{bug|741837}} Alpha quality. The B2G team has been working on getting it enabled and working through the implementation.
* BasicLayers (software only): {{bug|703484}} In initial implementation stages. Marco Castelluccio, a volunteer contributor, is working through implementing off-main-thread BasicLayers, which also involves making our Cairo library thread-safe.
* Windows:
** Direct3D 10: {{bug|756606}} Not started yet.
** Direct3D 9: {{bug|756608}} Not started yet.


* Desktop implementation (OS X)
==Future Work==
** Setup compositing thread. - Done
===WebGL===
** Setup ITC protocol, and pass messages. - Done
Currently, we read WebGL content back from the graphics card, and then send it to the compositor. This is terrible for performance.
** Make PLayers update async
* Sharing WebGL backing textures between threads {{bug|728524}} will let us avoid reading back.
** Setup Async Video
* Double-buffering WebGL {{bug|716859}} will let us hand off a texture to the compositor, then go back to drawing to the WebGL context.
** Get Cairo working in the new thread.
* Not using glFinish() to synchronize between threads {{bug|697831}} (most likely using the ARB_sync extension) will let us hand those textures off faster.


* Mobile implementation
'''Assignee''': Cody Brocious (Daeken)<br>
** Port the desktop implementation to Android. (In progress)
'''Status''': Being worked


* Other desktop platforms
===Video===
** Port the OS X implementation to Windows and Linux.
Video decoding is already off-main-thread, but it needs a synchronization point between the Gecko thread and the video thread to publish decoded frames. We need to be able to publish frames directly to the compositor thread, never involving the main thread.
* Asynchronous video composition is {{bug|706172}}
 
'''Assignee''': Nicolas Silva (nical)<br>
'''Status''': Blocked on Linux implementation of OMTC
 
===CSS Animations===
Currently CSS animations are driven by the main thread. It'd be better if the style and layout code could assign attributes to relevant layers that would then be interpolated using specified functions on the compositor thread.
* Asynchronous CSS animation is {{bug|706179}}
 
'''Assignee''': UNASSIGNED<br>
'''Status''': No work done
 
===Plugins===
Plugins are already drawn asynchronously (at least on OS X), similarly to how HTML5 video works. Publishing those plugin "frames" to the compositor is directly analagous.
* {{bug|706499}}
 
'''Assignee''': UNASSIGNED<br>
'''Status''': No work done
 
===Animated images===
Animated images are drawn using the main thread, but at least conceivably we could publish all the frames to the compositor and have the compositor know how to animate the images.
 
This could help us to animate throbbers, etc, but we don't currently have any layers support at all for animated images, so it will be more work for potentially less gain.
* {{bug|717872}}
 
'''Assignee''': UNASSIGNED<br>
'''Status''': No work done
 
===Off-main-thread layers implementations===
Features listed above won't work off the main thread unless we have a layers implementation that knows how to draw off the main thread.
 
==== Android ====
 
'''Assignee''': Benoit Girard/Ali Juma<br>
'''Status''': Beta quality
 
==== Mac OS X ====
 
'''Assignee''': UNASSIGNED<br>
'''Status''': Alpha quality
 
==== Linux ====
 
'''Assignee''': Nicolas Silva<br>
'''Status''': In development
 
==== Direct3D 9 ====
 
'''Assignee''': UNASSIGNED<br>
'''Status''': No work done
 
==== Direct3D 10 ====
 
'''Assignee''': UNASSIGNED<br>
'''Status''': No work done
 
==== Basic (Software-only) ====
This layers implementation is used when we have no hardware-accelerated layers backend; for example, when we're on a system with too-old drivers.
 
'''Assignee''': Marco Castelluccio <br>
'''Status''': In development


=Tracking bug=
=Tracking bug=

Revision as of 21:36, 18 May 2012

Current status

Running on Android and Mac, currently fixing Android bugs.

Outstanding bugs on Android

Builds

Latest: https://tbpl.mozilla.org/?tree=Maple Nightly: Pending

Goals

The main goal is to improve responsiveness. In the longer term, goals include:

  • Reduce main thread contention
  • Asynchronous scrolling
  • Preventing tearing
  • Asynchronous video
  • Asynchronous CSS animation

Design

Gfx layers class diagram

A new thread will handle composition. This thread will receive updates from the content thread, much as the chrome process receives updates from the content process in e10s. Indeed, as much as possible, the shadow layers machinery developed for e10s will be adapted for off-main-thread compositing. The design may need to diverge in some places from the e10s approach: for example, on Android, updates from the content thread to the compositing thread need to be asynchronous. (note: With the current code ... this might not be the case with the new frontend, and even in the current code, the cases that require async PLayers could be fixed at the widget/android level too. We'll need to make a tradeoff.)

Plan

Our highest priority platform for this project has been Android, but we want to extend this to all supported platforms: Mac, Windows (D3D9 + D3D10), Linux, B2G (Gonk) and software-only (BasicLayers).

Current status

Features

  • Only scrolling is asynchronous/off main thread. All other features require further work.

Operating systems

  • Android: Beta quality. We continue to iron out OMTC bugs as the Android test base expands, but we are already quite high-quality.
  • OS X: bug 756601 Alpha quality. We began development of OMTC on this platform, but we haven't worked on it much since then.
  • Linux: bug 722012 In initial implementation stages. We're still sorting out basic issues with X and GL, as we don't support OMTC with BasicLayers yet.
  • B2G (Gonk): bug 741837 Alpha quality. The B2G team has been working on getting it enabled and working through the implementation.
  • BasicLayers (software only): bug 703484 In initial implementation stages. Marco Castelluccio, a volunteer contributor, is working through implementing off-main-thread BasicLayers, which also involves making our Cairo library thread-safe.
  • Windows:

Future Work

WebGL

Currently, we read WebGL content back from the graphics card, and then send it to the compositor. This is terrible for performance.

  • Sharing WebGL backing textures between threads bug 728524 will let us avoid reading back.
  • Double-buffering WebGL bug 716859 will let us hand off a texture to the compositor, then go back to drawing to the WebGL context.
  • Not using glFinish() to synchronize between threads bug 697831 (most likely using the ARB_sync extension) will let us hand those textures off faster.

Assignee: Cody Brocious (Daeken)
Status: Being worked

Video

Video decoding is already off-main-thread, but it needs a synchronization point between the Gecko thread and the video thread to publish decoded frames. We need to be able to publish frames directly to the compositor thread, never involving the main thread.

Assignee: Nicolas Silva (nical)
Status: Blocked on Linux implementation of OMTC

CSS Animations

Currently CSS animations are driven by the main thread. It'd be better if the style and layout code could assign attributes to relevant layers that would then be interpolated using specified functions on the compositor thread.

Assignee: UNASSIGNED
Status: No work done

Plugins

Plugins are already drawn asynchronously (at least on OS X), similarly to how HTML5 video works. Publishing those plugin "frames" to the compositor is directly analagous.

Assignee: UNASSIGNED
Status: No work done

Animated images

Animated images are drawn using the main thread, but at least conceivably we could publish all the frames to the compositor and have the compositor know how to animate the images.

This could help us to animate throbbers, etc, but we don't currently have any layers support at all for animated images, so it will be more work for potentially less gain.

Assignee: UNASSIGNED
Status: No work done

Off-main-thread layers implementations

Features listed above won't work off the main thread unless we have a layers implementation that knows how to draw off the main thread.

Android

Assignee: Benoit Girard/Ali Juma
Status: Beta quality

Mac OS X

Assignee: UNASSIGNED
Status: Alpha quality

Linux

Assignee: Nicolas Silva
Status: In development

Direct3D 9

Assignee: UNASSIGNED
Status: No work done

Direct3D 10

Assignee: UNASSIGNED
Status: No work done

Basic (Software-only)

This layers implementation is used when we have no hardware-accelerated layers backend; for example, when we're on a system with too-old drivers.

Assignee: Marco Castelluccio
Status: In development

Tracking bug

The tracking bug is bug 598873. Tree view: [1]

Development

Experimental development is happening on the maple twig: https://hg.mozilla.org/projects/maple. m-c will be merged to Maple regularly; however, Maple will not merge with m-c. Patches should go through the regular review process on bugzilla and land on inbound. This user repo is simply for testing patch integration before patches are reviewed.

Test cases

http://people.mozilla.com/~ajuma/omtc/

Retrieved from "https://wiki.mozilla.org/index.php?title=Platform/GFX/OffMainThreadCompositing&oldid=432585"