Gecko:Layers

Layer API Proposals

Things we'd like to be able to do

• Accelerated compositing and scaling
• Accelerated color space conversion
• Off-main thread compositing and animation
• Implement CSS perspective transforms
• Keep a backing store for a chunk of rendered content
• Avoid having to read back WebGL surfaces to composite them
• Possibly hw accelerated windowless plugins

Requirements

• Should have an efficient software fallback
• Cross-platform

Roc

// Reference counted, safe-for-cross-thread-use layer class.
// Conceptually it's just something that knows how to composite itself
// onto some parent surface/layer.
// Layers are immutable. They might have time-varying rendering (animation),
// but you can't modify one once it's created. This makes them easy to use
// safely across threads.
class Layer {
};

class FunctionOfTime<T> {
  // We assume that we can access the frame rate to figure out how many
  // samples we should provide
  static FunctionOfTime createSeries(TimeStamp start, TimeDuration duration,
                                     int numSamples, T* values);
};

// Generic superclass for helper object that creates a layer. This can only
// be used on one thread.
class LayerBuilder {
  // Indicate that the given layer is related to this one, e.g., the new layer
  // corresponds to the same element as the given layer.
  // We can use this to predict that the new layer will be used in the
  // same way as the given layer, for example, the eventual rendering
  // destination(s) of the new layer can be predicted to be whatever the old
  // layer was rendered to.
  void setAffinity(Layer);

  // Set rendering properties
  void setOpacity(float);
  // Set the transform used to render this layer onto the destination surface
  void setTransform(matrix);
  void setColorSpaceConversion(...);
  void setProgram(...);

  // Set animated rendering properties. The layer is still immutable from
  // the API point of view, but its properties will change over time.
  // Specifying a function over time here lets a compositing thread change
  // the properties without blocking on the main thread.
  void setAnimatedOpacity(FunctionOfTime<float>);
  void setAnimatedTransform(FunctionOfTime<matrix>);

  // Finish building and return the Layer. This can only be called once,
  // nothing else can be done with this LayerBuilder afterward.
  // In some cases this may return no layer, in particular when the builder
  // was created by ContainerLayerBuilder::addContainerChild/addRenderedChild
  // (the layer system may have rendered the child's contents directly into the
  // parent).
  Layer finish();
};

class YUVLayerBuilder : LayerBuilder {
  // Create a YUV layer with given size and format, and adopt the memory buffer
  YUVLayerBuilder(size, format, bufferToAdopt);
};

class WebGLBufferLayerBuilder : LayerBuilder {
  // Create a layer that's a logical copy (ideally copy on write) of the
  // underlying buffer managed by a WebGL canvas
  WebGLLayerBuilder(webGLBuffer);
};

class ContainerLayerBuilder : LayerBuilder {
  // format can be RGB, ARGB (eventually ARAGAB?).
  // This constructs a container layer that can be used anywhere.
  ContainerLayerBuilder(size, format);

  // The following methods can only be called after all LayerBuilder property
  // setters are done.

  // Add an existing layer
  addLayer(Layer);

  // Open a child container layer. This child must finish()
  // before another child can be added or this builder finishes.
  ContainerLayerBuilder addContainerChild(size, format);
  // Open a child rendered layer. This child must finish()
  // before another child can be added or this builder finishes.
  // RenderedLayers constructed this way may not need a temporary surface.
  RenderedLayerBuilder addRenderedChild(size, format);
};

class RenderedLayerBuilder : LayerBuilder {
  // format can be RGB, ARGB (eventually ARAGAB?)
  // This constructs a layer rendered via gfx that can be used anywhere
  // (and therefore requires a temporary surface).
  RenderedLayerBuilder(size, format);

  // create a (conceptual) copy of the given RenderedLayer so we can modify its
  // parameters or draw into it. The underlying buffer can be managed with
  // copy on write so if we don't ever call getContext, the buffer need not
  // be copied.
  RenderedLayerBuilder(Layer layer);

  // This can only be called after all LayerBuilder property setters are
  // done. The context cannot be used after finish() is called.
  gfxContext* getContext();
};

Add a method gfxContext::SetSource(Layer).

Add a way to return a Layer from a paint event (or just set it directly on the widget), so it gets rendered, possibly asynchronously on another thread.

Clients can use a mixture of retained Layers and recursive painting with each recursion level delimited by ContainerLayerBuilder::addContainerChild followed by finish() on the child.

The goal is to allow a pure cairo implementation of this API that's as efficient as we have today. In that implementation RenderedLayerBuilder::getContext tries to return a context that renders directly into the underlying surface for some ancestor. Of course we also want to have a GL or D3D implementation that's fast, but will require more temporary surfaces if we're not using cairo-gl.

When we go to off-main-thread compositing we'll want to add support for animation and other stuff. For example we might want a YUVSeriesLayerBuilder that can select from a queue of timestamped frames based on the current time. The rendering property setters on LayerBuilder would be extended with animating setters that take a list of timestamped values, or perhaps the parameters of actual transition functions.

Jeff

Layers have two basic operations:

1. InvalidateRect()/Draw() - Gets content into a layer. Content is draw on the main thread using cairo, a video decoder etc.
2. Composite() - Composites a layer into the scene. Will perform color conversion, filter etc. as needed.

Possible example of how filter layers could work:

Assume we have 5 layers:

• 3 layers in the background
• 1 filter layer
• 1 layer on top

To render this scene we:

• create a texture the size of the filter layer
• set the render target to the texture
• composite the 3 background layers into the texture
• set the render target to the framebuffer
• composite the 3 background layers into the framebuffer (if the background layers are completely occluded by the filter layer, we can ommit this)
• composite the filter layer, applying a blur kernel to the texture we rendered earlier
• composite the top layer

A note about how to implement imperative animation in this world: CoreAnimation will ask a layer to redraw it's contents at particular timestamp. The layer can choose to do so, or ignore the request.

Bas

// Interface implemented by 'renderers'. This interface can for example
// be implemented by Cairo(possibly with cairo-gl), OpenGL, Direct3D or
// any other framework we want. This should allow easy switching between
// different renderers, and provide needed software fallback mechanisms.
class IRenderer
{
  // Set the widget this renders to.
  void SetWidget(widget);
}

// The controlling class that controls the composition of frames. This
// lives on a rectangular area on the client's screen, and controls the
// composition of all layers on the compositor. This runs its own thread
// internally from which all OpenGL/D3D operations are executed. All re-
// scheduling of drawing and invalidations are run based on operations
// executed on the compositor and its layers.
class Compositor
{
  // Create a layer that can be used to render to, the size here
  // describes the size in pixels. The format the format of the data,
  // This can be RGB, RGBA, YUV. The compositor will know what to do
  // with these layers, and how to render them properly. When the last
  // reference to the layer dies there will be only one left, and it's
  // ready to be destroyed. Type can be one of hardware or managed.
  // Only managed layers can be drawn to directly from software.
  // Any created layer can contain other layers inside, places anywhere
  // on its surface. The layer is initially locked, meaning it's not
  // shown until unlocked.
  Layer *CreateLayer(size, format, type);

  // This sets the renderer that this compositor uses, without a renderer
  // the compositor essentially does nothing.
  void SetRenderer(renderer)
};

// These are operations that can be executed on all layers.
class ILayer
{
  // Color by which the layers pixels are multiplied,
  // This contains an alpha value so opacity can implicitly
  // be controlled.
  void SetColor(color); 

  // Sets an affine transformation to place the layer with.
  void SetTransform(matrix);

  // Add a layer to this layer. This layer may be blitted onto
  // this layer's hardware surface.
  void AddLayer(ILayer);

  // Optional pixel shader program to run on this layer. This can be
  // used to apply a variety of effects to the layer when rendered.
  void SetShader(shader);

  // Lock the layer, this makes no changes take effect while in the
  // locked state.
  void Lock();

  // Unlock the layer, this will cause the compositor to traverse
  // passed this frame in the tree when compositing.
  void Unlock();

  // Clone an instance of this layer, it will contain a copy-on-write
  // reference to the contents of this layer. This layer will initially 
  // be locked.
  ILayer *Clone();
};

// Layers exposing this interface allow access to the surface. Double
// buffered, this means that if it's currently being drawn to the compositor
// will simply draw the texture. This will ensure rendering of the compositor
// area doesn't stall waiting on an expensive software render.
class ILockableLayer
{
  // Lock the surface of this layer. Returns a gfxContext to draw to.
  gfxContext *Lock();

  // Unlock the surface, this means we're done. And will signal the
  // compositor to update the associated texture and redraw.
  void Unlock();
};

// Layers exposing this interface can have their hardware surface accessed,
// which can then be used as a render target for other accelerated parts of
// the code.
class IHardwareLayer
{
  // Return hardware surface in whatever structure we pick. Might need
  // locking/unlocking logic.
  HardwareSurface *Surface();
};

// This class controls animations on objects, any class can be made to
// implement it, but we'd most likely provide some standard implementations.
// Any state it wants to maintain is contained on an implementation level.
class IAnimator
{
  // Called by the compositor when starting a rendering cycle, with
  // the elapsed time.
  virtual void AdvanceTime(double aTime);

  // This assigns the animator to a frame and registers with its compositor.
  void Assign(ILayer *aLayer);
}

Comparison with other APIs

Clutter

ClutterActor - Every actor is a 2D surface positioned and optionally transformed in 3D space. The actor is positioned relative to top left corner of it parent with the childs origin being its anchor point (also top left by default).

ClutterGroup - A group of Actors positioned relative to the group position ClutterStage - a place where Actors are composited

CoreAnimation

CALayer - CALayer is the model class for layer-tree objects. It encapsulates the position, size, and transform of a layer, which defines its coordinate system. It also encapsulates the duration and pacing of a layer and its animations by adopting the CAMediaTiming protocol, which defines a layer’s time space.

CARenderer - renders a layer tree