MozillaWiki

Platform/GFX/Surfaces: Difference between revisions

Page Discussion

Platform/GFX/Surfaces (view source)

Revision as of 03:00, 21 November 2013

3,689 bytes added ,  21 November 2013
→‎Concrete short-term plan to move towards "MozSurface"
No edit summary
 
(8 intermediate revisions by 2 users not shown)
Line 166: Line 166:
As explained in the previous section, we currently have two different abstractions, at two different levels, for "a surface that a producer can render to, and then pass over IPC". One is TextureClient, the other is SurfaceDescriptor.
As explained in the previous section, we currently have two different abstractions, at two different levels, for "a surface that a producer can render to, and then pass over IPC". One is TextureClient, the other is SurfaceDescriptor.


As explained above, TextureClient is closest to being the right one that we want everything to use; while SurfaceDescriptor is too low-level and has only become widely used because we didn't have anything else to do that job (it predates TextureClient). The key difference between TextureClient and SurfaceDescriptor is that TextureClient handles locking; so the key problem with code that uses SurfaceDescriptor's directly is that it is hard for such code to get locking right.
As explained above, TextureClient is closest to being the right one that we want everything to use; while SurfaceDescriptor is too low-level and has only become widely used because we didn't have anything else to do that job (it predates TextureClient). The key differences between TextureClient and SurfaceDescriptor is that TextureClient handles locking and cross-process memory management; so the key problem with code that uses SurfaceDescriptor's directly is that it is hard for such code to get locking and memory management right.


Let's use the term "MozSurface" to refer to that universal abstraction for "a surface that a producer can render to, and then pass over IPC".
Let's use the term "MozSurface" to refer to that universal abstraction for "a surface that a producer can render to, and then pass over IPC".


TextureClient isn't quite ready yet to be blessed as "MozSurface", for the following reasons:
TextureClient isn't quite ready yet to be blessed as "MozSurface", for the following reasons:
* 1. MozSurface should be a unique class used on both sides of IPC, while the TextureClient/TextureHost pair is two separate classes. It should be possible to unify TextureClient and TextureHost into a single class; that should be a prerequisite before it can claim to be MozSurface.
* 1. Not all layer types use the new TextureClient/Host API yet, some still use the DeprecatedTextureClient/Host. We can't use MozSurface with layers that are still on the deprecated API.
* 2. MozSurface should be free of any dependency on Layers classes, because even though ultimately graphics aren't visible to the user until they get composited as layers, we can have to start creating and even passing around surfaces before layers are created. A good example is a canvas that hasn't been inserted into the DOM yet. Currently TextureClient/Host refer to compositables to identify themselves. This is getting fixed with the introduction of the PTexture protocol in bug 897452.
* 2. MozSurface should be free of any dependency on Layers classes, because even though ultimately graphics aren't visible to the user until they get composited as layers, we can have to start creating and even passing around surfaces before layers are created. A good example is a canvas that hasn't been inserted into the DOM yet. Currently TextureClient/Host refer to compositables to identify themselves over IPC, and the lifetime of shared texture data is constrained by the lifetime of the Compositable. This is getting fixed with the introduction of the PTexture protocol in bug 897452.


Once the above items are done, we should be able to call MozSurface the class created from the merging of TextureClient and TextureHost, and declare it our universal surface abstraction to use in all producer/consumer/ipc contexts.
The goal is to make TextureClient evolve incrementally into an abstraction that fulfills all the requirements of MozSurface.
Note that the design of (the non-deprectaed) TextureClient was driven almost by the same needs as MozSurface. MozSurfaces pushes the requirement a little bit further, making the abstraction the "universal" surface abstraction, while TextureClient's goal was to be the required abstraction for any surface that may be shared with the compositor (potentially leaving out some use cases where we may use MozSurfaces that we know will not be used for compositing.


Here is a diagram that fits together the present TextureClient/TextureHost classes, the near-future PTexture protocol pairing them, and how they would fit in the future MozSurface abstraction:
As a longer term goal, we may want to merge some of TextureHost's functionalities into TextureClient/MozSurface, for instance the ability to expose one or several TextureSources for compositing. This will only be useful if we decide to prioritize being able to use the Compositor API outside the compositor process.


= A few design principles =
* We need a clear separation between abstractions that are about data, and abstractions that are about logic. MozSurface is strictly about Data. It's purpose is to handle safety through reference counting, locking and eventually IPC synchronization to ensure that data is always used in safe ways, without double-delete, use-after-free or other read-write races. By contrast, DrawTarget is an abstraction for the drawing logic. MozSurface may expose APIs like "GetAsDrawTarget" to delegate drawing to the DrawTarget abstraction, but should not be considered as a drawing tool.
* A MozSurface should be the only object owning its underlying buffer. Users of this buffer *must* go through a MozSurface. This is what makes it possible for MozSurface to safely control the access and the lifetime of the underlying memory.
* MozSurface should not compromise performances. No implicit surface copies under the hood, etc.
* MozSurface wraps one and only one texture/buffer/piece of memory. Tracking this the underlying memory should be equivalent to tracking the MozSurface.
= Concrete short-term plan to move towards "MozSurface" =
In the short term, we just iteratively switch more and more things to use TextureClient instead of less suitable or less generic surface abstractions (at least SurfaceDescriptor and probably SharedSurface), extending and fixing TextureClient along the way as needed.
Eventually, TextureClient will be "MozSurface" (or whatever it will be called then).


   *** Figure 4: The plan for MozSurface ***
   *** Figure 4: The plan for MozSurface ***
 
 
  Land PTexture
                    MozSurface
      897452
  +-------------------------------------------+
        |                Refcount ISurfaceAllocator
  |                                           |
        |              Split it away from ClientLayerManager
  | +-------------+             +-----------+ |
        |                  Allow checking if connection still up
   | |TextureClient| <---------> |TextureHost| |
        V                    933082
  | +-------------+   PTexture  +-----------+ |
       
   |                                          |
    Do not invoke            /
   +-------------------------------------------+
    RemoveTextureClient    /
    manually              /
      926745              /
                        /
        |               /
        |             /
        |            /
        |            /
        |          /
        V          V
       
  Replace all
  bad uses of            <------ Finish port to
  SurfaceDescriptor              new textures
  by TexureClient                893300
      941389
       
        |
        V
                     
  Replace SharedSurface
        by TextureClient
            941390    
    |               |
    |               |
    |               V
    |  
    |               Keep iterating on other
    |               existing "surface"-like
    |              abstractions as needed
    V
   
  Use TextureClient also
  in main-thread-compositing   <---     Merge TextureHost
   use cases                        (at least GetTextureSource)
   (if there still are any)            into TextureClient
                                      call that "MozSurface"


= Developing a standard MozStream abstraction =
= Developing a standard MozStream abstraction =


As explained earlier, having a good MozSurface abstraction is only half the way towards a good architecture. The other half is to share as much surface-handling code in a common abstraction, and the standard type of abstraction for doing this is that of a "stream", as described in the above-mentioned EGLStream specification, http://www.khronos.org/registry/egl/extensions/KHR/EGL_KHR_stream.txt.
The discussion below is about a separate abstraction. MozSurface and MozStream are two separate discussions, although MozStream would use MozSurface.
 
As explained earlier, having a good MozSurface abstraction is only half the way towards a good architecture. The other half is to share as much surface-handling code as possible in a common abstraction, and the standard type of abstraction for doing this is that of a "stream", as described in the above-mentioned EGLStream specification, http://www.khronos.org/registry/egl/extensions/KHR/EGL_KHR_stream.txt.


As we already said, we even have an implementation of a fairly standard stream, SurfaceStream, and are already using it for <canvas>.
As we already said, we even have an implementation of a fairly standard stream, SurfaceStream, and are already using it for <canvas>.


The question is how much more of our surface-passing code could we unify behind a shared stream mechanism, maybe by suitably extending SurfaceStream's capabilities?
The question is how much more of our surface-passing code could we unify behind a shared stream mechanism, maybe by suitably extending SurfaceStream's capabilities?
There is some low-hanging fruit: it is standard (e.g. on Android) to use such a stream to implement the queue of decoded frames for a <video>, so we could at least easily share that logic there.


Is there anything that we need to do, that fundamentally cannot be unified behind such a stream abstraction?
Is there anything that we need to do, that fundamentally cannot be unified behind such a stream abstraction?
Line 206: Line 255:
* 1. ThebesLayers need to get back the front buffer to do partial updates;
* 1. ThebesLayers need to get back the front buffer to do partial updates;
* 2. Drawing video frames into a canvas. It also seems that WEBGL_dynamic_texture would hit the same problem.
* 2. Drawing video frames into a canvas. It also seems that WEBGL_dynamic_texture would hit the same problem.
* 3. We currently do some screenshotting on the content side.
* 3. More importantly we have different plans for some of the <video> use cases which are based on a stream abstraction that is not a swap chain.
* a. We currently do some screenshotting on the content side.


These 3 use cases fall out of the scope of standard streams because they need a surface to be consumed by both compositor and content.
These use cases fall out of the scope of standard streams because they need a surface to be consumed by both compositor and content.


During the graphics sessions however, Jeff G and Dan proposed a solution to this problem.
During the graphics sessions however, Jeff G and Dan proposed a solution to some of these problems.


The reason why typical streams don't like the idea of a surface being consumed on two different processes, is that typical streams want to own the surfaces that they pass around. Having multiple processes hold references to the same surface makes that impossible (or would require one process to wait for the other to be done with that surface).
The reason why typical streams don't like the idea of a surface being consumed on two different processes, is that typical streams want to own the surfaces that they pass around. Having multiple processes hold references to the same surface makes that impossible (or would require one process to wait for the other to be done with that surface).
Line 369: Line 419:


B) is a stronger constraint but it is limited to video, which may have a separate implementation. A) could be solved by either letting the same MozSurface be in several swap chains, or making it possible for several layers to use the same swap chain (which probably makes more sense, but requires us to think about how we expose this functionality).
B) is a stronger constraint but it is limited to video, which may have a separate implementation. A) could be solved by either letting the same MozSurface be in several swap chains, or making it possible for several layers to use the same swap chain (which probably makes more sense, but requires us to think about how we expose this functionality).
= Reusable components =
Some of the video use cases are going to need a separate implementation than the MozStream swap chain. However these two streams could share some components.
* Client side buffer pool. We need faster surface allocation, especially with gralloc surfaces that can only be allocated on the compositor process. This can be helped with keeping surface pools to avoid the cost of actual allocation, and both video streams and MozStream can benefit from this.
* Both video and MozStream should be implemented on top of MozSurface.
Bjacob
Confirmed users
753

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/749950...767366"