Gecko:AcceleratedFilters - Revision history

Roc: /* Implementation */

2013-06-06T22:52:36Z

Implementation

← Older revision		Revision as of 22:52, 6 June 2013
Line 73:		Line 73:
	* Implement LayerFilter backends for our LayerManager implementations. Would mostly reuse Moz2D implementation code.		* Implement LayerFilter backends for our LayerManager implementations. Would mostly reuse Moz2D implementation code.
	* OMTA for LayerFilter parameters		* OMTA for LayerFilter parameters

			As soon as all Moz2D DrawTargets support the complete set of (non-custom) SVG filters, we can remove our existing SVG filter implementation.

	== Unaddressed Issues ==		== Unaddressed Issues ==

	Filters can reference the page contents behind a filtered element using BackgroundImage/BackgroundAlpha. Supporting this is somewhat orthogonal to the above concerns.		Filters can reference the page contents behind a filtered element using BackgroundImage/BackgroundAlpha. Supporting this is somewhat orthogonal to the above concerns.

Roc: /* Implementation */

2013-06-05T12:37:45Z

Implementation

← Older revision		Revision as of 12:37, 5 June 2013
Line 54:		Line 54:
	** Define an object which describes a graph of filter primitives. Let's call it the FilterDescription.		** Define an object which describes a graph of filter primitives. Let's call it the FilterDescription.
	** Moz2D DrawTargets provide an API to construct a backend-specific Filter object from the FilterDescription.		** Moz2D DrawTargets provide an API to construct a backend-specific Filter object from the FilterDescription.
	** The Filter object contains methods for setting parameters for each filter primitive. Parameters can be images (SourceSurfaces), scalars, lists of scalars. "~~Enum~~" parameters ~~could be baked~~ into the FilterDescription.		** The Filter object contains methods for setting parameters for each filter primitive. Parameters can be images (SourceSurfaces), scalars, lists of scalars.
			*** It might make sense to bake "enum" parameters, and the length of each list of scalars, into the FilterDescription since it doesn't make much sense to animate those (though they are technically animatable).
	** The Filter object does not allow changes to the graph structure.		** The Filter object does not allow changes to the graph structure.
	** Moz2D DrawTargets provide a DrawSurfaceWithFilter API similar to DrawSurfaceWithShadow (and will eventually replace DrawSurfaceWithShadow), which takes a Filter.		** Moz2D DrawTargets provide a DrawSurfaceWithFilter API similar to DrawSurfaceWithShadow (and will eventually replace DrawSurfaceWithShadow), which takes a Filter.

Roc: /* Implementation */

2013-06-05T12:36:15Z

Implementation

← Older revision		Revision as of 12:36, 5 June 2013
Line 49:		Line 49:
	== Implementation ==		== Implementation ==

	There are lots of ways we could organize this. At our filters meeting in Taipei, we agreed on an approach like the following. API names are just placeholders.		There are lots of ways we could organize this. At our filters meeting in Taipei, we agreed on an approach like the following (with some interpolation by roc on issues we skirted around). API names are just placeholders.

	* Start by adding a Moz2D API for filters.		* Start by adding a Moz2D API for filters.
	** Define an object which describes a graph of filter primitives. Let's call it the FilterDescription.		** Define an object which describes a graph of filter primitives. Let's call it the FilterDescription.
	** Moz2D DrawTargets provide an API to construct a backend-specific Filter object from the FilterDescription.		** Moz2D DrawTargets provide an API to construct a backend-specific Filter object from the FilterDescription.
	** The Filter object contains methods for setting parameters for each filter primitive. ~~Some of the parameters~~ can be images (SourceSurfaces).		** The Filter object contains methods for setting parameters for each filter primitive. Parameters can be images (SourceSurfaces), scalars, lists of scalars. "Enum" parameters could be baked into the FilterDescription.
	** The Filter object does not allow changes to the graph structure.		** The Filter object does not allow changes to the graph structure.
	** Moz2D DrawTargets provide a DrawSurfaceWithFilter API similar to DrawSurfaceWithShadow (and will eventually replace DrawSurfaceWithShadow), which takes a Filter.		** Moz2D DrawTargets provide a DrawSurfaceWithFilter API similar to DrawSurfaceWithShadow (and will eventually replace DrawSurfaceWithShadow), which takes a Filter.

Roc: /* Implementation */

2013-06-05T12:31:49Z

Implementation

← Older revision		Revision as of 12:31, 5 June 2013
Line 49:		Line 49:
	== Implementation ==		== Implementation ==

	There are lots of ways we could organize this.		There are lots of ways we could organize this. At our filters meeting in Taipei, we agreed on an approach like the following. API names are just placeholders.

	~~Perhaps the simplest approach meeting all our needs would be:~~		* Start by adding a Moz2D API for filters.
	* ~~A GLContext implementation that takes~~ a filter ~~object,~~ a GL object ~~for~~ the ~~source data, GL objects~~ for ~~the image~~ parameters~~, and composites the result~~ of the ~~filter into another GL~~ object.		** Define an object which describes a graph of filter primitives. Let's call it the FilterDescription.
	* ~~A CPU implementation that~~ takes a filter ~~object,~~ a ~~DataSourceSurface for the source data, DataSourceSurfaces for the image parameters,~~ and ~~computes the result~~ of ~~the~~ filter into a ~~new DataSourceSurface~~.		** Moz2D DrawTargets provide an API to construct a backend-specific Filter object from the FilterDescription.
	** Many filter ~~primitives would benefit from a SIMD implementation~~.		** The Filter object contains methods for setting parameters for each filter primitive. Some of the parameters can be images (SourceSurfaces).
			** The Filter object does not allow changes to the graph structure.
			** Moz2D DrawTargets provide a DrawSurfaceWithFilter API similar to DrawSurfaceWithShadow (and will eventually replace DrawSurfaceWithShadow), which takes a Filter.
			* Create a support library which works with DataSourceSurfaces that can be used by any Moz2D backend to implement Filter via readback. Skia code might be useful.
			** This library should use integer arithmetic as much as possible to avoid timing attacks.
			* Write code to convert an SVG filter element into a FilterDescription and set of parameter values.
			* Write code to convert a CSS filter value into a FilterDescription and set of parameter values. This would depend on the SVG filter converter for CSS filter values that reference an SVG filter.
			* Add some kind of canvas-2d 'globalFilter' attribute that takes a CSS filter value and uses this new filter code. This is good for writing tests and won't break any existing filter support.

			It probably makes sense to slice the work horizontally by just supporting a couple of filter primitives and doing all the above work for them before adding the rest.

	~~Each Moz2D~~ backend ~~would select either the GLContext implementation or the CPU implementation. Moz2D-D2D would need to use an ANGLE GLContext and wrap SourceSurfaceD2D and DrawTargetD2D objects into~~ GL ~~objects via some ANGLE hack.~~		Additional work required:
			* D2D/D3D Filter backend
	~~The BasicCompositor layer compositor would call down into the~~ Moz2D ~~API. Other layer compositors would use the GLContext~~ implementation. ~~D3D-based compositors would need to use ANGLE.~~		* GL Filter backend
			* Custom GLSL filters (GL/D3D only)
			* Define a LayerFilter object that a LayerManager constructs from a FilterDescription, that supports setting of filter parameters, and that can be attached to a Layer
			* Implement LayerFilter backends for our LayerManager implementations. Would mostly reuse Moz2D implementation code.
			* OMTA for LayerFilter parameters

	== Unaddressed Issues ==		== Unaddressed Issues ==

	Filters can reference the page contents behind a filtered element using BackgroundImage/BackgroundAlpha. Supporting this is somewhat orthogonal to the above concerns.		Filters can reference the page contents behind a filtered element using BackgroundImage/BackgroundAlpha. Supporting this is somewhat orthogonal to the above concerns.

Roc: /* Filter Objects */

2013-05-01T03:17:02Z

Filter Objects

← Older revision		Revision as of 03:17, 1 May 2013
Line 35:		Line 35:
	== Filter Objects ==		== Filter Objects ==

	We could try to map all filters down to GLSL and support only GLSL ~~custom~~ filters (using ANGLE on Windows). This would make it difficult to have a performant CPU implementation of ~~common~~ filters~~. We~~ could pattern-match particular GLSL filters in a CPU implementation (and reject the rest entirely, probably), but that could be more fragile than having ~~an actual~~ API that captures which filters are supported by CPU implementations. Also, because a single filter ~~could contain~~ a sequence of (vertex shader, fragment shader) custom filter primitives, and an SVG filter can be a DAG of filter primitives, we can't boil a filter down to a single (vertex shader, fragment shader) combination ~~anyway~~.		We could try to map all filters down to GLSL and support only GLSL filters (using ANGLE on Windows). This would make it difficult to have a performant CPU-only implementation of standard filters --- we could pattern-match particular GLSL filters in a CPU implementation (and reject the rest entirely, probably), but that could be more fragile than having a real API that captures which filters are supported by CPU implementations. Also, because a single filter can be a sequence of (vertex shader, fragment shader) custom filter primitives, and an SVG filter can be a DAG of filter primitives, we can't boil a filter down to a single (vertex shader, fragment shader) combination; we'd need something at least as complex as a DAG of GLSL filters.

	''Proposal: make a filter object a DAG of filter primitives, where each filter primitive is one of the SVG filter primitives (which could be a GLSL custom filter)''		''Proposal: make a filter object a DAG of filter primitives, where each filter primitive is one of the SVG filter primitives (which could be a GLSL custom filter)''

Roc: /* Implementation */

2013-05-01T03:14:27Z

Implementation

← Older revision		Revision as of 03:14, 1 May 2013
Line 55:		Line 55:
	* A CPU implementation that takes a filter object, a DataSourceSurface for the source data, DataSourceSurfaces for the image parameters, and computes the result of the filter into a new DataSourceSurface.		* A CPU implementation that takes a filter object, a DataSourceSurface for the source data, DataSourceSurfaces for the image parameters, and computes the result of the filter into a new DataSourceSurface.
	** Many filter primitives would benefit from a SIMD implementation.		** Many filter primitives would benefit from a SIMD implementation.


	Each Moz2D backend would select either the GLContext implementation or the CPU implementation. Moz2D-D2D would need to use an ANGLE GLContext and wrap SourceSurfaceD2D and DrawTargetD2D objects into GL objects via some ANGLE hack.		Each Moz2D backend would select either the GLContext implementation or the CPU implementation. Moz2D-D2D would need to use an ANGLE GLContext and wrap SourceSurfaceD2D and DrawTargetD2D objects into GL objects via some ANGLE hack.

Roc: /* Filter Objects */

2013-05-01T03:14:01Z

Filter Objects

← Older revision		Revision as of 03:14, 1 May 2013
Line 37:		Line 37:
	We could try to map all filters down to GLSL and support only GLSL custom filters (using ANGLE on Windows). This would make it difficult to have a performant CPU implementation of common filters. We could pattern-match particular GLSL filters in a CPU implementation (and reject the rest entirely, probably), but that could be more fragile than having an actual API that captures which filters are supported by CPU implementations. Also, because a single filter could contain a sequence of (vertex shader, fragment shader) custom filter primitives, and an SVG filter can be a DAG of filter primitives, we can't boil a filter down to a single (vertex shader, fragment shader) combination anyway.		We could try to map all filters down to GLSL and support only GLSL custom filters (using ANGLE on Windows). This would make it difficult to have a performant CPU implementation of common filters. We could pattern-match particular GLSL filters in a CPU implementation (and reject the rest entirely, probably), but that could be more fragile than having an actual API that captures which filters are supported by CPU implementations. Also, because a single filter could contain a sequence of (vertex shader, fragment shader) custom filter primitives, and an SVG filter can be a DAG of filter primitives, we can't boil a filter down to a single (vertex shader, fragment shader) combination anyway.

	''Proposal: make a filter object a DAG of filter primitives, where each filter primitive is one of the SVG filter primitives or a GLSL custom filter''		''Proposal: make a filter object a DAG of filter primitives, where each filter primitive is one of the SVG filter primitives (which could be a GLSL custom filter)''

	Many filter primitives take parameters. There are a few things to watch out for:		Many filter primitives take parameters. There are a few things to watch out for:

Roc: /* Filter Objects */

2013-05-01T03:13:05Z

Filter Objects

← Older revision		Revision as of 03:13, 1 May 2013
Line 41:		Line 41:
	Many filter primitives take parameters. There are a few things to watch out for:		Many filter primitives take parameters. There are a few things to watch out for:
	* The common animation infrastructure shared by the compositor and style system needs to be extended to support interpolation between filter objects, with their parameters.		* The common animation infrastructure shared by the compositor and style system needs to be extended to support interpolation between filter objects, with their parameters.
	* When a CSS transition or animation applies to parameters of filter primitives in an SVG <filter> element (not yet supported in Gecko, but will have to be eventually since the SVG and CSS WGs have resolved to promote some of those parameters from attributes to CSS properties), it can affect the rendering of the filtered content but there is no easy way to express this to the OMTC compositor so it can do OMTA of those parameters. ''Proposal: ignore this issue for now and not support OMTA of those parameters''		* When a CSS transition or animation applies to parameters of filter primitives in an SVG <filter> element (not yet supported in Gecko, but will have to be eventually since the SVG and CSS WGs have resolved to promote some of those parameters from attributes to CSS properties), it can affect the rendering of the filtered content but there is no easy way to express this to the OMTC compositor so it can do OMTA of those parameters.
			** ''Proposal: ignore this issue for now and not support OMTA of those parameters''
	* Some primitives (<feImage>, custom filters) take parameters that are images. These will have to be external to the filter object and supplied separately when the filter object is used.		* Some primitives (<feImage>, custom filters) take parameters that are images. These will have to be external to the filter object and supplied separately when the filter object is used.
	** ''Proposal: When using a filter object with Moz2D, pass in a list of Moz2D SourceSurfaces to use as the image parameters''		** ''Proposal: When using a filter object with Moz2D, pass in a list of Moz2D SourceSurfaces to use as the image parameters''

Roc: /* Requirements */

2013-05-01T03:12:29Z

Requirements

← Older revision		Revision as of 03:12, 1 May 2013
Line 17:		Line 17:
	* Some yet-to-be-proposed 'globalFilter' 2D canvas attribute, which applies the filter when rendering canvas primitives.		* Some yet-to-be-proposed 'globalFilter' 2D canvas attribute, which applies the filter when rendering canvas primitives.
	* The 'filter()' CSS image value, which applies a filter to an image wherever a CSS image value can be used (e.g. background-image).		* The 'filter()' CSS image value, which applies a filter to an image wherever a CSS image value can be used (e.g. background-image).


	We currently support SVG filters on all platforms, and we shouldn't regress that. It should be OK to limit support for GLSL custom filters to platforms where we support WebGL.		We currently support SVG filters on all platforms, and we shouldn't regress that. It should be OK to limit support for GLSL custom filters to platforms where we support WebGL.

Roc: Created page with "= Accelerated SVG And CSS Filters = == Requirements == Spec: https://dvcs.w3.org/hg/FXTF/raw-file/default/filters/index.html There are three kinds of filters we need to sup..."

2013-05-01T03:11:03Z

Created page with "= Accelerated SVG And CSS Filters = == Requirements == Spec: https://dvcs.w3.org/hg/FXTF/raw-file/default/filters/index.html There are three kinds of filters we need to sup..."

New page

= Accelerated SVG And CSS Filters =

== Requirements ==

Spec:
https://dvcs.w3.org/hg/FXTF/raw-file/default/filters/index.html

There are three kinds of filters we need to support:
* CSS filter shorthands (e.g. opacity(), grayscale())
* SVG filters
* Custom filters (vertex and fragment shaders defined in GLSL)

CSS filter shorthands are defined by mapping into SVG filters and custom filters in a fairly straightforward way, so we can limit our attention to the latter two if we want to.

There are a few places we will want to use filters (sooner or later):
* CSS 'filter' property applied to an element, which makes the element into a stacking context and filters the element and its descendants as a group. In general this requires that we support filters in the OMTC compositor *and* in non-compositor content-drawing contexts such as printing, drawWindow and -moz-element().
* Some yet-to-be-proposed 'globalFilter' 2D canvas attribute, which applies the filter when rendering canvas primitives.
* The 'filter()' CSS image value, which applies a filter to an image wherever a CSS image value can be used (e.g. background-image).

We currently support SVG filters on all platforms, and we shouldn't regress that. It should be OK to limit support for GLSL custom filters to platforms where we support WebGL.

Obviously we want to use the GPU for filter processing wherever possible, especially in situations where we don't need the results of filter processing to be read by the CPU.

One non-obvious requirement is that we want to minimize the dependence of filter processing time on the pixel values being filtered, to protect against timing-based data extraction attacks.

=== Implications ===

We will want some kind of Layers API for attaching filters to layers. This is essential for OMTC to work with filters.

We will want some kind of Moz2D API for 2D drawing with filters. This is needed for canvas 'globalFilter' to be efficient, and will be very helpful for supporting filters in BasicCompositor (which implements OMTC using Moz2D).

These APIs should both consume the same kind of internal representation of filters, so we can share code for mapping CSS filters, SVG filters and custom filters into our internal representation. This representation needs to be IPDL-compatible so we can easily ship it as part of a layer transaction.

== Filter Objects ==

We could try to map all filters down to GLSL and support only GLSL custom filters (using ANGLE on Windows). This would make it difficult to have a performant CPU implementation of common filters. We could pattern-match particular GLSL filters in a CPU implementation (and reject the rest entirely, probably), but that could be more fragile than having an actual API that captures which filters are supported by CPU implementations. Also, because a single filter could contain a sequence of (vertex shader, fragment shader) custom filter primitives, and an SVG filter can be a DAG of filter primitives, we can't boil a filter down to a single (vertex shader, fragment shader) combination anyway.

''Proposal: make a filter object a DAG of filter primitives, where each filter primitive is one of the SVG filter primitives or a GLSL custom filter''

Many filter primitives take parameters. There are a few things to watch out for:
* The common animation infrastructure shared by the compositor and style system needs to be extended to support interpolation between filter objects, with their parameters.
* When a CSS transition or animation applies to parameters of filter primitives in an SVG <filter> element (not yet supported in Gecko, but will have to be eventually since the SVG and CSS WGs have resolved to promote some of those parameters from attributes to CSS properties), it can affect the rendering of the filtered content but there is no easy way to express this to the OMTC compositor so it can do OMTA of those parameters. ''Proposal: ignore this issue for now and not support OMTA of those parameters''
* Some primitives (<feImage>, custom filters) take parameters that are images. These will have to be external to the filter object and supplied separately when the filter object is used.
** ''Proposal: When using a filter object with Moz2D, pass in a list of Moz2D SourceSurfaces to use as the image parameters''
** ''Proposal: When attaching a filter object to a Layer, attach a list of special child Layers to use as the image parameters (similar to the way we attach a special child Layer to mask with)''

== Implementation ==

There are lots of ways we could organize this.

Perhaps the simplest approach meeting all our needs would be:
* A GLContext implementation that takes a filter object, a GL object for the source data, GL objects for the image parameters, and composites the result of the filter into another GL object.
* A CPU implementation that takes a filter object, a DataSourceSurface for the source data, DataSourceSurfaces for the image parameters, and computes the result of the filter into a new DataSourceSurface.
** Many filter primitives would benefit from a SIMD implementation.

Each Moz2D backend would select either the GLContext implementation or the CPU implementation. Moz2D-D2D would need to use an ANGLE GLContext and wrap SourceSurfaceD2D and DrawTargetD2D objects into GL objects via some ANGLE hack.

The BasicCompositor layer compositor would call down into the Moz2D API. Other layer compositors would use the GLContext implementation. D3D-based compositors would need to use ANGLE.

== Unaddressed Issues ==

Filters can reference the page contents behind a filtered element using BackgroundImage/BackgroundAlpha. Supporting this is somewhat orthogonal to the above concerns.