XUL:Templates Plan

XUL 2.0 Templates

The goal here is improve the XUL template builder to be more useful to developers. Several proposals have been made, some simple and others more complex. None that I know of have been specified to any level of detail that would make them possible to implement. In this proposal, I provide specifics of how work on the template builder should proceed. It is primarily based on the other proposals.

Some specific things that have been desired include:

• support for other types of data besides RDF
• better logging and debugging
• conditions which are more complex (negations, substring matching, etc...)

Some proposals have been large and difficult to implement quickly. While backwards compatibility with existing templates isn't a long term goal, it must be in the short term, since changing all the existing templates used in Mozilla is a lot of work. Instead, changes should be made incrementally. I've spent some time over the last while beginning the first of the steps outlined below and am quite sure that this is possible.

1. Break the Template Builder into two components, one to do rule parsing and content generation, and the other to do query and result generation. The former is not dependant on RDF, while the latter is.
2. Change the datasource handling to be able to handle different types of datasources and allow different Query Processors to be used.
3. Add a better means of logging and debugging templates.
4. Add a Query Processor that uses an XML datasource, probably using XPath queries.
5. Add some default handling syntax for other types of datasources where the queries aren't as complex.
6. Add more control over recursion
7. Add any additonal syntax that anyone desires to the Template Builder.

In this last step we can add other features, like externally referenced blocks for parts of the template body, specifying more complex recursion, modifying the DOM generation, or any of the extras others have suggested.

There's also the sort service stuff which should eventually be simplified.

Templates

A template is a container element with a <template> inside it. The <template> is hidden, but its contents specify how to generate content from a Datasource. The content is inserted as siblings after the <template>. (This might later be changed to be anonymous siblings in step 6 above.)

The Datasource is specified using the datasources attribute on the container element. Any form of Datasource may be used, in general it will be a URL of something. The datasource retrieval process is described in detail later.

A Template Builder is responsible for generating content from the Datasource and template. Two subtypes, the Content Builder for normal content, and the Tree Builder for non-content trees are used. The only difference is that while the Content Builder generates DOM nodes from the template, the Tree Builder stores the output internally as a tree of results.

A second object, called a Query Processor, is responsible for generating results to be displayed. However, the Template Builder does the work of generating content from the results.

The Query Processor that is used will probably be dependant on the datasource, but one may specified explicitly in the template, including one implemented by an application in native code or JavaScript. This allows different Query Processors to be used, even for the same kind of data, although only one may be used per template. For instance, one might implement a Query Processor which processes XQuery on an XML datasource.

The Query Processor generally is expected to act as a layer between the Template Builder and the Datasource. The Template Builder doesn't interact with the data in the Datasource directly.

Template building consists of several steps:

The Compilation Step

The Template Builder compiles the queries and rules into an internal form. For the query this is done by calling the Query Processor.

The Query Processor takes a query node, compiles into into whatever form is necessary and then later uses it to generate a set of results.

The syntax of the query node designated by a <query> tag is determined by the Query Processor and has no predefined semantics. However, some form of default syntax will probably be available for simple cases so that Query Processors may be implemented easier.

In the RDF case, the query will contain the <content>, <member> and <triple> tags and will be compiled by the RDF Query Processor into a rule network, as is the case with the current template system.

The Query Step

Given an initial reference point and a query, generate a set of results. The actual specifics of the query and generation are left to a separate object called a Query Processor. The Query Processor takes a query node, compiles into into whatever form is necessary and then uses it to generate a set of results. The reference point may be used as a context for generating results. At the top level of the template, this is specified using the ref attribute.

Each result generated by the Query Processor consists of the following:

• an id, which is unique for a set of results
• an indication of whether a result is a container or not
• an indication of whether a result is empty or not
• a set of (variable -> value) pairs

How this information is determined is dependant on the Query Processor. For example, for an SQL processor, the variables might be the column names and the values of the variables are the values in those columns for each result.

Example:

 <query>
   select name, address from Users
 </query>

In this example, ?name and ?address will be available in each result. Naturally, for SQL output, there will be no containers. The id of each record might be constructed from the primary key.

Usually, one variable is designated as the member variable. This is typically used in the query to define what part of the query is most relevant to the results. For SQL, for instance, this would have little significance. For other types of data, for example a path-like query like XPath, it would be used to indicate what part of the navigation holds the results to separate it from parts which are used only to gather additional data.

The results from the Query Processor are expected to be returned in order and they will normaly be used in that order by the Template Builder, unless the Template Builder has been specified to use sorting in which case they will later be sorted.

The Query Processor itself may be a separate component, so one that is, for example, specific to SQL or mozStorage or web services would be included as part of those components instead of the template builder. Other simple Query Processors could even be implemented in XBL, for instance, for generating data from an array or JavaScript structure.

The Filter and Match Step

In this step, the Template Builder iterates through a set of rules, once for each result. For each result, an attempt is made to match the result to a set of rules. Each rule consists of three components:

• a set of match criteria (conditions)
• an additional set of (variable -> value) pairs to define (bindings)
• an action section containing the output to generate on a match (action)

The result is first examined using the match criteria. This is defined inside a <conditions> block using various forms. The actual conditions syntax and semantics are defined by the template builder, so a common simple language will be used. Most likely, this will be fairly simple, involving only a set of relational steps (such as ?x < ?y, ?y startswith "A", etc...)

The user may add additional conditions by suppling a Rule Filter which takes as an argument a result and returns true or false if there is a match. This Rule Filter may be written in native code or script for more complex situations. This allows greater flexibility when defining complex rules. However if a separate Rule Filter is not supplied, the default conditional processing will occur.

If a result is determined not to match a rule's conditions, the next rule is examined for that result. This continues until a rule is found which matches. If no rule is found, the result is dropped and does not get used in the output. Only one rule will match a result at a time. Once a rule is found where the conditions are satisfied, the other rules are not examined. Naturally, a rule with empty conditions will match by default.

If a result matches a rule, the bindings (or optional assignments) part adds additional (variable -> value) pairs. This is used to avoid calculating complex values before a rule is found to match. The Template Builder parses the <bindings> block, but the Query Processor assigns the bindings.

The Content Generation Step

The <action> section is used to specify the content that gets generated for the match. This works similarly to the existing XUL template mechanism. Each rule may have a different action, so different content may generated for each rule. One element is expected to have a uri attribute. The content above this element will be generated only once per query, while that element and it descendants will be generated once for each matching result. Naturally, since the content above the element will be generated only for the first match, the action for whichever rule first matches will apply. Later matches will not generate this additional content even if it is different in a different rule.

The element with the uri attribute is given an id attribute equivalent to the id of the corresponding result. The content and its descendants are examined for variable substitution. References to variables in attributes will be replaced by their corresponding values from the result.

The content generation is done entirely by the Template Builder. The Tree Builder is a subtype of the Template Builder which stores the data in a more optimal manner for large sets of data.

Once content generation is complete, the process may recurse to generate nested data. The value of the id of the result is used as the reference point (the ref) of the next iteration. The nested content will be inserted inside the element with the uri attribute. Note that this is how the existing XUL template mechanism works. However, there will need to be more direct control over how the recursion works. In particular, whether it recurses or not, where the nested content is generated and if a different set of rules should be used.

Content Updating

Some Query Processors may wish to support dynamic updating. That is, when the underlying datasource changes, the generated content should update to reflect this. The Query Processor is expected to determine when an update should occur and call the Template Builder indicating that a result needs to be updated. This would be the case in four situations:

• a new result is available
• a result has been removed
• a result has changed such that it may match a different rule
• a result has changed such that it will still match the same rule. This case applies when only a binding has changed, since the same rule would match.

While the Query Processor is responsible for determining when and which type of change has occured, the Template Builder is responsible for actually generating and removing the necessary content and checking which rule matches.

Note: a Query Processor is not required to support dynamic updating. In fact, there should be a means of turning it off even when it is supported. For instance, it is expected that most XML sources will be loaded from remote sources. It is likely that one would simply request a new document rather than modify an existing one. In this case, in makes sense to disable updating to reduce overhead.

General Syntax

The syntax is intended to be fairly similar to the existing syntax, at least at first. The goal is to implement something that is similar and backwards compatible first to avoid a lot of unnessesary additional work. Here is the general syntax, where items in curly braces are placeholders:

 <vbox datasources="{datasource}" ref="{reference point}">
   <template>
     <query>
       {query}
     </query>
     <rule>
       <conditions>
         {conditions}
       </conditions>
       <bindings>
         {bindings}
       </bindings>
       <action>
         {action}
       </action>
     </rule>
     ...
   </template>
 </vbox>

The <conditions> and <bindings> tags are optional, however the <action> is required. Naturally, without any conditions, the rule will match every result. As a shorthand in this case where there is only one rule, the <action> may be promoted to a location directly inside the template (as a sibling of the <query> tag); the syntax otherwise is the same.

The <queryset> tag which is used to group rules and queries together such that multiple queries may be done in one template, each with its own set of rules.

 <vbox datasources="{datasource}" ref="{reference point}">
   <template>
     <queryset>
       <query> ... </query>
       <rule> ... </rule>
     </queryset>
     <queryset>
       <query> ... </query>
       <rule> ... </rule>
     </queryset>
   </template>
 </vbox>

The Template Builder will process each queryset in sequence, generating results and content as it goes. However, if a later queryset has a query which causes a result to be generated that has the same id as one from an earlier queryset, the later result will be ignored. This is how the existing RDF template generation occurs.

The queryset syntax is intended to be a more formal way of specifying this behaviour, so that is more clear. For backwards compatibility with existing templates, if no <queryset> or <query> tags are present, a queryset is implied for each <rule> and the <conditions> tag inside a <rule> is considered to be the query.

Backwards Compatible Forms

There are several syntax forms uses only for backwards compatibility with existing XUL templates. They are used only when no <query> tag or query attribute is present. Things are determined as follows:

Each <rule> is considered to be wrapped inside a <queryset>. If there are no rules, consider there to be an implied rule containing the contents of the template.

If a <rule> contains a <conditions> element as a child, the <conditions> tag is considered to be the <query> and what would have been the conditions is empty. The Template Builder will invoke the Query Processor in the same way but using a different node. The <bindings> and <action> work as in normal usage.

If a <rule> does not contain a <conditions> element, a default query is used which generates one result for each child of the reference point and uses attributes specified on the <rule> to further specify constraints and variables. The content inside the <rule> in considered to be the action.

Here is an example. A <queryset> is added around the rule, and the <conditions> becomes the <query>.

 <vbox datasources="{datasource}" ref="{reference point}">
   <template>
     <rule>
       <conditions> {query} </conditions>
       <bindings> {bindings} </bindings>
       <action> {action} </action>
     </rule>
     ...
   </template>
 </vbox>

Only the RDF Query Processor needs to support these syntax forms.

Syntaxes Summary

• template with a <query> and a <rule>

-> Full syntax; the rule should have a <conditions> tag and an <action> tag and may optionally have a <bindings> tag.

• template with no <query> and a <rule> with no <action>

-> Simplified syntax (backwards compatibility only)

• template with no <query> and a <rule> with an <action>

-> Full syntax (backwards compatibility only)

Full Syntax

Here is a list of all the tags and attributes:

<vbox datasources="{datasource}"
     ref="{reference point}"
     flags="dont-test-empty dont-build-content dont-recurse"
     template="{template node}"
     containment="{containment predicates}"
     infer="{infer source}"
     sortDirection="ascending"
     sortActive="true"
     sort="?var">
 <template type="{query-processor-name}"
           container="?ref"
           member="?member"
           query="{query string}"/>
   <queryset>
     <query>
       {query}
     </query>
     <rule>
       <conditions>
         <where var="?var"
                rel="contains"
                value="value"
                ignorecase="true"
                negate="true"/>
         <filter expression="{script}"/>
       </conditions>
       <bindings>
         <binding subject="?ref"
                  predicate="{expr}"
                  object="?var"/>
         <assign var="?var"
                 ref="?ref"
                 expr="{expr}"/>
       </bindings>
       <action>
         <label uri="?member"/>
       </action>
     </rule>
     ...
   </queryset>
   ...
 </template>
</vbox>

I'm using some declarative tags for the conditions here. Another possibility is to use a simple expression language in an attribute. The disadvantage of that is that to a developer, one might just as well use JavaScript as the expression language rather than inventing some other language. I'd imagine that using script would be less efficient for simple comparison checks, so a simple <where> tag is used instead. However, the <filter> tag or Rule Filters allow script processing for more complex situations.

The <assign> tag inside the <bindings> tag is equivalent to the <binding> tag but uses a different set of attributes to be more clearer for non-RDF usage.

Conditions

Read about Template Conditions

Recursion

Recursion is controled in one of two ways. The 'dont-recurse' flag disables it for all generated results. If the flag is omitted, the results are queried for a default value. This allows RDF to still be recursive while other types are not. There should also be some determination based on the element tag used.

The second method is to add an element or attribute (to be determined) inside the action, pointing to another <template> via its id. The causes that template to be parsed and used for child generation.

It isn't clear at this point whether recursion which generates a result with the same id should be allowed or rejected. The current template builder doesn't handle this gracefully for the non-lazily built case (generation of things that aren't menus or trees) and the existing Tree Builder seems to drop the duplicated items. Maybe someone could come up with a reason to have duplicated items, but I don't think this needs to be supported.

Datasources

Datasources are specified using the datasources attribute (singular datasource may be better but might be confusing to switch to a similar name). It specifies the datasource to use in the template. The attribute is a URL or id reference.

A URL reference (<listbox datasources="http://www.example.com/data") will cause that URL to be retrieved and parsed into an XML document. This document is used as the datasource to the template builder. In some to-be-determined manner, the document might instead be parsed into an RDF datasource which would then be used by the template builder. For RDF, multiple URLs may be specified which will be combined into an nsIRDFCompositeDataSource. Non-RDF sources may only specify a single URL. For XML this isn't an issue since one could always append several documents or nodes together before using them in a template.

An id reference is a hash (#) followed by the id of a <datasource> element in the same document. The <datasource> element handles the datasource loading and supplies it to the template builder.

 <datasource id="thesource" type="xml" src="http://www.example.com/data"/>

The datasource element will look up a datasource loader of the given type (in the example "xml"), probably by appending the type to a contract id as rdf: datasources are currently done. This will retrieve the datasource from the src attribute and supply it to the template builder. Multiple templates may refer to the same <datasource> element; in this case the source is only loaded once and shared by all templates that refer to it. Datasources may load asynchronously and will callback to the template builder to start generation.

If there is no src attribute, the tag itself it used as the xml datasource.

An error event (onerror) may be used for error handling. Other attributes on the <datasource> tag may be used for datasource-specific purposes.

Examples

Here is an example for generating XML, assuming some simple query syntax. (note: people should be a different namespace -- there will be issues here)

 <datasource id="data" type="xml">
   <people>
     <person name="Fred" gender="male"/>
     <person name="Sarah" gender="female"/>
   </people>
 </datasource>
 <listbox datasources="#data">
   <template>
     <query>
       <results expr="person"/>
       <assign var="?name" expr="@name"/>
       <assign var="?gender" expr="@gender"/>
     </query>
     <rule>
       <conditions>
         <where var="?gender" rel="equals" value="male"/>
       </conditions>
       <action>
         <listitem uri="?" class="isMale" label="?name"/>
       </action>
     </rule>
     <rule>
       <action>
         <listitem uri="?" label="?name"/>
       </action>
     </rule>
   </template>
 </vbox>

An SQL example:

 <vbox datasources="{sql-datasource-reference}">
   <template>
     <query>
       select name, artist, cost from Albums
     </query>
     <action>
       <hbox uri="?">
         <label value="?name"/>
         <label value="?artist"/>
         <label value="?cost"/>
       </action>
     </rule>
   </template>
 </vbox>

API

See the XUL Templates API

Sort Service

See the XUL Sort Service