Platform/XML Rewrite

From MozillaWiki
Jump to: navigation, search

("I" in this page refers to hsivonen.)

Goals

  • Get rid of nsParser
  • Get rid of nsScanner
  • Get rid of nsIContentSink and related nsI stuff
  • Get rid of nsIParser
  • Get rid of content-initiated flushes
  • Move Web content XML parsing off the main thread
  • For Web content, reuse code from the HTML side
  • Less COMtamination

Non-Goals

  • Replacing expat
  • Hiding expat from sinks
  • Moving XUL/XBL1/SAX/RDF/XSLT off the main thread

Background observations

The HTML5 parser has a design that works. When document.write handling complexity is not considered, the HTML5 parser has these major parts:

  • A parser object (nsHtml5Parser) that nsDocument sees and that holds the rest together.
  • An IO driver (nsHtml5StreamParser) that can receive bytes from a network stream, manages the character encoding conversion and pushes UTF-16 code units to the portable parser core.
  • The portable parser core (nsHtml5Tokenizer and nsHtml5TreeBuilder).
  • Glue code that produces tree ops from what the portable core does (nsHtml5TreeBuilderCppSupplement)
  • An executor for the tree ops (nsHtml5TreeOpExecutor)

XML Web content loading

I propose making the XML Web content load path have the same structure as the HTML loads path (with document.write simplified out). That is, it would have these major parts:

  • A parser object (mozilla::parser::xml::Parser) that nsDocument sees and that holds the rest together.
  • An IO driver (mozilla::parser::xml::StreamParser) that can receive bytes from a network stream, manages the character encoding conversion and pushes UTF-16 code units to expat.
  • expat (portable parser core)
  • An object that implements handler callback for expat and produces tree ops. (mozilla::parser::xml::TreeOpGenerator)
  • The same executor for the tree ops an on the HTML side (nsHtml5TreeOpExecutor, eventually to be named mozilla::parser::TreeOpExecutor)

Character encodings

expat has built-in capability to decode US-ASCII, ISO-8859-1, UTF-8 and UTF-16 and has an API for plugging in support for other decoders. So why bother with putting bytes to UTF-16 conversion in mozilla::parser::xml::StreamParser outside expat?

Unfortunately, expat has an unconventional API for encoding pluggability. Instead of having an API where byte buffers go in and UTF-16 or UTF-8 buffers come out, expat has an API for loading conversion tables into expat in the format that expat wants. Our pre-existing decoders don't expose their internals in that format. Therefore, to be able to use our pre-existing converters, we can't let expat manage the conversion.

Encoding sniffing should be handled the same way nsHtml5StreamParser handles it in the XML View Source mode: mozilla::parser::xml::StreamParser itself should handle UTF-8 and UTF-16 BOM sniffing. If there's no BOM, an instance of expat itself should be used for extracting the encoding name from the XML declaration.

Connecting handlers to expat

Looking at the existing sinks, it looks like there's no real value in having an abstraction between expat and code that does the actual work in response to expat's callbacks. If we switched away from expat today, we'd have to change the current abstraction layer anyway. That is, I think it doesn't make sense to have a single class (like the old nsExpatDriver) that provides a set of expat callbacks and then provides another abstraction for concrete handler classes that do the real work. I propose we make the concrete handler classes set themselves as expat callbacks directly. That is, mozilla::parser::xml::TreeOpGenerator should know how to register itself as the handler of various expat callbacks. This way, we don't need a layer of virtual calls on right on top of expat's function pointer-based calls.

I think it would make sense to have a common superclass for classes that can handle expat callbacks, but instead of being anything like nsIXMLContentSink or nsIExpatSink, I think the only commonality the classes need to have is the ability to register themselves as expat callback handler. So the common superclass could look like this:

class mozilla::parser::xml::AExpatHandler {
  virtual void RegisterCallbacksInto(XML_Parser aParser) = 0;
};

Dealing with stream data off the main thread

mozilla::parser::xml::StreamParser should implement nsIStreamListener on the main thread and copy data over to the parser thread the way nsHtml5StreamParser does.

Dealing with entity references off the main thread

Currently, we map a small set of magic public ids to a DTD file that we actually feed to expat so that it gets parsed every time the user loads a document that references one of the magic public ids, such as the public ids for the XHTML 1.0 DTDs. This way, entities defined in the XHTML 1.0 DTDs are available to documents.

Since our IO APIs are meant to be called on the main thread, starting IO for the local DTD file from the parser thread is not good. And in any case, it's rather silly to parse an actual file when we know in advance what the file will contain.

Instead of parsing a special file in this case, expat should be hacked in such a way that its internal entity tables can be mutated to a state that's equivalent with the state they'd end up in by parsing the special DTD without actually parsing anything.

Lack of actual speculation

In the HTML case, the only thing that can cause a speculation fail is document.write. Since XML has no document.write, the off-the-main-thread XML parser can parse its input to completion and doesn't need to support stream rewinding. All the tree ops can be queued up and they just need to be executed in chunks that end at a script execution op so that the world experienced by scripts looks as though the parts of the document after the current script didn't exist yet.

Parsing chrome: XML

Chrome documents in Firefox are localized using external DTDs that define named entities. This needs to work with the new implementation. Since initiating DTD IO from off the main thread is trouble, chrome: documents should be parsed on the main thread. To enable this, there should be an on-the-main-thread alternative for mozilla::parser::xml::StreamParser: mozilla::parser::xml::MainThreadStreamParser. To get assertions about which methods should run on which thread right, it is probably useful to actually have two classes instead of having one class with a flag that picks different code paths within the class. The two classes should probably share encoding sniffing code in a common superclass.

Since chrome: documents can be XHTML, it follows that mozilla::parser::xml::TreeOpGenerator needs to work on the main thread, too. This shouldn't be a big deal considering that tree op generation in the HTML case can run on either thread.

Parsing XML that's not Web content-like

We have various uses of expat that don't currently target nsXMLContentSink. XUL, XBL1, XSLT (transformation program compilation), RDF and SAX all have special sinks. It's probably not worthwhile to move these off the main thread or to otherwise make substantial changes here. I propose making these cases use mozilla::parser::xml::MainThreadStreamParser for feeding content to expat and to deCOMtaminate the special sinks so that they inherit from mozilla::parser::xml::AExpatHandler (presented above) and know how to set themselves directly as expat callback handlers.

Fragment parsing

On the HTML side, nsHtml5Parser also supports fragment parsing, but that functionality doesn't really benefit from being in the class that's oriented towards full page loading, so I think even on the HTML side, the fragment parsing functionality should be separated from nsHtml5Parser. I think it's not worthwhile to avoid using the tree op executor mechanism, though, since avoiding it would lead to a lot of code duplication. I think both HTML and XML should have distinct fragment parsing entry points in separate classes but the code for generating tree ops and executing them should be shared with the Web content network stream parsing code paths.

XSLT and XHR

The document that is given as input to an XSLT program and the document loaded via XHR are currently built by nsXMLContentSink. In the new world, they'd use the tree op mechanism instead like other things that currently use nsXMLContentSink.

Showing a common interface to nsDocument

nsIParser is pointlessly crufty and COMtaminated. There should be a new non-XPCOM abstract class that shows the commonality of XML and HTML parsing to nsDocument but no cruft. The new interface should look something like this:

class mozilla::parser::AParser {
  virtual void setCharsetAndSource(nsACString aCharset, PRInt32 aSource) = 0;
  virtual nsIStreamListener* GetStreamListener() = 0;
  virtual void UnblockParser() = 0;
  virtual void Start(nsIRequestObserver* aObserver) = 0; // Replaces Parse()
  virtual void Terminate() = 0;
  // possibly document.write and script execution stuff that'd be no-op in the XML case
};

Both mozilla::parser::xml::Parser and nsHtml5Parser (to be renamed mozilla::parser::html::Parser) would inherit from this abstract class.