View source for LZMA2 Compression

This page contains some notes about the LZMA2 compression patch set for Firefox proposed in [https://bugzilla.mozilla.org/show_bug.cgi?id=366559 bug 366559]

== What does this patch do? ==

This patch allows Firefox to accept content from a web server which is compressed using the LZMA2 compression algorithm, using the [http://tukaani.org/xz/format.html xz format].  This algorithm compresses better than gzip/deflate, at the expense of slower compression.  Decompression is much faster than compression.

The browser will send a request header listing supported “content encodings”
  Accept-Encoding: gzip, deflate, xz

This gives the web server license to send the response compressed using any of the listed algorithms.  If the web server selects xz compression, this will be indicated by the following response header:
  Content-Encoding: xz

This patch uses the [http://tukaani.org/xz/embedded.html XZ Embedded] library.

== How do I set up a web server to send LZMA2 compressed content? ==

No web server patches are necessary to try this: Apache 2.x's [http://httpd.apache.org/docs/2.4/content-negotiation.html content negotiation] features may be used to serve static content that is compressed using LZMA2.

  # Ensure the following modules are loaded:
  LoadModule mime_module /path/to/apache2/modules/mod_mime.so
  LoadModule negotiation_module /path/to/apache2/modules/mod_negotiation.so
  
  AddEncoding gzip .gz
  AddEncoding xz .xz
  # Ensure that no MIME type is defined for .gz / .xz files:
  # AddType application/x-gzip .gz # comment this out!
  RemoveType .gz # use this if in doubt
  
  <Directory /var/www/somewhere/>
      Options +MultiViews
  </Directory>

Then replace each static file with three files as follows:
  somewhere # ls
  jquery-ui.min.js
  somewhere # gzip -c -9 jquery-ui.min.js > jquery-ui.min.js.gz
  somewhere # xz --keep jquery-ui.min.js
  somewhere # mv jquery-ui.min.js jquery-ui.min.js.js
  somewhere # ls -l
  total 336
  -rw-r--r-- 1 root root  60228 May 30 22:24 jquery-ui.min.js.gz
  -rw-r--r-- 1 root root 228077 May  7 19:09 jquery-ui.min.js.js
  -rw-r--r-- 1 root root  52512 May  7 19:09 jquery-ui.min.js.xz

* The “xz” command is available in current Linux distros, coming from [http://tukaani.org/xz/ XZ Utils].
* The original filename (jquery-ui.min.js) must not exist for the MultiView feature to be activated.


Now when your browser requests jquery-ui.min.js, Apache will select the smallest of these three files which is supported by your browser.

For heavily loaded servers, consider using type map files rather than MultiView so that Apache does not need to do a directory listing - see [http://httpd.apache.org/docs/2.4/content-negotiation.html#negotiation Apache documentation] for details.

== Building Firefox with LZMA2 Support ==

First get set up building Firefox from source - see [https://developer.mozilla.org/en/docs/Developer_Guide/Build_Instructions Build Instructions]. Then apply the patch set and build:

  curl -L https://bugzilla.mozilla.org/attachment.cgi?id=754466 | hg import -
  curl -L https://bugzilla.mozilla.org/attachment.cgi?id=8359763 | hg import -
  curl -L https://bugzilla.mozilla.org/attachment.cgi?id=8359769 | hg import -
  curl -L https://bugzilla.mozilla.org/attachment.cgi?id=8359780 | hg import -
  ./mach clobber # seems to be needed to make changes to all.js effective
  ./mach build

== Interoperability Notes ==
The [http://tukaani.org/xz/format.html xz file format] is quite flexible and extensible.  This is no good for interoperability between HTTP clients unless everyone agrees what features must be supported by a client advertising “Accept-Encoding: xz”.  The patch for Firefox supports only the following:
* The only supported filter is LZMA2 compression
* Maximum dictionary size of 32Mb, enough for the command xz -8 but not xz -9 (this is currently configurable with a pref, but needs to be standardized)
* The only supported integrity checks are CRC32, CRC64 or none


Alternative container formats could have been:
* The older .lzma format (apparently supported by [http://elinks.or.cz/ elinks]) which is now deprecated by xz.  It has a slightly smaller header, and uses the original LZMA algorithm.
* Raw LZMA2 with no header.  This would require the precise dictionary size to be fixed in the definition of “Content-Encoding: xz”, which seems a bit inflexible.