LZMA2 Compression

This page contains some notes about the LZMA2 compression patch set for Firefox proposed in bug 366559

What does this patch do?

This patch allows Firefox to accept content from a web server which is compressed using the LZMA2 compression algorithm, using the xz format. This algorithm compresses better than gzip/deflate, at the expense of slower compression. Decompression is much faster than compression.

The browser will send a request header listing supported “content encodings”

 Accept-Encoding: gzip, deflate, xz

This gives the web server license to send the response compressed using any of the listed algorithms. If the web server selects xz compression, this will be indicated by the following response header:

 Content-Encoding: xz

This patch uses the XZ Embedded library.

How do I set up a web server to send LZMA2 compressed content?

No web server patches are necessary to try this: Apache 2.x's content negotiation features may be used to serve static content that is compressed using LZMA2.

 # Ensure the following modules are loaded:
 LoadModule mime_module /path/to/apache2/modules/mod_mime.so
 LoadModule negotiation_module /path/to/apache2/modules/mod_negotiation.so
 
 AddEncoding gzip .gz
 AddEncoding xz .xz
 # Ensure that no MIME type is defined for .gz / .xz files:
 # AddType application/x-gzip .gz # comment this out!
 RemoveType .gz # use this if in doubt
 
 <Directory /var/www/somewhere/>
     Options +MultiViews
 </Directory>

Then replace each static file with three files as follows:

 somewhere # ls
 jquery-ui.min.js
 somewhere # gzip -c -9 jquery-ui.min.js > jquery-ui.min.js.gz
 somewhere # xz --keep jquery-ui.min.js
 somewhere # mv jquery-ui.min.js jquery-ui.min.js.js
 somewhere # ls -l
 total 336
 -rw-r--r-- 1 root root  60228 May 30 22:24 jquery-ui.min.js.gz
 -rw-r--r-- 1 root root 228077 May  7 19:09 jquery-ui.min.js.js
 -rw-r--r-- 1 root root  52512 May  7 19:09 jquery-ui.min.js.xz

• The “xz” command is available in current Linux distros, coming from XZ Utils.
• The original filename (jquery-ui.min.js) must not exist for the MultiView feature to be activated.

Now when your browser requests jquery-ui.min.js, Apache will select the smallest of these three files which is supported by your browser.

For heavily loaded servers, consider using type map files rather than MultiView so that Apache does not need to do a directory listing - see Apache documentation for details.

Building Firefox with LZMA2 Support

First get set up building Firefox from source - see Build Instructions. Then apply the patch set and build:

 curl -L https://bugzilla.mozilla.org/attachment.cgi?id=754466 | hg import -
 curl -L https://bugzilla.mozilla.org/attachment.cgi?id=8359763 | hg import -
 curl -L https://bugzilla.mozilla.org/attachment.cgi?id=8359769 | hg import -
 curl -L https://bugzilla.mozilla.org/attachment.cgi?id=8359780 | hg import -
 ./mach clobber # seems to be needed to make changes to all.js effective
 ./mach build

Interoperability Notes

The xz file format is quite flexible and extensible. This is no good for interoperability between HTTP clients unless everyone agrees what features must be supported by a client advertising “Accept-Encoding: xz”. The patch for Firefox supports only the following:

• The only supported filter is LZMA2 compression
• Maximum dictionary size of 32Mb, enough for the command xz -8 but not xz -9 (this is currently configurable with a pref, but needs to be standardized)
• The only supported integrity checks are CRC32, CRC64 or none

Alternative container formats could have been:

• The older .lzma format (apparently supported by elinks) which is now deprecated by xz. It has a slightly smaller header, and uses the original LZMA algorithm.
• Raw LZMA2 with no header. This would require the precise dictionary size to be fixed in the definition of “Content-Encoding: xz”, which seems a bit inflexible.