DXR

DXR is a smart source code browser, along the lines of MXR, that uses instrumented compilers to determine information about every variable, type, and function in your code for an enhanced browsing experience. It is primarily developed by Joshua Cranmer and David Humphrey; for more information, you can check the #static channel on irc.mozilla.org or the static analysis mailing list (as a newsgroup).

Obtaining source code

The official source code for DXR can be found on hg.mozilla.org or on github, the clang branch (the two should be direct mirrors of each other). The current main development fork of DXR is this branch, which contains insufficiently-baked features not considered ready for production use.

Prerequisites

In addition to the source code, the following programs are necessary:

• python (with python-sqlite package)
• LLVM and clang (if using clang)
• A web server capable of running python as cgi (a standard apache2 install should work)

Setting up

For the purposes of the rest of this documentation, dxrsrc refers to the directory which contains the source code for DXR, and wwwdir the directory from which the web information will be stored. These should not be the same directory.

Configuring the webserver

The following is an example configuration in use for apache2 for the webserver:

<VirtualHost *:80>
  DocumentRoot wwwdir # Non-root installs currently have issues
  AddHandler cgi-script .cgi
  <Directory wwwdir>
    Options Indexes FollowSymLinks MultiViews
    Options +ExecCGI
    AllowOverride None
    Order allow,deny
    allow from all
  </Directory>
</VirtualHost>

The most important thing here is that wwwdir should be readable by the web server.

Set up wwwdir

In the wwwdir, you will need to copy or symlink the following files and directory:

export DXRSRC=dxrsrc
ln -s -t wwwdir $DXRSRC/www/*

Create dxr.config

The example dxr.config in dxrsrc is a guideline for how to start, but it is not immediately adaptable to all systems. I recommend you put this file in the wwwdir directory. Some notes on important values:

The [DXR] section contains files that it needs to run itself. The dxrroot variable contains dxrsrc, and templates is a list of templates, in case you want to override the default templates with your own versions.

The [Web] section contains setup information for your webserver, so that DXR knows how to generate links.

Any other section contains information on a build-tree. The sourcedir is the full path to the source root, the objdir the full path to the build root.

An example configuration is as follows:

[DXR]
templates=/src/dxr/dxr-clang/templates
dxrroot=/src/dxr/dxr-clang

[Web]
wwwdir=/src/dxr/www
virtroot=/
hosturl=http://xochiquetzal

[git]
sourcedir=/src/dxr/git-1.7.5.3
objdir=/src/dxr/build-git

Producing DXR output

The following instructions require the dxr-clang development branch.

You will need to do the following to set up your environment before building:

export DXRSRC=dxrsrc # Not necessary, but it can save some typing
. $DXRSRC/setup-env.sh srcdir

Now configure and build your program as you want. Most build systems should sensibly handle the exported $CC and $CXX, however some crazier build systems (tmake?) appear not to. After building, you need to generate the web-directory:

cd wwwdir
python $DXRSRC/dxr-index.py

When that is done, you can then point your web browser at your DXR installation and start navigating the source code. Your source file will be located at virtroot/tree/path/to/file.c.html, e.g., http://<server>/git/ws.c.html.

Deploying DXR

1. Run as user mozbuild
2. Update the mozilla-central: need to run hg pull -u
3. For comm-central: python client.py checkout
4. Update clang and the webserver with the new indexes if there is a source change:

   . setup-env.sh /home/mozbuild/tools/llvm/tools/clang/ /home/mozbuild/tools/llvm-build2/tools/clang/
   cd /home/mozbuild/tools/llvm-build2/tools/clang
   rm *.csv
   make clean
   make
   cd /var/www/html/dxr/clang/
   python26 /home/mozbuild/dxr-clang/dxr-index.py
   cp -uR /home/mozbuild/dxr-clang/www/* . 

Note: post-processing takes a lot of memory and if there are errors (which there probably will be), typically you must catch the index and pass on it if the errors are safe (ie: ignore it).

1. Run the daily script to rerun dxr:

   in /home/moz-build, run:
       ./update-trees.sh

If this script is unavailable, then run the following commands in a bash script:

   export CFLAGS=-std=gnu89
   srcdirs=(/var/www/html/dxr/mozilla-trees/mozilla-central /var/www/html/dxr/mozilla-trees/comm-central)
   objdirs=(/var/www/html/dxr/mozilla-trees/objdir-mc-opt /var/www/html/dxr/mozilla-trees/objdir-cc-opt)
   for i in $(seq 0 $((${#srcdirs[@]} - 1))); do
     srcdir=${srcdirs[i]}
     objdir=${objdirs[i]}
     pushd $HOME/dxr-clang
     . setup-env.sh $srcdir $objdir
     popd
     rm -rf $objdir
     pushd $srcdir
     make -f client.mk configure
     pushd $objdir
     for f in $(find -name 'autoconf.mk'); do
       echo '-include $(DXRSRC)/xref-tools/moztools/myrules.mk' >> ${f/autoconf/myrules}
     done
     popd
     find $objdir -name '*.csv' | xargs rm
     make -f client.mk build
     popd
   done
   
   pushd /var/www/html/dxr/mozilla/
   python26 $HOME/dxr-clang/dxr-index.py
   popd
   pushd /var/www/html/dxr/comm-central/
   python26 $HOME/dxr-clang/dxr-index.py
   popd

Note: clang indexing must be done on 64bit machine or else you run out of addresses

1. Update the www files in each of the trees with the newly built ones from dxr-clang:

   in /home/moz-build, run:
       ./update-dxr.sh

If this script is unavailable, then use the following commands in a bash script:

   wwwdir=/var/www/html/dxr/
   trees=(clang mozilla comm-central)
   for i in $(seq 0 $((${#trees[@]} - 1))); do
     echo "Updating ${trees[i]}"
     cp -uR /home/mozbuild/dxr-clang/www/*  "$wwwdir/${trees[i]}"
   done

Help wanted

DXR is still an infant project, and more help is always appreciated. We have a list of open bugs for DXR, as well as a list of enhancement ideas that could be fixed. Feel free to add support for more languages as well!