DXR: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
(Link to Intern job description.)
 
(25 intermediate revisions by 2 users not shown)
Line 1: Line 1:
DXR is a smart source code browser, along the lines of MXR, that uses instrumented compilers to determine information about every variable, type, and function in your code for an enhanced browsing experience. It is primarily developed by [http://quetzalcoatal.blogspot.com Joshua Cranmer] and [http://vocamus.net/dave/ David Humphrey]; for more information, you can check the [irc://irc.mozilla.org/static #static channel on irc.mozilla.org] or [https://lists.mozilla.org/listinfo/dev-static-analysis the static analysis mailing list] ([news://news.mozilla.org/mozilla.dev.static.analysis as a newsgroup]).
DXR is Mozilla's code search and navigation tool, aimed at making sense of large projects like Firefox. It supports full-text and regex searches as well as structural queries like "Find all the callers of this function." Behind the scenes, it uses trigram indices, the re2 library, and structural data collected by a custom compiler plugin.


== Obtaining source code ==
Try it here: http://dxr.mozilla.org/
The official source code for DXR can be found <!--on [http://hg.mozilla.org/webtools/dxr hg.mozilla.org] or NOTE: I haven't kept this up-to-date.--> on [http://github.com/mozilla/dxr github].


== Prerequisites ==
== Getting Started ==
In addition to the source code, the following programs are necessary:
* python (with python-sqlite package)
<!--* gcc-4.5 with dehydra plugin (if using dehydra)-->
* LLVM and clang (if using clang)
* A web server capable of running python as cgi (a standard apache2 install should work)


== Setting up ==
# Stop by the [irc://irc.mozilla.org/static #static channel on irc.mozilla.org] and say hi. Some of the folks you might see include the main developers on the project. You can ping them if you have questions:
#* [https://github.com/erikrose Erik Rose] (ErikRose)
#* [http://quetzalcoatal.blogspot.com Joshua Cranmer] (jcranmer)
#* [http://vocamus.net/dave/ David Humphrey]
# Pop over to our [https://dxr.readthedocs.org/en/latest/getting-started.html Getting Started docs] and follow the instructions.


For the purposes of the rest of this documentation, ''dxrsrc'' refers to the directory which contains the source code for DXR, and ''wwwdir'' the directory from which the web information will be stored. These should not be the same directory.
== Contributing ==


=== Configuring the webserver ===
DXR is an exciting project with lots of room for growth; more help is always appreciated. Grab a ticket you find interesting:


The following is an example configuration in use for apache2 for the webserver:
=== [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=whiteboard%3Aeasy%20component%3ADXR&list_id=6127603 Easy ones to start with] ===
<VirtualHost *:80>
<bugzilla>
  DocumentRoot ''wwwdir'' # Non-root installs currently have issues
    {
  AddHandler cgi-script .cgi
        "product": "Webtools",
  <Directory ''wwwdir''>
        "component": "DXR",
    Options Indexes FollowSymLinks MultiViews
        "whiteboard":"easy",
    Options +ExecCGI
        "resolution": "---",
    AllowOverride None
        "include_fields": "id, summary, assigned_to, target_milestone, priority, status, resolution, whiteboard"
    Order allow,deny
    }
    allow from all
</bugzilla>
  </Directory>
</VirtualHost>


The most important thing here is that ''wwwdir'' should be readable by the web server.
=== All open bugs ===
[https://bugzilla.mozilla.org/buglist.cgi?order=Importance&resolution=---&query_format=advanced&component=DXR&product=Webtools Here they are.]


=== Set up ''wwwdir'' ===
Whatever you pick, be sure to stop by the IRC channel to make sure it's still relevant; there is always Bugzilla gardening to do.
In the ''wwwdir'', you will need to copy or symlink the following files and directory:
export DXRSRC=''dxrsrc''
ln -s -t ''wwwdir'' $DXRSRC/www/*


=== Create dxr.config ===
== Mailing List ==


The example dxr.config in ''dxrsrc'' is a guideline for how to start, but it is not immediately adaptable to all systems. I recommend you put this file in the ''wwwdir'' directory. Some notes on important values:
[https://lists.mozilla.org/listinfo/dev-static-analysis The static analysis mailing list] or [news://news.mozilla.org/mozilla.dev.static.analysis newsgroup] is where the wider Mozilla static analysis community hangs out. Big DXR announcements happen there.


The <nowiki>[DXR]</nowiki> section contains files that it needs to run itself. The dxrroot variable contains ''dxrsrc'', and templates is a list of templates, in case you want to override the default templates with your own versions.
== Current Projects ==


The <nowiki>[Web]</nowiki> section contains setup information for your webserver, so that DXR knows how to generate links.
* '''[[DXR Roadmap]]'''
* [[DXR JS Analysis]]
* [[DXR Query Language Refresh]]
* [[DXR Result Mixing]]
* [[DXR Storages]] (to replace SQLite)
* [[DXR Intern Job Description]]


Any other section contains information on a build-tree. The sourcedir is the full path to the source root, the objdir the full path to the build root.
== Obsolete Stuff ==


An example configuration is as follows:
Of historical interest at best:
[DXR]
templates=/src/dxr/dxr-clang/templates
dxrroot=/src/dxr/dxr-clang
[Web]
wwwdir=/src/dxr/www
virtroot=/
hosturl=http://xochiquetzal
[git]
sourcedir=/src/dxr/git-1.7.5.3
objdir=/src/dxr/build-git


== Producing DXR output ==
* [[DXR UI Refresh]]
You will need to do the following to set up your environment before building:
* [https://etherpad.mozilla.org/DXRplan Old development plans] and [http://wiki.mozilla.org/DXR_Future_Work_Plan enhancement ideas]
export DXRSRC=''dxrsrc'' # Not necessary, but it can save some typing
* Completed goals toward [[Better DXR Testing]]
. $DXRSRC/setup-env.sh ''srcdir''
* [[DXR Language-Independent Schema]]
 
Now configure and build your program as you want. Most build systems should sensibly handle the exported $CC and $CXX, however some crazier build systems (tmake?) appear not to. After building, you need to generate the web-directory:
cd ''wwwdir''
python $DXRSRC/dxr-index.py
 
When that is done, you can then point your web browser at your DXR installation and start navigating the source code. Your source file will be located at virtroot/tree/path/to/file.c.html, e.g., <nowiki>http://<server>/git/ws.c.html</nowiki>.
 
<!-- This is only applicable to internal mozilla stuff
== Deploying DXR ==
#Run as user mozbuild
#Update the mozilla-central: need to run hg pull -u
#For comm-central: python client.py checkout
#Update clang and the webserver with the new indexes if there is a source change:
 
    . setup-env.sh /home/mozbuild/tools/llvm/tools/clang/ /home/mozbuild/tools/llvm-build2/tools/clang/
    cd /home/mozbuild/tools/llvm-build2/tools/clang
    rm *.csv
    make clean
    make
    cd /var/www/html/dxr/clang/
    python26 /home/mozbuild/dxr-clang/dxr-index.py
    cp -uR /home/mozbuild/dxr-clang/www/* .
 
Note: post-processing takes a lot of memory and if there are errors (which there probably will be), typically you must catch the index and pass on it if the errors are safe (ie: ignore it).
#Run the daily script to rerun dxr:
    in /home/moz-build, run:
        ./update-trees.sh
 
If this script is unavailable, then run the following commands in a bash script:
 
    export CFLAGS=-std=gnu89
    srcdirs=(/var/www/html/dxr/mozilla-trees/mozilla-central /var/www/html/dxr/mozilla-trees/comm-central)
    objdirs=(/var/www/html/dxr/mozilla-trees/objdir-mc-opt /var/www/html/dxr/mozilla-trees/objdir-cc-opt)
    for i in $(seq 0 $((${#srcdirs[@]} - 1))); do
      srcdir=${srcdirs[i]}
      objdir=${objdirs[i]}
      pushd $HOME/dxr-clang
      . setup-env.sh $srcdir $objdir
      popd
      rm -rf $objdir
      pushd $srcdir
      make -f client.mk configure
      pushd $objdir
      for f in $(find -name 'autoconf.mk'); do
        echo '-include $(DXRSRC)/xref-tools/moztools/myrules.mk' >> ${f/autoconf/myrules}
      done
      popd
      find $objdir -name '*.csv' | xargs rm
      make -f client.mk build
      popd
    done
   
    pushd /var/www/html/dxr/mozilla/
    python26 $HOME/dxr-clang/dxr-index.py
    popd
    pushd /var/www/html/dxr/comm-central/
    python26 $HOME/dxr-clang/dxr-index.py
    popd
 
Note: clang indexing must be done on 64bit machine or else you run out of addresses
#Update the www files in each of the trees with the newly built ones from dxr-clang:
    in /home/moz-build, run:
        ./update-dxr.sh
 
If this script is unavailable, then use the following commands in a bash script:
    wwwdir=/var/www/html/dxr/
    trees=(clang mozilla comm-central)
    for i in $(seq 0 $((${#trees[@]} - 1))); do
      echo "Updating ${trees[i]}"
      cp -uR /home/mozbuild/dxr-clang/www/*  "$wwwdir/${trees[i]}"
    done-->
 
== Tracking Development ==
Recent notes are up [https://etherpad.mozilla.org/DXRplan here]
 
== Help wanted ==
DXR is still an infant project, and more help is always appreciated.  We have a list of [https://bugzilla.mozilla.org/buglist.cgi?order=Importance&resolution=---&query_format=advanced&component=DXR&product=Webtools open bugs] for DXR, as well as a list of [http://wiki.mozilla.org/DXR_Future_Work_Plan enhancement ideas] that could be fixed. Feel free to add support for more languages as well!

Latest revision as of 19:22, 29 October 2014

DXR is Mozilla's code search and navigation tool, aimed at making sense of large projects like Firefox. It supports full-text and regex searches as well as structural queries like "Find all the callers of this function." Behind the scenes, it uses trigram indices, the re2 library, and structural data collected by a custom compiler plugin.

Try it here: http://dxr.mozilla.org/

Getting Started

  1. Stop by the #static channel on irc.mozilla.org and say hi. Some of the folks you might see include the main developers on the project. You can ping them if you have questions:
  2. Pop over to our Getting Started docs and follow the instructions.

Contributing

DXR is an exciting project with lots of room for growth; more help is always appreciated. Grab a ticket you find interesting:

Easy ones to start with

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


All open bugs

Here they are.

Whatever you pick, be sure to stop by the IRC channel to make sure it's still relevant; there is always Bugzilla gardening to do.

Mailing List

The static analysis mailing list or newsgroup is where the wider Mozilla static analysis community hangs out. Big DXR announcements happen there.

Current Projects

Obsolete Stuff

Of historical interest at best:

Retrieved from "https://wiki.mozilla.org/index.php?title=DXR&oldid=1029291"