What is the progress so far!!
Results would be discussed in a separate section.
We have generated a lot of trend diagrams to study the interaction between code coverage data for a given file and the number of bug fixes went into the file; code coverage Vs the number of times the file is modified and file churn Vs bug fix rate.
Results would be discussed below.
The plan is to observe the temporal changes in the above trends as we fix more bugs as well as develop new test cases.
This is what is planned for the immediate future
- We are going to identify top 3 modules that needs further analysis
- We will identify at least 6 files in each of those modules that have active churn but not good coverage
- We will work with DEV and QA to come up with scenarios to improve coverage on those files
Results and Findings
Code Coverage report generated using lcov looks like the following URL
Code Coverage report generated using jscoverage looks like the following URL
Please be patient while loading jscoverage report. It loads a huge json file in the back end.
How to instrument the Mozilla Firefox Browser
This is what you do to instrument the Mozilla Firefox browser.
Create a .mozconfig-gold file in your home directory with the following statements
export CFLAGS="-fprofile-arcs -ftest-coverage" export CXXFLAGS="-fprofile-arcs -ftest-coverage" export LDFLAGS="-lgcov -static-libgcc" export EXTRA_DSO_LDFLAGS="-lgcov -static-libgcc" mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../mozcentral-dbg mk_add_options MOZ_MAKE_FLAGS="-j3" mk_add_options AUTOCONF=autoconf2.13 ac_add_options --enable-application=browser ac_add_options --enable-debug ac_add_options --enable-tests ac_add_options --enable-mochitest ac_add_options --disable-optimize ac_add_options --enable-chrome-format=flat
For more detailed build options and prerequisites, please visit the following URL
Checkout the source code from the mozilla-central code repository.
rm -rf /home/user/src rm -rf /home/user/mozcentral-dbg hg clone http://hg.mozilla.org/mozilla-central/ src cd src cp ../.mozconfig-gold .mozconfig
TIP: Save the above commands into a file named ‘startover’ and make it executable.
TIP:Change the ‘user’ part to your login account name.
The standard build command to build instrumented mozilla firefox is
make -f client.mk build
Execute some Unit tests
from $OBJDIR run the following
from SRC directory run the following
./jsDriver.pl -e smdebug -L lc2 -L lc3 -L spidermonkey-n-1.9.1.tests -s ~/mozcentral-dbg/dist/bin/js
./jsDriver.pl -e smdebug -L lc2 -L lc3 -L spidermonkey-n-1.9.1.tests -s ~/mozcentral-dbg/dist/bin/js -o '-j'
Please keep the focus on the Firefox browser , when running your tests, at all times or else your tests will fail.
Gather the data
After you are done with the test runs, you will end up with a bunch of *.gcda and *.gcno files in your build directory.
Now, come back into your home directory and execute the following command
lcov -c -d mozcentral-dbg -o app.info
The lcov tool will go through the entire build directory and generates an app.info file in your home directory
TIP: If you don’t know what is an ‘lcov’ tool, please feel free to ‘google’ the term.
TIP: Please use lcov 1.7.x or later since it provides coverage statistics at both the line level and also at the function level.
Then, execute the following command
If by any chance the above command fails to complete, use the following option.
genhtml --no-source app.info
The resulting index.html is the landing page for all your code coverage data for the C/C++ code in the Firefox browser.
HINT: Look at the advanced tweaks section on how to modify the geninfo and genhtml scripts in order to generate full code coverage reports.
When we run the instrumented Firefox, there is now a "JSCoverage" menu in the "Tools" menu.
Tools -> JSCoverage -> View Coverage loads the jscoverage.html file
Tools -> JSCoverage -> Store Coverage saves the coverage data into a directory named "./jscoverage-report.
Scripts and Works around required
Please make sure that you have all the build dependencies installed properly before attempting the build.
You'll need lcov.
on os X the bin/install.sh needs to be modified on the last line there is a install -D remove the -D switch as it's not supported on OS X.
The following needs to be done :
sudo vi /usr/bin/geninfo Press the following key sequence: Esc -> : -> %s/die/print/g -> Enter
sudo vi /usr/bin/genhtml Press the following key sequence: Esc -> : -> %s/die/print/g -> Enter
I wish to fully acknowledge the great help Ed Kelly , who maintains the JSCoverage tool , provided in this instrumentation exercise. Ed was very generous in spending cycles to provide enhancements to the JSCoverage tool to meet Firefox instrumentation requirements.
Do not download the binary of the JScoverage from http://siliconforks.com/jscoverage/
You should check out the source code from its public Subversion repository:
svn co http://svn.siliconforks.com/jscoverage/trunk jscoverage
To compile, run bootstrap.sh (requires Automake and Autoconf):
./bootstrap.sh Then run make as usual. Finally execute the make install command so that jscoverage is installed in the /usr/local/bin
Why not use the binary from the site and why to build our own jscoverage !!
The following enhancements are present in our own build with latest revision.
1. Instrumenting symbolic links in the case where the destination is a regular file.
2. It copies file permissions (so that executable files stay executable).
3. It has the --mozilla option to copy the jscoverage.html file directly into Firefox's chrome.
NOTE: --js-version=180 would allow instrumenting the symbolic links
So, starting from the top of the Firefox build (~/mozcentral-dbg )directory , here are the commands to instrument the whole Firefox directory and run the mochitests with the instrumented version:
cd /dist mv bin bin-original jscoverage \ --mozilla \ --no-instrument=defaults \ --no-instrument=greprefs \ bin-original bin
How to consolidate / merge results and generate graphs
TO BE ADDED