QA/Automation/Projects/WebRTC/Crashtest Guide
Contents
Step by Step Guide to WebRTC Crashtests
Setup your system for WebRTC
There are two ways to get ready for WebRTC testing. The first one (Short-term setup) is most likely what you want to do for a testday but if you want to contribute code more often to the project you should have your own debug build of Firefox (Long-term setup). In any case, the first thing you should do in either case is to install mozdownload.
Download and install mozdownload
Go to PyPI and install it or run 'pip install mozdownload'. If you don't want to install it globally you can use a virtualenv.
For information on how to use mozdownload, please see the above website.
Short-term Setup
Download one of the tinderbox builds of the alder branch:
- Download a platform appropriate build calling the following from command line: mozdownload -t tinderbox --branch=alder
- Extract the alder-build into a suitable directory of your choice
- Make a note of the URL that mozdownload uses to download the build from and obtain the tests.zip archive from the same location
- e.g. mozdownload output
Downloading build: https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/alder-linux64/1355348662/firefox-20.0a1.en-US.linux-x86_64.tar.bz2
Therefor get the tests.zip archive from: https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/alder-linux64/1355348662/
Long-term setup
Build Firefox on your own (be warned that this will take longer)
- Follow the instructions on https://developer.mozilla.org/En/Simple_Firefox_build to prepare your system and to build Firefox
- As best create a debug, non-optimized build of Firefox. Here the content of a possible .mozconfig file
. $topsrcdir/browser/config/mozconfig mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj mk_add_options MOZ_MAKE_FLAGS="-s -j8" # the number behind -j depends on the available cores of your machine ac_add_options --enable-debug ac_add_options --enable-debugger-info-modules ac_add_options --enable-debug-symbols ac_add_options --disable-optimize export MOZ_WEBRTC_LEAKING_TESTS=1 - Build Firefox off the mozilla-central branch aka Nightly
Other useful webpages are:
- https://developer.mozilla.org/en-US/docs/Simple_Firefox_build
- https://developer.mozilla.org/en-US/docs/Configuring_Build_Options
- https://developer.mozilla.org/en-US/docs/Developer_Guide/Mozilla_build_FAQ#Making_builds_faster
- https://developer.mozilla.org/en-US/docs/Developer_Guide/mach
Running existing Tests
Another thing to note for later is how to run existing tests. Depending on how you got your Firefox build there are two ways in running the tests for WebRTC. For now we will only focus on crashtests because mochitests aren't ready yet.
With packaged tests (Short-term setup):
python reftest/runreftest.py reftest/tests/dom/media/tests/crashtests/crashtests.list --appname=%path_to_binary%
With a self-build Firefox (Long-term setup):
./mach crashtest dom/media/tests/crashtests
Creating a new Crash Test
The following process is best explained with an example. There is a brief section at the end on how to best choose a suitable bug candidate for a crashtest, so if you only want to use this as a guideline rather than work your way through it first time, feel free to look at the Section on 'How to find a suitable bug to write a crashtest for'.
We take a 803535 as an example (WebRTC crash [@mozilla::MediaEngineDefault::EnumerateAudioDevices]).
Take the submitted testcase and download it to your local harddrive. Then obtain a build from the day this bug was submitted using mozdownload. In this case it was: 2012-10-19 07:31 PDT (top right of the bug report)
Now call
mozdownload --type=daily –date=2012-10-19
from the command line. This should download a file called 2012-10-19-03-05-51-mozilla-central-firefox-19.0a1.en-US.linux-x86_64.tar.bz2.
Unpack this version of firefox and start it from command line with
home:~/[path]/firefox-2012-10-19$ ./firefox -no-remote -p
This will ask you to choose a profile. I would suggest creating a new one with a fitting title such 'WebRTC'. Also you have to make sure that the preferences media.navigator.enabled and media.peerconnection.enabled in about:config are set to true before testing for the crash. If those are not enabled, nothing will happen.
Now access the previously downloaded testcase file from this instance of Firefox and check if the application crashes. In this specific case, the nightly becomes unresponsive but does not crash. If any of this does not happen, then download a build earlier in date and repeat until you can reproduce the error.
Note: If it crashes straight away, it is possible to check the stacktrace by starting the nightly build and going on the page about:crashes.
Recreate the testcase in a firefox-debug build and compare to the callstack
Building the debug
We now need to test this scenario in a full debug build. In this case, firefox only froze, so we need to make sure it crashes in order to compare the given stacktrace (in the bug) and the one obtained. We will now revert our firefox nightly build to the version that it was before that specific bug report. We shall do this by backing out of this particular patch with a simple command line entry. In our case, go to this page and save the file on to your hard drive.
https://bug803535.bugzilla.mozilla.org/attachment.cgi?id=673403
In general, the file can be found by adjusting this webpage to reflect the current bug you are working on and the attachment number of the patch.
Now run the following command from the command line
(home):~/path/to/dir$ patch -p1 -R < %filename%
as an example, I ran in this case patch -p1 -R < revert-patch-bug-803535.patch, however the file can be called whatever you want.
Note: In order to revert this change call either hg revert –all or patch -p1 < %filename%. It might also be wise to make sure at this point that the changes were truly implemented.
Now it is time, to (re-)build our firefox debug build. To do this, call the following from command line
nice -19 ./mach build
This should only take a couple of minutes given that we are only changing a couple of files from the previous build. The time taken to build this, however, may take longer depending on the amount of changes that were made since then.
Note: If the building process takes a very long time, you may want to consult the following websites to potentially adjust the running time:
- https://developer.mozilla.org/en-US/docs/Developer_Guide/Mozilla_build_FAQ#Making_builds_faster
- https://developer.mozilla.org/en-US/docs/Developer_Guide/mach
Check the testcase in the debug build
Here we have two possibilities: If the testcase crashes the browser, then we can retrieve the callstack from about:crashes in the browser. In this case, follow to the next subheading “Compare the callstack”. It needs to crash, a frozen screen like in our case is no good.
Note: Make sure that the preferences media.navigator.enabled and media.peerconnection.enabled in about:config are set to true before testing for the crash. If those are not enabled, nothing will happen.
If this, however, fails the next option is to start firefox using gdb (for more information, please consult the following webpage: https://developer.mozilla.org/en-US/docs/Debugging_Mozilla_with_gdb ).
Therefore from the command line call:
gdb %path_to_firefox_binary%
and once you see the prompt, type:
r -p %profilename%
Note: If you don't know your profile name, call ./firefox with the -ProfileManager flag, to see the available profiles, i.e. ./firefox -ProfileManager.
Once the browser becomes unresponsive or crashes, check the output in the gdb prompt and look for an exception. In our case, you would see something like this:
Assertion failure: i < Length() (invalid array index), at ../../dist/include/nsTArray.h:538 Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fffc76fe700 (LWP 15827)] 0x00007ffff16d5420 in nsTArray<nsRefPtr<mozilla::MediaEngineAudioSource>, nsTArrayDefaultAllocator>::ElementAt (this=0x7fffd03ad850, i=0) at ../../dist/include/nsTArray.h:538 538 MOZ_ASSERT(i < Length(), "invalid array index");
At this point, the gdb prompt should reappear. Now type
bt
and observe the output. This leads us then to the next section.
Compare the output of the crash to the callstack given in the bug
The output obtained in this via gdb, should look something like this (NOTE: If you arrived here via a crash, then consult about:crashes, find the report and scroll down. You should find the same output in pretty tabular form)
#0 0x00007ffff16d5420 in nsTArray<nsRefPtr<mozilla::MediaEngineAudioSource>, nsTArrayDefaultAllocator>::ElementAt (this=0x7fffd6ea5b90, i=0)
at ../../dist/include/nsTArray.h:538
#1 0x00007ffff19ce182 in mozilla::MediaEngineDefault::EnumerateAudioDevices (
this=0x7fffd6ea5b80, aASources=0x7fffd14fec80)
at /home/nebelhom/Mozilla-Dev/mozilla-central/content/media/webrtc/MediaEngineDefault.cpp:367
#2 0x00007ffff16d1155 in mozilla::GetUserMediaRunnable::SelectDevice (
this=0x7fffc8efcb60)
at /home/nebelhom/Mozilla-Dev/mozilla-central/dom/media/MediaManager.cpp:505
#3 0x00007ffff16d0b36 in mozilla::GetUserMediaRunnable::Run (
this=0x7fffc8efcb60)
at /home/nebelhom/Mozilla-Dev/mozilla-central/dom/media/MediaManager.cpp:408
#4 0x00007ffff25b7735 in nsThread::ProcessNextEvent (this=0x7fffc76a47e0,
mayWait=true, result=0x7fffd14fedff)
at /home/nebelhom/Mozilla-Dev/mozilla-central/xpcom/threads/nsThread.cpp:627
#5 0x00007ffff2544dec in NS_ProcessNextEvent_P (thread=0x7fffc76a47e0,
mayWait=true)
at /home/nebelhom/Mozilla-Dev/mozilla-central/obj/xpcom/build/nsThreadUtils---Type <return> to continue, or q <return> to quit---
.cpp:221
#6 0x00007ffff25b6590 in nsThread::ThreadFunc (arg=0x7fffc76a47e0)
at /home/nebelhom/Mozilla-Dev/mozilla-central/xpcom/threads/nsThread.cpp:265
#7 0x00007ffff69e4f19 in _pt_root (arg=0x7fffc774f370)
at /home/nebelhom/Mozilla-Dev/mozilla-central/nsprpub/pr/src/pthreads/ptthread.c:156
#8 0x00007ffff73c4e9a in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007ffff70f1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()
This output should be similar to the output given in the callstack of the bug (There may be some slight differences in the numbers). You have now successfully re-created the bug in question. Time to create a crashtest for this bug.
Creating a crashtest
Minimising the testcase
Now it is time to create the crashtest. The first thing to do, is to minimize the testcase.html to a minimal example in which the error still occurs. Of course, this is different for each testcase and has to be assessed on a case by case basis. In our case, it is possible to remove the following lines from the <body></body> tags:
var v2 = document.getElementById("pc1v");
var v3 = document.getElementById("pc2v");
var v4 = document.getElementById("pc3v");
var v5 = document.getElementById("pc4a");
var v6 = document.getElementById("pc5i");
...
<video id="pc1v" width="100" height="100" controls></video>
<video id="pc2v" width="100" height="100" controls></video>
<video id="pc3v" width="100" height="100" controls></video>
Note: Make sure the “same” errors occur. Not just any error.
Preparing the crashtest
All the details about how to create a crashtest can be found at: https://dxr.mozilla.org/mozilla-central/source/layout/tools/reftest/README.txt
Now that we have minimized our testcase, we can prepare it to be used as a crashtest. In order to do this, we must add the following to the <html> tag of the testcase file:
<html class="reftest-wait">
Furthermore, we must add
document.documentElement.removeAttribute("class");
to the javascript part of the testcase in order to let the crashtest framework know that the test is done now. Where to exactly put this can be gathered from the other crashtest files but will depend on the nature of the crashtest you are working on. If still in doubt, it is always a possibility to ask someone in IRC in the #automation channel. They are always happy to help.
To round it off, give the file a meaningful title, that explains what this test should actually test and transfer it to $HOME/mozilla-central/dom/media/tests/crashtests
To introduce the test to the framework, navigate to $OBJ_DIR/dom/media and run
make
This should set everything up to test your crashtest.
Testing your crashtest
Now is the time to test your crashtest in a real case scenario. In order to do this, add the following line to crastest.list in the $HOME/mozilla-central/dom/media/tests/crashtests directory.
load %filename%
Note: If you just want to test your crashtest, you can comment out the other lines by preceding each line with a '#' (without inverted commas).
Now go to $HOME/mozilla-central/ and run the crasthest
./mach crashtest dom/media/tests/crashtests
If you use the Short-term build, you will have to call
python reftest/runreftest.py reftest/tests/dom/media/tests/crashtests/crashtests.list --appname=%path_to_binary%
What we should now see, is a failed crashtest with the error message that you have seen earlier. To give you a guideline example again, I have shown the excerpt from the crashtest for this bug.
++DOMWINDOW == 9 (0x2ad2f35e7480) [serial = 9] [outer = 0x2ad2f1110c00] WARNING: NS_ENSURE_TRUE(frame) failed: file /home/nebelhom/Mozilla-Dev/mozilla-central/layout/base/nsPresContext.cpp, line 1179 WARNING: NS_ENSURE_TRUE(frame) failed: file /home/nebelhom/Mozilla-Dev/mozilla-central/layout/base/nsPresContext.cpp, line 1179 Assertion failure: i < Length() (invalid array index), at ../../dist/include/nsTArray.h:538 TEST-UNEXPECTED-FAIL | file:///home/nebelhom/Mozilla-Dev/mozilla-central/dom/media/tests/crashtests/803535.html | Exited with code 11 during test run INFO | automation.py | Application ran for: 0:00:07.677545 INFO | automation.py | Reading PID log: /tmp/tmpMHhXprpidlog … REFTEST INFO | runreftest.py | Running tests: end. crashtest failed: TEST-UNEXPECTED-FAIL | file:///home/nebelhom/Mozilla-Dev/mozilla-central/dom/media/tests/crashtests/803535.html | Exited with code 11 during test run TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | missing output line for total leaks! make: *** [crashtest] Error 1 make: Leaving directory `/home/nebelhom/Mozilla-Dev/mozilla-central/obj'
As you can see from the highlighted section, the error was raised again and we successfully created a crashtest! Well done!!!
Testing on the current version
Now all that is left to be done, is to test the crashtest on the current version of firefox and if everything is ok, create a patch file which can be submitted.
In order to revert this change call either hg revert –all (if you are using mercurial) or patch -p1 < %filename%. It might also be wise to make sure at this point that the changes were truly implemented. At this juncture just run another crashtest and see if the output is expected (in our case, the crashtest should pass)
./mach crashtest dom/media/tests/crashtests
or
python reftest/runreftest.py reftest/tests/dom/media/tests/crashtests/crashtests.list --appname=%path_to_binary%
Create a patch
Now, all that is left is to create a patch and submit it to the bug report to be implemented.
How to find a good bug to write a crashtest for?
These are only guidelines and there is no easy rule to obey here.
- Find a bug that crashes straight away. If you have to do reloads or anything like this to re-create the crash, then it is probably not good for a start.
- If the bug has the keywords crash or in-testsuite, that is a good sign (in-testsuite may also mean mochitest, so be careful there).
- If you are unsure, you can always ask ;)
