Auto-tools/AutoTestingGuide: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
(Replaced content with "Moved to [https://developer.mozilla.org/en/Mozilla_automated_testing MDN].")
 
(22 intermediate revisions by 4 users not shown)
Line 1: Line 1:
'''''This page should be moved to MDN when we have a reasonable first version.'''''
Moved to [https://developer.mozilla.org/en/Mozilla_automated_testing MDN].
 
= Automated Testing at Mozilla =
 
You've just written a feature and (hopefully!) want to test it.  Or you've decided that an existing feature doesn't have enough tests and want to contribute some.  But where do you start? You've looked around and found references to things like "xpcshell" or "mozmill" or "talos".  What do they all do?  What's the overlap?  In short, where should your new tests go?
 
This document aims to answer that question.  There's a very short summary of each framework, and a bit of Q&A to help you pick your framework.  This may only narrow down your choices, however, in which case you should read more about the frameworks and/or hop on #ateam, #qa, or one of the development forums and ask questions.
 
Generally, you should pick the lowest-level framework that you can.  If you are testing JavaScript but don't need a window, use xpcshell.  If you're testing page layout, try to use reftest.  The advantage is that you don't drag in a lot of other components that might have their own problems, so you can home in quickly on any bugs in what you are specifically testing.
 
= In production =
 
== buildbot tests ==
 
These tests are found within the mozilla-central tree, along with the product code.  They are all run when a changeset is pushed to mozilla-central, mozilla-inbound, or try, with the results showing up on [[http://tbpl.mozilla.org tbpl]].  They can also be run on their own.
 
The letters in parentheses are the abbreviations used by tbpl.
 
=== compiled-code (B) ===
 
Written in C++, [https://developer.mozilla.org/en/Compiled-code_automated_tests compiled-code tests] can test pretty much anything but are difficult to write properly.  In general, this should be your last option for a new test.
 
=== xpcshell (B) ===
 
[https://developer.mozilla.org/en/XPConnect/xpcshell xpcshell] are console JavaScript tests.  There is no chrome, no content, no window.  xpcshell is useful for testing low-level things, such as XPCOM components. If you don't need a window, use this.
 
=== JS engine regression (J) ===
 
Tests specifically for the JavaScript engine, and that's about it.
 
=== crashtest (C) ===
 
Really simple: open a web page and see if it causes a crash.  If you've found pages that crash Firefox, add a test here to make sure future version don't experience this crash again.
 
=== reftest (R) ===
 
A [https://developer.mozilla.org/en/Creating_reftest-based_unit_tests reftest] verifies that two web pages are rendered identically.  There is generally more than one way to write a web page, and a reftest can be used to make sure that they are rendered identically.  If you want to test layout or graphics, reftest can be a very easy way to do so.
 
=== Mochitest (M, Moth) ===
 
[https://developer.mozilla.org/en/Mochitest Mochitest] uses JavaScript to test features.  Anything piece that has its functionality exposed in JavaScript can theoretically be tested with Mochitest.  A subsection of Mochitest, [https://developer.mozilla.org/en/Chrome_tests chrome tests], run with high privileges and hence can test a lot of the browser's functionality.  And then there are [https://developer.mozilla.org/en/Browser_chrome_tests browser chrome tests], which run in the scope of the browser window. (FIXME: what are the differences between chrome and browser-chrome tests?).
 
If you can use JavaScript to test your feature, Mochitests are not a bad place to start. (FIXME: or are there now better places?)
 
=== Talos (T) ===
 
Talos is a framework for performance testing.  It is a more complicated beast than other Mozilla frameworks, but there is a [https://wiki.mozilla.org/StandaloneTalos#How_to_set_up_Talos_for_testing_at_home standalone] version which is easier to set up than the full [https://wiki.mozilla.org/Auto-tools/Tools/Talos automated system].  If you're measuring performance, Talos is the place to go.
 
== Mozmill ==
 
[https://developer.mozilla.org/en/Mozmill Mozmill] is an extensive framework for browser automation. It is an extension with an extensive API to help you write functional tests that simulate user interactions, as well as a full unit test API.  It is not yet part of the main automated test suite.
 
== Speedtests ==
 
== Bughunter ==
 
= In progress =
 
== Eideticker ==
 
== Marionette ==
 
== Robocop ==
 
== Peptest ==

Latest revision as of 22:37, 10 January 2012

Moved to MDN.

Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/AutoTestingGuide&oldid=385743"