|
|
| Line 1: |
Line 1: |
| {{draft}}
| |
| We all want quality to happen. How can we help? What is being run? What can I run something with? What does any of this mean? Not all questions are answered, but some.
| |
|
| |
| This document has three sections:
| |
|
| |
| # Test Harnesses - tools, designed to run tests. A harness may be targeted to a specific set of tests, or general to many kinds of tests. What test do you want to see run? Different harnesses could be used to run a given test. Which harness is better for which kinds of testing? <i>kinds of testing?</i>
| |
| # Test Suites - arbitrary groupings of test cases, roughly organized by functional area
| |
| # Test Infrastructure - the environments the test tools and test cases run in. This includes machine, operating system and physical location (ex: QA farm vs QA lab vs someone's desktop)
| |
|
| |
| There are questions to be answered about any of our tests. If anyone has information or suggestions, please [mailto:rkiddy@mozilla.com let us know].
| |
|
| |
| # What tests do we have now, how current are they and how they should be run? Should they be expected to pass?
| |
| # What features or modules need tests to be written for them? What are the requirements of the testing that should happen?
| |
| # What tools do we have now that can be used to write tests? What tools outside of mozilla have people used to build tests and test tools.
| |
| # Of the tests currently in existence, which of them are outside the "mozilla.org" umbrella? Which use resources from outside? What is the reliability and accessibility of these resources? Are there issues which keep us from keeping local copies?
| |
|
| |
| Any questions and suggestions are welcome!
| |
|
| |
|
| |
| = Test Harnesses =
| |
|
| |
| {| class="fullwidth-table"
| |
| | style="background:#efefef" | '''Test Harness Name'''
| |
| | style="background:#efefef" | '''Brief Description'''
| |
| | style="background:#efefef" | '''Language Used'''
| |
| | style="background:#efefef" | '''Status of Harness'''
| |
| | style="background:#efefef" | '''Notes'''
| |
| |-
| |
| | MochiTest
| |
| | based on the MochiKit framework
| |
| | JavaScript
| |
| | Invocation by buildbot on QA machines is being set up. Invocation of tests by inidividuals on built trunk is possible with setup. Currently being made to work in automated test harnesses.
| |
| | [http://developer.mozilla.org/en/docs/Writing_MochiTest-based_unit_tests MochiTest on devmo]
| |
| |-
| |
| | jsunit
| |
| |
| |
| | JavaScript
| |
| |
| |
| |
| |
| |-
| |
| | Acid2
| |
| | Web Standards Project compliance test page(s)
| |
| |
| |
| |
| |
| | [http://www.webstandards.org/action/acid2/ Stds page]
| |
| |-
| |
| | litmus
| |
| | Catalog of manual test cases
| |
| | English, German, French, etc.
| |
| | The place to be on Community Test Day!
| |
| | [http://litmus.mozilla.org main site]
| |
| |-
| |
| | xpcshell
| |
| | lots of under-the-hood XPCOM smarts!
| |
| | some JavaScript, much C++
| |
| |
| |
| | [http://developer.mozilla.org/en/docs/Writing_xpcshell-based_unit_tests xpcshell on devmo]
| |
| |-
| |
| | SpiderMonkey
| |
| | used only for the js/tests by bc, et al, for JS testing
| |
| | JavaScript
| |
| |
| |
| | [http://bclary.com/projects/spider/ bc's page]
| |
| |-
| |
| | reftest
| |
| | comparison of two pages, via JavaScript and Canvas, for visual equivalency
| |
| | JavaScript
| |
| | runnable on built trunk
| |
| | [http://developer.mozilla.org/en/docs/Creating_reftest-based_unit_tests reftest on devmo]
| |
| |-
| |
| | jssh
| |
| | a javascript shell. What more can you say?
| |
| | JavaScript
| |
| | described as "largely defunct"
| |
| |
| |
| |-
| |
| | Tp
| |
| | test of page loading
| |
| |
| |
| | currently runs in tinderbox
| |
| |
| |
| |-
| |
| | Tp2
| |
| | test of page loading
| |
| |
| |
| | being developed to run in buildbot
| |
| |
| |
| |-
| |
| | T*
| |
| | includes Tdhtml, Ts, Txul, Trender (aka Tgfx), Z, and mZ
| |
| |
| |
| |
| |
| | [http://wiki.mozilla.org/Performance:Tinderbox_Tests most T] [http://wiki.mozilla.org/Mozilla2:Trender Trender]
| |
| |-
| |
| | 'Browser and Eyeballs'
| |
| | The most common test harness in use today.
| |
| | wetware
| |
| | There is a new tester born every day! :-)
| |
| |
| |
| |}
| |
|
| |
| = Test Suites = | | = Test Suites = |
|
| |
|
| Line 223: |
Line 122: |
| |- | | |- |
| | Layout/HTML | | | Layout/HTML |
| | | | | 1800+ files. These can be transitioned, probably to reftest. |
| | "the old regression test suite" and reftest. Older tests will go to reftest. | | | "the old regression test suite" and reftest. Older tests will go to reftest. |
| | | | | |
| | There are a lot of test files here. It is hard to say how current or correct they are. They can be transitioned, probably to reftest. | | | HTML file groups: CellMapTest, FrameSetNav, SlideShow, attributes, block, css, formctls, frameset, printer, style, table, tags, xbl. |
| | | | | |
| |- | | |- |
| Line 244: |
Line 143: |
| |- | | |- |
| | Networking | | | Networking |
| | | | | 25 test executables, but all must be given parameters |
| | [[Custom Test Executables|custom executable/C++]] | | | [[Custom Test Executables|custom executable/C++]] |
| | | | | |
| Line 260: |
Line 159: |
| | | | | |
| | [[Custom Test Executables|custom executable/C++]] | | | [[Custom Test Executables|custom executable/C++]] |
| | | | | dsds (not being built), rdfcat, rdfpoll, triplescat (doc for invoke?) |
| | dir = rdf | | | dir = rdf |
| | | | | |
| |- | | |- |
| | Security/NSS | | | Security/NSS |
| | | | | 26 tests, but not runnable without info |
| | [[Custom Test Executables|custom executable/C++]] and "make check" | | | [[Custom Test Executables|custom executable/C++]] and "make check" |
| | | | | |
| | dirs = security/dbm, security/nss | | | dir = security/nss |
| | | | | |
| |- | | |- |
| | XPCOM | | | XPCOM |
| | | | | run -> "FAILED to get a JSRuntime" |
| | [[Custom Test Executables|custom executable/C++]] | | | [[Custom Test Executables|custom executable/C++]] |
| | | | | |
| Line 279: |
Line 178: |
| |- | | |- |
| | XPFE | | | XPFE |
| | | | | winopen.xul, runs but does not show final results |
| | | | | |
| | | | | |
| Line 286: |
Line 185: |
| |- | | |- |
| | XPInstall | | | XPInstall |
| | | | | pre_checkin_trigger.html, runs in browser, passes |
| | | | | |
| | | | | |
| | dir = xpinstall | | | dir = xpinstall |
| | | | | |
| |}
| |
|
| |
| = Test Infrastructures =
| |
|
| |
| {| class="fullwidth-table"
| |
| | style="background:#efefef" | '''Test Infrastructure Name'''
| |
| | style="background:#efefef" | '''Brief Description'''
| |
| | style="background:#efefef" | '''Harnesses and Test Suites in the Infrastructure'''
| |
| | style="background:#efefef" | '''Status of the Instructure'''
| |
| | style="background:#efefef" | '''Notes'''
| |
| |-
| |
| | tinderboxen
| |
| | Continuous build engine for Mozilla source, much chronicled in song and fable.
| |
| | "make check", the results of which may not be verified
| |
| | see dynamic [http://tinderbox.mozilla.org/showbuilds.cgi Tinderbox] display
| |
| |
| |
| |-
| |
| | buildbots
| |
| | Continuous build engines, written in Python, cycling rapidly based on Bonsai checkins.
| |
| | "make check", reftest and MochiTest
| |
| | reported via [http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest MozillaTest] tinderbox
| |
| |-
| |
| | QA Farm
| |
| | some number of Windows, Linux and Macs. Details are not important. They should be a black box resource for developers to say "test this".
| |
| | "make check", reftest and MochiTest
| |
| |
| |
| |
| |
| |} | | |} |