QA/TDAI/Inventory

From MozillaWiki
< QA‎ | TDAI
Jump to: navigation, search

« QA/TDAI

Draft-template-image.png THIS PAGE IS A WORKING DRAFT Pencil-emoji U270F-gray.png
The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly.
If you have any questions or ideas, please add them as a new topic on the discussion page.

Suites? Harnesses? Tests?

We all want quality to happen. How can we help? What is being run? What can I run something with? What does any of this mean? Not all questions are answered, but some.

This document has three sections:

  1. Test Harnesses - tools, designed to run tests. A harness may be targeted to a specific set of tests, or general to many kinds of tests. What test do you want to see run? Different harnesses could be used to run a given test. Which harness is better for which kinds of testing? kinds of testing?
  2. Test Suites - arbitrary groupings of test cases, roughly organized by functional area
  3. Test Infrastructure - the environments the test tools and test cases run in. This includes machine, operating system and physical location (ex: QA farm vs QA lab vs someone's desktop)

There are questions to be answered about any of our tests. If anyone has information or suggestions, please let us know.

  1. What tests do we have now, how current are they and how they should be run? Should they be expected to pass?
  2. What features or modules need tests to be written for them? What are the requirements of the testing that should happen?
  3. What tools do we have now that can be used to write tests? What tools outside of mozilla have people used to build tests and test tools?
  4. Of the tests currently in existence, which of them are outside the "mozilla.org" umbrella? Which use resources from outside? What is the reliability and accessibility of these resources? Are there issues which keep us from keeping local copies?

Any questions and suggestions are welcome!

Criteria for Colors

NOTE: that judgments are subjective and not "hard and fast". Caring enough to discuss will change a rating. Not caring enough to discuss is explanation enough.
  • Green: Generously defined now. Info is current or findable. Something is buildable and runnable by someone. There may still be obvious improvements needed.
  • Yellow: Little may be known. Probably not completely broken. Work to bring current is unknown.
  • Red: Obviously broken or missing. Info not findable. People not locatable.

Test Harnesses

Test Harness Name Brief Description Language Used Status of Harness Notes
MochiTest built by sayrer using the MochiKit framework JavaScript Automation being set up on QA Farm. Exec by community is possible. mochitest
reftest comparison of two pages, via JavaScript and Canvas, for visual equivalency JavaScript runnable on built trunk. There is a list of reftest opportunities, but it is not clear if we want to use them. reftest
xpcshell used by 40+ tests, plans for use by many more JavaScript recent care and feeding, very usable xpcshell
eggplant uses VNC for device-level visual comparisons Being used for automation of Smoke Tests suites in litmus redstone
litmus Catalog of manual test cases. Being developed to allow community to run the manual smoke tests, BFT, FFT and L10n tests. GUI to capture results and provide feedback to the QA team. Reporting function to check test and platform coverage. English, German, French, etc. The place to be on Community Test Day! Still in refinement by zlipton and coop. 2445 test cases, 1994 tester accounts with 845 active litmus
T* perf tests, runs in tinderbox running, passing, run by tinderbox wiki
test.js custom harnesses JavaScript an opportunity for consolidation [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16]
Spider crawls a site to specified depth, loading CSS, executing JS along the way. JavaScript used only for the js/tests by bc, et al, for JS testing bc
build/make a standard for invoking stand-alone test executables, and for integrating invocation of other harnesses into MoCo automated systems "make check" is accepted standard, being integrated with buildbot tinderbox
XULUnit "XulUnit is a simple, yet powerful and easy to use framework for Unit Testing in XUL applications (XPFE)." any info known? please share. xulunit.m.o
litmus2 redesign/rework of the test case management tool. in idea phase notes

Test Suites

Test Suite Name Brief Description Test Harness(es) Status Last Info
js/tests well-developed suite of tests of the JavaScript language in js/tests SpiderMonkey Runs on QA Farm. Stable and not necessary to use or share. Semi-automated. Plans to fully automate with Alice and Rob's help in Q1 2007 Info is current. See bc, alice, robcee.
Security tests Custom framework. Runs on QA Farm. When bugs were filed, regression test cases are usually added. Semi-automated. Plans to fully automate with Eggplant in Q1 2007 Info is current. See bc, jruderman. Definite opportunity to mine bugs for test cases.
Automated Smoke Tests A "Smoke Test" is a quick, basic qualification. Eggplant. Runs on Tracy's machines. Tests for FFx 1.5 and 2.0 Fully automated. Completed porting to Mac and Linux in Q4 2006. Plan to move execution to QA Farm in Q1 2007 Info is current. See tracy.
Basic Functionality Tests (BFT) 130+ manual test cases, broadly covering major features litmus for 1.5 and 2.0 will be updated for FF3 Info is current. See marcia, timr.
Full Functionality Tests (FFT) approx. 280 manual test cases, covering most features litmus for 1.0, 1.5 and 2.0 will be updated for FF3 Info is current. See marcia, timr.
Localization Tests 15 test cases litmus for 1.5 and 2.0 engaged and open to design Info is current. Proposed designs and issues current in newsgroups. See axel
Software Update Suite approx. 50 tests, targeted to update process itself litmus Info is current. See bc.
T* includes Tdhtml, Ts, Txul, Trender (aka Tgfx), Z, and mZ, wiki for most T running, but not growing Info is current. Tests need transition to buildbot. See robcee, timr.
Accessibility Accessibility/qa Info is current. A lot is going on, but opportunities to automate are not as developed as they could be. See Aaron Leventhal.
Update Checker validity of update mar files currently automated on release machines, moving to buildbot, wiki Info is out of date. See rhelmer.
L10n checker/metadiff verify that :
  • en-US build matches locale build
  • ensure no read-only files
  • show differences from last release
currently automated on release machines, moving to buildbot, wiki Info is out of date. See rhelmer.
Tp2 new page load tests "make check"/buildbot nearing complete automation Info is current. Being integrated into automation. See alice, robcee.
Download Checker verify integrity of major released manual, wiki runnable, passing Info is current. See bc.
L10N Search Verifier manual, wiki runnable, passing Info is current, but current plans are unclear.
content/xslt/tests/buster re-purposing of the Adobe Xalan tests. custom executable/C++ Driver is in mozilla, but test content is at Apache. Info is current, but current run occasionally. Need automation.
DOM being transitioned into MochiTest browser and eyeballs? Info is currrent. More tests to be transitioned. See roc, robcee.
CSS verify spec compliance reftest running, passing, needing much to be added Info is current. More tests needed. See robcee, dbaron.
XBL2 wiki XBL2 still in design Info is current. jonas
Browser/Content May be transitioned to reftest or MochiTest, as appropriate. custom executable/C++ Info is current. Several groups of tests need to be seperated, given TLC, automation.
Embedding Tests seem fairly current but not building, why? custom executable/C++ Info is in source, but not sure of source for more info.
Extensions custom executable/C++ Info is in source, but not sure of source for more info.
Java Applets list suggested by Java team at Apple browser and eyeballs, wiki Just a list. No automation is currently in process. Info is current. See ray.
I18N custom executable/C++ Info is current, but not sure of approach to automation plans.
Graphics custom executable/C++ Info is current, but tests will need to be made current. Match to external Cairo testing for relevance.
HTML Parsing Will need to deal with WHATWG/html5lib tests bug 366936. The htmlparser/tests/html test vs a baseline, but a baseline has not been stored. custom executable/C++ Info is in source. Other source of info is unclear (perhaps mrbkap?)
Acid2 compliance tests from outside standards "watcher" manually executed, Acid2 Browser Test currently passing Info is current. See timr.
XPConnect TestXPC -> "FAILED to get a JSRuntime" custom executable/C++ Info is in source, but not sure of source for more info.
Layout/HTML 1800+ files. These can be transitioned, probably to reftest. HTML file groups: CellMapTest, FrameSetNav, SlideShow, attributes, block, css, formctls, frameset, printer, style, table, tags, xbl. "the old regression test suite" and reftest. Older tests will go to reftest. Info is current, but a lot of tests need TLC, automation.
MathML Do mathematical glyphs display correctly on all platforms? "browsers and eyeballs" also using w3c.org site content Info is current, May be able to automate with reftest.
Plugins Not known Info is in source, but not sure of source for more info.
Java Plugin not current, dev may feel out of loop Info is current, but not sure of approach to automation plans.
Networking 25 test executables, but all must be given parameters custom executable/C++ Info is in source, but not sure of source for more info.
NSPR approx 150 test executables, most pass. custom executable/c Info is in source, but not sure of source for more info.
RDF custom executable/C++ dsds (not being built), rdfcat, rdfpoll, triplescat (doc for invoke?) Info is in source, but not sure of source for more info.
Security/NSS 26 tests, but not runnable without info custom executable/C++ and "make check" Info is current. NSS tests run externally. Need tests relevant to MoCo.
XPCOM 28 custom executables, most pass, can be better documented, xpcom/tests/unit/test_pipe.js (xpcshell) custom executable/C++ Info is current, but not sure of source for more info.
XPFE winopen.xul, runs but does not show final results Info is in source, but not sure of source for more info.
XPInstall pre_checkin_trigger.html, runs in browser, passes Info is in source, but not sure of source for more info.

External Resources/Tests

Test Resource Brief Description Status of the Instructure Notes
http://www.propagandism.org/tests/domts/ Mentioned in Bug 346627. Runnable with FF 2.0.0.8 but shows errors. Does not seem to run on FF 3 (custom build).
http://test.bclary.com/ Not sure. Anybody can say? Mentioned in Bug 346627.
http://www.w3.org/DOM/Test/ W3. What else is there to say? Mentioned in Bug 346627.
http://fantasai.inkedblade.net/style/tests/css2.1/
http://www.w3.org/International/tests/#rtl-chrome
http://www.mozilla.org/projects/mathml/demo/tester.html
http://cgi.din.or.jp/~hagi3/JavaScript/Mozilla/SampleList.cgi?fmt=html List of Mozilla DOM Samples
http://mochikit.com/examples/effects/full.html
http://www.maths-informatique-jeux.com/international/mathml_test/index.xml
http://cgi.din.or.jp/~hagi3/JavaScript/Mozilla/SampleList.cgi?fmt=html
http://bclary.com/projects/spider/
http://www.webstandards.org/action/acid2/ Acid2 Browser Test
http://hixie.ch/tests/
http://www.cynthiasays.com/ Cynthia Says™ Portal
http://webxact.watchfire.com/ watchfire WebXACT
http://www.tawdis.net/taw3/cms/en TAW (Web Accessibility Test)

Test Infrastructures

TestInfrastructure Name Brief Description Harnesses and Test Suites in the Infrastructure Status of the Instructure Notes
tinderboxen Continuous build engine for Mozilla source, much chronicled in song and fable. "make check", the results of which may not be verified see dynamic Tinderbox display
buildbots Continuous build engines, written in Python, cycling rapidly based on Bonsai checkins. "make check", reftest and MochiTest reported via Firefox tinderbox
QA Farm some number of Windows, Linux and Macs. Details are not important. They should be a black box resource for developers to say "test this". "make check", reftest and MochiTest