Methodologies (Black box vs. white box)

Definitions

BLACK BOX TESTING 

Testing without knowledge of the internal workings of the item being tested. Tests are usually functional.

WHITE BOX TESTING 

Testing based on an analysis of internal workings and structure of a piece of software. Includes techniques such as Branch Testing and Path Testing. Also known as Structural Testing and Glass Box Testing.

Experiences

1. Visit http://www.example.com/ in your web browser and observe the layout. What works? What doesn't look the way you would expect? Can you give good reasons why something looks broken?
2. Visit the same site again, and use 'View Source' in your browser. Can you draw a connection between elements you can see in the source of the website and the things on the site that look broken?
3. What differences in your approach do you notice between steps 1 and 2?

Small-scale testing (Unit tests vs. functional tests vs. regression tests)

Definitions

UNIT TESTING 

Functional and reliability testing in an Engineering environment. Producing tests for the behavior of components of a product to ensure their correct behavior prior to system integration.

FUNCTIONAL TESTING 

Validating that software conforms to its specifications and correctly performs all its required functions. This entails a series of tests which perform a feature by feature validation of behavior, using a wide range of normal and erroneous input data. This can involve testing of the software's user interface, APIs, database management, security, installation, networking, etc. Functional testing can be performed on an automated or manual basis using black box or white box methodologies.

REGRESSION TESTING 

Similar in scope to a functional test, a regression test allows a consistent, repeatable validation of each new release of software. Such testing ensures reported defects have been corrected for each new release and that no new quality problems were introduced in the maintenance process. Though regression testing can be performed manually, an automated test suite is often used to reduce the time and resources needed to perform the required testing.

Experiences

1. On your phone, add a new contact called 'Mozilla Testing.' Experiment with how to add phone numbers. If you add 4085551212, does it appear as (408) 555-1212 after you save? Does it automatically save a country code with the number?
2. Add an email address to your new contact. Don't put an @ in the email address. Does it fail? Does it give you an error message?
3. What else can you do to make adding the contact fail?
4. What feels different about this testing from the black-box vs white-box testing above?

Larger-scale testing (Integration and system-level tests)

Definitions

INTEGRATION TESTING 

Testing in which functional units of software are combined and tested as a group. A funcitonal unit of software has well-defined inputs and outputs: typically this includes code modules, individual applications, client and server applications on a network, etc. Integration Testing falls between UNIT TESTING and SYSTEM TESTING in scale.

SYSTEM TESTING 

Testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing generaly uses black box testing methodologies, and as such, should require no knowledge of the inner design of the code or logic.

COMPATIBILITY TESTING 

Testing to ensure compatibility of software with different browsers, OSes, and hardware platforms. Compatibility testing can be performed manually or can be driven by an automated functional or regression test suite.

ACCEPTANCE TESTING 

Testing to verify a product meets customer specified requirements. A customer usually does this type of testing on a product that is developed externally. This is distinguished from CONFORMANCE TESTING in that the requirements are specified by the customer rather than by a widely-published standard.

SMOKE TESTING 

A quick-and-dirty test that the major functions of a piece of software work without testing finer details. Meant to evoke the hardware testing practice of turning on a new piece of hardware for the first time and considering it a success if it does not catch on fire. Technically, this is a subset of SYSTEM TESTING.

CONFORMANCE TESTING 

Verifying implementation conformance to one or more industry standards. Producing tests for the behavior of an implementation to be sure it provides the portability, interoperability, and/or compatibility a standard defines. Distinguished from ACCEPTANCE TESTING.

Experiences

1. Point your web browser to http://validator.w3.org/
2. Type the URL for your home page (or any web page you know the URL for) in the form.
3. Examine the output of the validator and compare it to what others in your group got back. See if you can identify the parts of the page cause which messages, and discuss which kinds of testing from the above list you think the validator is doing.

Load tests, performance tests, and stress tests

Definitions

LOAD TESTING 

Load testing generally includes performance testing and stress testing.

PERFORMANCE TESTING 

Performance testing can be applied to understand your application or web site's scalability, or to benchmark the performance in an environment of third party products such as servers and middleware for potential purchase. This sort of testing is particularly useful to identify performance bottlenecks in high use applications. Performance testing generally involves an automated test suite as this allows easy simulation of a variety of normal, peak, and exceptional load conditions.

STRESS TESTING 

Testing conducted to evaluate a system or component at or beyond the limits of its specified requirements to determine the load under which it fails and how. A graceful degradation under load leading to non-catastrophic failure is the desired result. Often Stress Testing is performed using the same process as Performance Testing but employing a very high level of simulated load.

Experiences

1. Point your web browser to http://www.speedtest.net/
2. Pick a server and run the test.
3. Pick another server and see if you get different results.
4. Discuss amongst your group: How is this being measured? How would you go about verifying that the numbers given back are true?

Commonly used abbreviations

STR 

Steps to reproduce: A series of detailed steps to follow in order to demonstrate a defect or an anomalous behaviour of a functional unit.

WFM 

Works for me: Can be used in two senses: 1) I have followed your steps to reproduce and I do not see the defect you describe, or 2) Now that a fix has been applied to this functional unit, I no longer observe the defect previously shown.

LGTM 

Looks good to me: An expression conveying approval or endorsement.