Changes

Jump to: navigation, search

B2G/QA/Automation/UI/Strategy/Integration vs End to end

41 bytes added, 00:13, 13 February 2015
The Problem
But not all automation is the same just because it targets the UI, just as not all testing is the same just because it targets the SUT.
In particular, integration automation run before landing a commit is generally subject to a series of restrictions meant to ensure that the test results are returned quickly and in an unambiguous fashion. Those restrictions reduce the usefulness of the automation for use in acceptance testing, which must often accept a higher level of fragility and longer runtime to be more comprehensive.
When viewed through the lens of the other purpose, each type of automation is inferior:
Integration Commit integration automation is incomplete and sloppy, trading expedience for coverage, and is generally written from a developer's perspective to verify ensure the behavior of what they did write--which might not actually meet the requirements of what they should have written. As such, it frequently falls short of the needs of acceptance.
Conversely, build acceptance automation is too slow, device-bound, and sometimes has spurious failures due to non-determinism inherent in the SUT. As the user scenarios it tests often bridge multiple parts of the system, it's very fragile with poor isolation. The need to check after every step slows it down. The scope means it often can't be written adequately until entire subsystems are in place. So it's often wholly unsuitable for the incremental growth, quick land-or-not decisions, and debugging assistance that a good continuous integration suite provides.
However, these are weaknesses in perception. When used for their own specific purposes, each type of automation provides great value. Further, the compromises each makes are compensated for by the other.
Canmove, confirm
2,041
edits

Navigation menu