Canmove, confirm
2,041
edits
Changes
→The Solution
== Two different suites of UI automation ==
* Continuous Gaia Integration
** Must be quick-running with no non-deterministic factors.
** Results must be absolutely unambiguous.
** Runs on B2G Desktop currently, will run on device as well soon.
* User Acceptance TestingGaia UAT
** Can have longer-running tests and have reasonable amounts of fragility due to non-determinism
** Results can be occasionally ambiguous as a trade off for higher coverage
The main differences are:
* CI Gaia Integration tests run faster and are generally more isolated.* Gaia UATs have more coverage than CI due to less rules.
* CI Gaia Integration tests can never fail unless there's a defective code change. They must be unambiguous.* Gaia UATs can still fail non-deterministically if they are reliable enough to be a net gain.
* CI Gaia Integration tests are maintained by development to give themselves confidence in code changes.* Gaia UATs are maintained by QA to replace or extend manual regression testing.
* CI Gaia Integration tests frequently test fragments of UI behavior and may be based on mocks or other low-level objects.* Gaia UATs are always written as complete user-like scenarios, and generally operate and verify as the user would.
* It is more important that CI Gaia Integration tests be solid than that they're complete. Any incremental gain is valuable.* It is more important that Gaia UATs be complete than solid. All acceptance criteria must be tested to accept a build.
== Ownership and Overlap ==
Ownership is separated because these differences can lead to variant needs in terms of breadth and depth of verification.
Tests can and should overlap between CI Gaia Integration and UATsGaia UAT. It is unacceptable process complication for QA to expect developers to always consult their needs and vice versa for every change. If the suites are united, it's all too easy for one group to inadvertently make a change that damages the purpose of the other. For acceptance, in particular, this is too risky.
Avoiding this requires separate tests, even to the point where there might be tests whose code is entirely duplicated between the two suites.
