B2G/QA/Automation/UI/Integration suite criteria
In order to make a clearer distinction between the Integration suite and the End-to-end one, here is a list which helps to define where a test belongs depending on its nature.
Should be in Integration suite rather than in the End-to-end one:
- Tests that don't represent a whole user story, such as boundary/edge cases, negative cases, and highly specific or unusual bug verifications.
- Tests that perform small, isolated checks of the UI (such as a test that only verifies the sort order of a table)
- Tests that use mocks to simulate a portion of the system
- Tests that validate binding between the UI state and back end state
- Tests that verify something which is invisible for an end-user (example: "Is the gaia setting 'ril.data.enabled' correctly set?"). An exception could be made for an End-to-end test if this verification is done during the setUp/tearDown phases.
Should not be in Integration suite:
- Tests that are flaky by design (networking tests for example)
- Tests that check a device specific issue.
- Tests that check the behavior with an external service without being mocked (like a test which interacts with the actual Gmail).
- Tests that takes too long to be executed.
Firefox OS current specific considerations
A test cannot currently be in the Integration suite, if the test:
- relies on hardware which can't be currently mocked (like Camera, SIM/DSDS, FM antenna)
- depends on external website/service which can't be mocked (as Carrier network, Firefox Account, some Email providers)
- Is blocked by a bug in b2g desktop, (for instance: MP4/3GP not recognized, emergency dialer cannot be open)
- Is blocked by design issues in b2g desktop:
- Different FTUs on b2gdesktop
- Can't set the time on b2gdesktop without changing the time on the computer
- Some settings are hidden on b2g desktop