B2G/QA/Automation/UI/Integration suite criteria

This document is currently in work in progress

In order to make a clearer distinction between the Integration suite and the Acceptance one, here is a list which helps to define where a test belongs depending on its nature.

General considerations

Should be in Integration suite rather than in the Acceptance one:

• Tests which require to mock an actor of the system
• General edge cases tests *should we use the term negative test case instead?*
• Tests coming with a regression fix with unusual STR
• Tests that verify something which is invisible for an end-user (example: "Is the gaia setting "ril.data.enabled" correctly set?")
• Tests that perform an action which can't be done by an end-user (example: )

Should not be in Integration suite:

• Tests that are flaky by design (networking tests for example)
• Tests that check a device specific issue.
• Tests that check the behavior with an external service without being mocked (like a test which interacts with the actual Gmail).
• Tests that takes too long to be executed (more than 120 *other value?* seconds)

Firefox OS current specific considerations

A test cannot currently be in the Integration suite, if the test:

• relies on hardware which can't be currently mocked (like Camera, SIM/DSDS, FM antenna)
• depends on external website/service which can't be mocked (as Carrier network, Firefox Account, some Email providers)
• Is blocked by a bug in b2g desktop, for instance:
  • Airplane mode *link to bug*
  • MP4/3GP *link to bug*
  • Can't access emergency dialer *link to bug*
  • Scrolling issue on the time picker with b2gdesktop *link to bug*
• Is blocked by design issues in b2g desktop:
  • Different FTUs on b2gdesktop
  • Can't set the time on b2gdesktop without changing the time on the computer
  • Some settings are hidden on b2gdesktop