B2G/QA/Automation/UI/Xfail and Disable
- 1 X-Failing and Disabling Tests
- 1.1 Where To Do What?
- 1.2 When To Do What?
- 1.3 Conditional Vs. Unconditional X-Failing/Disabling of Tests
X-Failing and Disabling Tests
Where To Do What?
To disable or x-fail a test, you would make an update to the manifest.ini file that is for that particular app.
When To Do What?
X-Fail (Expected Fail)
If there is a problem with the application under test (such as a bug that causes the test to always fail) then you would X-Fail the test.
View this bug’s pull request to see an example of x-failing a test.
Disable (Turn off)
If there is a problem with the test that causes it to fail, then you would Disable the test. Things that fall under this category would be:
- The selectors were changed and the test needs to be updated
- Intermittent failures (Never X-Fail an intermittent test because you will wind up with upasses (unexpected passes))
View this bug’s pull request to see an example of a test being disabled.
Conditional Vs. Unconditional X-Failing/Disabling of Tests
Conditionally failing or disabling tests mean that the test is to be failed or disabled pending a certain condition is true.
skip-if = device == "desktop" && os == "mac"
fail-if = os = "mac"
Unconditionally x-failing or disabling tests mean that the test will fail on every OS, Device and scenario. Be sure to do thorough investigation before unconditionally x-failing a test.
disabled = Bug <Bug#> - <Bug Description Verbatim>
expected = fail