X-Failing and Disabling Tests

Where To Do What?

To disable or x-fail a test, you would make an update to the manifest.ini file that is for that particular app.

When To Do What?

X-Fail (Expected Fail)

If there is a problem with the application under test (such as a bug that causes the test to always fail) then you would X-Fail the test.

View this bug’s pull request to see an example of x-failing a test.

Disable (Turn off)

If there is a problem with the test that causes it to fail, then you would Disable the test. Things that fall under this category would be:

• The selectors were changed and the test needs to be updated
• Intermittent failures (Never X-Fail an intermittent test because you will wind up with upasses (unexpected passes))

View this bug’s pull request to see an example of a test being disabled.

Conditional Vs. Unconditional X-Failing/Disabling of Tests

Conditional

Conditionally failing or disabling tests mean that the test is to be failed or disabled pending a certain condition is true.

Conditional Disable

skip-if = device == "desktop" && os == "mac"

Conditional X-Fail

fail-if = os = "mac"

Unconditional

Unconditionally x-failing or disabling tests mean that the test will fail on every OS, Device and scenario. Be sure to do thorough investigation before unconditionally x-failing a test.

Unconditional Disable

disabled = Bug <Bug#> - <Bug Description Verbatim>

Unconditional X-Fail

expected = fail