B2G/QA/Automation/UI/Xfail and Disable

From MozillaWiki
< B2G‎ | QA‎ | Automation‎ | UI
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

X-Failing and Disabling Tests

Where To Do What?

To disable or x-fail a test, you would make an update to the manifest.ini file that is for that particular app.

When To Do What?

X-Fail (Expected Fail)

If there is a problem with the application under test (such as a bug that causes the test to always fail) then you would X-Fail the test.

View this bug’s pull request to see an example of x-failing a test.

Disable (Turn off)

If there is a problem with the test that causes it to fail, then you would Disable the test. Things that fall under this category would be:

  • The selectors were changed and the test needs to be updated
  • Intermittent failures (Never X-Fail an intermittent test because you will wind up with upasses (unexpected passes))

View this bug’s pull request to see an example of a test being disabled.

Conditional Vs. Unconditional X-Failing/Disabling of Tests

Conditional

Conditionally failing or disabling tests mean that the test is to be failed or disabled pending a certain condition is true.

Conditional Disable

skip-if = device == "desktop" && os == "mac"

Conditional X-Fail

fail-if = os = "mac"

Unconditional

Unconditionally x-failing or disabling tests mean that the test will fail on every OS, Device and scenario. Be sure to do thorough investigation before unconditionally x-failing a test.

Unconditional Disable

disabled = Bug <Bug#> - <Bug Description Verbatim>

Unconditional X-Fail

expected = fail
Retrieved from "https://wiki.mozilla.org/index.php?title=B2G/QA/Automation/UI/Xfail_and_Disable&oldid=1061512"