B2G/QA/Automation/UI/Xfail and Disable: Difference between revisions
(X-Failing and Disabling Tests) |
|||
| Line 28: | Line 28: | ||
Conditionally failing or disabling tests mean that the test is to be failed or disabled pending a certain condition is true. | Conditionally failing or disabling tests mean that the test is to be failed or disabled pending a certain condition is true. | ||
=== Conditional Disable === | ==== Conditional Disable ==== | ||
<pre>skip-if = device == "desktop" && os == "mac"</pre> | <pre>skip-if = device == "desktop" && os == "mac"</pre> | ||
=== Conditional X-Fail === | ==== Conditional X-Fail ==== | ||
<pre>fail-if = os = "mac"</pre> | <pre>fail-if = os = "mac"</pre> | ||
| Line 38: | Line 38: | ||
Unconditionally x-failing or disabling tests mean that the test will fail on every OS, Device and scenario. Be sure to do thorough investigation before unconditionally x-failing a test. | Unconditionally x-failing or disabling tests mean that the test will fail on every OS, Device and scenario. Be sure to do thorough investigation before unconditionally x-failing a test. | ||
=== Unconditional Disable === | ==== Unconditional Disable ==== | ||
<pre>disabled = Bug <Bug#> - <Bug Description Verbatim></pre> | <pre>disabled = Bug <Bug#> - <Bug Description Verbatim></pre> | ||
=== Unconditional X-Fail === | ==== Unconditional X-Fail ==== | ||
<pre>expected = fail</pre> | <pre>expected = fail</pre> | ||
Revision as of 19:10, 6 March 2015
X-Failing and Disabling Tests
Where To Do What?
To disable or x-fail a test, you would make an update to the manifest.ini file that is for that particular app.
When To Do What?
X-Fail (Expected Fail)
If there is a problem with the application under test (such as a bug that causes the test to always fail) then you would X-Fail the test.
View this bug’s pull request to see an example of x-failing a test.
Disable (Turn off)
If there is a problem with the test that causes it to fail, then you would Disable the test. Things that fall under this category would be:
- The selectors were changed and the test needs to be updated
- Intermittent failures (Never X-Fail an intermittent test because you will wind up with upasses (unexpected passes))
View this bug’s pull request to see an example of a test being disabled.
Conditional Vs. Unconditional X-Failing/Disabling of Tests
Conditional
Conditionally failing or disabling tests mean that the test is to be failed or disabled pending a certain condition is true.
Conditional Disable
skip-if = device == "desktop" && os == "mac"
Conditional X-Fail
fail-if = os = "mac"
Unconditional
Unconditionally x-failing or disabling tests mean that the test will fail on every OS, Device and scenario. Be sure to do thorough investigation before unconditionally x-failing a test.
Unconditional Disable
disabled = Bug <Bug#> - <Bug Description Verbatim>
Unconditional X-Fail
expected = fail