Confirmed users
603
edits
Sheriffing/How To/Updating test expectations: Difference between revisions
Jump to navigation
Jump to search
Sheriffing/How To/Updating test expectations (view source)
Revision as of 14:35, 9 February 2022
, 9 February 2022add reftests
(→web-platform-tests: highlight to not copy the job symbol with the job name) |
(add reftests) |
||
| Line 2: | Line 2: | ||
* web platform tests (wpt) are regularly imported from an external repository and the features tested might not have been implemented in Firefox yet. The import script sets the expectations based on Try pushes but the expected result can have changed because the feature got implemented between the Try push and the wpt test synchronization. | * web platform tests (wpt) are regularly imported from an external repository and the features tested might not have been implemented in Firefox yet. The import script sets the expectations based on Try pushes but the expected result can have changed because the feature got implemented between the Try push and the wpt test synchronization. | ||
* Tests developed by Mozilla are expected to pass but can be disabled for some platform configurations if they fail, e.g. new versions of operating systems often fail some tests and get added with tier 2 priority. The tests are initially skipped and bugs filed to fix the issue and reenable the test. | * Tests developed by Mozilla are expected to pass but can be disabled for some platform configurations if they fail, e.g. new versions of operating systems often fail some tests and get added with tier 2 priority. The tests are initially skipped and bugs filed to fix the issue and reenable the test. | ||
== reftests == | |||
Reftests display content ("paint it") with one technology and once again with others and compare if the results match. | |||
A failure message could be: | |||
<code>REFTEST TEST-UNEXPECTED-FAIL | layout/reftests/transform/partial-prerender-in-svg-1.html == layout/reftests/transform/partial-prerender-in-svg-1-ref.html | image comparison, max difference: 255, number of differing pixels: 400</code> | |||
The first number is the maximum difference for a pixel byte while the second one is the count of different pixels. These get used when a test gets set as intermittent: | |||
<code>fuzzy-if(Android,0-255,0-400)</code> | |||
The first parameter is the condition. The second one allows between 0 and 255 as difference for a byte, and the third parameter allows between 0 and 400 different pixels. | |||
* The functions available for annotations can be found in the [https://searchfox.org/mozilla-central/source/layout/tools/reftest/README.txt documentation in the repository]. | |||
* The variables available and their values are listed in each reftest log near the end of the setup. Search for "Dumping representation of sandbox which can be used for expectation annotations" in the log. | |||
== web-platform-tests == | == web-platform-tests == | ||