Sheriffing/How To/Updating test expectations

Sometimes test expectations (if they pass, fail, time out etc.) need to be adjusted:

• web platform tests (wpt) are regularly imported from an external repository and the features tested might not have been implemented in Firefox yet. The import script sets the expectations based on Try pushes but the expected result can have changed because the feature got implemented between the Try push and the wpt test synchronization.
• Tests developed by Mozilla are expected to pass but can be disabled for some platform configurations if they fail, e.g. new versions of operating systems often fail some tests and get added with tier 2 priority. The tests are initially skipped and bugs filed to fix the issue and reenable the test.

reftests

Reftests display content ("paint it") with one technology and once again with others and compare if the results match.

A failure message could be:

REFTEST TEST-UNEXPECTED-FAIL | layout/reftests/transform/partial-prerender-in-svg-1.html == layout/reftests/transform/partial-prerender-in-svg-1-ref.html | image comparison, max difference: 255, number of differing pixels: 400

The first number is the maximum difference for a pixel byte while the second one is the count of different pixels. These get used when a test gets set as intermittent:

fuzzy-if(Android,0-255,0-400)

The first parameter is the condition. The second one allows between 0 and 255 as difference for a byte, and the third parameter allows between 0 and 400 different pixels.

• The functions available for annotations can be found in the documentation in the repository.
• The variables available and their values are listed in each reftest log near the end of the setup. Search for "Dumping representation of sandbox which can be used for expectation annotations" in the log.

web-platform-tests

Expectation update from the command line

1. Open a console window.
2. Switch to the parent folder of the mozilla-unified directory with the cd command.
3. Create a folder named logs:
   mkdir logs
4. Install the following dependency:
   sudo apt-get install libssl-dev
5. Clone the script to download several logs for wpt tasks:
   git clone https://github.com/jgraham/fetchlogs.git
   You can also download it as .zip file but won't be able to update it from the command line in that case.
6. Switch to the folder with newly downloaded source code:
   cd fetchlogs
7. In Treeherder, identify the tasks whose unexpected test results should be used to update test expectations.
8. In the console, download the logs which a command similar to the following one:
   cargo run mozilla-central e40a136dc8760f3fdd8a29fad47581d16d646c80 --out-dir ../logs/ --filter-jobs test-linux1804-32-shippable-qr/opt-web-platform-tests-e10s-10
   Replace mozilla-central with the name of the tree and e40a136dc8760f3fdd8a29fad47581d16d646c80 with the revision of the push from which the logs shall be downloaded.
   --filter-jobs expects the name of the task as parameter (found at the bottom left in Treeherder) - or a regular expression to match multiple task names. If omitted, the log for every wpt task for that push will be downloaded. Don't copy the job symbol mentioned at the end.
   This depends on the Rust programming language which should be installed since ./mach bootstrap had been run in mozilla-unified
9. Switch to mozilla-unified
cd ../mozilla-unified/
10. Update the files with the test expectations:
    ./mach wpt-update ../logs/* --update-intermittent
    The log output should be similar to:
    0:00.03 INFO Skipping manifest download because existing file is recent
Processing log 1/3
Processing log 2/3
Processing log 3/3
11. Check which files have been updated:
    hg status
    There are very likely more files modified (M) or added (A) than desired.
12. Revert the unintentional changes:
    hg revert -r . -X "path:testing/web-platform/meta/<continuation_of_path>/"
    Multiple paths can be kept in the current state by using the -X ("exclude") pattern multiple times.
13. Check hg status again.
14. Save the changes with hg commit -e