What is test-verify

Test-Verify is a tool that runs a single test (or set of tests) multiple times in various configs to ensure the test is stable. The goal is to provide a solution for reducing intermittents over time by ensuring tests are not affected by or affecting other tests and they are not subject to basic timing issues.

This is run when we detect testfiles that are edited (including added) on all trees and shows up with the TV symbol on treeherder (TV-x for cases of >10 edited cases, and TVw[-x] for web platform).

We have the ability to run this on-demand with test-verify-backfill (TV-bf on treeherder) which is more experimental than TV is right now. The intention here is to make test-verify output a strong signal and we can prove that it fails on a given commit, but passes on the previous commit, therefore identifying a regression.

How to run TV-bf

A quick summary of how to use TV-bf:

• select a job with a failure
• in the summary pane below, next to the retrigger button, click the "..." button
• this drops down a list, select "Custom Action..." from the bottom of the list
• choose the "Backfill" action
• edit the payload text area:

depth: 1
inclusive: true
times: 3
testPath: <path>

• here you have to add the testPath line and change out <path> for the failing test. In most cases it is this format in the failure message: TEST-UNEXPECTED-FAIL | <path> | reason...

Once you have a <path> entered, click "Trigger" and TV-bf will be there on your push as well as the previous push. If the job is green, please make sure the test ran, do this by:

• click on the green TV-bf job
• in the preview pane scroll to the bottom of the "Job Details" tab, and look for "per-test run...", if that isn't there then the test <path> was incorrect or there were other errors.