Sheriffing/How:To:Comm-central

This page is mostly just a collection of notes for managing comm-central in treeherder, https://treeherder.mozilla.org/#/jobs?repo=comm-central.

Starring a test failure

Select the build you want to star, and then click the pin button (looks like a push pin) in the top of the lower-left panel. The bottom half of the panel displays the test failures along with suggestions for matching bugs if available.
You can select more builds by Ctrl-clicking them. Please do so if you are starring multiple builds with the same bug.
Add a bug number by either hitting the "+" icon in the top-right of the panel and typing it in, or hitting the pin icon next to the bug number in the summary in the failure summary.
Just because a bug number is suggested doesn't mean it's correct. Do ascertain that it is relevant first.

Click on the orange/red test build
Under "Failure Summary", find the new test failure you want to file.
Copy the test failure to a new bug. (Thunderbird::Test Infrastructure or MailNews Core::Test Infrastructure is a good place to dump issues with infa, you can also use Thunderbird::General if it's not an infra issue and you don't know exactly where it belongs)
Add the intermittent-failure keyword. This is important, it will not show up in suggestions if you do not have this.
Copy sufficient details from the bug to diagnose the results. Use the log viewer, it actually works better than TBPL's.
Make sure to add the new bug to the tree status etherpad.

Note: Newly-filed bugs do not show up in the test failure suggestions box immediately.

Deny approval if there is not at least 1 "successful" debug and opt version of tests on the latest checkin. This isn't a "everything before must go green" but "the previous checkin did not break everybody" check.
Triage all problems. Get the unstarred failure count down to 0, or at least check the the tree status etherpad to ensure all failures are known and filed.
Do we have builds on all platforms? If no, and the patch is not a bustage fix, deny approval.
Are the test failure counts going up? If there are too many reports, you can check the number of tests that failed by looking in the lower-left panel: it will say something like mozmill virtualenv 1090/7. If that count goes up, there are probably new mozilla-central failures to be triaged.
New mozilla-central failures? Confirm regression range by looking for the mozilla-central revision numbers printed on the build steps (not currently working on Treeherder). If those are the same between the previous build and the current one, then it isn't a mozilla-central regression. If those are different, then it could very well be.
If there are new failures, please file bugs (see above) or back out patches.
If the last checkin contains a platform-specific patch, wait for tests on that platform to complete before approving the patch.

The primary purpose of moving to APPROVAL NEEDED is to throttle patches and ensure that new regressions are not introduced when checking in patches.