canmove, Confirmed users
1,126
edits
Sheriffing/Job Visibility Policy: Difference between revisions
Jump to navigation
Jump to search
m
→6) Outputs failures in a TBPL-starrable format: Add link, tweak wording
m (→5) Easily run on try server: Wording) |
m (→6) Outputs failures in a TBPL-starrable format: Add link, tweak wording) |
||
| Line 35: | Line 35: | ||
=== 6) Outputs failures in a TBPL-starrable format === | === 6) Outputs failures in a TBPL-starrable format === | ||
* It is highly recommended that new test harnesses do not reinvent the wheel and instead use parts of MozBase (eg: mozcrash) if at all possible - speak to the A-Team for more info. | * It is highly recommended that new test harnesses do not reinvent the wheel and instead use parts of MozBase (eg: mozcrash) if at all possible - speak to the [[Auto-tools|A-Team]] for more info. | ||
* Failures must appear in the TBPL annotated summary (ie: | * Failures must appear in the TBPL annotated summary (ie: matches the [https://hg.mozilla.org/webtools/tbpl/file/tip/php/inc/GeneralErrorFilter.php log parsing regexp]), otherwise the full log will have to be opened for every failure. | ||
* Failure output must be in the format expected by TBPL's [https://hg.mozilla.org/webtools/tbpl/file/tip/php/inc/AnnotatedSummaryGenerator.php bug suggestion generator] (otherwise sheriffs have to manually search Bugzilla when starring intermittent failures): | * Failure output must be in the format expected by TBPL's [https://hg.mozilla.org/webtools/tbpl/file/tip/php/inc/AnnotatedSummaryGenerator.php bug suggestion generator] (otherwise sheriffs have to manually search Bugzilla when starring intermittent failures): | ||
** For in-tree/product issues (eg: test failures, crashes): | ** For in-tree/product issues (eg: test failures, crashes): | ||
| Line 44: | Line 44: | ||
*** 3rd token: The specific failure message (eg: the test part that failed, the top frame of a crash or the leaked objects list for a leak). | *** 3rd token: The specific failure message (eg: the test part that failed, the top frame of a crash or the leaked objects list for a leak). | ||
** For non test-specific issues (eg: infra/automation/harness): | ** For non test-specific issues (eg: infra/automation/harness): | ||
*** TBPL falls back to searching Bugzilla for the entire failure line (excluding mozharness logging prefix), so it should be both unique to that failure type & repeatable (ie: no use of process IDs for which there will rarely be a repeat match against a bug summary). | *** TBPL falls back to searching Bugzilla for the entire failure line (excluding mozharness logging prefix), so it should be both unique to that failure type & repeatable (ie: no use of process IDs or timestamps, for which there will rarely be a repeat match against a bug summary). | ||
** Exceptions & timeouts must be handled with appropriate log output (eg: the failure line must state in which test the timeout occurred, not just that the entire run has timed out). | ** Exceptions & timeouts must be handled with appropriate log output (eg: the failure line must state in which test the timeout occurred, not just that the entire run has timed out). | ||
* The sheriffs will be happy to advise regarding | * The sheriffs will be happy to advise regarding the above. | ||
=== 7) Low intermittent failure rate === | === 7) Low intermittent failure rate === | ||