Sheriffing/How To/Treeherder - Revision history

Archaeopteryx: /* Pushes and tasks not ingested or with delay */ fix typo

2025-11-17T19:52:23Z

Pushes and tasks not ingested or with delay: fix typo

← Older revision		Revision as of 19:52, 17 November 2025
Line 121:		Line 121:
	== Pushes and tasks not ingested or with delay ==		== Pushes and tasks not ingested or with delay ==
	* Open [https://sql.telemetry.mozilla.org/queries/92157 this query] and press "Execute". This will show the latest time any task got requested, started to run or finished. If there is no recent time, the task ingestion is delayed or stopped.		* Open [https://sql.telemetry.mozilla.org/queries/92157 this query] and press "Execute". This will show the latest time any task got requested, started to run or finished. If there is no recent time, the task ingestion is delayed or stopped.
	* In [https://yardstick.mozilla.org/d/bh8mbjf/taskcluster-treeherder-task-latency?orgId=1&from=now-6h&to=now&timezone=browser this dashboard], e.g. the ~~graphs~~ "Treeherder Processing: *" charts would show lines dropping close to zero or disappearing if there is an issue to receive or process messages.		* In [https://yardstick.mozilla.org/d/bh8mbjf/taskcluster-treeherder-task-latency?orgId=1&from=now-6h&to=now&timezone=browser this dashboard], e.g. the "Treeherder Processing: *" charts would show lines dropping close to zero or disappearing if there is an issue to receive or process messages.

	== Logs not parsed or with delay ==		== Logs not parsed or with delay ==
	Open [https://sql.telemetry.mozilla.org/queries/85738 this query] and press "Execute". This will show the time when the newest task ended whose logs have been parsed - not the last time a log got parsed. The difference to the current time is the log parsing delay.		Open [https://sql.telemetry.mozilla.org/queries/85738 this query] and press "Execute". This will show the time when the newest task ended whose logs have been parsed - not the last time a log got parsed. The difference to the current time is the log parsing delay.

Archaeopteryx: /* Pushes and tasks not ingested or with delay */ add Grafana dashboard for Pulse messages

2025-11-17T19:11:02Z

Pushes and tasks not ingested or with delay: add Grafana dashboard for Pulse messages

← Older revision		Revision as of 19:11, 17 November 2025
Line 120:		Line 120:

	== Pushes and tasks not ingested or with delay ==		== Pushes and tasks not ingested or with delay ==
	Open [https://sql.telemetry.mozilla.org/queries/92157 this query] and press "Execute". This will show the latest time any task got requested, started to run or finished. If there is no recent time, the task ingestion is delayed or stopped.		* Open [https://sql.telemetry.mozilla.org/queries/92157 this query] and press "Execute". This will show the latest time any task got requested, started to run or finished. If there is no recent time, the task ingestion is delayed or stopped.
			* In [https://yardstick.mozilla.org/d/bh8mbjf/taskcluster-treeherder-task-latency?orgId=1&from=now-6h&to=now&timezone=browser this dashboard], e.g. the graphs "Treeherder Processing: *" charts would show lines dropping close to zero or disappearing if there is an issue to receive or process messages.

	== Logs not parsed or with delay ==		== Logs not parsed or with delay ==
	Open [https://sql.telemetry.mozilla.org/queries/85738 this query] and press "Execute". This will show the time when the newest task ended whose logs have been parsed - not the last time a log got parsed. The difference to the current time is the log parsing delay.		Open [https://sql.telemetry.mozilla.org/queries/85738 this query] and press "Execute". This will show the time when the newest task ended whose logs have been parsed - not the last time a log got parsed. The difference to the current time is the log parsing delay.

Archaeopteryx: fix link format for issue handling

2024-03-11T13:57:59Z

fix link format for issue handling

← Older revision		Revision as of 13:57, 11 March 2024
Line 120:		Line 120:

	== Pushes and tasks not ingested or with delay ==		== Pushes and tasks not ingested or with delay ==
	[~~Open this query~~ https://sql.telemetry.mozilla.org/queries/92157] and press "Execute". This will show the latest time any task got requested, started to run or finished. If there is no recent time, the task ingestion is delayed or stopped.		Open [https://sql.telemetry.mozilla.org/queries/92157 this query] and press "Execute". This will show the latest time any task got requested, started to run or finished. If there is no recent time, the task ingestion is delayed or stopped.

	== Logs not parsed or with delay ==		== Logs not parsed or with delay ==
	[~~Open this query~~ https://sql.telemetry.mozilla.org/queries/85738] and press "Execute". This will show the time when the newest task ended whose logs have been parsed - not the last time a log got parsed. The difference to the current time is the log parsing delay.		Open [https://sql.telemetry.mozilla.org/queries/85738 this query] and press "Execute". This will show the time when the newest task ended whose logs have been parsed - not the last time a log got parsed. The difference to the current time is the log parsing delay.

Archaeopteryx: add queries for log parsing delay and push & task ingestion delay

2023-06-28T14:23:05Z

add queries for log parsing delay and push & task ingestion delay

← Older revision		Revision as of 14:23, 28 June 2023
Line 116:		Line 116:

	More information about the Treeherder project can be found [[Auto-tools/Projects/Treeherder\|here]].		More information about the Treeherder project can be found [[Auto-tools/Projects/Treeherder\|here]].

			= Issue handling =

			== Pushes and tasks not ingested or with delay ==
			[Open this query https://sql.telemetry.mozilla.org/queries/92157] and press "Execute". This will show the latest time any task got requested, started to run or finished. If there is no recent time, the task ingestion is delayed or stopped.

			== Logs not parsed or with delay ==
			[Open this query https://sql.telemetry.mozilla.org/queries/85738] and press "Execute". This will show the time when the newest task ended whose logs have been parsed - not the last time a log got parsed. The difference to the current time is the log parsing delay.

Sclements: Changed OrangeFactor link to Intermittent Failures View (OF is being decommissioned)

2018-06-04T16:26:56Z

Changed OrangeFactor link to Intermittent Failures View (OF is being decommissioned)

← Older revision		Revision as of 16:26, 4 June 2018
Line 79:		Line 79:

	== What does it do? ==		== What does it do? ==
	# Submits the failure and classification to [https://~~brasstacks~~.mozilla.~~com~~/~~orangefactor/ OrangeFactor~~]. This dashboard helps track trends in classified failures, so failures that happen frequently (or suddenly start failing frequently) can be prioritized.		# Submits the failure and classification to [https://treeherder.mozilla.org/intermittent-failures.html Intermittent Failures View]. This dashboard helps track trends in classified failures, so failures that happen frequently (or suddenly start failing frequently) can be prioritized.
	# Stores the classification comment and any associated bugs in Treeherders's database, so everyone viewing that tree/push can see that the failure has been starred (ie annotated with a comment and the star icon added next to the job symbol), so they know it has been dealt with.		# Stores the classification comment and any associated bugs in Treeherders's database, so everyone viewing that tree/push can see that the failure has been starred (ie annotated with a comment and the star icon added next to the job symbol), so they know it has been dealt with.

ChrisCooper: /* Starring many failures at once */

2017-08-30T20:31:41Z

Starring many failures at once

← Older revision		Revision as of 20:31, 30 August 2017
Line 106:		Line 106:
	* a system configuration change goes awry		* a system configuration change goes awry

	~~This manifests~~ as lots on concurrent failures displayed in Treeherder. Luckily, Treeherder allows you to pin up to 500 jobs at one time, greatly speeding up the starring process.		These types of issues manifest as lots on concurrent failures as displayed in Treeherder. Luckily, Treeherder allows you to pin up to 500 jobs at one time, greatly speeding up the starring process. The process you use is similar for each situation, and the specifics depend on how pervasive the failure is.

	If the failure is pervasive, i.e. platform-independent, you can use Treeherder to show unclassified failures. Pin all the failures for a single push, and then mark them as "fixed by commit." Repeat until it's done for that single push, then move on to the next push.		* If the failure is pervasive, i.e. platform-independent, you can use Treeherder to show all unclassified failures. Pin all the failures for a single push, and then mark them as "fixed by commit." Repeat until it's done for that single push, then move on to the next push.
			* If only a subset of platforms are affected, i.e. all on one build target or test suites, you can filter on the top right of the Treeherder interface with the search field and also filter for the job name (substring) which is shown at the bottom left if you select a job
	If only a subset of platforms are affected, i.e. all on one build target or test suites, you can filter on the top right of the Treeherder interface with the search field and also filter for the job name (substring) which is shown at the bottom left if you select a job		* If the failure doesn't affect all test jobs of one test suite on the same platform and build target, you will need to manually go through the list and add them to the pinboard before marking them as "fixed by commit." This reduces the likelihood of mis-starring.
			** If it's an older regression which has been resolved many pushes ago, selecting and starring all these test failures as "fixed by commit" might outweigh the time needed to do it manually. Other regressions should have been caught by the later pushes after the backout of the obvious test failure.
	If the failure doesn't affect all test jobs of one test suite on the same platform and build target, you will need to manually go through the list and add them to the pinboard before marking them as "fixed by commit." This reduces the likelihood of mis-starring.

	If it's an older regression which has been resolved many pushes ago, selecting and starring all these test failures as "fixed by commit" might outweigh the time needed to do it manually. Other regressions should have been caught by the later pushes after the backout of the obvious test failure.

	= More Treeherder Resources =		= More Treeherder Resources =

	More information about the Treeherder project can be found [[Auto-tools/Projects/Treeherder\|here]].		More information about the Treeherder project can be found [[Auto-tools/Projects/Treeherder\|here]].

ChrisCooper: /* Starring many failures at once */

2017-08-30T19:56:56Z

Starring many failures at once

← Older revision		Revision as of 19:56, 30 August 2017
Line 108:		Line 108:
	This manifests as lots on concurrent failures displayed in Treeherder. Luckily, Treeherder allows you to pin up to 500 jobs at one time, greatly speeding up the starring process.		This manifests as lots on concurrent failures displayed in Treeherder. Luckily, Treeherder allows you to pin up to 500 jobs at one time, greatly speeding up the starring process.

	If the failure is pervasive, i.e. platform-independent, you can use Treeherder to show unclassified failures. Pin all the failures for a single push, and then mark them as "fixed by commit." ~~so repeat~~ until it's done for that single push, then move on to the next push.		If the failure is pervasive, i.e. platform-independent, you can use Treeherder to show unclassified failures. Pin all the failures for a single push, and then mark them as "fixed by commit." Repeat until it's done for that single push, then move on to the next push.

	If only a subset of platforms are affected, i.e. all on one build target or test suites, you can filter on the top right of the Treeherder interface with the search field and also filter for the job name (substring) which is shown at the bottom left if you select a job		If only a subset of platforms are affected, i.e. all on one build target or test suites, you can filter on the top right of the Treeherder interface with the search field and also filter for the job name (substring) which is shown at the bottom left if you select a job

ChrisCooper: /* Classifying/Starring jobs */

2017-08-30T19:53:08Z

Classifying/Starring jobs

@@ Line 99: / Line 99: @@
 That's it! You've classified/starred a job! You can move on to a different failed job.
 = More Treeherder Resources =
 More information about the Treeherder project can be found [[Auto-tools/Projects/Treeherder|here]].

ChrisCooper: ChrisCooper moved page Sheriffing/How:To:Treeherder to Sheriffing/How To/Treeherder: More intuitive naming scheme

2017-07-28T16:53:10Z

ChrisCooper moved page Sheriffing/How:To:Treeherder to Sheriffing/How To/Treeherder: More intuitive naming scheme

← Older revision	Revision as of 16:53, 28 July 2017
(No difference)

ChrisCooper at 16:46, 28 July 2017

2017-07-28T16:46:02Z

← Older revision		Revision as of 16:46, 28 July 2017
Line 1:		Line 1:
	~~[[Category:~~Sheriffing~~:HowTo]]~~		{{Sheriffing How To\|Treeherder}}

	Treeherder is the dashboard used by sheriffs and the rest of Mozilla to see build and test results. When you first open [https://treeherder.mozilla.org Treeherder], there will be a lot of information thrown at you all at once. This document should help explain what to do with it all.		Treeherder is the dashboard used by sheriffs and the rest of Mozilla to see build and test results. When you first open [https://treeherder.mozilla.org Treeherder], there will be a lot of information thrown at you all at once. This document should help explain what to do with it all.
	[[File:Treeherder_intermittent.png\|thumbnail\|right\|Treeherder interface showing an intermittent failure]]		[[File:Treeherder_intermittent.png\|thumbnail\|right\|Treeherder interface showing an intermittent failure]]