Confirmed users
600
edits
Sheriffing/How To/Data Ingestion Backlog: Difference between revisions
Sheriffing/How To/Data Ingestion Backlog (view source)
Revision as of 14:38, 25 February 2025
, 25 Februaryadd Treeherder dashboards
(add category Sheriffing How To) |
(add Treeherder dashboards) |
||
| Line 6: | Line 6: | ||
* Treeherder fails to access the Pulse queue’s notification, process it or is slowed down, e.g. by too many or too expensive database operations. | * Treeherder fails to access the Pulse queue’s notification, process it or is slowed down, e.g. by too many or too expensive database operations. | ||
== | == Dashboard-based identification of data pipeline part in which the issue starts == | ||
* [https://yardstick.mozilla.org/d/ZPy4RBX7z/treeherder-rabbitmq?orgId=1&from=now-6h&to=now&timezone=browser&var-environment=prod&var-datasource=cdq6ttvymu4g0c&var-app=treeherder&var-queue=log_parser Treeherder (RabbitMQ)]: RabbitMQ is the system which sends Treeherder notifications about tasks switching state, e.g. when they start to run, or when logs are ready to parse. This dashboard shows the backlog of unprocessed messages which answers the question if tasks not being shown in Treeherder is an issue with Treeherder or the Taskcluster instance at https://firefox-ci-tc.services.mozilla.com/ . If it shows a backlog, Treeherder is the bottleneck else the Taskcluster instance does not emit the messages. | |||
* [https://yardstick.mozilla.org/d/rBBO6ajMz/treeherder?orgId=1&from=now-30m&to=now&timezone=browser&var-environment=nonprod&var-data_source=bdq6tqxmbgef4f Treeherder general]: A more general dashboard which will be extended. | |||
== Manual identification of data pipeline part in which the issue starts == | |||
* Taskcluster or not: Shortly after a task got shown with delay (e.g. a gecko decision task shown much later than the push), select it in Treeherder and then click at the bottom left on the link to its Taskcluster page. | * Taskcluster or not: Shortly after a task got shown with delay (e.g. a gecko decision task shown much later than the push), select it in Treeherder and then click at the bottom left on the link to its Taskcluster page. | ||