EngineeringProductivity/Projects/Treeherder: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
m (→‎TBPL switch-off tasks: Add product & component columns to tbpl-eol bugs table)
(Clean up out of date sections)
 
(85 intermediate revisions by 9 users not shown)
Line 1: Line 1:
== Overview ==
== About ==
[https://treeherder.mozilla.org/ Treeherder] is the successor to [[Sheriffing/TBPL|TBPL]].
[https://treeherder.mozilla.org/ Treeherder] is a reporting dashboard for checkins to Mozilla projects (for example, [https://developer.mozilla.org/en-US/docs/mozilla-central mozilla-central]). It allows users to see the results of automatic builds and their respective tests. Treeherder also provides a rich set of APIs that can be used by other projects interested in this information.


The goal of the first version of TBPL was to correlate checkins with Buildbot results, so that developers and sheriffs could quickly see test results across multiple platforms for automatic builds associated with a developer's specific Firefox changesets. As more test types, platforms, products and features were added to TBPL, a variety of architectural bottle necks became apparent.  
Treeherder is the successor to [[Sheriffing/TBPL|TBPL]].


The purpose of Treeherder is to support longer term use-cases for managing and visualizing data and rewrite the existing application from scratch, so that it can scale appropriately in the future.
For tracking performance data, see Treeherder's sister project, [[EngineeringProductivity/Projects/Perfherder|Perfherder]].
 
For tracking intermittent test failures, see [[EngineeringProductivity/Projects/IntermittentFailuresView|Intermittent Failures View]].


== Contributing ==
== Contributing ==
* Good first bugs for new developers: [http://www.joshmatthews.net/bugsahoy/?reporting=1 Bugs Ahoy]
To make UI-only changes, you only need to perform a very simple setup running a local webserver pointing at the production backend, described [https://treeherder.readthedocs.io/installation.html#ui-development here]. If you wish to hack on the backend, or the UI and backend together, you will instead need to set up a Vagrant environment, using [https://treeherder.readthedocs.io/installation.html#server-and-full-stack-development these steps].
* Bugzilla: [https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree+Management&component=Treeherder File a bug] / [https://bugzilla.mozilla.org/buglist.cgi?product=Tree+Management&component=Treeherder&resolution=--- All open bugs] / [https://bugzilla.mozilla.org/buglist.cgi?component=Treeherder&resolution=FIXED&chfieldto=Now&chfield=resolution&chfieldfrom=-14d&chfieldvalue=FIXED Recently fixed] / [[#Bug_Tables|Bug tables]]
 
* IRC: [irc://irc.mozilla.org/treeherder #treeherder] / [http://logs.glob.uno/?c=treeherder logs]
* [https://ateam-bootcamp.readthedocs.io/en/latest/ A-Team Bootcamp]: Best practices for working on A-Team projects (of which Treeherder is one), including valuable information on using Git and Bugzilla. If you're new to Mozilla or the A-Team, please read this guide thoroughly before proceeding.
* Meetings: [[Auto-tools/Projects/Treeherder/Meetings|Notes & dial-in details]]
* Good first bugs for new developers: [https://codetribute.mozilla.org/projects/treeherder Codetribute]
* Mozilla instances: [http://treeherder-dev.allizom.org Dev] / [https://treeherder.allizom.org Staging] / [https://treeherder.mozilla.org Production]
* Issue tracker: [https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree+Management&component=Treeherder Report a bug] / [https://bugzilla.mozilla.org/query.cgi?query_format=advanced&product=Tree+Management&f1=component&o1=substring&v1=Treeherder&resolution=--- search open bugs]
* Mozilla Treeherder instances: [https://treeherder.allizom.org Staging] / [https://treeherder.mozilla.org Production] / [https://whatsdeployed.io/s-dqv Deployed changes info]


== Source and Docs ==
== Source and Docs ==
* Treeherder backend: [https://github.com/mozilla/treeherder-service Source] / [https://treeherder-service.readthedocs.org Docs] / [http://treeherder-dev.allizom.org/docs/ API Docs]
* UI & backend: [https://github.com/mozilla/treeherder Source] / [https://treeherder.readthedocs.io Docs]
* Treeherder UI: [https://github.com/mozilla/treeherder-ui Source] / [https://treeherder-ui.readthedocs.org Docs]
* Treeherder data submission clients:
* Treeherder data submission clients:
** Python: [https://github.com/mozilla/treeherder-client Source] / [https://github.com/mozilla/treeherder-client/blob/master/README.md Readme]
** Pulse: [https://hg.mozilla.org/automation/mozillapulse/ Source] / [http://treeherder.readthedocs.io/submitting_data.html#submitting-using-pulse docs]
** NodeJS: [https://github.com/lightsofapollo/treeherder-node Source] / [https://github.com/lightsofapollo/treeherder-node/blob/master/README.md Readme]
** (Python and Node submission support has been removed.)
 
== Roadmap ==
==== Milestone M1: Functional Prototype ====
* Deliver a functional prototype that sheriffs can use for E2E testing.
* Status: {{done|}} delivered 2014 Q1.
==== Milestone M2: Production Deployment ====
* Complete a security review and deploy into production with IT.
* Status: {{done|}} delivered 2014 Q2.
==== Milestone M3: Sheriff Transition from TBPL ====
* Work with the sheriffs to address bugs and workflow issues so that Treeherder is a viable alternate for TBPL for their workflow, and transition the sheriffs to using Treeherder for their day-to-day work. Bugs: see dependents of {{bug|1030636}}.
* Status: {{done|}} delivered 2014 Q3 ([https://groups.google.com/forum/#!topic/mozilla.dev.tree-management/WSQY2EYgxz8 newsgroup post]).
 
==== Milestone M4: Make TBPL redundant for remaining users ====
* Primary goal:
** M4.1: Work to implement other developer use cases, refine sheriffing work flows, and make TBPL redundant for all of its users. (Note: Switching TBPL off will come later). Bugs: see [[#TBPL_regressions]] below.
* Secondary goals (may be deferred to later milestones):
** M4.2: Improve support for submitting correctness data from non-buildbot sources such as TaskCluster and Autophone. Bugs: see [[#Data_submission_bugs]] below.
** M4.3: General cleanup to make Treeherder development more accessible to others. Bugs: see [[#Hacking_UX_bugs]] below.
* Status: {{ok|}} for delivery 2014 Q4.


==== Milestone M5: Performance Data ====
== Getting in touch ==
* We want to obsolete Graphserver and Datazilla, and make Treeherder the dashboard that is used to visualize performance data generated by Talos and other sources.  This will be comprised of at least the following:
* Chat on IRC: [irc://irc.mozilla.org/treeherder #treeherder] / [[IRC|learn about IRC]] / [https://mozilla.logbot.info/treeherder channel logs]
** A performance panel in pushlog view that can be used to see details about the performance data submitted for a particular push, plus recent antecedents.
* Mailing list: [https://lists.mozilla.org/listinfo/tools-treeherder tools-treeherder] (or [https://groups.google.com/forum/#!forum/mozilla.tools.treeherder via Google groups])
** A performance summary tab that gives a high-level overview by branch, platform, and suite.
* Weekly meetings: [[EngineeringProductivity/Projects/Treeherder/Meetings|Notes & dial-in details]]
* Status: {{proposed|}} detailed planning to take place in late 2014 Q4, with delivery in 2015 Q1.
==== Possible Future Milestones ====
* Annotation of backouts, tree closures
* Better performance graphs
* Integration of performance alerts
* Integration of OrangeFactor, GoFaster dashboards
* Create a display for code coverage data
* Additional metrics and visualizations
 
== Bug Tables ==
 
=== [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=blocked%3A1059400 TBPL regressions] ===
Remaining regressions/pain points compared to TBPL that were not blocking the sheriff transition.
<bugzilla>
{
"resolution": "---",
"blocked": "1059400",
"include_fields": "id, priority, summary, assigned_to",
"order": "priority"
}
</bugzilla>
=== [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=blocked%3A1080757 Operational and scaling bugs] ===
Open bugs affecting the day to day operation of the Treeherder infrastructure, including its ability to scale well and work correctly under load.
<bugzilla>
{
"resolution": "---",
"blocked": "1080757",
"include_fields": "id, priority, summary, assigned_to",
"order": "priority"
}
</bugzilla>
=== [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=blocked%3A1072676 Data submission bugs] ===
Open bugs affecting the ability to report correctness tests to Treeherder from sources other than buildbot.
<bugzilla>
{
"resolution": "---",
"blocked": "1072676",
"include_fields": "id, priority, summary, assigned_to",
"order": "priority"
}
</bugzilla>
=== [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=blocked%3A1072681 Hacking UX bugs] ===
Open bugs related to the cleanup of Treeherder's docs and development environment that will make Treeherder more accessible to other developers, both those hacking on Treeherder as well as those making use of Treeherder's APIs.
<bugzilla>
{
"resolution": "---",
"blocked": "1072681",
"include_fields": "id, priority, summary, assigned_to",
"order": "priority"
}
</bugzilla>
=== [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=blocked%3A1054977 TBPL switch-off tasks] ===
Remaining tasks not covered in [[#TBPL_regressions]] that will need to be completed before the TBPL crons/DB/accounts can be switched off/deleted. Many of these issues will be have to be fixed by other projects/teams, since they involve switching consumers of TBPL's API over to Treeherder's API.
<bugzilla>
{
"resolution": "---",
"blocked": "1054977",
"id": "1030636,1059400", "id_mode": "exclude",
"include_fields": "id, product, component, summary, assigned_to",
"order": "priority"
}
</bugzilla>


=== [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=%3Atreeherder+-blocked%3A1030636+-blocked%3A1080757+-blocked%3A1059400+-blocked%3A1072676+-blocked%3A1072681+-blocked%3A1054977+-bug_id%3A1030636+-bug_id%3A1059400+-bug_id%3A1072676+-bug_id%3A1072681+-bug_id%3A1054977+-bug_id%3A1080757 All others] ===
== What we're working on ==
All other open bugs not included above.
Assigned Treeherder bugs that have been modified in the last month.
<bugzilla>
[https://bugzilla.mozilla.org/buglist.cgi?quicksearch=%3Atreeherder+-assignee%3Anobody%40mozilla.org+delta_ts%3E-1m View on Bugzilla]
<bugzilla stats="hide">
{
{
"component": "Treeherder",
  "component": "Treeherder", "component_type": "contains",
"resolution": "---",
  "resolution": "---",
"id": "1030636,1059400,1072676,1072681,1054977,1080757", "id_mode": "exclude",
  "email1": "nobody@mozilla.org", "email1_type": "not_equals", "email1_assigned_to": "1",
"blocked": "1030636 1059400,1072676,1072681,1054977,1080757", "blocked_type": "not_contains_any_words",
  "changed_after": "-1m",
"include_fields": "id, priority, summary, assigned_to",
  "changed_before": "Now",
"order": "priority"
  "include_fields": "id,priority,component,summary,assigned_to",
  "order": "priority,assigned_to"
}
}
</bugzilla>
</bugzilla>
== Archive ==
* Original design notes:
** [[Auto-tools/Projects/Treeherder/Current_TBPL_Feature_List]]
** [[Auto-tools/Projects/Treeherder/Design_Draft]] (quite out of date now)
* [https://www.pivotaltracker.com/projects/749519 Pivotal Tracker project] (Deprecated)
* Etherpads: [https://etherpad.mozilla.org/sheriffing-with-treeherder sheriff-feedback-april] / [https://sheriffs.etherpad.mozilla.org/move-to-treeherder-requirements sheriff-feedback-june] / [https://etherpad.mozilla.org/treeherder-transition sheriff-transition-meeting-notes]

Latest revision as of 06:42, 11 March 2019

About

Treeherder is a reporting dashboard for checkins to Mozilla projects (for example, mozilla-central). It allows users to see the results of automatic builds and their respective tests. Treeherder also provides a rich set of APIs that can be used by other projects interested in this information.

Treeherder is the successor to TBPL.

For tracking performance data, see Treeherder's sister project, Perfherder.

For tracking intermittent test failures, see Intermittent Failures View.

Contributing

To make UI-only changes, you only need to perform a very simple setup running a local webserver pointing at the production backend, described here. If you wish to hack on the backend, or the UI and backend together, you will instead need to set up a Vagrant environment, using these steps.

Source and Docs

  • UI & backend: Source / Docs
  • Treeherder data submission clients:
    • Pulse: Source / docs
    • (Python and Node submission support has been removed.)

Getting in touch

What we're working on

Assigned Treeherder bugs that have been modified in the last month. View on Bugzilla

Full Query
ID Priority Component Summary Assigned to
1991905 -- Treeherder: Data Ingestion do not treat log lines with only 'Assertion failure' as failure lines Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)
1990742 -- Treeherder Ingest perfherder_data from JSON artifacts instead of parsing logs Myeongjun Go
1995932 -- Treeherder Add detailed log parsing status display in job detail panel Myeongjun Go
Retrieved from "https://wiki.mozilla.org/index.php?title=EngineeringProductivity/Projects/Treeherder&oldid=1208668"