EngineeringProductivity/Projects/Perfherder

About

Perfherder is a dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android).

Perfherder is part of the Treeherder project. The production version of it lives at https://treeherder.mozilla.org/perf.html

Team

• Talos Sheriffs (:jmaher)
• Developers (:wlach, :rwood)

Goals

Develop tools and integration of them to assist sheriffs to find and report regressions to developers, and to assist developers in understanding the impact of their changes and potential fixes.

This includes:

• Graphical display of performance data over time
• An easy way to compare all performance tests for a given revision to another revision (or time range)
• An alert system to provide notification when there has been a level shift in performance data, along with tools for filing bugs when such events occur
• Integration with normal developer work flows for using the Try server and landing code

Non Goals

• fixing tests
• fixing harnesses
• adding new tests
• bisection of a range of commits to find the root cause

Dependencies / Who will use this

We have a dependency upon tools to generate data. Currently this is Talos, but the design is to allow all other types of tools to post here as well.

The other big dependency we have is TreeHerder. Right now we are tightly integrated into the project and use a lot of common data (branches, revisions, history, pushlog). We fully expect this to remain the case and will do our best to integrate as much as possible if it helps the workflow of the users.

The primary users will be:

• sheriffs
• developers

Secondary users will be:

• manager (project, release)
• qa

Contribution

We welcome contributions to Perfherder! We track issues inside the Perfherder component on bugzilla, see below for a list of open bugs.

The contribution section of the Treeherder page has some handy resources for those just getting started. In general, if you have a development version of treeherder running, you're all set to hack on Perfherder! Just jump to:

• http://local.treeherder.mozilla.org/perf.html (full copy inside Vagrant)
• http://localhost:8000/perf.html (if just running the UI using the node server)

If you have a question, jump on #perfherder on irc.mozilla.org and look for wlach or jmaher (mostly online in EST timezone).

Source and docs

See the Treeherder page.

Bug Table

ID	Priority	Component	Summary	Assigned to
1266451	--	Perfherder	Add a unit test for import_perf_data	Anjana Vakil (:vakila)
1285322	--	Perfherder	Implement auto-classification of "downstream" alerts	crosscent
1332323	--	Perfherder	"file bug" in perfherder alerts should auto-suggest blocking bug + component based on commit range of regression	George Veneel Dogga [::TheLayman]
1170639	--	Perfherder	Provide a way of using compareperf with locally produced talos data
1170648	--	Perfherder	Provide some kind of documentation of what the calculations in compareperf are
1171703	--	Perfherder	Indicate whether higher or lower is better in perfherder compare summary even for those tests without enough data to detect a statistically significant change
1202718	--	Perfherder	Add ability to retrigger talos tests directly from the "compare performance" UI
1202731	--	Perfherder	Add a way to link to a particular set of results in comparechooser UI
1210509	--	Perfherder	perfherder compare view should do a better job dealing with high std dev sets of data
1224571	--	Perfherder	when viewing data in perfherder, allow custom platforms, right now this is auto detected
1229275	--	Perfherder	Perfherder doesn't distinguish between linux64 debug builds and linux64 debug static analysis builds
1229504	--	Perfherder	small tweaks to alert view for maximum productivity - adding a compare view
1231368	--	Perfherder	Reassigning alerts doesn't check for errors
1239821	--	Perfherder	on alerts page of perfherder, there is a link to the jobs, this should filter by talos
1247997	--	Perfherder	Not immediately obvious whether a result in perfherder compare view is a regression/improvement without being able to see the colour
1252814	--	Perfherder	perfherder pgo alerts should have an easy way to see the non pgo alerts
1253218	--	Perfherder	when using compare view in perfherder it is unclear that the base revision needs retriggered jobs
1255001	--	Perfherder	The 'important changes' and 'uncertain results' checkboxes don't behave as advertised
1258845	--	Perfherder	when two alerts have the same target revision but different revision ranges we have different alerts and it is confusing
1261969	--	Perfherder	[Meta] Features necessary for using Perfherder for sheriffing AWFY data
1265381	--	Perfherder	Consider alerting with less than 12 future data points if we're very confident there is a regression
1265709	--	Perfherder	The performance data API should be rewritten to not be tied to signature hashes
1266183	--	Perfherder	Submissions with varying numbers of subtests end up in different graphs
1268247	--	Perfherder	Alert management UI doesn't really scale to uplifts
1269489	--	Perfherder	Automatically track standard deviation in performance tests over time
1272716	--	Perfherder	Perfherder - subtest alerts should use historical vs forward comparisons instead of revision comparisons
1277954	--	Perfherder	links to perfherder graphs are relative to the date of opening the graph, not the date when the link was shared
1283819	--	Perfherder	[tracking bug] - dashboard and/or tool changes to accommodate data needs of release management
1289540	--	Perfherder	alert pagination make it difficult to find alerts
1316590	--	Perfherder	Compare view should allow viewing subtests across all tests
1324859	--	Perfherder	post build_metrics alerts to mozilla.dev.builds
1329240	--	Perfherder	Add a link to the pushlog for the visible changeset range
1330948	--	Perfherder	Add an option to show trend lines in graph view
1335784	--	Perfherder	Use qparent for default original revision in perf hearder links in try server emails
1363967	--	Perfherder	Automatic base revision is wrong
1382371	--	Perfherder	Compare performance result link should take me to a comparison for that test
1383123	--	Perfherder	Distribution view gives mixed active content warnings when served over HTTPs
1329224	--	Perfherder	Treeherder graph URLs are unnecessarily long, making them difficult to share on irc	Shruti Jasoria [:ShrutiJ]
1350384	--	Perfherder	Add ability to see a visualization of the test replicates in Perfherder	Shruti Jasoria [:ShrutiJ]
1376829	--	Perfherder	Add notes to alerts	William Lachance (:wlach) (use needinfo!)
1381904	--	Perfherder	Viewing subtests never loads	William Lachance (:wlach) (use needinfo!)
1173399	P2	Perfherder	Selecting Compare in Perf complains in the console about invalid plot dimensions
1308122	P2	Perfherder	PerformanceSignatureViewSet.list "ValueError: invalid literal for int() with base 10: '1 and 2'"	MikeLing
1288857	P3	Perfherder	Create a Perfherder dashboard for Stylo similar to the e10s one	Chris Peterson [:cpeterson]
1200217	P5	Perfherder	UX improvement for Compare - provide valid, default landing values for Submit
1343328	P5	Perfherder	Add data expiration for performance_datum or switch to bigint to protect against auto_increment overflow

46 Total; 46 Open (100%); 0 Resolved (0%); 0 Verified (0%);