EngineeringProductivity/Projects/Perfherder

About

Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android).

Perfherder is part of the Treeherder project. The production version of it lives at https://treeherder.mozilla.org/perf.html

Team

• Sheriffs (:igoldan, :bebe)
• Developers (:sclements, :igoldan)

Goals

Develop tools and integration of them to assist sheriffs to find and report regressions to developers, and to assist developers in understanding the impact of their changes and potential fixes.

This includes:

• Graphical display of performance data over time
• An easy way to compare all performance tests for a given revision to another revision (or time range)
• An alert system to provide notification when there has been a level shift in performance data, along with tools for filing bugs when such events occur
• Integration with normal developer work flows for using the Try server and landing code

Non Goals

• fixing tests
• fixing harnesses
• adding new tests
• bisection of a range of commits to find the root cause

Dependencies / Who will use this

We have a dependency upon tools to generate data. Currently this is Talos, but the design is to allow all other types of tools to post here as well.

The other big dependency we have is TreeHerder. Right now we are tightly integrated into the project and use a lot of common data (branches, revisions, history, pushlog). We fully expect this to remain the case and will do our best to integrate as much as possible if it helps the workflow of the users.

The primary users will be:

• sheriffs
• developers

Secondary users will be:

• manager (project, release)
• qa

Contribution

We welcome contributions to Perfherder! We track issues inside the Perfherder component on bugzilla, see below for a list of open bugs.

The contribution section of the Treeherder page has some handy resources for those just getting started. In general, if you have a development version of treeherder running, you're all set to hack on Perfherder! Just jump to:

• http://localhost:5000/perf.html

If you have a question, jump on #perfherder on irc.mozilla.org and look for :igoldan or :davehunt.

Source and docs

See the Treeherder page.

Bug Table

Full Query

ID	Priority	Component	Summary	Assigned to
1609059	P1	Perfherder	Remove "My alerts" in detailed Alerts view	Ionuț Goldan [:igoldan]
1609081	P1	Perfherder	Encapsulate client side search for filter text	Ionuț Goldan [:igoldan]
1595018	P1	Perfherder	Temporarily allow maxLength of 100 for extra options	Octavian Negru
1609079	P1	Perfherder	Add “all” item to framework dropdown	Octavian Negru
1593815	P1	Perfherder	Perfherder doesn't reload if you change the URL	rishav.21m
1609363	P1	Perfherder	Order platform names alphabetically	Subramanian
1564833	P2	Perfherder	[meta] Server side search in Alerts view
1569166	P2	Perfherder	Measure machine time of backfilled perf jobs
1587018	P2	Perfherder	Display the public variant of suite & tests, if available
1595111	P2	Perfherder	Alerts view gets stuck on last page
1596372	P2	Perfherder	How should we surface measurement units in Graphs view?
1600004	P2	Perfherder	Replace current push from alert summaries with previous push & current push tuple
1607710	P2	Perfherder	Input filter from TestDataModal doesn't properly update signature list
1609064	P2	Perfherder	[meta] Encapsulate existing client side search
1609080	P2	Perfherder	[meta] Provide server side search strategy
1609082	P2	Perfherder	Encapsulate client side search for “Hide improvements”
1609083	P2	Perfherder	Encapsulate client side search for “Hide downstream / reassigned to / invalid”
1609085	P2	Perfherder	Encapsulate client side search for “My alerts”
1609086	P2	Perfherder	Provide server side search for filter text
1609087	P2	Perfherder	Provide server side search for “Hide improvements”
1609088	P2	Perfherder	Provide server side search for “Hide downstream / reassigned to / invalid”
1609089	P2	Perfherder	Provide server side search for “My alerts”
1609359	P2	Perfherder	Provide framework name in alert summary headers
1565516	P3	Perfherder	[meta] Improvements to compare view	Alexandru Irimovici
1582710	P3	Perfherder	[meta] UI/UX improvements to Graphs view	Ionuț Goldan [:igoldan]
1265381	P3	Perfherder	Consider alerting with less than 12 future data points if we're very confident there is a regression	Kyle Lahnakoski [:ekyle]
1268247	P3	Perfherder	Alert management UI doesn't really scale to uplifts
1316590	P3	Perfherder	Compare view should allow viewing subtests across all tests
1343328	P3	Perfherder	Add data expiration for performance_datum or switch to bigint to protect against auto_increment overflow
1452238	P3	Perfherder	test_alerts_post doesn't set or test the PerformanceDatum's job_id
1481728	P3	Perfherder	Improvement alerts aren't generated automatically
1481821	P3	Perfherder	perfherder compare is very limited when developers are investigating a regression or making their test relevant or stable
1484714	P3	Perfherder	make it easier to push to try and compare against a base revision
1500528	P3	Perfherder	Make perfherder comparisons consistent across benchmarks
1501239	P3	Perfherder	Test validity dashboard
1501241	P3	Perfherder	[meta] Make states more manageable
1501247	P3	Perfherder	Integrate tagging functionality
1502805	P3	Perfherder	Integrate browser notifications
1506590	P3	Perfherder	add ability in alert manager to search by bug
1515338	P3	Perfherder	State machine diagram
1515681	P3	Perfherder	Perfherder Compare should not allow (or highly warn) when comparing a try build against a mozilla-central build
1519289	P3	Perfherder	Remove usage of signature_hash in Perfherder
1520726	P3	Perfherder	Check fixes for perf regressions
1520731	P3	Perfherder	Directly retrigger & backfill
1520735	P3	Perfherder	Warn about platforms with limited capacity
1522871	P3	Perfherder	It would be very useful to have a link for the pushlog for each set of data visible in the graph
1523037	P3	Perfherder	Page titles for subtests are confusing
1528740	P3	Perfherder	Notify on new alerts/no more data from existing alerts
1529090	P3	Perfherder	No way to easily examine all the performance data for a try push
1532283	P3	Perfherder	Improve & re enable nudge
1532995	P3	Perfherder	Cover compare replicates view with FE tests
1536018	P3	Perfherder	Alert summary's status doesn't properly update
1539854	P3	Perfherder	Timeranges in Perfherder URL are limited to 1, 2, 7, 14, 30, 60, 90 days
1543016	P3	Perfherder	Android PGO platforms generating OPT metrics
1543780	P3	Perfherder	Make https://treeherder.mozilla.org/api/project/{repo}/performance/signatures faster
1546054	P3	Perfherder	Add new "Top alerts" view
1550014	P3	Perfherder	Allow sorting by table column on compare subtest page
1551556	P3	Perfherder	Add support to enable/disable alerting on suite and sub-test level
1558746	P3	Perfherder	Reflect confirmed commits on alerts
1562162	P3	Perfherder	Measure machine time of retriggered/backfilled jobs
1562178	P3	Perfherder	[meta] Automate backfills and retriggers for alerts
1565480	P3	Perfherder	Add pagination at the top of the alerts page
1569018	P3	Perfherder	Graphs of try vs try pushes show irrelevant comparison points
1569022	P3	Perfherder	graph backdrop for pushes should be based on base revision, not push date
1569024	P3	Perfherder	select the right test for the graph backdrop when adding more test data to graph
1569165	P3	Perfherder	Measure machine time of retriggered perf jobs
1569590	P3	Perfherder	Mention pending retriggers in compare view
1570225	P3	Perfherder	Automatically retrigger jobs for new perf regressions
1570226	P3	Perfherder	[meta] Automatically backfill jobs for new alerts
1570906	P3	Perfherder	Performance alert created from a [tier 3] job
1570952	P3	Perfherder	Distinguish between manual & automated padfills
1570956	P3	Perfherder	Measure machine time of automated backfilled perf jobs
1571372	P3	Perfherder	Integrate backfill bot
1571377	P3	Perfherder	Decouple backfill reports from backfill bot
1571378	P3	Perfherder	Provide CRUD endpoint for backfill reports
1571379	P3	Perfherder	Integrate mailing service
1571381	P3	Perfherder	Provide basic landing page for backfill endpoint
1571382	P3	Perfherder	Email backfill reports
1571383	P3	Perfherder	Provide backfill endpoint
1572722	P3	Perfherder	[meta] Improve accessibility
1573131	P3	Perfherder	Automatically identify backouts
1581533	P3	Perfherder	Discussion: Improving performance regression detection
1582711	P3	Perfherder	[meta] Introduce noise alerts for perf tests
1583539	P3	Perfherder	Provide mechanism for renaming suite/test names
1584900	P3	Perfherder	[meta] Tag infrastructure changes into Graphs view
1585650	P3	Perfherder	[meta] Provide boxplots in Graphs view
1585656	P3	Perfherder	[meta] Improve search capabilities of "Add Test Data" modal
1585969	P3	Perfherder	Tab titles should sync with the editable titles from Compare views
1587382	P3	Perfherder	Provide application as a new perf signature property
1587766	P3	Perfherder	build times should be grouped by dimension sccache hit rate
1588026	P3	Perfherder	Endpoint for test validity dashboard must provide tiers
1590627	P3	Perfherder	Clean databases of remnants of old unused frameworks
1591044	P3	Perfherder	New Tier column to Test Validity Dashboard
1591045	P3	Perfherder	Backfill reports should consider indirect alerts also
1591053	P3	Perfherder	Change logic for the Graphs view "Add related configurations" hyperlinks
1593206	P3	Perfherder	Make application fields mandatory in validation schema
1596347	P3	Perfherder	Investigate perf data produced by job_resource_usage
1533002	P3	Perfherder	Add proper error handling in ReplicatesGraph	parshva.barbhaya
1574628	P3	Perfherder	Improve graph colors and add symbols to better distinguish between tests	Mellina Yonashiro
1600668	--	Perfherder	Error out if PERFHERDER_DATA is malformed
1600759	--	Perfherder	Investigate creation of an alternative view to the Graphs view for screen reader users
1601952	--	Perfherder	Use The Mann-Whitney-U test
1604868	--	Perfherder	Unable to remove series from graph view
1604871	--	Perfherder	Unable to add data to graph when there are no recent results
1605240	--	Perfherder	Vertical graph labels are truncated on most significant digits
1605247	--	Perfherder	Consider adding more options for date ranges
1605748	--	Perfherder	Perfherder regression filing should inform the group/builder of a talos regression
1611158	--	Perfherder	Subtest view - include the platform in the subtest name
1611203	--	Perfherder	Make it possible to embed Perfherder graphs in other pages

109 Total; 109 Open (100%); 0 Resolved (0%); 0 Verified (0%);