EngineeringProductivity/Projects/Perfherder

About

Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android).

Perfherder is part of the Treeherder project. The production version of it lives at https://treeherder.mozilla.org/perf.html

Team

• Sheriffs (:igoldan, :bebe)
• Developers (:sclements, :igoldan)

Goals

Develop tools and integration of them to assist sheriffs to find and report regressions to developers, and to assist developers in understanding the impact of their changes and potential fixes.

This includes:

• Graphical display of performance data over time
• An easy way to compare all performance tests for a given revision to another revision (or time range)
• An alert system to provide notification when there has been a level shift in performance data, along with tools for filing bugs when such events occur
• Integration with normal developer work flows for using the Try server and landing code

Non Goals

• fixing tests
• fixing harnesses
• adding new tests
• bisection of a range of commits to find the root cause

Dependencies / Who will use this

We have a dependency upon tools to generate data. Currently this is Talos, but the design is to allow all other types of tools to post here as well.

The other big dependency we have is TreeHerder. Right now we are tightly integrated into the project and use a lot of common data (branches, revisions, history, pushlog). We fully expect this to remain the case and will do our best to integrate as much as possible if it helps the workflow of the users.

The primary users will be:

• sheriffs
• developers

Secondary users will be:

• manager (project, release)
• qa

Contribution

We welcome contributions to Perfherder! We track issues inside the Perfherder component on bugzilla, see below for a list of open bugs.

The contribution section of the Treeherder page has some handy resources for those just getting started. In general, if you have a development version of treeherder running, you're all set to hack on Perfherder! Just jump to:

• http://localhost:5000/perf.html

If you have a question, jump on #perfherder on irc.mozilla.org and look for :igoldan or :davehunt.

Source and docs

See the Treeherder page.

Bug Table

Full Query

ID	Priority	Component	Summary	Assigned to
1564833	P1	Perfherder	[meta] Server side search in Alerts view	Alexandru Irimovici
2007109	P1	Perfherder	Add an interface that we can use to run detection techniques against arbitrary groups of data	Mohamed Bilel Besbes[:bilelbesbes]
2007112	P1	Perfherder	Perform a binary search backfill with the sheriffing bot with change detection technique integration	Myeongjun Go
2007009	P1	Perfherder	Change the bot used for bug filling and commenting to a "PerfSheriff Bot"	Myeongjun Go
2007005	P1	Perfherder	Automatically mirror the bug status to the alert status in our database	Myeongjun Go
2030053	P1	Perfherder	[Perfherder][FileBug] Regression column in comment 0 contains incorrect URL for alert GraphsView
2009180	P1	Perfherder	Determine change detection techniques, and new detection methodology
2008944	P1	Perfherder	Add new field to the performance alert model to specify if an alert is a prototype
2008943	P1	Perfherder	Test new alerting techniques in Treeherder production
2008942	P1	Perfherder	Implement new change point detection techniques for performance test alerts
2008940	P1	Perfherder	[meta] Implement and test new change detection point techniques for performance test alerts
2007002	P1	Perfherder	[meta] Implement an automated smart backfill system for performance tests
2006999	P1	Perfherder	[meta] Improve Sheriffing Automation and Tooling
2017562	P2	Perfherder	Performance Audit for Graphs View	Andra Esanu (needinfo me)
1960354	P2	Perfherder	Broken links in performance improvement/regression bugzilla comments due to wrong anchor name	Andra Esanu (needinfo me)
2012477	P2	Perfherder	Filter by regressions or improvements in the Perfherder Alerts view	Alex Finder
2006399	P2	Perfherder	Alert comment 0 profile links for before after sometimes lead to the same profile	Alex Finder
1651432	P2	Perfherder	Perfherder UI tests need to be updated	Acasandrei Beatrice (needinfo me)
1694335	P2	Perfherder	Delete the remove_vcs_data management command	Acasandrei Beatrice (needinfo me)
1929249	P2	Perfherder	Build a process for taking action on performance regressions that were resolved as WONTFIX/FIXED	Greg Mierzwinski [:sparky]
2031333	P2	Perfherder	Perf Alerts: Subcritical and critical Performance tests should be in a seperate table in alert bug comment 0	Greg Mierzwinski [:sparky]
1689621	P2	Perfherder	[meta] Monitor backfill bot’ stability on production	Ionuț Goldan [:igoldan]
1723719	P2	Perfherder	Link to devtools test documentation	Julian Descottes [:jdescottes]
1831943	P2	Perfherder	Expand replicate storage to other projects/branches	Myeongjun Go
2008333	P2	Perfherder	IrrelevantDataRemoval may stop early depending on repository order	Myeongjun Go
1982829	P2	Perfherder	Automate leaving improvement/backout comments on bugzilla from Alerts View	Myeongjun Go
2009828	P2	Perfherder	Investigate performance issues with Perfherder/Treeherder API
1990829	P2	Perfherder	Remove all PERFHERDER_DATA log outputs, and disable perfherder data log-based ingestion
1995357	P2	Perfherder	Add a tool to remove data from bad devices, and retrigger alert generation
2031842	P2	Perfherder	Add links to expanded row in perfcompare from the alert bug comment 0
1983826	P2	Perfherder	Perfherder: Fix intermittent failing test in test_identify_retriggerables.py
2037200	P2	Perfherder	Add a quick summary for alert resolution options in comment 0
2040701	P2	Perfherder	Increase retention period for performance data
1940556	P2	Perfherder	Enable perfdocs documentation links for mozperftest tests
1935268	P2	Perfherder	Add replicates to the subtest performance artifact schema
2011256	P2	Perfherder	Graphs View - Don't highlight alerts by default in graphs view
1992965	P2	Perfherder	Use BugBot to auto-close regression bugs once they are resolved
1635127	P2	Perfherder	Alert summaries should show the status of linked bug
1722092	P2	Perfherder	comparison browsertime (Btime) metrics of Try push with mozilla-central fails to load data, Perfherder request of m-c data fails with server error
1837203	P2	Perfherder	Add a way to tag infra changes
1870682	P2	Perfherder	Add links to docs for mozperftest tests in alerts view
1904551	P2	Perfherder	Incorrect autogenerated summary on performance alert https://treeherder.mozilla.org/perfherder/alerts?id=853
1907909	P2	Perfherder	Alert summaries with large number of alerting subtests show very large header
1912030	P2	Perfherder	Magnitude sorting on the alert summary page is confusing
1914167	P2	Perfherder	Provide a link to compareview/perfcompare in the alert comment
1915403	P2	Perfherder	Add a small dashboard or redash query to keep up-to-date with alerts
1919432	P2	Perfherder	AWFY -Motionmark graphs are not updating since 16sep2024 (maybe because the old version of Motionmark was replaced by Motionmark1.3.1)
1922517	P2	Perfherder	Add documentation links for build_metrics alerts
1927867	P2	Perfherder	Perfherder Graphs view fails to load properly when values include negative values
1930128	P2	Perfherder	Perfherder ingestion error: duplicate key values for performance_alert_summary table insertion
1853410	P3	Perfherder	Reduce the radius of the plot points (graphs are too crowded)	Andra Esanu (needinfo me)
1756774	P3	Perfherder	Tests View - Changing the framework should reset the table to page 1	Alex Finder
1830472	P3	Perfherder	Pull time duration for the sheriffing KPIs alert categories	Acasandrei Beatrice (needinfo me)
2034090	P3	Perfherder	Remove dead code from AlertsViewControl	David Miculit [:dmiculit]
1933686	P3	Perfherder	Automatic status update not working properly	Greg Mierzwinski [:sparky]
1689584	P3	Perfherder	Provide POC for integrating Mann-Whitney-U	Greg Mierzwinski [:sparky]
1689586	P3	Perfherder	Provide actual plan for integrating MWU	Greg Mierzwinski [:sparky]
1839150	P3	Perfherder	Add side-by-side API endpoint	Greg Mierzwinski [:sparky]
1500528	P3	Perfherder	Make perfherder comparisons consistent across benchmarks
1501241	P3	Perfherder	[meta] Make states more manageable
1515338	P3	Perfherder	State machine diagram
1519289	P3	Perfherder	Remove usage of signature_hash in Perfherder
1520726	P3	Perfherder	Check fixes for perf regressions
1520731	P3	Perfherder	Directly retrigger & backfill
1528740	P3	Perfherder	Notify on new alerts/no more data from existing alerts
1532995	P3	Perfherder	Cover compare replicates view with FE tests
1539854	P3	Perfherder	Timeranges in Perfherder URL are limited to 1, 2, 7, 14, 30, 60, 90 days
1543780	P3	Perfherder	Make https://treeherder.mozilla.org/api/project/{repo}/performance/signatures faster
1546054	P3	Perfherder	Automatically email top alerts on a weekly basis
1558746	P3	Perfherder	Reflect confirmed commits on alerts
1569018	P3	Perfherder	Graphs of try vs try pushes show irrelevant comparison points
1569022	P3	Perfherder	graph backdrop for pushes should be based on base revision, not push date
1569024	P3	Perfherder	select the right test for the graph backdrop when adding more test data to graph
1569590	P3	Perfherder	Mention pending retriggers in compare view
1573131	P3	Perfherder	Automatically identify backouts
1581533	P3	Perfherder	Discussion: Improving performance regression detection
1582711	P3	Perfherder	[meta] Introduce noise alerts for perf tests
1585650	P3	Perfherder	[meta] Provide boxplots in Graphs view
1587766	P3	Perfherder	build times should be grouped by dimension sccache hit rate
1591045	P3	Perfherder	Backfill reports should consider indirect alerts also
1591053	P3	Perfherder	Change logic for the Graphs view "Add related configurations" hyperlinks
1600668	P3	Perfherder	Error out if PERFHERDER_DATA is malformed
1601952	P3	Perfherder	Use The Mann-Whitney-U test
1615267	P3	Perfherder	Tests View: Consistent renaming from health/validity to Tests View
1615268	P3	Perfherder	[meta] Improvements to Tests View
1616191	P3	Perfherder	Provide server side filtering by option
1620736	P3	Perfherder	Graphs View - Add Table view param in URL
1620737	P3	Perfherder	Graphs View - When toggle to Table View, it should focus on table
1620739	P3	Perfherder	Graphs View - TableView and GraphTooltip component share helpers functions
1620741	P3	Perfherder	Graphs View - Table view could have trends and outliers navigation to improve accessibility
1622564	P3	Perfherder	Improve performance of graphs view when showing a large date range
1622567	P3	Perfherder	Add trend lines to graphs
1622569	P3	Perfherder	Add graphical plots for subtests
1622578	P3	Perfherder	Add the ability to use the shortened hash for a commit in the compare view
1633660	P3	Perfherder	Attribute inconsistency in alert summary serializer
1635085	P3	Perfherder	Provide test coverage for when Compare view is changing its URL
1639112	P3	Perfherder	Consider making SecretaryTool a singleton
1639248	P3	Perfherder	Feature request to save perfherder comparisons for future lookup
1639277	P3	Perfherder	tp6 tests unnecessarily require multiple tasks to increase confidence on compare chooser
1639308	P3	Perfherder	Show the last commit message when comparing two revisions
1641623	P3	Perfherder	Add a numerical summary of Perfherder improvement/regression to the Compare Revisions view
1646259	P3	Perfherder	Add/modify perfherder graph view to better communicate noise
1649156	P3	Perfherder	Distinguish subtest items from test items in alert summaries
1649551	P3	Perfherder	add job_type_id to performance_signature table
1650153	P3	Perfherder	Expose performance properties for simple (or complex!) filtering
1665100	P3	Perfherder	Perherder should be able to show task name for a graph
1684946	P3	Perfherder	browsertime vismet tasks with --rebuild flag all reuse a single test result. Perfherder says Confidence is Infinity (high).
1700970	P3	Perfherder	Perfherder compare Retrigger Jobs button no longer retriggers jobs
1701615	P3	Perfherder	Group alert items in tests+subtests to visually distinguish them easier.
1727787	P3	Perfherder	Perfherder should show `unit` and `lower_is_better` in addition to existing color-coding on e.g. the Alert view
1728295	P3	Perfherder	Improve test coverage for report_backfill_outcome command
1729665	P3	Perfherder	Add a performance tab in Push Health
1734898	P3	Perfherder	Add support for filtering out matches in perfherder compare
1744024	P3	Perfherder	fission/non-fission awsy jobs are mixed up in the display, and also in retrigger
1773562	P3	Perfherder	[perfdocs] Integrate perfdocs test owner information into our alerts
1824750	P3	Perfherder	Add an API endpoint for getting the application version
1838924	P3	Perfherder	Integrate sitespeedio on treeherder for performance testing
1839320	P3	Perfherder	Hide triaged alerts from untriaged queue
1842195	P3	Perfherder	Set status flag when filing regression bugs
1855574	P3	Perfherder	Error when clicking "View" for a talos-profiling task: `Something went wrong TypeError: can't access property "y", e[0] is undefined`
1855728	P3	Perfherder	Add Slack and/or Element notifications for performance alerts
1869245	P3	Perfherder	Show common alerts along with the normal alert in graph tooltip
1874894	P3	Perfherder	Allow us to annotate data points in the graph view
1877784	P3	Perfherder	Hovering over a replicate doesn't show you the value for that replicate
1900303	P3	Perfherder	Two different values for the same alert metric in comment 0 when filling a sheriffing regression bug
1902168	P3	Perfherder	Show reassigned alerts in graphs view
1905930	P3	Perfherder	Filling alert bug should only use test names that regressed in bug title
1907925	P3	Perfherder	Annotate profiles in before/after with before/after to denote which is which
1914258	P3	Perfherder	Add test owner/team in the cc for when their tests alert
1987436	P3	Perfherder	Investigate how we should handle backed out patches in perfherder
1985314	P3	Perfherder	Alert summary has a side-by-side button displayed but there's no side-by-side available
2006137	P3	Perfherder	Graphs view - Remove framework option from test selection window and show all available tests/subtests by default
1975449	P3	Perfherder	Common alerts for the same framework are not displayed on the graph
1967765	P3	Perfherder	Graphs View - Make the original test run task more visible
1962337	P3	Perfherder	Tooling occasionally misclassifies regressions as improvements
1955647	P3	Perfherder	Side-by-side videos should only be generated for regressing tests
1955640	P3	Perfherder	Use bug-template for copySummary alert view method instead of a string
1954049	P3	Perfherder	Update UI test integration run command for Perfherder
1940375	P3	Perfherder	Make https://treeherder.mozilla.org/perfherder/graphs more ergonomic to use when trying to filter/select large number of tests/metrics/subtests
1940232	P3	Perfherder	links to perfherder data series from Treeherder not working if older than 28 days because purged? Something went wrong TypeError: can't access property "y", e[0] is undefined
1935199	P3	Perfherder	UI on perfherder doesnt work correctly when linked via AWFY-Browsertime-Android-SP3
1850759	P5	Perfherder	Rectangle drag selection gets stuck if the mouse is released outside the graph	Andra Esanu (needinfo me)
2033791	P5	Perfherder	Implement lifecycle metrics query for performance alerts (including triage and resolution timings)	Florin Bilt
1931234	P5	Perfherder	On perfherder, datapoints generated by backfill for AWSY tests do not have the "delta" number
1946767	P5	Perfherder	Determine how we can provide better ordering for the extra-options in the alert bug tables
1982029	P5	Perfherder	Kraken subtests show "score" but say "lower is better"
1502805	P5	Perfherder	Integrate browser notifications
1593206	P5	Perfherder	Make application fields mandatory in validation schema
1605247	P5	Perfherder	Consider adding more options for date ranges
1611203	P5	Perfherder	Make it possible to embed Perfherder graphs in other pages
1616242	P5	Perfherder	Provide frontend test coverage for server side search
1622829	P5	Perfherder	Consider changing default display to more than 2 days
1627922	P5	Perfherder	Reassign Alerts dialog doesn't have autofocus
1655019	P5	Perfherder	Display a name/unit for summary results
1723073	P5	Perfherder	Have properties to distinguish manual/auto backfilled jobs
1741786	P5	Perfherder	Performance regression detected between m-c build and try build with the same revision
1838025	P5	Perfherder	Provide a method to filter out noisy/outlier data points in the compare view
1894252	P5	Perfherder	AWSY test is showing subtest in the suite name with subtests under it as well
1901067	P5	Perfherder	Button to open all alerts on perfherder
1908641	P5	Perfherder	Perfherder's Tests View is currently broken
2012482	--	Perfherder	When opening the graphs view link, highlight the datapoint of the alert I’ve clicked on	Andra Esanu (needinfo me)
2012475	--	Perfherder	Spike - Backfill/Retriggers directly from the graphs	Andra Esanu (needinfo me)
2018876	--	Perfherder	Add section to comment 0 on how to run tests that have regressed locally	Alex Finder
1901066	--	Perfherder	When clicking a link from perfherder you should have the treeherder view filtered to just that job	Andrej (:aglavic)
2012476	--	Perfherder	Spike - Highlight gap between data points, especially where there are alerts	Acasandrei Beatrice (needinfo me)
2039815	--	Perfherder	Alert summary not updating status correctly	David Miculit [:dmiculit]
2038599	--	Perfherder	Add bisection as a new tag option	David Miculit [:dmiculit]
2038574	--	Perfherder	Add a new feature to mark alerts as infra instead of using the tag	David Miculit [:dmiculit]
2027671	--	Perfherder	Alerts View: Display the <alert id> for duplicated summaries with the status reassigned and add a copy id feature	David Miculit [:dmiculit]
2017561	--	Perfherder	Performance Audit for Alerts View	Florin Bilt
2037299	--	Perfherder	Display mirrored Bugzilla status on alert page	Myeongjun Go
2021696	--	Perfherder	Use a fixed chunk size in the Perfherder data cycler	Myeongjun Go
2027670	--	Perfherder	Improve test coverage for absolute and relative paths in Perfherder
2027669	--	Perfherder	Some documentation links from test names in https://firefox-source-docs.mozilla.org/testing/perfdocs are broken
2030046	--	Perfherder	Performance bug template broken perfherder link
2027667	--	Perfherder	Update the sheriffing workflow diagram for the documentation
2006136	--	Perfherder	[meta] Make the framework selection optional or hidden
2031461	--	Perfherder	Perf Alerts: Subcritical and critical Performance tests should be in a seperate table in alerts summary interface
2005310	--	Perfherder	Tooltip close button is outside the tooltip and almost invisible
2033106	--	Perfherder	Consider removing the no_retriggers option for PerformanceDatumViewSet performance/data endpoint
2005270	--	Perfherder	Graphs view - Very slow to switch between data points selected
2027662	--	Perfherder	Some documentation links from the test names in Alerts View are broken
2035204	--	Perfherder	Prevent the creation of duplicate summaries
2035210	--	Perfherder	Manage duplicate alerts within an alert summary due to reassignements
1980743	--	Perfherder	intermittent test_identify_retriggerables_selects_even_single_data_point - assert {4} == {92}
2017593	--	Perfherder	Alerts View: Improve Summary Status Autodetermination Logic
2017582	--	Perfherder	Alerts View: Improve the reassignement process
2038586	--	Perfherder	Migrate all summaries tagged as infra to the new status infra
2017571	--	Perfherder	Alerts View: Limit the number of alerts returned within a summary to 50 alerts
2017566	--	Perfherder	Backfilling Process Causing Alert Generation
2040018	--	Perfherder	Display Sherlock iterator information on the alert page
2040400	--	Perfherder	Investigate adaptive slice sizing strategies for iterative smart backfill convergence
2016740	--	Perfherder	PerformanceAlertSummary can group alerts with different sheriffed values
2040949	--	Perfherder	Remove "raptor" option in perfherder test data selection
2041743	--	Perfherder	Colour dots by machine in perfherder graphs
2041948	--	Perfherder	Optimize slow ingestion queries
1689373	--	Perfherder	[meta] Provide plan for using Mann-Whitney-U alongside student' s t-test
1698545	--	Perfherder	[meta] Link to test documentation
1709778	--	Perfherder	[meta] Distinguish between auto/manual backfills
1809467	--	Perfherder	frequent intermittent failures on collapsable_rows_tests.jsx and test_perfcompare_api.py, disabled for now
1818857	--	Perfherder	If retriggers fail on PerfHerder due to authentication issues, the error message gets thrown off the top of the viewport
1819632	--	Perfherder	Hitting enter/return in test filter on perfherder graphs page shouldn't reload the page
1819634	--	Perfherder	It's too difficult to look at m-c results for try graphs
1823531	--	Perfherder	arewefastyet.com speedometer toplevel seems to be inverted
1836081	--	Perfherder	perfherder graph does not show older data for damp signatures even if they are in the displayed timeframe
1860800	--	Perfherder	Intermittent test_identify_retriggerables.py test_identify_retriggerables_selects_all_data_points assert {1, 2, 4, 7, 9} == {2, 4, 7, 9, 11}

206 Total; 206 Open (100%); 0 Resolved (0%); 0 Verified (0%);