Auto-tools/Projects/Perfherder

Team

• Talos Sheriffs (:jmaher, :vaibhav1994)
• Developers (:wlach, :jmaher, :mishravikas)
• Others (:avih)

Problem

Mozilla generates a lot of data from performance tests and other metrics collected during the process of development. Most of this data is useful to see on a graph and look for trends over time. It becomes difficult to understand the data when there is a lot and we need different tools for different levels of data.

In some cases we want a graphical display, but understanding all the data points for a given change to the codebase might be better represented in a table summary. Most of the time these tools are useful for investigating specific issues, which means that we need tools to analyze the data and report regressions automatically.

Goals

Develop tools and integration of them to assist sheriffs to find and report regressions to developers, and to assist developers in understanding the impact of their changes and potential fixes.

This includes:

• graphical display of data over time
• finding specific data and related data for the graphs
• an easy way to compare all tests for a given revision to the previous revision (or specified one)
• A basic alerting system
• integration with normal developer work flows for using the Try server and landing code.

Perfherder is part of the Treeherder project.

Non Goals

• fixing tests
• fixing harnesses
• adding new tests
• bisection of a range of commits to find the root cause

Contribution

We welcome contributions to Perfherder! We track issues inside the Perfherder component on bugzilla, see below for a list of open bugs.

The contribution section of the Treeherder page has some handy resources for those just getting started. In general, if you have a development version of treeherder running, you're all set to hack on Perfherder! Just jump to:

• http://local.treeherder.mozilla.org/perf.html (full copy inside Vagrant)
• http://localhost:8000/app/perf.html (if just running the UI using the node server)

If you have a question, jump on #treeherder on irc.mozilla.org and look for wlach or jmaher (mostly online in EST timezone).

Source and docs

See the Treeherder page.

Bug Table

Full Query

ID	Priority	Component	Summary	Assigned to
1564833	P1	Perfherder	[meta] Server side search in Alerts view	Alexandru Irimovici
1830472	P1	Perfherder	Pull time duration for the sheriffing KPIs alert categories	Acasandrei Beatrice (needinfo me)
1523037	P2	Perfherder	Page titles for subtests are confusing	Acasandrei Beatrice (needinfo me)
1651432	P2	Perfherder	Perfherder UI tests need to be updated	Acasandrei Beatrice (needinfo me)
1694335	P2	Perfherder	Delete the remove_vcs_data management command	Acasandrei Beatrice (needinfo me)
1711883	P2	Perfherder	Compare View call to backend responds with 502 Bad Gateway	Acasandrei Beatrice (needinfo me)
1689621	P2	Perfherder	[meta] Monitor backfill bot’ stability on production	Ionuț Goldan [:igoldan]
1723719	P2	Perfherder	Link to devtools test documentation	Julian Descottes [:jdescottes]
1722092	P2	Perfherder	comparison browsertime (Btime) metrics of Try push with mozilla-central fails to load data, Perfherder request of m-c data fails with server error
1824750	P2	Perfherder	Add an API endpoint for getting the application version
1831943	P2	Perfherder	Expand replicate storage to other projects/branches
1837970	P2	Perfherder	Decrease try performance data expiration from 6 weeks to 3 weeks
1838025	P2	Perfherder	Provide a method to filter out noisy/outlier data points in the compare view
1839150	P2	Perfherder	Add side-by-side API endpoint
1870682	P2	Perfherder	Add links to docs for mozperftest tests in alerts view
1894252	P2	Perfherder	AWSY test is showing subtest in the suite name with subtests under it as well
1900606	P2	Perfherder	Gather replicates during alert generation if they are available
1929249	P2	Perfherder	Build a process for taking action on performance regressions that were resolved as WONTFIX/FIXED
1995357	P2	Perfherder	Add a tool to remove data from bad devices, and retrigger alert generation
1993279	P2	Perfherder	Update readthedocs on treeherder with more accurate information about accessing the staging DB
1904551	P2	Perfherder	Incorrect autogenerated summary on performance alert https://treeherder.mozilla.org/perfherder/alerts?id=853
1905930	P2	Perfherder	Filling alert bug should only use test names that regressed in bug title
1907909	P2	Perfherder	Alert summaries with large number of alerting subtests show very large header
1907925	P2	Perfherder	Annotate profiles in before/after with before/after to denote which is which
1954049	P2	Perfherder	Update UI test integration run command for Perfherder
1990829	P2	Perfherder	Remove all PERFHERDER_DATA log outputs, and disable perfherder data log-based ingestion
1914167	P2	Perfherder	Provide a link to compareview/perfcompare in the alert comment
1914258	P2	Perfherder	Add test owner/team in the cc for when their tests alert
1915403	P2	Perfherder	Add a small dashboard or redash query to keep up-to-date with alerts
1955647	P2	Perfherder	Side-by-side videos should only be generated for regressing tests
1944375	P2	Perfherder	Improve data cycling/deletion for performance datum replicates
1927867	P2	Perfherder	Perfherder Graphs view fails to load properly when values include negative values
1940556	P2	Perfherder	Enable perfdocs documentation links for mozperftest tests
1929962	P2	Perfherder	Add replicates as an option/field to the suites in the performance artifact schema
1930128	P2	Perfherder	Perfherder ingestion error: duplicate key values for performance_alert_summary table insertion
1931234	P2	Perfherder	On perfherder, datapoints generated by backfill for AWSY tests do not have the "delta" number
1932360	P2	Perfherder	Remove the raptor framework
1933686	P2	Perfherder	Automatic status update not working properly
1962337	P2	Perfherder	Tooling occasionally misclassifies regressions as improvements
1935268	P2	Perfherder	Add replicates to the subtest performance artifact schema
1976036	P2	Perfherder	Add an API endpoint that can reduce the two treeherder performance tab requests into a single one
1985314	P3	Perfherder	Alert summary has a side-by-side button displayed but there's no side-by-side available
1935199	P3	Perfherder	UI on perfherder doesnt work correctly when linked via AWFY-Browsertime-Android-SP3
1946767	P3	Perfherder	Determine how we can provide better ordering for the extra-options in the alert bug tables
1922517	P3	Perfherder	Add documentation links for build_metrics alerts
1987436	P3	Perfherder	Investigate how we should handle backed out patches in perfherder
1992965	P3	Perfherder	Use BugBot to auto-close regression bugs once they are resolved
1955640	P3	Perfherder	Use bug-template for copySummary alert view method instead of a string
1265381	P3	Perfherder	Consider alerting with less than 12 future data points if we're very confident there is a regression
1500528	P3	Perfherder	Make perfherder comparisons consistent across benchmarks
1501241	P3	Perfherder	[meta] Make states more manageable
1502805	P3	Perfherder	Integrate browser notifications
1515338	P3	Perfherder	State machine diagram
1519289	P3	Perfherder	Remove usage of signature_hash in Perfherder
1520726	P3	Perfherder	Check fixes for perf regressions
1520731	P3	Perfherder	Directly retrigger & backfill
1528740	P3	Perfherder	Notify on new alerts/no more data from existing alerts
1532995	P3	Perfherder	Cover compare replicates view with FE tests
1539854	P3	Perfherder	Timeranges in Perfherder URL are limited to 1, 2, 7, 14, 30, 60, 90 days
1543780	P3	Perfherder	Make https://treeherder.mozilla.org/api/project/{repo}/performance/signatures faster
1546054	P3	Perfherder	Automatically email top alerts on a weekly basis
1558746	P3	Perfherder	Reflect confirmed commits on alerts
1562178	P3	Perfherder	[meta] Automate backfills and retriggers for alerts
1569018	P3	Perfherder	Graphs of try vs try pushes show irrelevant comparison points
1569022	P3	Perfherder	graph backdrop for pushes should be based on base revision, not push date
1569024	P3	Perfherder	select the right test for the graph backdrop when adding more test data to graph
1569590	P3	Perfherder	Mention pending retriggers in compare view
1573131	P3	Perfherder	Automatically identify backouts
1581533	P3	Perfherder	Discussion: Improving performance regression detection
1582711	P3	Perfherder	[meta] Introduce noise alerts for perf tests
1585650	P3	Perfherder	[meta] Provide boxplots in Graphs view
1587766	P3	Perfherder	build times should be grouped by dimension sccache hit rate
1591045	P3	Perfherder	Backfill reports should consider indirect alerts also
1591053	P3	Perfherder	Change logic for the Graphs view "Add related configurations" hyperlinks
1593206	P3	Perfherder	Make application fields mandatory in validation schema
1600668	P3	Perfherder	Error out if PERFHERDER_DATA is malformed
1601952	P3	Perfherder	Use The Mann-Whitney-U test
1605247	P3	Perfherder	Consider adding more options for date ranges
1611203	P3	Perfherder	Make it possible to embed Perfherder graphs in other pages
1615267	P3	Perfherder	Tests View: Consistent renaming from health/validity to Tests View
1615268	P3	Perfherder	[meta] Improvements to Tests View
1616191	P3	Perfherder	Provide server side filtering by option
1616240	P3	Perfherder	Provide backend test coverage for server side search
1616242	P3	Perfherder	Provide frontend test coverage for server side search
1620736	P3	Perfherder	Graphs View - Add Table view param in URL
1620737	P3	Perfherder	Graphs View - When toggle to Table View, it should focus on table
1620739	P3	Perfherder	Graphs View - TableView and GraphTooltip component share helpers functions
1620741	P3	Perfherder	Graphs View - Table view could have trends and outliers navigation to improve accessibility
1622564	P3	Perfherder	Improve performance of graphs view when showing a large date range
1622567	P3	Perfherder	Add trend lines to graphs
1622569	P3	Perfherder	Add graphical plots for subtests
1622578	P3	Perfherder	Add the ability to use the shortened hash for a commit in the compare view
1622829	P3	Perfherder	Consider changing default display to more than 2 days
1627922	P3	Perfherder	Reassign Alerts dialog doesn't have autofocus
1633660	P3	Perfherder	Attribute inconsistency in alert summary serializer
1635085	P3	Perfherder	Provide test coverage for when Compare view is changing its URL
1635127	P3	Perfherder	Alert summaries should show the status of linked bug
1639112	P3	Perfherder	Consider making SecretaryTool a singleton
1639248	P3	Perfherder	Feature request to save perfherder comparisons for future lookup
1639277	P3	Perfherder	tp6 tests unnecessarily require multiple tasks to increase confidence on compare chooser
1639308	P3	Perfherder	Show the last commit message when comparing two revisions
1641623	P3	Perfherder	Add a numerical summary of Perfherder improvement/regression to the Compare Revisions view
1644707	P3	Perfherder	Perfherder: Add test for platform dropdown options order
1646259	P3	Perfherder	Add/modify perfherder graph view to better communicate noise
1649156	P3	Perfherder	Distinguish subtest items from test items in alert summaries
1650153	P3	Perfherder	Expose performance properties for simple (or complex!) filtering
1655019	P3	Perfherder	Display a name/unit for summary results
1665100	P3	Perfherder	Perherder should be able to show task name for a graph
1684946	P3	Perfherder	browsertime vismet tasks with --rebuild flag all reuse a single test result. Perfherder says Confidence is Infinity (high).
1689584	P3	Perfherder	Provide POC for integrating Mann-Whitney-U
1689586	P3	Perfherder	Provide actual plan for integrating MWU
1693051	P3	Perfherder	Update tests with suite-level values to produce replicates at the suite-level
1700970	P3	Perfherder	Perfherder compare Retrigger Jobs button no longer retriggers jobs
1701615	P3	Perfherder	Group alert items in tests+subtests to visually distinguish them easier.
1723073	P3	Perfherder	Have properties to distinguish manual/auto backfilled jobs
1727787	P3	Perfherder	Perfherder should show `unit` and `lower_is_better` in addition to existing color-coding on e.g. the Alert view
1728295	P3	Perfherder	Improve test coverage for report_backfill_outcome command
1729665	P3	Perfherder	Add a performance tab in Push Health
1734898	P3	Perfherder	Add support for filtering out matches in perfherder compare
1741786	P3	Perfherder	Performance regression detected between m-c build and try build with the same revision
1744024	P3	Perfherder	fission/non-fission awsy jobs are mixed up in the display, and also in retrigger
1751158	P3	Perfherder	Last page is not removed from pagination control when resolving a previous alert
1754151	P3	Perfherder	Perfherder subtest in Compare View misses the first letter of the test name
1756774	P3	Perfherder	Tests View - Changing the framework should reset the table to page 1
1773562	P3	Perfherder	[perfdocs] Integrate perfdocs test owner information into our alerts
1837203	P3	Perfherder	Add a way to tag infra changes
1838924	P3	Perfherder	Integrate sitespeedio on treeherder for performance testing
1855728	P3	Perfherder	Add Slack and/or Element notifications for performance alerts
1867284	P3	Perfherder	Add tooltip to "Use replicates" button
1869245	P3	Perfherder	Show common alerts along with the normal alert in graph tooltip
1874894	P3	Perfherder	Allow us to annotate data points in the graph view
1894726	P3	Perfherder	Compare View should allow us to change the method used to summarize the values
1901066	--	Perfherder	When clicking a link from perfherder you should have the treeherder view filtered to just that job	Andrej (:aglavic)
1971105	--	Perfherder	Graphs View - Flickering when hovering over data point while already having one selected	Florin Bilt
1986824	--	Perfherder	Ensure PERFHERDER_DATA is also uploaded as perfherder-data.json artifacts across all test harnesses	Myeongjun Go
1996563	--	Perfherder	Generate perfherder_data artifacts for jobs running fetch-content	Myeongjun Go
2001364	--	Perfherder	Ensure perfherder data is output to a JSON file for Firefox iOS performance tests	Myeongjun Go
1960354	--	Perfherder	Broken links in performance improvement/regression bugzilla comments due to wrong anchor name
1940375	--	Perfherder	Make https://treeherder.mozilla.org/perfherder/graphs more ergonomic to use when trying to filter/select large number of tests/metrics/subtests
1962733	--	Perfherder	Highlight summaries that include critical signatures in Alerts View
1963918	--	Perfherder	Highlight infra changes button should be unselected by default when opening an alert
1967765	--	Perfherder	Graphs View - Make the original test run task more visible
1975449	--	Perfherder	Common alerts for the same framework are not displayed on the graph
1940232	--	Perfherder	links to perfherder data series from Treeherder not working if older than 28 days because purged? Something went wrong TypeError: can't access property "y", e[0] is undefined
1980743	--	Perfherder	intermittent test_identify_retriggerables_selects_even_single_data_point - assert {4} == {92}
1982029	--	Perfherder	Kraken subtests show "score" but say "lower is better"
1982829	--	Perfherder	Automate leaving improvement comments on bugzilla from Alerts View
1983826	--	Perfherder	Perfherder: Fix intermittent failing test in test_identify_retriggerables.py
1938332	--	Perfherder	Add mechanism to change "from" revision for alert summaries
1919432	--	Perfherder	AWFY -Motionmark graphs are not updating since 16sep2024 (maybe because the old version of Motionmark was replaced by Motionmark1.3.1)
1912030	--	Perfherder	Magnitude sorting on the alert summary page is confusing
1991089	--	Perfherder	Add PerfCompare link to regression bug in comment 0
1908641	--	Perfherder	Perfherder's Tests View is currently broken
1902168	--	Perfherder	Show reassigned alerts in graphs view
1901067	--	Perfherder	Button to open all alerts on perfherder
2000858	--	Perfherder	incorrectly placed popup does not point to performance data point
2001125	--	Perfherder	Set up Monitoring for Decision task
1649551	--	Perfherder	add job_type_id to performance_signature table
1689373	--	Perfherder	[meta] Provide plan for using Mann-Whitney-U alongside student' s t-test
1698545	--	Perfherder	[meta] Link to test documentation
1709778	--	Perfherder	[meta] Distinguish between auto/manual backfills
1809467	--	Perfherder	frequent intermittent failures on collapsable_rows_tests.jsx and test_perfcompare_api.py, disabled for now
1818857	--	Perfherder	If retriggers fail on PerfHerder due to authentication issues, the error message gets thrown off the top of the viewport
1819632	--	Perfherder	Hitting enter/return in test filter on perfherder graphs page shouldn't reload the page
1819634	--	Perfherder	It's too difficult to look at m-c results for try graphs
1823531	--	Perfherder	arewefastyet.com speedometer toplevel seems to be inverted
1831319	--	Perfherder	Add percentage indicator to standard deviation in compare view
1836081	--	Perfherder	perfherder graph does not show older data for damp signatures even if they are in the displayed timeframe
1839320	--	Perfherder	Hide triaged alerts from untriaged queue
1842195	--	Perfherder	Set status flag when filing regression bugs
1850748	--	Perfherder	Add documentation about how to ingest performance data locally
1850756	--	Perfherder	At certain window widths, the perfherder layout is broken
1850759	--	Perfherder	Rectangle drag selection gets stuck if the mouse is released outside the graph
1853410	--	Perfherder	Reduce the radius of the plot points (graphs are too crowded)
1855574	--	Perfherder	Error when clicking "View" for a talos-profiling task: `Something went wrong TypeError: can't access property "y", e[0] is undefined`
1860800	--	Perfherder	Intermittent test_identify_retriggerables.py test_identify_retriggerables_selects_all_data_points assert {1, 2, 4, 7, 9} == {2, 4, 7, 9, 11}
1877784	--	Perfherder	Hovering over a replicate doesn't show you the value for that replicate
1885014	--	Perfherder	The "Use replicates" button on the graph view never works the first time you click it
1900303	--	Perfherder	Two different values for the same alert metric in comment 0 when filling a sheriffing regression bug

179 Total; 179 Open (100%); 0 Resolved (0%); 0 Verified (0%);