From MozillaWiki
Jump to: navigation, search


Perfherder is a dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android).

Perfherder is part of the Treeherder project. The production version of it lives at


  • Talos Sheriffs (:jmaher)
  • Developers (:wlach, :rwood)


Develop tools and integration of them to assist sheriffs to find and report regressions to developers, and to assist developers in understanding the impact of their changes and potential fixes.

This includes:

  • Graphical display of performance data over time
  • An easy way to compare all performance tests for a given revision to another revision (or time range)
  • An alert system to provide notification when there has been a level shift in performance data, along with tools for filing bugs when such events occur
  • Integration with normal developer work flows for using the Try server and landing code

Non Goals

  • fixing tests
  • fixing harnesses
  • adding new tests
  • bisection of a range of commits to find the root cause

Dependencies / Who will use this

We have a dependency upon tools to generate data. Currently this is Talos, but the design is to allow all other types of tools to post here as well.

The other big dependency we have is TreeHerder. Right now we are tightly integrated into the project and use a lot of common data (branches, revisions, history, pushlog). We fully expect this to remain the case and will do our best to integrate as much as possible if it helps the workflow of the users.

The primary users will be:

  • sheriffs
  • developers

Secondary users will be:

  • manager (project, release)
  • qa


We welcome contributions to Perfherder! We track issues inside the Perfherder component on bugzilla, see below for a list of open bugs.

The contribution section of the Treeherder page has some handy resources for those just getting started. In general, if you have a development version of treeherder running, you're all set to hack on Perfherder! Just jump to:

If you have a question, jump on #perfherder on and look for wlach or jmaher (mostly online in EST timezone).

Source and docs

See the Treeherder page.

Bug Table

Full Query
ID Priority Component Summary Assigned to
1519290 -- Perfherder Convert compare replicates view to React Ionuț Goldan [:igoldan], Performance Sheriffing
1445252 -- Perfherder Add test data dialog should allow resizing
1459623 -- Perfherder Improve noticing alerts for big time ranges
1467182 -- Perfherder Filtering talos tests for uppercase "ARES" doesn't work
1481817 -- Perfherder [meta] - tracking bug for tier2 alerts / self managed alerts / sub test alerts
1481821 -- Perfherder perfherder compare is very limited when developers are investigating a regression or making their test relevant or stable
1484714 -- Perfherder make it easier to push to try and compare against a base revision
1487761 -- Perfherder Display try push message in PerfHerder
1490447 -- Perfherder Alerts with subtests should hint which subtests have regressed
1496035 -- Perfherder "better" and "worse" are backwards for octane
1500528 -- Perfherder Make perfherder comparisons consistent across benchmarks
1501247 -- Perfherder Integrate tagging functionality
1502088 -- Perfherder Alerts subtest links should point to comparison corresponding to their original summary in case they are reassigned
1506590 -- Perfherder add ability in alert manager to search by bug
1507490 -- Perfherder Alert action bar interferes with status dropdown
1509216 -- Perfherder Convert compare view to React
1509217 -- Perfherder Convert comparesubtest view to React
1515681 -- Perfherder Perfherder Compare should not allow (or highly warn) when comparing a try build against a mozilla-central build
1515847 -- Perfherder Perfherder UI is slow to load / times out
1516630 -- Perfherder delete old frameworks and alerts that are not used
1517436 -- Perfherder Add more Selenium tests for Graphs view
1517437 -- Perfherder Selenium tests for Alerts view
1517512 -- Perfherder Long subtest names are not readable when adding graph data
1518492 -- Perfherder Error when hovering over downstreamed alerts
1518497 -- Perfherder Big arrow buttons
1519289 -- Perfherder Remove usage of signature_hash in Perfherder
1519610 -- Perfherder Label vertical axis in graph
1519991 -- Perfherder Convert graph modal to a React component
1519995 -- Perfherder Convert graphs view to React
1520712 -- Perfherder Implement Ranking system
1520714 -- Perfherder Provide alert summary assignment
1520739 -- Perfherder Show measurement units
1521047 -- Perfherder Subtest comparison page is broken (at least on DAMP for try runs) Sarah Clements [:sclements]
1501239 P1 Perfherder Test validity dashboard Ionuț Goldan [:igoldan], Performance Sheriffing
1510944 P1 Perfherder Add unit tests for PhAlerts' functions Ionuț Goldan [:igoldan], Performance Sheriffing
1511821 P1 Perfherder Replace old .then() syntax with async/await Ionuț Goldan [:igoldan], Performance Sheriffing
1519323 P1 Perfherder Convert revisionInformation to ReactJS Ionuț Goldan [:igoldan], Performance Sheriffing
1423607 P1 Perfherder Decide whether less Perfherder data needs to be retained indefinitely
1481728 P1 Perfherder Improvement alerts aren't generated automatically
1496321 P1 Perfherder Alert summaries with unknown revisions
1503510 P1 Perfherder Add unit tests for Perfherder
1509052 P1 Perfherder Compare view breaks
1450044 P1 Perfherder [meta] Convert Perfherder to ReactJS Sarah Clements [:sclements]
1509214 P1 Perfherder Convert PhCompareTable to ReactJS Sarah Clements [:sclements]
1518408 P2 Perfherder Move test documentation from wiki to readthedocs Ionuț Goldan [:igoldan], Performance Sheriffing
1518534 P2 Perfherder Hide 'tests with no results' section Ionuț Goldan [:igoldan], Performance Sheriffing
1515576 P2 Perfherder Starring too slow
1521025 P2 Perfherder Add extra timestamp fields on perf alerts
1426714 P3 Perfherder Switch remaining Perfherder graphs from jQuery Flot to metrics-graphics Yunhao Zhang [:wangjie]
1501241 P3 Perfherder [meta] Make states more manageable Ionuț Goldan [:igoldan], Performance Sheriffing
1512382 P3 Perfherder Provide a well structured tutorial documentation for Perfherder Ionuț Goldan [:igoldan], Performance Sheriffing
1170639 P3 Perfherder Provide a way of using compareperf with locally produced talos data
1170648 P3 Perfherder Provide some kind of documentation of what the calculations in compareperf are
1202718 P3 Perfherder Add ability to retrigger talos tests directly from the "compare performance" UI
1224571 P3 Perfherder when viewing data in perfherder, allow custom platforms, right now this is auto detected
1247997 P3 Perfherder Not immediately obvious whether a result in perfherder compare view is a regression/improvement without being able to see the colour
1266451 P3 Perfherder Add a unit test for import_perf_data
1268247 P3 Perfherder Alert management UI doesn't really scale to uplifts
1272716 P3 Perfherder Perfherder - subtest alerts should use historical vs forward comparisons instead of revision comparisons
1277954 P3 Perfherder links to perfherder graphs are relative to the date of opening the graph, not the date when the link was shared
1283819 P3 Perfherder [tracking bug] - dashboard and/or tool changes to accommodate data needs of release management
1285322 P3 Perfherder Implement auto-classification of "downstream" alerts
1316590 P3 Perfherder Compare view should allow viewing subtests across all tests
1329240 P3 Perfherder Add a link to the pushlog for the visible changeset range
1330948 P3 Perfherder Add an option to show trend lines in graph view
1332323 P3 Perfherder "file bug" in perfherder alerts should auto-suggest blocking bug + component based on commit range of regression
1335784 P3 Perfherder Use qparent for default original revision in perf hearder links in try server emails
1343328 P3 Perfherder Add data expiration for performance_datum or switch to bigint to protect against auto_increment overflow
1382371 P3 Perfherder Compare performance result link should take me to a comparison for that test
1383901 P3 Perfherder make a simpler tool to compare memory profiles from AWSY
1404865 P3 Perfherder Add CSV Export feature for Perfherder Compare Revisions page
1416861 P3 Perfherder Do away with the signature hash (for real this time)
1418929 P3 Perfherder g2/DAMP tests are not displayed on PerfHerder/compare page
1425493 P3 Perfherder Perfherder's alerting algorithm doesn't handle multi-modal data well
1429809 P3 Perfherder import_perf_data management command is currently broken
1436819 P3 Perfherder generate alerts for performance tests that only run on mozilla-central
1452238 P3 Perfherder test_alerts_post doesn't set or test the PerformanceDatum's job_id
1470473 P3 Perfherder [meta] Add more basic perfherder selenium tests
1470601 P3 Perfherder Navigating back from subtests causes full reload of comparison view
1470851 P3 Perfherder Disable build_metrics tests from devedition builds
1499319 P3 Perfherder Allow for setting states from inside Perfherder popup
1500841 P3 Perfherder Nudging alerts breaks some links
1501245 P3 Perfherder Duplicate rest of states to tags
1501249 P3 Perfherder Entirely replace rest of states to tags
1501252 P3 Perfherder Bring the tags into our graphs
1501309 P3 Perfherder Design custom tags
1502805 P3 Perfherder Integrate browser notifications
1515338 P3 Perfherder State machine diagram
1520709 P3 Perfherder [meta] Scalable perf sheriffing
1520719 P3 Perfherder Provide high overview of available alerts
1520720 P3 Perfherder [meta] Perfherder enhancements
1520726 P3 Perfherder Check fixes for perf regressions
1520731 P3 Perfherder Directly retrigger & backfill
1520735 P3 Perfherder Warn about platforms with limited capacity
1520736 P3 Perfherder Default to Alerts view
1410430 P3 Perfherder Consider adding a visual summary of subtests on top-level compare view Robert Wood [:rwood]
1383514 P4 Perfherder Add ability to view sorted data in replicate distribution view
1200217 P5 Perfherder UX improvement for Compare - provide valid, default landing values for Submit
1202731 P5 Perfherder Add a way to link to a particular set of results in comparechooser UI
1231368 P5 Perfherder Reassigning alerts doesn't check for errors
1239821 P5 Perfherder on alerts page of perfherder, there is a link to the jobs, this should filter by talos
1252814 P5 Perfherder perfherder pgo alerts should have an easy way to see the non pgo alerts
1253218 P5 Perfherder when using compare view in perfherder it is unclear that the base revision needs retriggered jobs
1255001 P5 Perfherder The 'important changes' and 'uncertain results' checkboxes don't behave as advertised
1258845 P5 Perfherder when two alerts have the same target revision but different revision ranges we have different alerts and it is confusing
1265381 P5 Perfherder Consider alerting with less than 12 future data points if we're very confident there is a regression
1266183 P5 Perfherder Submissions with varying numbers of subtests end up in different graphs
1269489 P5 Perfherder Automatically track standard deviation in performance tests over time
1289540 P5 Perfherder alert pagination make it difficult to find alerts
1468943 P5 Perfherder Mark confirmed alerts automatically

110 Total; 110 Open (100%); 0 Resolved (0%); 0 Verified (0%);