About
Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android).
Perfherder is part of the Treeherder project.
The production version of it lives at https://treeherder.mozilla.org/perf.html
Team
- Sheriffs (:igoldan, :bebe)
- Developers (:igoldan)
Goals
Develop tools and integration of them to assist sheriffs to find and report regressions to developers, and to assist developers in understanding the impact of their changes and potential fixes.
This includes:
- Graphical display of performance data over time
- An easy way to compare all performance tests for a given revision to another revision (or time range)
- An alert system to provide notification when there has been a level shift in performance data, along with tools for filing bugs when such events occur
- Integration with normal developer work flows for using the Try server and landing code
Non Goals
- fixing tests
- fixing harnesses
- adding new tests
- bisection of a range of commits to find the root cause
Dependencies / Who will use this
We have a dependency upon tools to generate data. Currently this is Talos, but the design is to allow all other types of tools to post here as well.
The other big dependency we have is TreeHerder. Right now we are tightly integrated into the project and use a lot of common data (branches, revisions, history, pushlog). We fully expect this to remain the case and will do our best to integrate as much as possible if it helps the workflow of the users.
The primary users will be:
Secondary users will be:
- manager (project, release)
- qa
Contribution
We welcome contributions to Perfherder! We track issues inside the Perfherder component on bugzilla, see below for a list of open bugs.
The contribution section of the Treeherder page has some handy resources for those just getting started. In general, if you have a development version of treeherder running, you're all set to hack on Perfherder! Just jump to:
If you have a question, jump on #perfherder on irc.mozilla.org and look for :igoldan or :davehunt.
Source and docs
See the Treeherder page.
Bug Table
Full Query
| ID |
Priority |
Component |
Summary |
Assigned to |
| 1510944 |
P1 |
Perfherder |
Add unit tests for PhAlerts' functions |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1511821 |
P1 |
Perfherder |
Replace old .then() syntax with async/await |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1524521 |
P1 |
Perfherder |
Provisioning local perf data is hard |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1423607 |
P1 |
Perfherder |
Decide whether less Perfherder data needs to be retained indefinitely |
|
| 1481728 |
P1 |
Perfherder |
Improvement alerts aren't generated automatically |
|
| 1496321 |
P1 |
Perfherder |
Alert summaries with unknown revisions |
|
| 1503510 |
P1 |
Perfherder |
Add unit tests for Perfherder |
|
| 1517512 |
P1 |
Perfherder |
Long subtest names are not readable when adding graph data |
|
| 1523330 |
P1 |
Perfherder |
Compare subtest view error |
|
| 1450044 |
P1 |
Perfherder |
[meta] Convert Perfherder to ReactJS |
Sarah Clements [:sclements] |
| 1509216 |
P1 |
Perfherder |
Convert compare view to React |
Sarah Clements [:sclements] |
| 1509217 |
P1 |
Perfherder |
Convert comparesubtest view to React |
Sarah Clements [:sclements] |
| 1532995 |
P2 |
Perfherder |
Cover compare replicates view with FE tests |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1533002 |
P2 |
Perfherder |
Add proper error handling in ReplicatesGraph |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1496035 |
P2 |
Perfherder |
"better" and "worse" are backwards for octane |
|
| 1507490 |
P2 |
Perfherder |
Alert action bar interferes with status dropdown |
|
| 1515576 |
P2 |
Perfherder |
Starring too slow |
|
| 1517436 |
P2 |
Perfherder |
Add more Selenium tests for Graphs view |
|
| 1517437 |
P2 |
Perfherder |
Selenium tests for Alerts view |
|
| 1518408 |
P2 |
Perfherder |
Move test documentation from wiki to readthedocs |
|
| 1520712 |
P2 |
Perfherder |
Implement Ranking system |
|
| 1523037 |
P2 |
Perfherder |
Page titles for subtests are confusing |
|
| 1523301 |
P2 |
Perfherder |
Noise metric in comparison view should be green when it decreases |
|
| 1523710 |
P2 |
Perfherder |
Convert Alerts section/controllers to React |
|
| 1524732 |
P2 |
Perfherder |
perfherder graphs y-axis are hard to read for large values |
|
| 1528740 |
P2 |
Perfherder |
Notify on new alerts/no more data from existing alerts |
|
| 1536018 |
P2 |
Perfherder |
Alert summary's status doesn't properly update |
|
| 1426714 |
P3 |
Perfherder |
Switch remaining Perfherder graphs from jQuery Flot to metrics-graphics |
Yunhao Zhang [:wangjie] |
| 1316590 |
P3 |
Perfherder |
Compare view should allow viewing subtests across all tests |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1170639 |
P3 |
Perfherder |
Provide a way of using compareperf with locally produced talos data |
|
| 1170648 |
P3 |
Perfherder |
Provide some kind of documentation of what the calculations in compareperf are |
|
| 1202718 |
P3 |
Perfherder |
Add ability to retrigger talos tests directly from the "compare performance" UI |
|
| 1224571 |
P3 |
Perfherder |
when viewing data in perfherder, allow custom platforms, right now this is auto detected |
|
| 1247997 |
P3 |
Perfherder |
Not immediately obvious whether a result in perfherder compare view is a regression/improvement without being able to see the colour |
|
| 1266451 |
P3 |
Perfherder |
Add a unit test for import_perf_data |
|
| 1268247 |
P3 |
Perfherder |
Alert management UI doesn't really scale to uplifts |
|
| 1272716 |
P3 |
Perfherder |
Perfherder - subtest alerts should use historical vs forward comparisons instead of revision comparisons |
|
| 1277954 |
P3 |
Perfherder |
links to perfherder graphs are relative to the date of opening the graph, not the date when the link was shared |
|
| 1283819 |
P3 |
Perfherder |
[tracking bug] - dashboard and/or tool changes to accommodate data needs of release management |
|
| 1285322 |
P3 |
Perfherder |
Implement auto-classification of "downstream" alerts |
|
| 1329240 |
P3 |
Perfherder |
Add a link to the pushlog for the visible changeset range |
|
| 1330948 |
P3 |
Perfherder |
Add an option to show trend lines in graph view |
|
| 1332323 |
P3 |
Perfherder |
"file bug" in perfherder alerts should auto-suggest blocking bug + component based on commit range of regression |
|
| 1335784 |
P3 |
Perfherder |
Use qparent for default original revision in perf hearder links in try server emails |
|
| 1343328 |
P3 |
Perfherder |
Add data expiration for performance_datum or switch to bigint to protect against auto_increment overflow |
|
| 1382371 |
P3 |
Perfherder |
Compare performance result link should take me to a comparison for that test |
|
| 1383901 |
P3 |
Perfherder |
make a simpler tool to compare memory profiles from AWSY |
|
| 1404865 |
P3 |
Perfherder |
Add CSV Export feature for Perfherder Compare Revisions page |
|
| 1416861 |
P3 |
Perfherder |
Do away with the signature hash (for real this time) |
|
| 1418929 |
P3 |
Perfherder |
g2/DAMP tests are not displayed on PerfHerder/compare page |
|
| 1425493 |
P3 |
Perfherder |
Perfherder's alerting algorithm doesn't handle multi-modal data well |
|
| 1429809 |
P3 |
Perfherder |
import_perf_data management command is currently broken |
|
| 1436819 |
P3 |
Perfherder |
generate alerts for performance tests that only run on mozilla-central |
|
| 1452238 |
P3 |
Perfherder |
test_alerts_post doesn't set or test the PerformanceDatum's job_id |
|
| 1459623 |
P3 |
Perfherder |
Improve noticing alerts for big time ranges |
|
| 1467182 |
P3 |
Perfherder |
Filtering talos tests for uppercase "ARES" doesn't work |
|
| 1470473 |
P3 |
Perfherder |
[meta] Add more basic perfherder selenium tests |
|
| 1470599 |
P3 |
Perfherder |
Compare result against another revision should take you to the right framework |
|
| 1470601 |
P3 |
Perfherder |
Navigating back from subtests causes full reload of comparison view |
|
| 1470851 |
P3 |
Perfherder |
Disable build_metrics tests from devedition builds |
|
| 1481821 |
P3 |
Perfherder |
perfherder compare is very limited when developers are investigating a regression or making their test relevant or stable |
|
| 1484714 |
P3 |
Perfherder |
make it easier to push to try and compare against a base revision |
|
| 1487761 |
P3 |
Perfherder |
Display try push message in PerfHerder |
|
| 1490447 |
P3 |
Perfherder |
Alerts with subtests should hint which subtests have regressed |
|
| 1499319 |
P3 |
Perfherder |
Allow for setting states from inside Perfherder popup |
|
| 1500528 |
P3 |
Perfherder |
Make perfherder comparisons consistent across benchmarks |
|
| 1500841 |
P3 |
Perfherder |
Nudging alerts breaks some links |
|
| 1501239 |
P3 |
Perfherder |
Test validity dashboard |
|
| 1501241 |
P3 |
Perfherder |
[meta] Make states more manageable |
|
| 1501245 |
P3 |
Perfherder |
Duplicate rest of states to tags |
|
| 1501247 |
P3 |
Perfherder |
Integrate tagging functionality |
|
| 1501249 |
P3 |
Perfherder |
Entirely replace rest of states to tags |
|
| 1501252 |
P3 |
Perfherder |
Bring the tags into our graphs |
|
| 1501309 |
P3 |
Perfherder |
Design custom tags |
|
| 1502805 |
P3 |
Perfherder |
Integrate browser notifications |
|
| 1506590 |
P3 |
Perfherder |
add ability in alert manager to search by bug |
|
| 1512382 |
P3 |
Perfherder |
Provide a well structured tutorial documentation for Perfherder |
|
| 1515338 |
P3 |
Perfherder |
State machine diagram |
|
| 1515681 |
P3 |
Perfherder |
Perfherder Compare should not allow (or highly warn) when comparing a try build against a mozilla-central build |
|
| 1516630 |
P3 |
Perfherder |
delete old frameworks and alerts that are not used |
|
| 1518492 |
P3 |
Perfherder |
Error when hovering over downstreamed alerts |
|
| 1518497 |
P3 |
Perfherder |
Big arrow buttons |
|
| 1520709 |
P3 |
Perfherder |
[meta] Scalable perf sheriffing |
|
| 1520714 |
P3 |
Perfherder |
Provide alert summary assignment |
|
| 1520719 |
P3 |
Perfherder |
Provide high overview of available alerts |
|
| 1520720 |
P3 |
Perfherder |
[meta] Perfherder enhancements |
|
| 1520726 |
P3 |
Perfherder |
Check fixes for perf regressions |
|
| 1520731 |
P3 |
Perfherder |
Directly retrigger & backfill |
|
| 1520735 |
P3 |
Perfherder |
Warn about platforms with limited capacity |
|
| 1520736 |
P3 |
Perfherder |
Default to Alerts view |
|
| 1520739 |
P3 |
Perfherder |
Show measurement units |
|
| 1522871 |
P3 |
Perfherder |
It would be very useful to have a link for the pushlog for each set of data visible in the graph |
|
| 1525997 |
P3 |
Perfherder |
Please add a way to label perfherder compare pages |
|
| 1529090 |
P3 |
Perfherder |
No way to easily examine all the performance data for a try push |
|
| 1532283 |
P3 |
Perfherder |
Improve & re enable nudge |
|
| 1536040 |
P3 |
Perfherder |
Create a script to measure time-to-bug |
|
| 1410430 |
P3 |
Perfherder |
Consider adding a visual summary of subtests on top-level compare view |
Robert Wood [:rwood] |
| 1383514 |
P4 |
Perfherder |
Add ability to view sorted data in replicate distribution view |
|
| 1200217 |
P5 |
Perfherder |
UX improvement for Compare - provide valid, default landing values for Submit |
|
| 1202731 |
P5 |
Perfherder |
Add a way to link to a particular set of results in comparechooser UI |
|
| 1231368 |
P5 |
Perfherder |
Reassigning alerts doesn't check for errors |
|
| 1239821 |
P5 |
Perfherder |
on alerts page of perfherder, there is a link to the jobs, this should filter by talos |
|
| 1252814 |
P5 |
Perfherder |
perfherder pgo alerts should have an easy way to see the non pgo alerts |
|
| 1253218 |
P5 |
Perfherder |
when using compare view in perfherder it is unclear that the base revision needs retriggered jobs |
|
| 1255001 |
P5 |
Perfherder |
The 'important changes' and 'uncertain results' checkboxes don't behave as advertised |
|
| 1258845 |
P5 |
Perfherder |
when two alerts have the same target revision but different revision ranges we have different alerts and it is confusing |
|
| 1265381 |
P5 |
Perfherder |
Consider alerting with less than 12 future data points if we're very confident there is a regression |
|
| 1266183 |
P5 |
Perfherder |
Submissions with varying numbers of subtests end up in different graphs |
|
| 1269489 |
P5 |
Perfherder |
Automatically track standard deviation in performance tests over time |
|
| 1289540 |
P5 |
Perfherder |
alert pagination make it difficult to find alerts |
|
| 1468943 |
P5 |
Perfherder |
Mark confirmed alerts automatically |
|
| 1519289 |
-- |
Perfherder |
Remove usage of signature_hash in Perfherder |
|
| 1519991 |
-- |
Perfherder |
Convert graph modal to a React component |
|
| 1519995 |
-- |
Perfherder |
Convert graphs view to React |
|
| 1523352 |
-- |
Perfherder |
Track libxul export table size in perfherder |
|
115 Total;
115 Open (100%);
0 Resolved (0%);
0 Verified (0%);
