About
Perfherder is a dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android).
Perfherder is part of the Treeherder project.
The production version of it lives at https://treeherder.mozilla.org/perf.html
Team
- Talos Sheriffs (:jmaher)
- Developers (:wlach, :rwood)
Goals
Develop tools and integration of them to assist sheriffs to find and report regressions to developers, and to assist developers in understanding the impact of their changes and potential fixes.
This includes:
- Graphical display of performance data over time
- An easy way to compare all performance tests for a given revision to another revision (or time range)
- An alert system to provide notification when there has been a level shift in performance data, along with tools for filing bugs when such events occur
- Integration with normal developer work flows for using the Try server and landing code
Non Goals
- fixing tests
- fixing harnesses
- adding new tests
- bisection of a range of commits to find the root cause
Dependencies / Who will use this
We have a dependency upon tools to generate data. Currently this is Talos, but the design is to allow all other types of tools to post here as well.
The other big dependency we have is TreeHerder. Right now we are tightly integrated into the project and use a lot of common data (branches, revisions, history, pushlog). We fully expect this to remain the case and will do our best to integrate as much as possible if it helps the workflow of the users.
The primary users will be:
Secondary users will be:
- manager (project, release)
- qa
Contribution
We welcome contributions to Perfherder! We track issues inside the Perfherder component on bugzilla, see below for a list of open bugs.
The contribution section of the Treeherder page has some handy resources for those just getting started. In general, if you have a development version of treeherder running, you're all set to hack on Perfherder! Just jump to:
If you have a question, jump on #perfherder on irc.mozilla.org and look for wlach or jmaher (mostly online in EST timezone).
Source and docs
See the Treeherder page.
Bug Table
Full Query
| ID |
Priority |
Component |
Summary |
Assigned to |
| 1519290 |
-- |
Perfherder |
Convert compare replicates view to React |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1445252 |
-- |
Perfherder |
Add test data dialog should allow resizing |
|
| 1459623 |
-- |
Perfherder |
Improve noticing alerts for big time ranges |
|
| 1467182 |
-- |
Perfherder |
Filtering talos tests for uppercase "ARES" doesn't work |
|
| 1481817 |
-- |
Perfherder |
[meta] - tracking bug for tier2 alerts / self managed alerts / sub test alerts |
|
| 1481821 |
-- |
Perfherder |
perfherder compare is very limited when developers are investigating a regression or making their test relevant or stable |
|
| 1484714 |
-- |
Perfherder |
make it easier to push to try and compare against a base revision |
|
| 1487761 |
-- |
Perfherder |
Display try push message in PerfHerder |
|
| 1490447 |
-- |
Perfherder |
Alerts with subtests should hint which subtests have regressed |
|
| 1496035 |
-- |
Perfherder |
"better" and "worse" are backwards for octane |
|
| 1500528 |
-- |
Perfherder |
Make perfherder comparisons consistent across benchmarks |
|
| 1501247 |
-- |
Perfherder |
Integrate tagging functionality |
|
| 1502088 |
-- |
Perfherder |
Alerts subtest links should point to comparison corresponding to their original summary in case they are reassigned |
|
| 1506590 |
-- |
Perfherder |
add ability in alert manager to search by bug |
|
| 1507490 |
-- |
Perfherder |
Alert action bar interferes with status dropdown |
|
| 1509216 |
-- |
Perfherder |
Convert compare view to React |
|
| 1509217 |
-- |
Perfherder |
Convert comparesubtest view to React |
|
| 1515681 |
-- |
Perfherder |
Perfherder Compare should not allow (or highly warn) when comparing a try build against a mozilla-central build |
|
| 1515847 |
-- |
Perfherder |
Perfherder UI is slow to load / times out |
|
| 1516630 |
-- |
Perfherder |
delete old frameworks and alerts that are not used |
|
| 1517436 |
-- |
Perfherder |
Add more Selenium tests for Graphs view |
|
| 1517437 |
-- |
Perfherder |
Selenium tests for Alerts view |
|
| 1517512 |
-- |
Perfherder |
Long subtest names are not readable when adding graph data |
|
| 1518492 |
-- |
Perfherder |
Error when hovering over downstreamed alerts |
|
| 1518497 |
-- |
Perfherder |
Big arrow buttons |
|
| 1519289 |
-- |
Perfherder |
Remove usage of signature_hash in Perfherder |
|
| 1519610 |
-- |
Perfherder |
Label vertical axis in graph |
|
| 1519991 |
-- |
Perfherder |
Convert graph modal to a React component |
|
| 1519995 |
-- |
Perfherder |
Convert graphs view to React |
|
| 1520712 |
-- |
Perfherder |
Implement Ranking system |
|
| 1520714 |
-- |
Perfherder |
Provide alert summary assignment |
|
| 1520739 |
-- |
Perfherder |
Show measurement units |
|
| 1521047 |
-- |
Perfherder |
Subtest comparison page is broken (at least on DAMP for try runs) |
Sarah Clements [:sclements] |
| 1501239 |
P1 |
Perfherder |
Test validity dashboard |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1510944 |
P1 |
Perfherder |
Add unit tests for PhAlerts' functions |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1511821 |
P1 |
Perfherder |
Replace old .then() syntax with async/await |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1519323 |
P1 |
Perfherder |
Convert revisionInformation to ReactJS |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1423607 |
P1 |
Perfherder |
Decide whether less Perfherder data needs to be retained indefinitely |
|
| 1481728 |
P1 |
Perfherder |
Improvement alerts aren't generated automatically |
|
| 1496321 |
P1 |
Perfherder |
Alert summaries with unknown revisions |
|
| 1503510 |
P1 |
Perfherder |
Add unit tests for Perfherder |
|
| 1509052 |
P1 |
Perfherder |
Compare view breaks |
|
| 1450044 |
P1 |
Perfherder |
[meta] Convert Perfherder to ReactJS |
Sarah Clements [:sclements] |
| 1509214 |
P1 |
Perfherder |
Convert PhCompareTable to ReactJS |
Sarah Clements [:sclements] |
| 1518408 |
P2 |
Perfherder |
Move test documentation from wiki to readthedocs |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1518534 |
P2 |
Perfherder |
Hide 'tests with no results' section |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1515576 |
P2 |
Perfherder |
Starring too slow |
|
| 1521025 |
P2 |
Perfherder |
Add extra timestamp fields on perf alerts |
|
| 1426714 |
P3 |
Perfherder |
Switch remaining Perfherder graphs from jQuery Flot to metrics-graphics |
Yunhao Zhang [:wangjie] |
| 1501241 |
P3 |
Perfherder |
[meta] Make states more manageable |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1512382 |
P3 |
Perfherder |
Provide a well structured tutorial documentation for Perfherder |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1170639 |
P3 |
Perfherder |
Provide a way of using compareperf with locally produced talos data |
|
| 1170648 |
P3 |
Perfherder |
Provide some kind of documentation of what the calculations in compareperf are |
|
| 1202718 |
P3 |
Perfherder |
Add ability to retrigger talos tests directly from the "compare performance" UI |
|
| 1224571 |
P3 |
Perfherder |
when viewing data in perfherder, allow custom platforms, right now this is auto detected |
|
| 1247997 |
P3 |
Perfherder |
Not immediately obvious whether a result in perfherder compare view is a regression/improvement without being able to see the colour |
|
| 1266451 |
P3 |
Perfherder |
Add a unit test for import_perf_data |
|
| 1268247 |
P3 |
Perfherder |
Alert management UI doesn't really scale to uplifts |
|
| 1272716 |
P3 |
Perfherder |
Perfherder - subtest alerts should use historical vs forward comparisons instead of revision comparisons |
|
| 1277954 |
P3 |
Perfherder |
links to perfherder graphs are relative to the date of opening the graph, not the date when the link was shared |
|
| 1283819 |
P3 |
Perfherder |
[tracking bug] - dashboard and/or tool changes to accommodate data needs of release management |
|
| 1285322 |
P3 |
Perfherder |
Implement auto-classification of "downstream" alerts |
|
| 1316590 |
P3 |
Perfherder |
Compare view should allow viewing subtests across all tests |
|
| 1329240 |
P3 |
Perfherder |
Add a link to the pushlog for the visible changeset range |
|
| 1330948 |
P3 |
Perfherder |
Add an option to show trend lines in graph view |
|
| 1332323 |
P3 |
Perfherder |
"file bug" in perfherder alerts should auto-suggest blocking bug + component based on commit range of regression |
|
| 1335784 |
P3 |
Perfherder |
Use qparent for default original revision in perf hearder links in try server emails |
|
| 1343328 |
P3 |
Perfherder |
Add data expiration for performance_datum or switch to bigint to protect against auto_increment overflow |
|
| 1382371 |
P3 |
Perfherder |
Compare performance result link should take me to a comparison for that test |
|
| 1383901 |
P3 |
Perfherder |
make a simpler tool to compare memory profiles from AWSY |
|
| 1404865 |
P3 |
Perfherder |
Add CSV Export feature for Perfherder Compare Revisions page |
|
| 1416861 |
P3 |
Perfherder |
Do away with the signature hash (for real this time) |
|
| 1418929 |
P3 |
Perfherder |
g2/DAMP tests are not displayed on PerfHerder/compare page |
|
| 1425493 |
P3 |
Perfherder |
Perfherder's alerting algorithm doesn't handle multi-modal data well |
|
| 1429809 |
P3 |
Perfherder |
import_perf_data management command is currently broken |
|
| 1436819 |
P3 |
Perfherder |
generate alerts for performance tests that only run on mozilla-central |
|
| 1452238 |
P3 |
Perfherder |
test_alerts_post doesn't set or test the PerformanceDatum's job_id |
|
| 1470473 |
P3 |
Perfherder |
[meta] Add more basic perfherder selenium tests |
|
| 1470601 |
P3 |
Perfherder |
Navigating back from subtests causes full reload of comparison view |
|
| 1470851 |
P3 |
Perfherder |
Disable build_metrics tests from devedition builds |
|
| 1499319 |
P3 |
Perfherder |
Allow for setting states from inside Perfherder popup |
|
| 1500841 |
P3 |
Perfherder |
Nudging alerts breaks some links |
|
| 1501245 |
P3 |
Perfherder |
Duplicate rest of states to tags |
|
| 1501249 |
P3 |
Perfherder |
Entirely replace rest of states to tags |
|
| 1501252 |
P3 |
Perfherder |
Bring the tags into our graphs |
|
| 1501309 |
P3 |
Perfherder |
Design custom tags |
|
| 1502805 |
P3 |
Perfherder |
Integrate browser notifications |
|
| 1515338 |
P3 |
Perfherder |
State machine diagram |
|
| 1520709 |
P3 |
Perfherder |
[meta] Scalable perf sheriffing |
|
| 1520719 |
P3 |
Perfherder |
Provide high overview of available alerts |
|
| 1520720 |
P3 |
Perfherder |
[meta] Perfherder enhancements |
|
| 1520726 |
P3 |
Perfherder |
Check fixes for perf regressions |
|
| 1520731 |
P3 |
Perfherder |
Directly retrigger & backfill |
|
| 1520735 |
P3 |
Perfherder |
Warn about platforms with limited capacity |
|
| 1520736 |
P3 |
Perfherder |
Default to Alerts view |
|
| 1410430 |
P3 |
Perfherder |
Consider adding a visual summary of subtests on top-level compare view |
Robert Wood [:rwood] |
| 1383514 |
P4 |
Perfherder |
Add ability to view sorted data in replicate distribution view |
|
| 1200217 |
P5 |
Perfherder |
UX improvement for Compare - provide valid, default landing values for Submit |
|
| 1202731 |
P5 |
Perfherder |
Add a way to link to a particular set of results in comparechooser UI |
|
| 1231368 |
P5 |
Perfherder |
Reassigning alerts doesn't check for errors |
|
| 1239821 |
P5 |
Perfherder |
on alerts page of perfherder, there is a link to the jobs, this should filter by talos |
|
| 1252814 |
P5 |
Perfherder |
perfherder pgo alerts should have an easy way to see the non pgo alerts |
|
| 1253218 |
P5 |
Perfherder |
when using compare view in perfherder it is unclear that the base revision needs retriggered jobs |
|
| 1255001 |
P5 |
Perfherder |
The 'important changes' and 'uncertain results' checkboxes don't behave as advertised |
|
| 1258845 |
P5 |
Perfherder |
when two alerts have the same target revision but different revision ranges we have different alerts and it is confusing |
|
| 1265381 |
P5 |
Perfherder |
Consider alerting with less than 12 future data points if we're very confident there is a regression |
|
| 1266183 |
P5 |
Perfherder |
Submissions with varying numbers of subtests end up in different graphs |
|
| 1269489 |
P5 |
Perfherder |
Automatically track standard deviation in performance tests over time |
|
| 1289540 |
P5 |
Perfherder |
alert pagination make it difficult to find alerts |
|
| 1468943 |
P5 |
Perfherder |
Mark confirmed alerts automatically |
|
110 Total;
110 Open (100%);
0 Resolved (0%);
0 Verified (0%);
