About
Perfherder is a dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android).
Perfherder is part of the Treeherder project.
The production version of it lives at https://treeherder.mozilla.org/perf.html
Team
- Talos Sheriffs (:jmaher)
- Developers (:wlach, :rwood)
Goals
Develop tools and integration of them to assist sheriffs to find and report regressions to developers, and to assist developers in understanding the impact of their changes and potential fixes.
This includes:
- Graphical display of performance data over time
- An easy way to compare all performance tests for a given revision to another revision (or time range)
- An alert system to provide notification when there has been a level shift in performance data, along with tools for filing bugs when such events occur
- Integration with normal developer work flows for using the Try server and landing code
Non Goals
- fixing tests
- fixing harnesses
- adding new tests
- bisection of a range of commits to find the root cause
Dependencies / Who will use this
We have a dependency upon tools to generate data. Currently this is Talos, but the design is to allow all other types of tools to post here as well.
The other big dependency we have is TreeHerder. Right now we are tightly integrated into the project and use a lot of common data (branches, revisions, history, pushlog). We fully expect this to remain the case and will do our best to integrate as much as possible if it helps the workflow of the users.
The primary users will be:
Secondary users will be:
- manager (project, release)
- qa
Contribution
We welcome contributions to Perfherder! We track issues inside the Perfherder component on bugzilla, see below for a list of open bugs.
The contribution section of the Treeherder page has some handy resources for those just getting started. In general, if you have a development version of treeherder running, you're all set to hack on Perfherder! Just jump to:
If you have a question, jump on #perfherder on irc.mozilla.org and look for wlach or jmaher (mostly online in EST timezone).
Source and docs
See the Treeherder page.
Bug Table
Full Query
| ID |
Priority |
Component |
Summary |
Assigned to |
| 1470473 |
-- |
Perfherder |
Add more basic perfherder selenium tests |
Florin Strugariu [:Bebe] |
| 1470851 |
-- |
Perfherder |
Disable build_metrics tests from devedition builds |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1506553 |
-- |
Perfherder |
Convert PhAlerts model to ReactJS |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1445252 |
-- |
Perfherder |
Add test data dialog should allow resizing |
|
| 1459623 |
-- |
Perfherder |
Improve noticing alerts for big time ranges |
|
| 1467182 |
-- |
Perfherder |
Filtering talos tests for uppercase "ARES" doesn't work |
|
| 1481817 |
-- |
Perfherder |
[meta] - tracking bug for tier2 alerts / self managed alerts / sub test alerts |
|
| 1481821 |
-- |
Perfherder |
perfherder compare is very limited when developers are investigating a regression or making their test relevant or stable |
|
| 1484714 |
-- |
Perfherder |
make it easier to push to try and compare against a base revision |
|
| 1487761 |
-- |
Perfherder |
Display try push message in PerfHerder |
|
| 1490447 |
-- |
Perfherder |
Alerts with subtests should hint which subtests have regressed |
|
| 1496035 |
-- |
Perfherder |
"better" and "worse" are backwards for octane |
|
| 1500528 |
-- |
Perfherder |
Make perfherder comparisons consistent across benchmarks |
|
| 1501247 |
-- |
Perfherder |
Integrate tagging functionality |
|
| 1502088 |
-- |
Perfherder |
Alerts subtest links should point to comparison corresponding to their original summary in case they are reassigned |
|
| 1505955 |
-- |
Perfherder |
Convert Perf models to ReactJS |
|
| 1506590 |
-- |
Perfherder |
add ability in alert manager to search by bug |
|
| 1507490 |
-- |
Perfherder |
Alert action bar interferes with status dropdown |
|
| 1505796 |
-- |
Perfherder |
Perfherder Comparison view is slow |
Sarah Clements [:sclements] |
| 1501239 |
P1 |
Perfherder |
Test validity dashboard |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1507397 |
P1 |
Perfherder |
/performance/data/ endpoint is frequently timing out |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1423607 |
P1 |
Perfherder |
Decide whether less Perfherder data needs to be retained indefinitely |
|
| 1436819 |
P1 |
Perfherder |
generate alerts for performance tests that only run on mozilla-central |
|
| 1481728 |
P1 |
Perfherder |
Improvement alerts aren't generated automatically |
|
| 1496321 |
P1 |
Perfherder |
Alert summaries with unknown revisions |
|
| 1503510 |
P1 |
Perfherder |
Add unit tests for Perfherder |
|
| 1450044 |
P1 |
Perfherder |
[meta] Convert Perfherder to ReactJS |
Sarah Clements [:sclements] |
| 1506671 |
P1 |
Perfherder |
Convert compareChooser to React |
Sarah Clements [:sclements] |
| 1499319 |
P2 |
Perfherder |
Allow for setting states from inside Perfherder popup |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1426714 |
P3 |
Perfherder |
Switch remaining Perfherder graphs from jQuery Flot to metrics-graphics |
Yunhao Zhang [:wangjie] |
| 1501241 |
P3 |
Perfherder |
[meta] Make states more manageable |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1170639 |
P3 |
Perfherder |
Provide a way of using compareperf with locally produced talos data |
|
| 1170648 |
P3 |
Perfherder |
Provide some kind of documentation of what the calculations in compareperf are |
|
| 1202718 |
P3 |
Perfherder |
Add ability to retrigger talos tests directly from the "compare performance" UI |
|
| 1224571 |
P3 |
Perfherder |
when viewing data in perfherder, allow custom platforms, right now this is auto detected |
|
| 1247997 |
P3 |
Perfherder |
Not immediately obvious whether a result in perfherder compare view is a regression/improvement without being able to see the colour |
|
| 1266451 |
P3 |
Perfherder |
Add a unit test for import_perf_data |
|
| 1268247 |
P3 |
Perfherder |
Alert management UI doesn't really scale to uplifts |
|
| 1272716 |
P3 |
Perfherder |
Perfherder - subtest alerts should use historical vs forward comparisons instead of revision comparisons |
|
| 1277954 |
P3 |
Perfherder |
links to perfherder graphs are relative to the date of opening the graph, not the date when the link was shared |
|
| 1283819 |
P3 |
Perfherder |
[tracking bug] - dashboard and/or tool changes to accommodate data needs of release management |
|
| 1285322 |
P3 |
Perfherder |
Implement auto-classification of "downstream" alerts |
|
| 1316590 |
P3 |
Perfherder |
Compare view should allow viewing subtests across all tests |
|
| 1329240 |
P3 |
Perfherder |
Add a link to the pushlog for the visible changeset range |
|
| 1330948 |
P3 |
Perfherder |
Add an option to show trend lines in graph view |
|
| 1332323 |
P3 |
Perfherder |
"file bug" in perfherder alerts should auto-suggest blocking bug + component based on commit range of regression |
|
| 1335784 |
P3 |
Perfherder |
Use qparent for default original revision in perf hearder links in try server emails |
|
| 1343328 |
P3 |
Perfherder |
Add data expiration for performance_datum or switch to bigint to protect against auto_increment overflow |
|
| 1382371 |
P3 |
Perfherder |
Compare performance result link should take me to a comparison for that test |
|
| 1383901 |
P3 |
Perfherder |
make a simpler tool to compare memory profiles from AWSY |
|
| 1404865 |
P3 |
Perfherder |
Add CSV Export feature for Perfherder Compare Revisions page |
|
| 1416861 |
P3 |
Perfherder |
Do away with the signature hash (for real this time) |
|
| 1418929 |
P3 |
Perfherder |
g2/DAMP tests are not displayed on PerfHerder/compare page |
|
| 1425493 |
P3 |
Perfherder |
Perfherder's alerting algorithm doesn't handle multi-modal data well |
|
| 1429809 |
P3 |
Perfherder |
import_perf_data management command is currently broken |
|
| 1452238 |
P3 |
Perfherder |
test_alerts_post doesn't set or test the PerformanceDatum's job_id |
|
| 1470601 |
P3 |
Perfherder |
Navigating back from subtests causes full reload of comparison view |
|
| 1501245 |
P3 |
Perfherder |
Duplicate rest of states to tags |
|
| 1501249 |
P3 |
Perfherder |
Entirely replace rest of states to tags |
|
| 1501252 |
P3 |
Perfherder |
Bring the tags into our graphs |
|
| 1501309 |
P3 |
Perfherder |
Design custom tags |
|
| 1502805 |
P3 |
Perfherder |
Integrate browser notifications |
|
| 1410430 |
P3 |
Perfherder |
Consider adding a visual summary of subtests on top-level compare view |
Robert Wood [:rwood] |
| 1383514 |
P4 |
Perfherder |
Add ability to view sorted data in replicate distribution view |
|
| 1200217 |
P5 |
Perfherder |
UX improvement for Compare - provide valid, default landing values for Submit |
|
| 1202731 |
P5 |
Perfherder |
Add a way to link to a particular set of results in comparechooser UI |
|
| 1231368 |
P5 |
Perfherder |
Reassigning alerts doesn't check for errors |
|
| 1239821 |
P5 |
Perfherder |
on alerts page of perfherder, there is a link to the jobs, this should filter by talos |
|
| 1252814 |
P5 |
Perfherder |
perfherder pgo alerts should have an easy way to see the non pgo alerts |
|
| 1253218 |
P5 |
Perfherder |
when using compare view in perfherder it is unclear that the base revision needs retriggered jobs |
|
| 1255001 |
P5 |
Perfherder |
The 'important changes' and 'uncertain results' checkboxes don't behave as advertised |
|
| 1258845 |
P5 |
Perfherder |
when two alerts have the same target revision but different revision ranges we have different alerts and it is confusing |
|
| 1265381 |
P5 |
Perfherder |
Consider alerting with less than 12 future data points if we're very confident there is a regression |
|
| 1266183 |
P5 |
Perfherder |
Submissions with varying numbers of subtests end up in different graphs |
|
| 1269489 |
P5 |
Perfherder |
Automatically track standard deviation in performance tests over time |
|
| 1289540 |
P5 |
Perfherder |
alert pagination make it difficult to find alerts |
|
| 1468943 |
P5 |
Perfherder |
Mark confirmed alerts automatically |
|
77 Total;
77 Open (100%);
0 Resolved (0%);
0 Verified (0%);
