About
Perfherder is a dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android).
Perfherder is part of the Treeherder project.
The production version of it lives at https://treeherder.mozilla.org/perf.html
Team
- Talos Sheriffs (:jmaher)
- Developers (:wlach, :rwood)
Goals
Develop tools and integration of them to assist sheriffs to find and report regressions to developers, and to assist developers in understanding the impact of their changes and potential fixes.
This includes:
- Graphical display of performance data over time
- An easy way to compare all performance tests for a given revision to another revision (or time range)
- An alert system to provide notification when there has been a level shift in performance data, along with tools for filing bugs when such events occur
- Integration with normal developer work flows for using the Try server and landing code
Non Goals
- fixing tests
- fixing harnesses
- adding new tests
- bisection of a range of commits to find the root cause
Dependencies / Who will use this
We have a dependency upon tools to generate data. Currently this is Talos, but the design is to allow all other types of tools to post here as well.
The other big dependency we have is TreeHerder. Right now we are tightly integrated into the project and use a lot of common data (branches, revisions, history, pushlog). We fully expect this to remain the case and will do our best to integrate as much as possible if it helps the workflow of the users.
The primary users will be:
Secondary users will be:
- manager (project, release)
- qa
Contribution
We welcome contributions to Perfherder! We track issues inside the Perfherder component on bugzilla, see below for a list of open bugs.
The contribution section of the Treeherder page has some handy resources for those just getting started. In general, if you have a development version of treeherder running, you're all set to hack on Perfherder! Just jump to:
If you have a question, jump on #perfherder on irc.mozilla.org and look for wlach or jmaher (mostly online in EST timezone).
Source and docs
See the Treeherder page.
Bug Table
Full Query
| ID |
Priority |
Component |
Summary |
Assigned to |
| 1470851 |
-- |
Perfherder |
Disable build_metrics tests from devedition builds |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1445252 |
-- |
Perfherder |
Add test data dialog should allow resizing |
|
| 1459623 |
-- |
Perfherder |
Improve noticing alerts for big time ranges |
|
| 1465588 |
-- |
Perfherder |
Consider removing dashboard code |
|
| 1465589 |
-- |
Perfherder |
Convert perfherder services to static functions |
|
| 1467182 |
-- |
Perfherder |
Filtering talos tests for uppercase "ARES" doesn't work |
|
| 1470473 |
-- |
Perfherder |
Add more basic perfherder selenium tests |
|
| 1481817 |
-- |
Perfherder |
[meta] - tracking bug for tier2 alerts / self managed alerts / sub test alerts |
|
| 1481821 |
-- |
Perfherder |
perfherder compare is very limited when developers are investigating a regression or making their test relevant or stable |
|
| 1484714 |
-- |
Perfherder |
make it easier to push to try and compare against a base revision |
|
| 1487761 |
-- |
Perfherder |
Display try push message in PerfHerder |
|
| 1490447 |
-- |
Perfherder |
Alerts with subtests should hint which subtests have regressed |
|
| 1496035 |
-- |
Perfherder |
"better" and "worse" are backwards for octane |
|
| 1500528 |
-- |
Perfherder |
Make perfherder comparisons consistent across benchmarks |
|
| 1499319 |
P1 |
Perfherder |
Allow for setting states from inside Perfherder popup |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1500841 |
P1 |
Perfherder |
Nudging alerts breaks some links |
Ionuț Goldan [:igoldan], Performance Sheriffing |
| 1423607 |
P1 |
Perfherder |
Decide whether less Perfherder data needs to be retained indefinitely |
|
| 1431085 |
P1 |
Perfherder |
Perfherder log parsing "django.db.utils:DataError: (1406, "Data too long for column 'extra_options' at row 1")"" |
|
| 1436819 |
P1 |
Perfherder |
generate alerts for performance tests that only run on mozilla-central |
|
| 1481728 |
P1 |
Perfherder |
Improvement alerts aren't generated automatically |
|
| 1496321 |
P1 |
Perfherder |
Alert summaries with unknown revisions |
|
| 1426714 |
P3 |
Perfherder |
Switch remaining Perfherder graphs from jQuery Flot to metrics-graphics |
Yunhao Zhang [:wangjie] |
| 1170639 |
P3 |
Perfherder |
Provide a way of using compareperf with locally produced talos data |
|
| 1170648 |
P3 |
Perfherder |
Provide some kind of documentation of what the calculations in compareperf are |
|
| 1202718 |
P3 |
Perfherder |
Add ability to retrigger talos tests directly from the "compare performance" UI |
|
| 1224571 |
P3 |
Perfherder |
when viewing data in perfherder, allow custom platforms, right now this is auto detected |
|
| 1247997 |
P3 |
Perfherder |
Not immediately obvious whether a result in perfherder compare view is a regression/improvement without being able to see the colour |
|
| 1266451 |
P3 |
Perfherder |
Add a unit test for import_perf_data |
|
| 1268247 |
P3 |
Perfherder |
Alert management UI doesn't really scale to uplifts |
|
| 1272716 |
P3 |
Perfherder |
Perfherder - subtest alerts should use historical vs forward comparisons instead of revision comparisons |
|
| 1277954 |
P3 |
Perfherder |
links to perfherder graphs are relative to the date of opening the graph, not the date when the link was shared |
|
| 1283819 |
P3 |
Perfherder |
[tracking bug] - dashboard and/or tool changes to accommodate data needs of release management |
|
| 1285322 |
P3 |
Perfherder |
Implement auto-classification of "downstream" alerts |
|
| 1316590 |
P3 |
Perfherder |
Compare view should allow viewing subtests across all tests |
|
| 1329240 |
P3 |
Perfherder |
Add a link to the pushlog for the visible changeset range |
|
| 1330948 |
P3 |
Perfherder |
Add an option to show trend lines in graph view |
|
| 1332323 |
P3 |
Perfherder |
"file bug" in perfherder alerts should auto-suggest blocking bug + component based on commit range of regression |
|
| 1335784 |
P3 |
Perfherder |
Use qparent for default original revision in perf hearder links in try server emails |
|
| 1343328 |
P3 |
Perfherder |
Add data expiration for performance_datum or switch to bigint to protect against auto_increment overflow |
|
| 1382371 |
P3 |
Perfherder |
Compare performance result link should take me to a comparison for that test |
|
| 1383901 |
P3 |
Perfherder |
make a simpler tool to compare memory profiles from AWSY |
|
| 1404865 |
P3 |
Perfherder |
Add CSV Export feature for Perfherder Compare Revisions page |
|
| 1416861 |
P3 |
Perfherder |
Do away with the signature hash (for real this time) |
|
| 1418929 |
P3 |
Perfherder |
g2/DAMP tests are not displayed on PerfHerder/compare page |
|
| 1425493 |
P3 |
Perfherder |
Perfherder's alerting algorithm doesn't handle multi-modal data well |
|
| 1429809 |
P3 |
Perfherder |
import_perf_data management command is currently broken |
|
| 1450044 |
P3 |
Perfherder |
[meta] Convert Perfherder to ReactJS |
|
| 1452238 |
P3 |
Perfherder |
test_alerts_post doesn't set or test the PerformanceDatum's job_id |
|
| 1470601 |
P3 |
Perfherder |
Navigating back from subtests causes full reload of comparison view |
|
| 1410430 |
P3 |
Perfherder |
Consider adding a visual summary of subtests on top-level compare view |
Robert Wood [:rwood] |
| 1383514 |
P4 |
Perfherder |
Add ability to view sorted data in replicate distribution view |
|
| 1200217 |
P5 |
Perfherder |
UX improvement for Compare - provide valid, default landing values for Submit |
|
| 1202731 |
P5 |
Perfherder |
Add a way to link to a particular set of results in comparechooser UI |
|
| 1231368 |
P5 |
Perfherder |
Reassigning alerts doesn't check for errors |
|
| 1239821 |
P5 |
Perfherder |
on alerts page of perfherder, there is a link to the jobs, this should filter by talos |
|
| 1252814 |
P5 |
Perfherder |
perfherder pgo alerts should have an easy way to see the non pgo alerts |
|
| 1253218 |
P5 |
Perfherder |
when using compare view in perfherder it is unclear that the base revision needs retriggered jobs |
|
| 1255001 |
P5 |
Perfherder |
The 'important changes' and 'uncertain results' checkboxes don't behave as advertised |
|
| 1258845 |
P5 |
Perfherder |
when two alerts have the same target revision but different revision ranges we have different alerts and it is confusing |
|
| 1265381 |
P5 |
Perfherder |
Consider alerting with less than 12 future data points if we're very confident there is a regression |
|
| 1266183 |
P5 |
Perfherder |
Submissions with varying numbers of subtests end up in different graphs |
|
| 1269489 |
P5 |
Perfherder |
Automatically track standard deviation in performance tests over time |
|
| 1289540 |
P5 |
Perfherder |
alert pagination make it difficult to find alerts |
|
| 1468943 |
P5 |
Perfherder |
Mark confirmed alerts automatically |
|
64 Total;
64 Open (100%);
0 Resolved (0%);
0 Verified (0%);
