EngineeringProductivity/Projects/Perfherder

From MozillaWiki
Jump to: navigation, search

About

Perfherder is a dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android).

Perfherder is part of the Treeherder project. The production version of it lives at https://treeherder.mozilla.org/perf.html

Team

  • Talos Sheriffs (:jmaher)
  • Developers (:wlach, :rwood)

Goals

Develop tools and integration of them to assist sheriffs to find and report regressions to developers, and to assist developers in understanding the impact of their changes and potential fixes.

This includes:

  • Graphical display of performance data over time
  • An easy way to compare all performance tests for a given revision to another revision (or time range)
  • An alert system to provide notification when there has been a level shift in performance data, along with tools for filing bugs when such events occur
  • Integration with normal developer work flows for using the Try server and landing code

Non Goals

  • fixing tests
  • fixing harnesses
  • adding new tests
  • bisection of a range of commits to find the root cause

Dependencies / Who will use this

We have a dependency upon tools to generate data. Currently this is Talos, but the design is to allow all other types of tools to post here as well.

The other big dependency we have is TreeHerder. Right now we are tightly integrated into the project and use a lot of common data (branches, revisions, history, pushlog). We fully expect this to remain the case and will do our best to integrate as much as possible if it helps the workflow of the users.

The primary users will be:

  • sheriffs
  • developers

Secondary users will be:

  • manager (project, release)
  • qa

Contribution

We welcome contributions to Perfherder! We track issues inside the Perfherder component on bugzilla, see below for a list of open bugs.

The contribution section of the Treeherder page has some handy resources for those just getting started. In general, if you have a development version of treeherder running, you're all set to hack on Perfherder! Just jump to:

If you have a question, jump on #perfherder on irc.mozilla.org and look for wlach or jmaher (mostly online in EST timezone).

Source and docs

See the Treeherder page.

Bug Table

ID Priority Component Summary Assigned to
1266451 -- Perfherder Add a unit test for import_perf_data Anjana Vakil (:vakila)
1285322 -- Perfherder Implement auto-classification of "downstream" alerts crosscent
1332323 -- Perfherder "file bug" in perfherder alerts should auto-suggest blocking bug + component based on commit range of regression George Veneel Dogga [::TheLayman]
1170639 -- Perfherder Provide a way of using compareperf with locally produced talos data
1170648 -- Perfherder Provide some kind of documentation of what the calculations in compareperf are
1171703 -- Perfherder Indicate whether higher or lower is better in perfherder compare summary even for those tests without enough data to detect a statistically significant change
1202718 -- Perfherder Add ability to retrigger talos tests directly from the "compare performance" UI
1202731 -- Perfherder Add a way to link to a particular set of results in comparechooser UI
1210509 -- Perfherder perfherder compare view should do a better job dealing with high std dev sets of data
1224571 -- Perfherder when viewing data in perfherder, allow custom platforms, right now this is auto detected
1229275 -- Perfherder Perfherder doesn't distinguish between linux64 debug builds and linux64 debug static analysis builds
1229504 -- Perfherder small tweaks to alert view for maximum productivity - adding a compare view
1231368 -- Perfherder Reassigning alerts doesn't check for errors
1239821 -- Perfherder on alerts page of perfherder, there is a link to the jobs, this should filter by talos
1247997 -- Perfherder Not immediately obvious whether a result in perfherder compare view is a regression/improvement without being able to see the colour
1252814 -- Perfherder perfherder pgo alerts should have an easy way to see the non pgo alerts
1253218 -- Perfherder when using compare view in perfherder it is unclear that the base revision needs retriggered jobs
1255001 -- Perfherder The 'important changes' and 'uncertain results' checkboxes don't behave as advertised
1258845 -- Perfherder when two alerts have the same target revision but different revision ranges we have different alerts and it is confusing
1261969 -- Perfherder [Meta] Features necessary for using Perfherder for sheriffing AWFY data
1265381 -- Perfherder Consider alerting with less than 12 future data points if we're very confident there is a regression
1265709 -- Perfherder The performance data API should be rewritten to not be tied to signature hashes
1266183 -- Perfherder Submissions with varying numbers of subtests end up in different graphs
1268247 -- Perfherder Alert management UI doesn't really scale to uplifts
1269489 -- Perfherder Automatically track standard deviation in performance tests over time
1272716 -- Perfherder Perfherder - subtest alerts should use historical vs forward comparisons instead of revision comparisons
1277954 -- Perfherder links to perfherder graphs are relative to the date of opening the graph, not the date when the link was shared
1283819 -- Perfherder [tracking bug] - dashboard and/or tool changes to accommodate data needs of release management
1289540 -- Perfherder alert pagination make it difficult to find alerts
1316590 -- Perfherder Compare view should allow viewing subtests across all tests
1324859 -- Perfherder post build_metrics alerts to mozilla.dev.builds
1329240 -- Perfherder Add a link to the pushlog for the visible changeset range
1330948 -- Perfherder Add an option to show trend lines in graph view
1335784 -- Perfherder Use qparent for default original revision in perf hearder links in try server emails
1363967 -- Perfherder Automatic base revision is wrong
1382371 -- Perfherder Compare performance result link should take me to a comparison for that test
1383123 -- Perfherder Distribution view gives mixed active content warnings when served over HTTPs
1329224 -- Perfherder Treeherder graph URLs are unnecessarily long, making them difficult to share on irc Shruti Jasoria [:ShrutiJ]
1350384 -- Perfherder Add ability to see a visualization of the test replicates in Perfherder Shruti Jasoria [:ShrutiJ]
1376829 -- Perfherder Add notes to alerts William Lachance (:wlach) (use needinfo!)
1381904 -- Perfherder Viewing subtests never loads William Lachance (:wlach) (use needinfo!)
1173399 P2 Perfherder Selecting Compare in Perf complains in the console about invalid plot dimensions
1308122 P2 Perfherder PerformanceSignatureViewSet.list "ValueError: invalid literal for int() with base 10: '1 and 2'" MikeLing
1288857 P3 Perfherder Create a Perfherder dashboard for Stylo similar to the e10s one Chris Peterson [:cpeterson]
1200217 P5 Perfherder UX improvement for Compare - provide valid, default landing values for Submit
1343328 P5 Perfherder Add data expiration for performance_datum or switch to bigint to protect against auto_increment overflow

46 Total; 46 Open (100%); 0 Resolved (0%); 0 Verified (0%);