EngineeringProductivity/Projects/Perfherder

From MozillaWiki
Jump to: navigation, search

About

Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android).

Perfherder is part of the Treeherder project. The production version of it lives at https://treeherder.mozilla.org/perf.html

Team

  • Sheriffs (:igoldan, :bebe)
  • Developers (:sclements, :igoldan)

Goals

Develop tools and integration of them to assist sheriffs to find and report regressions to developers, and to assist developers in understanding the impact of their changes and potential fixes.

This includes:

  • Graphical display of performance data over time
  • An easy way to compare all performance tests for a given revision to another revision (or time range)
  • An alert system to provide notification when there has been a level shift in performance data, along with tools for filing bugs when such events occur
  • Integration with normal developer work flows for using the Try server and landing code

Non Goals

  • fixing tests
  • fixing harnesses
  • adding new tests
  • bisection of a range of commits to find the root cause

Dependencies / Who will use this

We have a dependency upon tools to generate data. Currently this is Talos, but the design is to allow all other types of tools to post here as well.

The other big dependency we have is TreeHerder. Right now we are tightly integrated into the project and use a lot of common data (branches, revisions, history, pushlog). We fully expect this to remain the case and will do our best to integrate as much as possible if it helps the workflow of the users.

The primary users will be:

  • sheriffs
  • developers

Secondary users will be:

  • manager (project, release)
  • qa

Contribution

We welcome contributions to Perfherder! We track issues inside the Perfherder component on bugzilla, see below for a list of open bugs.

The contribution section of the Treeherder page has some handy resources for those just getting started. In general, if you have a development version of treeherder running, you're all set to hack on Perfherder! Just jump to:

If you have a question, jump on #perfherder on irc.mozilla.org and look for :igoldan or :davehunt.

Source and docs

See the Treeherder page.

Bug Table

Full Query
ID Priority Component Summary Assigned to
1202718 P1 Perfherder Add ability to retrigger talos tests directly from the "compare performance" UI Alexandru Irimovici
1569584 P1 Perfherder Integrate retriggering dialog on Compare view Alexandru Irimovici
1571411 P1 Perfherder Initial test validity dashboard Alexandru Irimovici
1580687 P1 Perfherder Provide backend endpoint for test validity dashboard Alexandru Irimovici
1343328 P1 Perfherder Add data expiration for performance_datum or switch to bigint to protect against auto_increment overflow Ionuț Goldan [:igoldan], Performance Sheriff
1569588 P1 Perfherder Analyze how Treeherder retriggers selected jobs Ionuț Goldan [:igoldan], Performance Sheriff
1585966 P1 Perfherder Centered header titles from Compare views should be editable Octavian Negru
1520714 P2 Perfherder Provide alert summary assignment Alexandru Irimovici
1565516 P2 Perfherder [meta] Improvements to compare view Alexandru Irimovici
1565480 P2 Perfherder Add pagination at the top of the alerts page Alexandru Ionescu :alexandrui
1569165 P2 Perfherder Measure machine time of retriggered perf jobs Ionuț Goldan [:igoldan], Performance Sheriff
1516630 P2 Perfherder delete old frameworks and alerts that are not used Octavian Negru
1581061 P2 Perfherder Include perf-alert keyword into our bug templates Octavian Negru
1558746 P3 Perfherder Reflect confirmed commits on alerts Alexandru Irimovici
1584135 P3 Perfherder Expected bottom labels from Graphs view to date back from September, not January Albiona
1550014 P3 Perfherder Allow sorting by table column on compare subtest page Alexandru Ionescu :alexandrui
1316590 P3 Perfherder Compare view should allow viewing subtests across all tests Ionuț Goldan [:igoldan], Performance Sheriff
1503510 P3 Perfherder [meta] Add unit tests for Perfherder Ionuț Goldan [:igoldan], Performance Sheriff
1510944 P3 Perfherder Add unit tests for PhAlerts' functions Ionuț Goldan [:igoldan], Performance Sheriff
1520719 P3 Perfherder Provide high overview of available alerts Ionuț Goldan [:igoldan], Performance Sheriff
1524732 P3 Perfherder perfherder graphs y-axis are hard to read for large values Ionuț Goldan [:igoldan], Performance Sheriff
1532995 P3 Perfherder Cover compare replicates view with FE tests Ionuț Goldan [:igoldan], Performance Sheriff
1533002 P3 Perfherder Add proper error handling in ReplicatesGraph Ionuț Goldan [:igoldan], Performance Sheriff
1569166 P3 Perfherder Measure machine time of backfilled perf jobs Ionuț Goldan [:igoldan], Performance Sheriff
1582710 P3 Perfherder [meta] UI/UX improvements to Graphs view Ionuț Goldan [:igoldan], Performance Sheriff
1582711 P3 Perfherder [meta] Introduce noise alerts for perf tests Ionuț Goldan [:igoldan], Performance Sheriff
1170639 P3 Perfherder Provide a way of using compareperf with locally produced talos data
1170648 P3 Perfherder Provide some kind of documentation of what the calculations in compareperf are
1200217 P3 Perfherder UX improvement for Compare - provide valid, default landing values for Submit
1224571 P3 Perfherder when viewing data in perfherder, allow custom platforms, right now this is auto detected
1268247 P3 Perfherder Alert management UI doesn't really scale to uplifts
1272716 P3 Perfherder Perfherder - subtest alerts should use historical vs forward comparisons instead of revision comparisons
1277954 P3 Perfherder links to perfherder graphs are relative to the date of opening the graph, not the date when the link was shared
1285322 P3 Perfherder Implement auto-classification of "downstream" alerts
1329240 P3 Perfherder Add a link to the pushlog for the visible changeset range
1330948 P3 Perfherder Add an option to show trend lines in graph view
1332323 P3 Perfherder "file bug" in perfherder alerts should auto-suggest blocking bug + component based on commit range of regression
1335784 P3 Perfherder Use qparent for default original revision in perf hearder links in try server emails
1383901 P3 Perfherder make a simpler tool to compare memory profiles from AWSY
1404865 P3 Perfherder Add CSV Export feature for Perfherder Compare Revisions page
1416861 P3 Perfherder Remove signature hash use in API endpoints
1418929 P3 Perfherder g2/DAMP tests are not displayed on PerfHerder/compare page
1425493 P3 Perfherder Perfherder's alerting algorithm doesn't handle multi-modal data well
1436819 P3 Perfherder generate alerts for performance tests that only run on mozilla-central
1452238 P3 Perfherder test_alerts_post doesn't set or test the PerformanceDatum's job_id
1459623 P3 Perfherder Improve noticing alerts for big time ranges
1467182 P3 Perfherder Filtering talos tests for uppercase "ARES" doesn't work
1470473 P3 Perfherder [meta] Add more basic perfherder JS tests
1470851 P3 Perfherder Disable build_metrics tests from devedition builds
1481728 P3 Perfherder Improvement alerts aren't generated automatically
1481821 P3 Perfherder perfherder compare is very limited when developers are investigating a regression or making their test relevant or stable
1484714 P3 Perfherder make it easier to push to try and compare against a base revision
1487761 P3 Perfherder Display try push message in PerfHerder
1490447 P3 Perfherder Alerts with subtests should hint which subtests have regressed
1499319 P3 Perfherder Allow for setting states from inside Perfherder popup
1500528 P3 Perfherder Make perfherder comparisons consistent across benchmarks
1501239 P3 Perfherder Test validity dashboard
1501241 P3 Perfherder [meta] Make states more manageable
1501247 P3 Perfherder Integrate tagging functionality
1501252 P3 Perfherder Bring the tags into our graphs
1501309 P3 Perfherder Design custom tags
1502805 P3 Perfherder Integrate browser notifications
1506590 P3 Perfherder add ability in alert manager to search by bug
1512382 P3 Perfherder Provide a well structured tutorial documentation for Perfherder
1515338 P3 Perfherder State machine diagram
1515681 P3 Perfherder Perfherder Compare should not allow (or highly warn) when comparing a try build against a mozilla-central build
1517436 P3 Perfherder Add more JS tests for Graphs view
1517437 P3 Perfherder Add more JS tests for Alerts view
1519289 P3 Perfherder Remove usage of signature_hash in Perfherder
1520709 P3 Perfherder [meta] Scalable perf sheriffing
1520712 P3 Perfherder Implement Ranking system
1520720 P3 Perfherder [meta] Perfherder enhancements
1520726 P3 Perfherder Check fixes for perf regressions
1520731 P3 Perfherder Directly retrigger & backfill
1520735 P3 Perfherder Warn about platforms with limited capacity
1520739 P3 Perfherder Show measurement units
1522871 P3 Perfherder It would be very useful to have a link for the pushlog for each set of data visible in the graph
1523037 P3 Perfherder Page titles for subtests are confusing
1528740 P3 Perfherder Notify on new alerts/no more data from existing alerts
1529090 P3 Perfherder No way to easily examine all the performance data for a try push
1532283 P3 Perfherder Improve & re enable nudge
1536018 P3 Perfherder Alert summary's status doesn't properly update
1539854 P3 Perfherder Timeranges in Perfherder URL are limited to 1, 2, 7, 14, 30, 60, 90 days
1541299 P3 Perfherder [meta] New Alerts features
1543016 P3 Perfherder Android PGO platforms generating OPT metrics
1543780 P3 Perfherder Make https://treeherder.mozilla.org/api/project/{repo}/performance/signatures faster
1543952 P3 Perfherder Outliers mess up with the alert generation
1546054 P3 Perfherder Add new "Top alerts" view
1551556 P3 Perfherder Add support to enable/disable alerting on suite and sub-test level
1562162 P3 Perfherder Measure machine time of retriggered/backfilled jobs
1562178 P3 Perfherder [meta] Automate backfill and retrigger when regressions are detected
1564833 P3 Perfherder Switch to server side search
1567823 P3 Perfherder Provide markdown support for alert summary notes
1569018 P3 Perfherder Graphs of try vs try pushes show irrelevant comparison points
1569022 P3 Perfherder graph backdrop for pushes should be based on base revision, not push date
1569024 P3 Perfherder select the right test for the graph backdrop when adding more test data to graph
1569586 P3 Perfherder Plug the retriggering into the dialog
1569590 P3 Perfherder Also mention pending base/new jobs
1570225 P3 Perfherder Automatically retrigger jobs for new perf regressions
1570226 P3 Perfherder Automatically backfill jobs for new perf regressions
1570906 P3 Perfherder Performance alert created from a [tier 3] job
1570952 P3 Perfherder Distinguish between manual & automated padfills
1570956 P3 Perfherder Measure machine time of automated backfilled perf jobs
1571366 P3 Perfherder Integrate reporter cronjob
1571369 P3 Perfherder Store backfill reports per alert summary
1571372 P3 Perfherder Integrate backfill bot
1571377 P3 Perfherder Decouple backfill reports from backfill bot
1571378 P3 Perfherder Provide CRUD endpoint for backfill reports
1571379 P3 Perfherder Integrate mailing service
1571381 P3 Perfherder Provide basic landing page for backfill endpoint
1571382 P3 Perfherder Email backfill reports
1571383 P3 Perfherder Provide backfill endpoint
1572722 P3 Perfherder [meta] Improve accessibility
1573131 P3 Perfherder Automatically identify backouts
1574628 P3 Perfherder Improve graph colors and add symbols to better distinguish between tests
1581046 P3 Perfherder [meta] Perf tests pulse notifications
1585650 P3 Perfherder [meta] Provide boxplots in Graphs view
1585656 P3 Perfherder [meta] Improve search capabilities of "Add Test Data" modal
1585969 P3 Perfherder Tab titles should sync with the editable titles from Compare views
1525997 P3 Perfherder Please add a way to label perfherder compare pages Octavian Negru
1581497 P3 Perfherder The graph-tooltip is unaligned with its datapoint Sarah Clements [:sclements]
1583989 P3 Perfherder Adding new tests to graphs view plots data points in white in certain scenarios Sarah Clements [:sclements]
1537890 P3 Perfherder Perfherder shouldn't center numbers in their cells and should use thousands separators for formatting them Viktor Stojanov [:Draci]
1383514 P4 Perfherder Add ability to view sorted data in replicate distribution view
1231368 P5 Perfherder Reassigning alerts doesn't check for errors
1239821 P5 Perfherder on alerts page of perfherder, there is a link to the jobs, this should filter by talos
1253218 P5 Perfherder when using compare view in perfherder it is unclear that the base revision needs retriggered jobs
1255001 P5 Perfherder The 'important changes' and 'uncertain results' checkboxes don't behave as advertised
1258845 P5 Perfherder when two alerts have the same target revision but different revision ranges we have different alerts and it is confusing
1265381 P5 Perfherder Consider alerting with less than 12 future data points if we're very confident there is a regression
1266183 P5 Perfherder Submissions with varying numbers of subtests end up in different graphs
1269489 P5 Perfherder Automatically track standard deviation in performance tests over time
1289540 P5 Perfherder alert pagination make it difficult to find alerts
1581533 -- Perfherder Discussion: Improving performance regression detection
1583539 -- Perfherder Provide mechanism for renaming suite/test names
1584900 -- Perfherder [meta] Tag infrastructure changes into Graphs view
1584946 -- Perfherder Mirror mozilla-central schema change
1584947 -- Perfherder Display measurement units on the Graphs view
1587017 -- Perfherder Provide display variants for suite & test names
1587018 -- Perfherder Display the public variant of suite & tests, if available
1587019 -- Perfherder Provide web API for setting the display names for suites & tests
1587020 -- Perfherder Integrate web API that sets the display names for suites & tests
1587380 -- Perfherder Provide tagging mechanics for perf data
1587382 -- Perfherder Provide browser as a new perf signature property
1587393 -- Perfherder Provide backend support for tagging performance signatures
1587395 -- Perfherder Provide capability of ingesting performance signature tags
1587401 -- Perfherder Mirror new schema validations on Perfherder
1587404 -- Perfherder Integrate tags into all filter text inputs, so they're easily searcheable
1587766 -- Perfherder build times should be grouped by dimension sccache hit rate
1588026 -- Perfherder Endpoint for test validity dashboard must provide tiers

150 Total; 150 Open (100%); 0 Resolved (0%); 0 Verified (0%);

Retrieved from "https://wiki.mozilla.org/index.php?title=EngineeringProductivity/Projects/Perfherder&oldid=1213381"