EngineeringProductivity/Projects/Perfherder

From MozillaWiki
Jump to: navigation, search

About

Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android).

Perfherder is part of the Treeherder project. The production version of it lives at https://treeherder.mozilla.org/perf.html

Team

  • Sheriffs (:igoldan, :bebe)
  • Developers (:sclements, :igoldan)

Goals

Develop tools and integration of them to assist sheriffs to find and report regressions to developers, and to assist developers in understanding the impact of their changes and potential fixes.

This includes:

  • Graphical display of performance data over time
  • An easy way to compare all performance tests for a given revision to another revision (or time range)
  • An alert system to provide notification when there has been a level shift in performance data, along with tools for filing bugs when such events occur
  • Integration with normal developer work flows for using the Try server and landing code

Non Goals

  • fixing tests
  • fixing harnesses
  • adding new tests
  • bisection of a range of commits to find the root cause

Dependencies / Who will use this

We have a dependency upon tools to generate data. Currently this is Talos, but the design is to allow all other types of tools to post here as well.

The other big dependency we have is TreeHerder. Right now we are tightly integrated into the project and use a lot of common data (branches, revisions, history, pushlog). We fully expect this to remain the case and will do our best to integrate as much as possible if it helps the workflow of the users.

The primary users will be:

  • sheriffs
  • developers

Secondary users will be:

  • manager (project, release)
  • qa

Contribution

We welcome contributions to Perfherder! We track issues inside the Perfherder component on bugzilla, see below for a list of open bugs.

The contribution section of the Treeherder page has some handy resources for those just getting started. In general, if you have a development version of treeherder running, you're all set to hack on Perfherder! Just jump to:

If you have a question, jump on #perfherder on irc.mozilla.org and look for :igoldan or :davehunt.

Source and docs

See the Treeherder page.

Bug Table

Full Query
ID Priority Component Summary Assigned to
1595111 P1 Perfherder Alerts view gets stuck on last page Alexandru Ionescu :alexandrui (needinfo me)
1564833 P1 Perfherder [meta] Server side search in Alerts view Alexandru Irimovici
1616242 P1 Perfherder Provide frontend test coverage for server side search Alexandru Irimovici
1618840 P1 Perfherder Create a tool that checks and updates the status of all backfilled jobs Alexandru Irimovici
1612547 P1 Perfherder Define component for doing backfills using existing reports Ionuț Goldan [:igoldan]
1615540 P1 Perfherder Remove AWFY from Perfherder Kimberly Sereduck :kimberlythegeek
1616759 P1 Perfherder Provide test coverage for grouping of common repositories Kimberly Sereduck :kimberlythegeek
1618578 P1 Perfherder All perfherder dropdowns have the items in alphabetical order Kimberly Sereduck :kimberlythegeek
1611158 P1 Perfherder Subtest view - include the platform in the subtest name kumar.ishita
1617121 P1 Perfherder [meta] Mitigate race conditions between reporter bot & retrigger bot
1626011 P1 Perfherder Adjust AWSY perfherder templates to remove erahm
1616215 P1 Perfherder Warn when user filters by option or repository Octavian Negru
1615253 P1 Perfherder Graphs View: Add the Time range dropdown in `Add test data modal` persdre
1619724 P1 Perfherder Alert's name should not be inside the table shamar knibbs
1609363 P1 Perfherder Order platform names alphabetically Subramanian
1600759 P1 Perfherder Investigate creation of an alternative view to the Graphs view for screen reader users Mellina Yonashiro
1626548 P2 Perfherder Check outcome of backfills Alexandru Irimovici
1570226 P2 Perfherder [meta] Automatically backfill jobs for new alerts Ionuț Goldan [:igoldan]
1613925 P2 Perfherder Prepare credentials for backfill bot Ionuț Goldan [:igoldan]
1616240 P2 Perfherder Provide backend test coverage for server side search Ionuț Goldan [:igoldan]
1618825 P2 Perfherder Rename the synthesize_backfill_report management script to something more relevant Ionuț Goldan [:igoldan]
1571372 P2 Perfherder Setup Heroku Scheduler for retrigger bot
1587018 P2 Perfherder Display the public variant of suite & tests, if available
1593815 P2 Perfherder Perfherder doesn't reload if you change the URL
1596372 P2 Perfherder How should we surface measurement units in Graphs view?
1600004 P2 Perfherder Replace current push from alert summaries with previous push & current push tuple
1607710 P2 Perfherder Input filter from TestDataModal doesn't properly update signature list
1614586 P2 Perfherder Run measure noise tool over all sheriffed perf tests
1618823 P2 Perfherder [meta] PerfSheriffBot
1618829 P2 Perfherder Enable all feature flags defined
1618832 P2 Perfherder [meta] Backfill tool
1618836 P2 Perfherder Integrate the backfill logic in the PerfSheriffBot
1618838 P2 Perfherder [meta] Secretary tool
1618842 P2 Perfherder Integrate the secretary tool into the PerfSheriffBot
1618844 P2 Perfherder [meta] Use the provided Taskcluster credentials
1618845 P2 Perfherder Setup environment variable via Heroku
1627265 P2 Perfherder At data ingestion distinguish between firefox and chrome applications Octavian Negru
1605352 P3 Perfherder Update sheriffing regression templates to use the same template text for descriptions Alexandru Ionescu :alexandrui (needinfo me)
1617820 P3 Perfherder Blank pages at the end of the results queue Ionuț Goldan [:igoldan]
1265381 P3 Perfherder Consider alerting with less than 12 future data points if we're very confident there is a regression Kyle Lahnakoski [:ekyle]
1615365 P3 Perfherder Redirecting to correct url on changing the status or framework filter kumar.ishita
1268247 P3 Perfherder Alert management UI doesn't really scale to uplifts
1316590 P3 Perfherder Compare view should allow viewing subtests across all tests
1343328 P3 Perfherder Add data expiration for performance_datum or switch to bigint to protect against auto_increment overflow
1481728 P3 Perfherder Improvement alerts aren't generated automatically
1481821 P3 Perfherder perfherder compare is very limited when developers are investigating a regression or making their test relevant or stable
1484714 P3 Perfherder make it easier to push to try and compare against a base revision
1500528 P3 Perfherder Make perfherder comparisons consistent across benchmarks
1501239 P3 Perfherder Test validity dashboard
1501241 P3 Perfherder [meta] Make states more manageable
1501247 P3 Perfherder Integrate tagging functionality
1506590 P3 Perfherder add ability in alert manager to search by bug
1515338 P3 Perfherder State machine diagram
1515681 P3 Perfherder Perfherder Compare should not allow (or highly warn) when comparing a try build against a mozilla-central build
1519289 P3 Perfherder Remove usage of signature_hash in Perfherder
1520726 P3 Perfherder Check fixes for perf regressions
1520731 P3 Perfherder Directly retrigger & backfill
1520735 P3 Perfherder Warn about platforms with limited capacity
1522871 P3 Perfherder It would be very useful to have a link for the pushlog for each set of data visible in the graph
1528740 P3 Perfherder Notify on new alerts/no more data from existing alerts
1529090 P3 Perfherder No way to easily examine all the performance data for a try push
1532283 P3 Perfherder Improve & re enable nudge
1532995 P3 Perfherder Cover compare replicates view with FE tests
1536018 P3 Perfherder Alert summary's status doesn't properly update
1539854 P3 Perfherder Timeranges in Perfherder URL are limited to 1, 2, 7, 14, 30, 60, 90 days
1543016 P3 Perfherder Android PGO platforms generating OPT metrics
1543780 P3 Perfherder Make https://treeherder.mozilla.org/api/project/{repo}/performance/signatures faster
1546054 P3 Perfherder Add new "Top alerts" view
1550014 P3 Perfherder Allow sorting by table column on compare subtest page
1551556 P3 Perfherder Add support to enable/disable alerting on suite and sub-test level
1558746 P3 Perfherder Reflect confirmed commits on alerts
1562178 P3 Perfherder [meta] Automate backfills and retriggers for alerts
1565516 P3 Perfherder [meta] Improvements to compare view
1569018 P3 Perfherder Graphs of try vs try pushes show irrelevant comparison points
1569022 P3 Perfherder graph backdrop for pushes should be based on base revision, not push date
1569024 P3 Perfherder select the right test for the graph backdrop when adding more test data to graph
1569165 P3 Perfherder Measure machine time of retriggered perf jobs
1569590 P3 Perfherder Mention pending retriggers in compare view
1570225 P3 Perfherder Automatically retrigger jobs for new perf regressions
1570906 P3 Perfherder Performance alert created from a [tier 3] job
1570952 P3 Perfherder Distinguish between manual & automated padfills
1571378 P3 Perfherder Provide CRUD endpoint for backfill reports
1571379 P3 Perfherder Integrate mailing service
1571381 P3 Perfherder Provide basic landing page for backfill endpoint
1571382 P3 Perfherder Email backfill reports
1571383 P3 Perfherder Provide backfill endpoint
1572722 P3 Perfherder [meta] Improve accessibility
1573131 P3 Perfherder Automatically identify backouts
1581533 P3 Perfherder Discussion: Improving performance regression detection
1582710 P3 Perfherder [meta] UI/UX improvements to Graphs view
1582711 P3 Perfherder [meta] Introduce noise alerts for perf tests
1583539 P3 Perfherder Provide mechanism for renaming suite/test names
1584900 P3 Perfherder [meta] Tag infrastructure changes into Graphs view
1585650 P3 Perfherder [meta] Provide boxplots in Graphs view
1587766 P3 Perfherder build times should be grouped by dimension sccache hit rate
1588026 P3 Perfherder Endpoint for test validity dashboard must provide tiers
1590627 P3 Perfherder Clean databases of remnants of old unused frameworks
1591044 P3 Perfherder New Tier column to Test Validity Dashboard
1591045 P3 Perfherder Backfill reports should consider indirect alerts also
1591053 P3 Perfherder Change logic for the Graphs view "Add related configurations" hyperlinks
1593206 P3 Perfherder Make application fields mandatory in validation schema
1596347 P3 Perfherder Investigate perf data produced by job_resource_usage
1600668 P3 Perfherder Error out if PERFHERDER_DATA is malformed
1601952 P3 Perfherder Use The Mann-Whitney-U test
1604868 P3 Perfherder Unable to remove series from graph view
1604871 P3 Perfherder Unable to add data to graph when there are no recent results
1605240 P3 Perfherder Vertical graph labels are truncated on most significant digits
1605247 P3 Perfherder Consider adding more options for date ranges
1605748 P3 Perfherder Perfherder regression filing should inform the group/builder of a talos regression
1611203 P3 Perfherder Make it possible to embed Perfherder graphs in other pages
1615227 P3 Perfherder AlertsView: Add first and last page to pagination
1615267 P3 Perfherder Tests View: Consistent renaming from health/validity to Tests View
1615268 P3 Perfherder [meta] Improvements to Tests View
1616191 P3 Perfherder Provide server side filtering by option
1617871 P3 Perfherder Move long-series views to BigQuery?
1619645 P3 Perfherder Improve wording for Magnitude of Difference aria-label
1620736 P3 Perfherder Graphs View - Add Table view param in URL
1620737 P3 Perfherder Graphs View - When toggle to Table View, it should focus on table
1620739 P3 Perfherder Graphs View - TableView and GraphTooltip component share helpers functions
1620741 P3 Perfherder Graphs View - Table view could have trends and outliers navigation
1622564 P3 Perfherder Improve performance of graphs view when showing a large date range
1622567 P3 Perfherder Add trend lines to graphs
1622568 P3 Perfherder Allow base/new revisions to be edited from the compare view
1622569 P3 Perfherder Add graphical plots for subtests
1622570 P3 Perfherder Add test descriptions to all views
1622572 P3 Perfherder Enable re-triggering test runs with profiler enabled
1622573 P3 Perfherder Data Review of Compare View
1622575 P3 Perfherder Improve value of compare view
1622576 P3 Perfherder Add the ability to compare more than two revisions at the same time
1622578 P3 Perfherder Add the ability to use the shortened hash for a commit in the compare view
1622796 P3 Perfherder Add release tag to sheriffing template
1622829 P3 Perfherder Consider changing default display to more than 2 days
1623168 P3 Perfherder Expose information on modality of the results and alert when modality changes between results
1523037 P3 Perfherder Page titles for subtests are confusing parshva.barbhaya
1533002 P3 Perfherder Add proper error handling in ReplicatesGraph parshva.barbhaya
1622579 P3 Perfherder Improve UX of compare view persdre
1502805 P3 Perfherder Integrate browser notifications xuan pan

137 Total; 137 Open (100%); 0 Resolved (0%); 0 Verified (0%);

Retrieved from "https://wiki.mozilla.org/index.php?title=EngineeringProductivity/Projects/Perfherder&oldid=1213381"