Perfomatic:Requirements
From MozillaWiki
This page describes the main questions the perf app is supposed to answer.
Must Haves
Nice to have
- Making it easier to correlate the graph to actual events. For example it'd be great to be able to get a mercurial log of all checkins that happened during a range in the graph. Or even pull in the checkin data right into the graph, on request.
- Ability to annotate the graph to point out for example bugs filed for particular regressions, or explain why no bug needs to be filed.
- Better visability of the graphs on the main tinderbox page.
Random ideas
- Average data to 1 day?
-
Configure Talos to not send data for every second of a 30-45 min test. Only have it send the least # of data points. - It's rather unwieldy to sift through all the available data sets for a specific test. Multiple machines, multiple platforms, multiple branches. Not sure what a better UI would be, though.
- It would be nice to be able to add arbitrary performance tests, and have their results plotted. EG, suppose the Places team wanted to benchmark various operations (count bookmarks, get awesomebar results for an input string, etc), and be able to monitor that for regressions.
- There really needs to be some kind of overview page that a sheriff (or person who just checked in code) can watch for regressions. The graphs themselves are not too bad, they just are too tightly focused for broader uses. Johnath did something similar to this: http://people.mozilla.org/~johnath/pdb/, but it stopped working when the old graph server was retired.
- I tend to pick a set of machines (usually the fast running ones for each platform) then want to look at the Ts graphs for them, then Tp, then Twinopen etc. Right now I have to basically remove the existing plots, change the drop down for the new test, add in the machines to do that. Seems there might be some quick way to just switch tests for all the currently selected graphs.
- As far as sifting through data sets, the most useful graph display on the old graph server was the one that showed all graphs for a given test for a given branch. For example, all trunk Tp graphs. This allowed one to try to determine when the regression took place using all the graphs at once. A display like this would be great. If a single graph with multiple lines (in multiple colors) is used, then this plus item 1 is probably great. If we use one graph per line, ideally highlighting a time range on one graph would highlight it on all the others. There should be a link to this report from tinderbox, or from the individual Tp graphs.
- One of the biggest problems we have right now is that we don't have a quick way to check for "are there any performance regressions that should close the tree or require backouts?", so sometimes they're getting discovered days or weeks later.
- Please see bug 653961 for an example of some graphs that we absolutely should be able to get from graphserver and currently can't.
Questions
- Can we reduce the timeframe for #type=series to 15 days? (Temporarily at least)
