342
edits
TestEngineering/Performance/FAQ: Difference between revisions
TestEngineering/Performance/FAQ (view source)
Revision as of 13:05, 16 January 2020
, 16 January 2020→Tree
Alexandrui (talk | contribs) (→Tree) |
Alexandrui (talk | contribs) (→Tree) |
||
| Line 37: | Line 37: | ||
This affects the Performance Sheriffs because we will get a big pile of alerts for Mozilla-Beta. These need to be addressed rapidly. Luckily almost all the regressions seen on Mozilla-Beta will already have been tracked on Mozilla-Inbound or Autoland. | This affects the Performance Sheriffs because we will get a big pile of alerts for Mozilla-Beta. These need to be addressed rapidly. Luckily almost all the regressions seen on Mozilla-Beta will already have been tracked on Mozilla-Inbound or Autoland. | ||
=== What is a merge === | |||
Many times each day we merge code from the integration branches into the main branch and back. This is a common process in large projects. At Mozilla, this means that the majority of the code for Firefox is checked into Mozilla-Inbound and Autoland, then it is merged into Mozilla-Central (also referred to as Firefox) and then once merged, it gets merged back into the other branches. If you want to read more about this merge procedure, here are [[Sheriffing/How_To/Merges|the details]]. | |||
Here is an example of a view of what a merge looks like on [https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=126a1ec5c7c5 TreeHerder]: | |||
[[File:Merge.png]] | |||
Note that the topmost revision has the commit messsage of: "merge m-c to m-i". This is pretty standard and you can see that there are a series of [https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=126a1ec5c7c5 changesets], not just a few related patches. | |||
How this affects alerts is that when a regression lands on Mozilla-Inbound, it will be merged into Firefox, then Autoland. Most likely this means that you will see duplicate alerts on the other integration branch. | |||
* note: we do not generate alerts for the Firefox (Mozilla-Central) branch. | |||
=== What is a backout === | |||
Many times we backout or hotfix code as it is causing a build failure or unittest failure. The [[Sheriffing/Sheriff_Duty|Sheriff team]] handles this process in general and backouts/hotfixes are usually done within 3 hours (i.e. we won't have [[TestEngineering/Performance/Sheriffing/Noise_FAQ#Why_do_we_need_12_future_data_points|12 future changesets]]) of the original fix. As you can imagine we could get an alert 6 hours later and go to look at the graph and see there is no regression, instead there is a temporary spike for a few data points. | |||
While looking on TreeHerder for a backout, they all mention a backout in the commit message: | |||
[[File:Backout_tree.png]] | |||
* note ^ the above image mentions the bug that was backed out, sometimes it is the revisoin | |||
Backouts which affect [[TestEngineering/Performance/Sheriffing/Alerts|Perfherder alerts]] always generate a set of improvements and regressions. These are usually easy to spot on the graph server and we just need to annotate the set of alerts for the given revision to be a 'backout' with the bug to track what took place. | |||
Here is a view on graph server of what appears to be a backout (it could be a fix that landed quickly also): | |||
[[File:Backout_graph.png]] | |||
=== What is PGO === | |||
PGO is [https://developer.mozilla.org/en-US/docs/Building_with_Profile-Guided_Optimization Profile Guided Optimization] where we do a build, run it to collect metrics and optimize based on the output of the metrics. We only release PGO builds, and for the integration branches we do these periodically (6 hours) or as needed. For Mozilla-Central we follow the same pattern. As the builds take considerably longer (2+ times as long) we don't do this for every commit into our integration branches. | |||
How does this affect alerts? We care most about PGO alerts- that is what we ship! Most of the time an alert will be generated for a -Non-PGO build and then a few hours or a day later we will see alerts for the PGO build. | |||
Pay close attention to the branch the alerts are on, most likely you will see it on the non-pgo branch first (i.e. Mozilla-Inbound-Non-PGO), then roughly a day later you will see a similar alert show up on the PGO branch (i.e. Mozilla-Inbound). | |||
Caveats: | |||
* OSX does not do PGO builds, so we do not have -Non-PGO branches for those platforms. (i.e. we only have Mozilla-Inbound) | |||
* PGO alerts will probably have different regression percentages, but the overall list of platforms/tests for a given revision will be almost identical | |||
==Alert== | ==Alert== | ||