TestEngineering/Performance/Sheriffing - Revision history

Davehunt at 11:44, 16 March 2021

2021-03-16T11:44:50Z

Mraiciof: Added a video for Overview section

2020-02-26T12:03:22Z

Added a video for Overview section

← Older revision		Revision as of 12:03, 26 February 2020
Line 3:		Line 3:

	A Performance Sheriff's job is similar: to make sure that performance regressions in Firefox are detected and dealt with. They look at the data produced by performance test jobs (mainly Talos), find regressions, determine their root cause(s) and gets bugs on file to track all issues and make interested parties aware of what is going on.		A Performance Sheriff's job is similar: to make sure that performance regressions in Firefox are detected and dealt with. They look at the data produced by performance test jobs (mainly Talos), find regressions, determine their root cause(s) and gets bugs on file to track all issues and make interested parties aware of what is going on.

			A short demo describing the performance sheriffing workflow:
			https://youtu.be/VdTC7Maa6hE

	= What is an alert =		= What is an alert =

Alexandrui: /* Additional Resources */

2020-01-23T10:14:22Z

Additional Resources

← Older revision		Revision as of 10:14, 23 January 2020
Line 125:		Line 125:

	= Additional Resources =		= Additional Resources =
	* [[TestEngineering/Performance/~~FAQ Performance~~ FAQ]]		* [[TestEngineering/Performance/FAQ]]

Alexandrui: /* Additional Resources */

2020-01-23T10:14:06Z

Additional Resources

← Older revision		Revision as of 10:14, 23 January 2020
Line 125:		Line 125:

	= Additional Resources =		= Additional Resources =
	* [[/~~Alert_FAQ\|Alert FAQ]]~~		* [[TestEngineering/Performance/FAQ Performance FAQ]]
	* [[/Noise_FAQ\|Noise FAQ]]
	* [[/Perfherder_FAQ\|Perfherder FAQ]]
	* [[/Tree_FAQ\|Tree FAQ]]
	* [[/FAQ\| Performance FAQ]]

Alexandrui at 10:09, 23 January 2020

2020-01-23T10:09:31Z

← Older revision		Revision as of 10:09, 23 January 2020
Line 129:		Line 129:
	* [[/Perfherder_FAQ\|Perfherder FAQ]]		* [[/Perfherder_FAQ\|Perfherder FAQ]]
	* [[/Tree_FAQ\|Tree FAQ]]		* [[/Tree_FAQ\|Tree FAQ]]
			* [[/FAQ\| Performance FAQ]]

Mraiciof: Replaced "we need bugs to document these those" with "we need bugs to document these"

2020-01-21T09:53:08Z

Replaced "we need bugs to document these those" with "we need bugs to document these"

← Older revision		Revision as of 09:53, 21 January 2020
Line 78:		Line 78:
	* [[/Tree_FAQ#What_is_a_backout\|backout]] (usually within 1 week causing a similar regression/improvement)		* [[/Tree_FAQ#What_is_a_backout\|backout]] (usually within 1 week causing a similar regression/improvement)
	* [[/Tree_FAQ#What_is_PGO\|pgo/nonpgo]] (some errors are pgo only and might be a side effect of pgo). We only ship PGO, so these are the most important.		* [[/Tree_FAQ#What_is_PGO\|pgo/nonpgo]] (some errors are pgo only and might be a side effect of pgo). We only ship PGO, so these are the most important.
	* test/infrastructure change - once in a while we change big things about our tests or infrastructure and it affects our tests (we need bugs to document these ~~those~~)		* test/infrastructure change - once in a while we change big things about our tests or infrastructure and it affects our tests (we need bugs to document these)
	* [[/Tree_FAQ#What_is_a_merge\|Merged]] - sometimes the root cause looks to be a merge, this is a normall a side effect of [[/Tree_FAQ#What_is_coalescing\|Coalescing]].		* [[/Tree_FAQ#What_is_a_merge\|Merged]] - sometimes the root cause looks to be a merge, this is a normall a side effect of [[/Tree_FAQ#What_is_coalescing\|Coalescing]].
	* [[/Tree_FAQ#What_is_coalescing\|Coalesed]] - this is when we don't run every job on every platform on every push and sometimes we have a set of changes		* [[/Tree_FAQ#What_is_coalescing\|Coalesed]] - this is when we don't run every job on every platform on every push and sometimes we have a set of changes

Stephend: /* Filing a bug */

2020-01-09T17:35:17Z

Filing a bug

← Older revision		Revision as of 17:35, 9 January 2020
Line 91:		Line 91:

	= Filing a bug =		= Filing a bug =
	A lot of work has been done inside of Perfherder to make filing a bug easier. As each bug has unique attributes it is hard to handle this fully ~~programattic~~, but we can do our best. In fact, there is a 'File bug' clickable link which is underneath each Alert Summary section in Perfherder Alerts. Clicking it will bring you to Bugzilla with as many fields filled in as possible.		A lot of work has been done inside of Perfherder to make filing a bug easier. As each bug has unique attributes it is hard to handle this fully programmatic, but we can do our best. In fact, there is a 'File bug' clickable link which is underneath each Alert Summary section in Perfherder Alerts. Clicking it will bring you to Bugzilla with as many fields filled in as possible.

	[[File:File_bug_dropdown.png]]		[[File:File_bug_dropdown.png]]

Davehunt at 13:13, 6 August 2019

2019-08-06T13:13:55Z

Show changes

Davehunt: Davehunt moved page Performance sheriffing to TestEngineering/Performance/Sheriffing

2019-08-05T10:54:59Z

Davehunt moved page Performance sheriffing to TestEngineering/Performance/Sheriffing

← Older revision	Revision as of 10:54, 5 August 2019
(No difference)

Igoldan: correct link to old page

2018-02-13T13:50:58Z

correct link to old page

← Older revision		Revision as of 13:50, 13 February 2018
Line 129:		Line 129:
	* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Sheriffing/Perfherder_FAQ Perfherder FAQ]		* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Sheriffing/Perfherder_FAQ Perfherder FAQ]
	* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Sheriffing/Tree_FAQ Tree FAQ]		* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Sheriffing/Tree_FAQ Tree FAQ]
	* [https://wiki.mozilla.org/~~Performance_sheriffing~~/Talos/Sheriffing duplicated & updated from old page]		* [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing duplicated & updated from old page]

← Older revision		Revision as of 10:09, 23 January 2020
Line 129:		Line 129:
	* [[/Perfherder_FAQ\|Perfherder FAQ]]		* [[/Perfherder_FAQ\|Perfherder FAQ]]
	* [[/Tree_FAQ\|Tree FAQ]]		* [[/Tree_FAQ\|Tree FAQ]]
			* [[/FAQ\| Performance FAQ]]

← Older revision		Revision as of 09:53, 21 January 2020
Line 78:		Line 78:
	* [[/Tree_FAQ#What_is_a_backout\|backout]] (usually within 1 week causing a similar regression/improvement)		* [[/Tree_FAQ#What_is_a_backout\|backout]] (usually within 1 week causing a similar regression/improvement)
	* [[/Tree_FAQ#What_is_PGO\|pgo/nonpgo]] (some errors are pgo only and might be a side effect of pgo). We only ship PGO, so these are the most important.		* [[/Tree_FAQ#What_is_PGO\|pgo/nonpgo]] (some errors are pgo only and might be a side effect of pgo). We only ship PGO, so these are the most important.
	* test/infrastructure change - once in a while we change big things about our tests or infrastructure and it affects our tests (we need bugs to document these ~~those~~)		* test/infrastructure change - once in a while we change big things about our tests or infrastructure and it affects our tests (we need bugs to document these)
	* [[/Tree_FAQ#What_is_a_merge\|Merged]] - sometimes the root cause looks to be a merge, this is a normall a side effect of [[/Tree_FAQ#What_is_coalescing\|Coalescing]].		* [[/Tree_FAQ#What_is_a_merge\|Merged]] - sometimes the root cause looks to be a merge, this is a normall a side effect of [[/Tree_FAQ#What_is_coalescing\|Coalescing]].
	* [[/Tree_FAQ#What_is_coalescing\|Coalesed]] - this is when we don't run every job on every platform on every push and sometimes we have a set of changes		* [[/Tree_FAQ#What_is_coalescing\|Coalesed]] - this is when we don't run every job on every platform on every push and sometimes we have a set of changes

← Older revision		Revision as of 11:44, 16 March 2021
Line 14:		Line 14:
	** [[/Tree_FAQ#Branch_names_and_confusion\|branch]]		** [[/Tree_FAQ#Branch_names_and_confusion\|branch]]
	** % regressed, this is a range of the regressions (not improvements)		** % regressed, this is a range of the regressions (not improvements)
	** the [~~[TestEngineering~~/~~Performance~~/~~Talos~~/~~Tests\|~~tests]] which have regressed		** the [https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#test-definitions tests] which have regressed
	** the platforms we see this regression on		** the platforms we see this regression on
	* date of the suspect revision push		* date of the suspect revision push
Line 22:		Line 22:

	Below the summary will be a list of alerts, each alert will reference:		Below the summary will be a list of alerts, each alert will reference:
	* [~~[TestEngineering~~/~~Performance~~/~~Talos~~/~~Tests\|~~Test name]]		* [https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#test-definitions Test name]
	* platform (including build type, such as opt, pgo)		* platform (including build type, such as opt, pgo)
	* old score (median score of the previous 12 commits)		* old score (median score of the previous 12 commits)
Line 101:		Line 101:
	* Product/Component - this should be the same as the bug which is the root cause, if >1 bug, file in [https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Talos Talos]		* Product/Component - this should be the same as the bug which is the root cause, if >1 bug, file in [https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Talos Talos]
	* Dependent/Block bugs - For a new bug, add the [[#Tracking_bugs\|tracking bug]] (for the current version) and root cause bug(s) as blocking this bug		* Dependent/Block bugs - For a new bug, add the [[#Tracking_bugs\|tracking bug]] (for the current version) and root cause bug(s) as blocking this bug
	* CC list - cc patch author(s), reviewer(s) and owner of the tests as documented on the [~~[TestEngineering~~/~~Performance~~/~~Talos~~/~~Tests\|~~Talos ~~tests wiki]~~]; if we have >1 bug, we should cc everyone who worked on those bugs so we call pitch in an answer questions faster		* CC list - cc patch author(s), reviewer(s) and owner of the tests as documented in the [https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#test-definitions Talos test definition]; if we have >1 bug, we should cc everyone who worked on those bugs so we call pitch in an answer questions faster
	* Summary of bug should have a check to make sure the revision is accurate		* Summary of bug should have a check to make sure the revision is accurate
	* The description is auto suggested as well, please verify the revision here		* The description is auto suggested as well, please verify the revision here