View source for TestEngineering/Performance/Talos/Adding tests

{{DISPLAYTITLE:Adding Talos Tests}}
= At a glance =
Adding a new performance test or modifying an existing test is much easier than most people think. In general, we need to create a patch for Talos by:
* determining if this is a [[#Startup_tests|startup]] or [[#Pageloader_tests|pageloader]] test
* creating the appropriate folder for your test in [https://searchfox.org/mozilla-central/source/testing/talos/talos/tests testing/talos/talos/tests]
* adding all files and resources for the test (make sure there is no external network accesses) to that folder
* file a bug in the {{enterbug|Testing|Talos}} component with your patch

= Naming convention =
'''t''' is pre-pended to the names to represent '''t'''est
* ts = '''t'''est '''s'''tartup
* tp = '''t'''est '''p'''ageload
* tdhtml = '''t'''est '''dhtml'''

= What we need to know about tests =
When adding a new test, we really need to understand what we are doing. Here are some questions that you should know the answer to before adding a new test:
* What does this test measure?
* Does this test overlap with any existing test?
* What is the unit of measurement that we are recording?
* What would constitute a regression?  
* What is the expected range in the results over time?
* Are there variables or conditions which would affect this test?
** browser preferences, environment variables, etc?
** operating system, system resources, time of day, etc...?
* Indepenedent of observation? Will this test produce the same number regardless of what was run before it?
* What considerations are there for how this test should be run and what tools are required?

Please document your new test along with answers to these questions in the '''[https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html Firefox Source Docs]'''

= Pageloader tests =
A pageloader test is the most common. Using the pageloader extension allows for most of the reporting and talos integration to be done for you. Here is what you need to do:
* Create a new directory in [https://searchfox.org/mozilla-central/source/testing/talos/talos/tests testing/talos/talos/tests]
* Create a manifest file named {{code|<testname>.manifest}} file in your directory ([https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/svgx/svgx.manifest example])
** Raw page load will just be a single URL line ([https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/quantum_pageload/quantum_pageload_amazon.manifest#1 example])
** Internal measurements will have a % prior to the URL ([https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/tabpaint/tabpaint.manifest#1 example])
* Add your tests. For self reporting tests, use [[https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/pageloader/chrome/pageloader.js#656 tpRecordTime]]
** [https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/tests/svgx/hixie-001.xml#286 single page/value example]
** [https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/tests/tart/tart.html#148 multiple pages/values example]
* Add your test definition by adding a new class for your test to [https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/test.py testing/talos/talos/test.py] ([https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/test.py#792-812 example])
** Add an item for {{code|tpmanifest}}, {{code|<nowiki>${talos}</nowiki>}} is replaced with the current running directory ([https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/test.py#582 example]) 
** Add an item for {{code|tpcycles}} (we recommend 1) and {{code|tppagecycles}} (we recommend 25 for a simple page, and 5 for a internal recording benchmark)
** Add an item for {{code|filters}} ([https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/test.py#146 example])

= Startup tests =
A startup test is designed to load the browser with a single page many times. This is useful for tests which shouldn't have any extensions, can handle all loading and measuring themselves, or require the measurement of the browser startup. Here is what you need to do:
* Create a new directory in [https://searchfox.org/mozilla-central/source/testing/talos/talos/startup_test testing/talos/talos/startup_test]
* Add your tests to the folder, these will be accessed by a raw URL
** the tests need to report [https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/results.py#276 {{code|__start_report<value>__end_report}}] so Talos can find it ([https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/startup_test/tspaint_test.html#29 example])
** If you plan on doing shutdown times, you need to add in [https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/results.py#283 {{code|__startTimestamp<value>__endTimestamp}}]. ([https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/startup_test/tspaint_test.html#61 example])
** Include [https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/scripts/MozillaFileLogger.js MozillaFileLogger.js] and [https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/talos-powers/content/TalosPowersContent.js TalosPowersContent.js] in your script ([https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/startup_test/tspaint_test.html#9-10 example])
* Add your test definition by adding a new class for your test to [https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/test.py testing/talos/talos/test.py] ([https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/test.py#132-150 example])
** Add an item for {{code|cycles}}, we recommend 20
** Add an item for {{code|url}}, this is relative to the Talos directory and what Firefox will load
** Add {{code|tpmozafterpaint}} and set it to {{code|True}} by default, or {{code|False}} if your test does other internal measurements unrelated to rendering a page 
** Add an item for {{code|filters}} ([https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/test.py#146 example])

= Adding a test to production =
* {{enterbug|Testing|Talos|File a bug}} to add tests to [https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos/test.py testing/talos/talos/test.py] (see {{bug|893071}} for an example).
* For mozilla-central on desktop only - just land your new test to your in-tree commit (on inbound/autoland) and ensure that you adjust [https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/testing/talos/talos.json testing/talos/talos.json] to run the test
* If you add a new suite, edit [https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/taskcluster/ci/test/talos.yml taskcluster/ci/test/talos.yml] to add the testname to a new suite, then add it to [https://searchfox.org/mozilla-central/rev/30b01f4f60dbcbd6b01500a26b3100c28005cf62/taskcluster/ci/test/test-sets.yml taskcluster/ci/test/test-sets.yml] for all the different Talos test-sets.
* If this is an update to an existing test that could change the numbers, this needs to be treated as a new test and run side by side for a week to get a new baseline.
* Document the test in [https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html Firefox Source Docs]