Adding Talos Tests

From MozillaWiki
Jump to: navigation, search

At a glance

Adding a new performance test or modifying an existing test is much easier than most people think. In general, we need to create a patch for Talos by:

  • determining if this is a startup or pageloader test
  • creating the appropriate folder for your test in testing/talos/talos/tests
  • adding all files and resources for the test (make sure there is no external network accesses) to that folder
  • file a bug in the Testing::Talos component with your patch

Naming convention

t is pre-pended to the names to represent test

  • ts = test startup
  • tp = test pageload
  • tdhtml = test dhtml

What we need to know about tests

When adding a new test, we really need to understand what we are doing. Here are some questions that you should know the answer to before adding a new test:

  • What does this test measure?
  • Does this test overlap with any existing test?
  • What is the unit of measurement that we are recording?
  • What would constitute a regression?
  • What is the expected range in the results over time?
  • Are there variables or conditions which would affect this test?
    • browser preferences, environment variables, etc?
    • operating system, system resources, time of day, etc...?
  • Indepenedent of observation? Will this test produce the same number regardless of what was run before it?
  • What considerations are there for how this test should be run and what tools are required?

Please document your new test along with answers to these questions in the Firefox Source Docs

Pageloader tests

A pageloader test is the most common. Using the pageloader extension allows for most of the reporting and talos integration to be done for you. Here is what you need to do:

Startup tests

A startup test is designed to load the browser with a single page many times. This is useful for tests which shouldn't have any extensions, can handle all loading and measuring themselves, or require the measurement of the browser startup. Here is what you need to do:

Adding a test to production