View source for Performance/Evaluating Performance of New Features

<div><i>There are a lot of good tools available now for studying Firefox performance. This is a list of steps to follow when evaluating the performance of your next Firefox feature.</i>
</div>
<div>
  <ol>
    <li><b>Make sure to test your feature on a low-end or mid-range computer; our dev machines are uncommonly powerful. Think machines with spinning hard drives, not SSDs. Also make sure to test on Windows, as it is used by the vast majority of our users.</b>
      <ul>
        <li>The perf team, fx-team, and gfx team have Windows [http://www.amazon.com/Transformer-T100TA-C1-GR-Detachable-Touchscreen-Laptop/dp/B00K0G2XA4 Asus T100] tablets available in multiple offices just for this purpose. Contact me, Gavin, or Milan Sreckovic if you need one.</li>
      </ul>
    </li>

    <li><b>Ensure your feature does not touch storage on the main thread, either directly or indirectly</b>
      <ul>
        <li>If there&#39;s any chance it might cause main-thread IO, test it with the [https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Reporting_a_Performance_Problem Gecko profiler]. The profiler now has an option to show you all the IO done on the main thread, no matter how brief it is.</li>
        <li>Also be careful about using [https://wiki.mozilla.org/Performance/Avoid_SQLite_In_Your_Next_Firefox_Feature SQLite] </li>
      </ul>
    </li>

    <li><b>Make sure to add Telemetry probes that measure how well your feature performs on real user machines.</b>
      <ul>
        <li>Check the Telemetry numbers again after your feature reaches the release channel. The release channel has a diversity of configurations that simply don&#39;t exist on any of the pre-release channels.
          <ul>
            <li>You can check for regressions in the [https://telemetry.mozilla.org/ Telemetry dash], or you can ask the perf-team to show you how to do a custom analysis (e.g. performance on a particular gfx card type) using [http://mreid-moz.github.io/blog/2013/11/06/current-state-of-telemetry-analysis/ MapReduce] or [http://robertovitillo.com/2015/01/16/next-gen-data-analysis-framework-for-telemetry/ Spark].
              <ul>
                <li>The learning curve can be a bit steep, so the perf team can do one-off analyses for you.</li>
              </ul>
            </li>
            <li>We have additional performance dashboards; they are listed in the &quot;More Dashboards&quot; sidebar on [https://telemetry.mozilla.org/ telemetry.mozilla.org]</li>
          </ul>
        </li>
        <li>Always set the &quot;alert_mails&quot; field for your histogram in [http://mxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Histograms.json Histograms.json] so you get [https://groups.google.com/forum/#!forum/mozilla.dev.telemetry-alerts automatic e-mail notifications] of performance regressions and improvements.
          <ul>
            <li>Ideally, this email address should point to an alias for your team.</li>
            <li>Note that the Telemetry [http://mozilla.github.io/cerberus/dashboard/ regression detector] has an extremely low false-positive rate so you won&#39;t be getting any emails unless performance has changed significantly.</li>
          </ul>
        </li>
      </ul>
    </li>

    <li><b>Keep an eye out on the Talos scores.</b>
      <ul>
        <li>The Talos tests are much less noisy now than they used to be, and more sensitive as well. This is thanks to Avi Halachmi&#39;s, Joel Maher&#39;s, and others&#39; efforts.
          <ul>
            <li>Partly as a result of this, we now have a stricter Talos sheriffing policy. The patch author has 3 business days to respond to a Talos regression bug (before getting backed out), and two weeks to decide what to do with the regression.</li>
          </ul>
        </li>
        <li>Joel Maher will file a regression bug against you if you regress a Talos test.
          <ul>
            <li>The list of unresolved regressions in each release is tracked in the meta bugs: [https://bugzilla.mozilla.org/show_bug.cgi?id=1084461 Firefox 36], [https://bugzilla.mozilla.org/show_bug.cgi?id=1108235 Firefox 37], [https://bugzilla.mozilla.org/show_bug.cgi?id=1122690 Firefox 38], etc</li>
            <li>Joel tracks all the improvements together with all the regressions in a [http://alertmanager.allizom.org:8080/alerts.html?showAll=1 dashboard]</li>
          </ul>
        </li>
        <li>If you cause a regression that you can&#39;t reproduce on your own machine, you can capture a profile directly inside the Talos environment: [https://wiki.mozilla.org/Buildbot/Talos/Profiling https://wiki.mozilla.org/Buildbot/Talos/Profiling]
          <ul>
            <li>MattN has an excellent tool for comparing the scores of two Talos pushes: [http://compare-talos.mattn.ca/ http://compare-talos.mattn.ca/]</li>
          </ul>
        </li>
        <li>Some Talos tests can be run locally as extensions, others may require you to set up a [https://wiki.mozilla.org/Buildbot/Talos/Running#Running_locally_-_Source_Code Talos harness]. Instructions for doing this will be provided in the Talos regression bugs from now on.</li>
        <li>The [http://graphs.mozilla.org/graph.html graph server] can show you a history of test scores and test noise to help you determine if the reported regression is real.
          <ul>
            <li>William Lachance is working on a new &amp; improved graphing UI for treeherder.</li>
          </ul>
        </li>
      </ul>
    </li>

    <li><b>Consider adding a new Talos test</b>
      <ul>
        <li>Add [https://wiki.mozilla.org/Buildbot/Talos/Misc#Adding_a_new_test a new Talos test] if the performance of your feature is important and it is not covered by existing tests. The Perf team would be happy to help you design a meaningful and reliable test.</li>
        <li>Make sure your test measures the right things, isn't noisy and that it is is able to detect real regressions</li>
      </ul>
    </li>

  </ol>
</div>