TestEngineering/Performance

Who we are

• Dave Hunt [:davehunt] 🇬🇧
• Stephen Donner [:stephend] 🇺🇸
• Rob Wood [:rwood] 🇨🇦
• Greg Mierzwinski [:sparky] 🇨🇦
• Ionuț Goldan [:igoldan] 🇷🇴
• Florian Strugariu [:bebe] 🇷🇴
• Marian Raiciof [:marauder] 🇷🇴
• Alex Ionescu [:alexandrui] 🇷🇴
• Alex Irimovici [:air] 🇷🇴

Where to find us

• IRC: #perftest
• Slack: #perftest

What we do

We provide automated testing support for measuring the performance of Firefox products. Here are a few examples of what you can expect our team to be working on.

• Provide advice/troubleshooting related to performance testing or any of our tools and harnesses.
• Develop test plans that involve automation of performance testing.
• Prototype, build, and maintain test harnesses for performance testing.
• Monitoring and reporting of performance regressions.

What we don't do

• Own all performance tests. We work on the test harnesses and tools that are used for performance testing, but the tests themselves are often developed outside of our team. Every test should have an owner, who is responsible for responding to questions related to the test, and may be asked to assist when the test detects a regression.
• Review all performance tests. Similar to test ownership, we enable others to contribute performance tests. We can provide advice and reviews, but do not impose this as a restriction to landing test changes.
• Maintain the infrastructure the tests run on
• Maintain the continuous integration pipeline
• Maintain the reporting tools

Meetings

• When: Weekly on Wednesdays at 15:00 UTC (08:00 PST, 11:00 EST, 18:00 EET)
• Where: Zoom ID 110-819-485
• Notes: Google Docs

Onboarding

Welcome to the team! You are encouraged to improve the onboarding page. If you need to ask questions that are not already covered, please update the page so that the next person has a better onboarding experience.

Task Workflow

Visit our task workflow page for details on how we manage tasks.

Objectives

2019/Q3

• KR1.3: Fenix Startup Test Development [rwood] [NOT STARTED] [%]
• KR3.1: Video QoE testing for desktop [marauder] [ON TRACK] [90%]
• KR3.2: Video QoE testing for Fenix [marauder] [ON TRACK] [90%]
• KR3.4: Complete MVP for browsertime in CI [rwood] [ON TRACK] [100%]
• KR4.1: Migrate CI from Chromium to Chrome [sparky] [NOT STARTED] [80%]
• KR4.2: Power tests on macOS laptops [sparky] [ON TRACK] [100%]
• KR6.6: Improve perf sheriff efficiency [igoldan] [ON TRACK] [70%]

Projects

Raptor

Raptor is a Python testing framework used to run browser benchmark and browser page-load performance tests. Raptor is cross-browser compatible and is currently running in Mozilla taskcluster production on Firefox Desktop, Firefox Android, and on Google Chromium.

• Owner: Rob Wood [:rwood]
• Source: https://searchfox.org/mozilla-central/source/testing/raptor
• Good first bugs: https://codetribute.mozilla.org/projects/automation?project%3DRaptor
• Documentation: https://wiki.mozilla.org/Performance_sheriffing/Raptor

Talos

Talos is a Python performance testing framework that is usable on Windows, Mac and Linux. Talos is our versatile performance testing framework we use at Mozilla. It was created to serve as a test runner for the existing performance tests that Mozilla was running back in 2007 as well as providing an extensible framework for new tests as they were created.

• Owner: Rob Wood [:rwood], Joel Maher [:jmaher]
• Source: https://searchfox.org/mozilla-central/source/testing/talos
• Good first bugs: https://codetribute.mozilla.org/projects/automation?project%3DTalos
• Documentation: https://wiki.mozilla.org/Performance_sheriffing/Talos

WebPageTest

• Owner: Stephen Donner [:stephend]
• Source: https://github.com/mozilla/wpt-api
• Good first bugs: https://github.com/mozilla/wpt-api/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22
• Documentation: https://mozilla-wpt-api-docs.readthedocs.io/en/master/

Perfherder

Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android). Perfherder is part of the Treeherder project.

• Location: https://treeherder.mozilla.org/perf.html
• Owner: Ionuț Goldan [:igoldan]
• Source: https://github.com/mozilla/treeherder
• Good first bugs: https://codetribute.mozilla.org/projects/reporting?project%3DPerfherder
• User documentation: https://wiki.mozilla.org/EngineeringProductivity/Projects/Perfherder
• Developer documentation: https://treeherder.readthedocs.io/

Dashboards

Firefox Are We Fast Yet Dashboard

Shows a variety of benchmarks, run on a variety of platforms. Meant to be a detailed view of performance.

• Location: https://arewefastyet.com
• Owner: Armen Zambrano Gasparnian [:armen]
• Source: https://github.com/mozilla-frontend-infra/firefox-performance-dashboard

Firefox Health Dashboard

The health dashboard tracks metrics and statistics important for tracking performance improvements. Meant to be a high level view.

• Location: https://health.graphics/
• Owner: Armen Zambrano Gasparnian [:armen], Kyle Lahnakoski [:ekyle]
• Source: https://github.com/mozilla-frontend-infra/firefox-health-dashboard/
• Good first bugs: https://github.com/mozilla-frontend-infra/firefox-health-dashboard/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22

Resources

• Glossary
• Strategies for investigating intermittents
• Following up perf bugs
• Excessive Android device queue response plan
• Perf bisection workflows