TestEngineering/Performance: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
Line 1: Line 1:
{{DISPLAYTITLE:Firefox Performance Test Engineering πŸ”₯🦊⏱}}
{{DISPLAYTITLE:Firefox Performance Test Engineering πŸ”₯🦊⏱}}


== Who we are ==
= Who we are =
* Dave Hunt [:davehunt] πŸ‡¬πŸ‡§
* Dave Hunt [:davehunt] πŸ‡¬πŸ‡§
* Stephen Donner [:stephend] πŸ‡ΊπŸ‡Έ
* Stephen Donner [:stephend] πŸ‡ΊπŸ‡Έ
Line 12: Line 12:
* Alex Irimovici [:air] πŸ‡·πŸ‡΄
* Alex Irimovici [:air] πŸ‡·πŸ‡΄


== Where to find us ==
= Where to find us =
* [[IRC]]: #perftest
* [[IRC]]: #perftest
* Slack: #perftest
* Slack: #perftest


== What we do ==
= What we do =
We provide automated testing support for measuring the performance of Firefox products. Here are a few examples of what you can expect our team to be working on.
We provide automated testing support for measuring the performance of Firefox products. Here are a few examples of what you can expect our team to be working on.
* Provide advice/troubleshooting related to performance testing or any of our tools and harnesses.
* Provide advice/troubleshooting related to performance testing or any of our tools and harnesses.
Line 23: Line 23:
* Monitoring and reporting of performance regressions.
* Monitoring and reporting of performance regressions.


== What we don't do ==
= What we don't do =
* Own all performance tests. We work on the test harnesses and tools that are used for performance testing, but the tests themselves are often developed outside of our team. Every test should have an owner, who is responsible for responding to questions related to the test, and may be asked to assist when the test detects a regression.
* Own all performance tests. We work on the test harnesses and tools that are used for performance testing, but the tests themselves are often developed outside of our team. Every test should have an owner, who is responsible for responding to questions related to the test, and may be asked to assist when the test detects a regression.
* Review all performance tests. Similar to test ownership, we enable others to contribute performance tests. We can provide advice and reviews, but do not impose this as a restriction to landing test changes.
* Review all performance tests. Similar to test ownership, we enable others to contribute performance tests. We can provide advice and reviews, but do not impose this as a restriction to landing test changes.
Line 30: Line 30:
* Maintain the reporting tools
* Maintain the reporting tools


== Meetings ==
= Meetings =
{{/Meetings}}
{{/Meetings}}


== Onboarding ==
= Onboarding =
You are encouraged to improve the [[/Onboarding|onboarding page]]. If you need to ask questions that are not already covered, please update the page so that the next person has a better onboarding experience.
Welcome to the team! You are encouraged to improve the [[/Onboarding|onboarding page]]. If you need to ask questions that are not already covered, please update the page so that the next person has a better onboarding experience.


== Task Workflow ==
= Task Workflow =
Visit our [[/Task Workflow|task workflow]] page for details on how we manage tasks.
Visit our [[/Task Workflow|task workflow]] page for details on how we manage tasks.


== Objectives ==
= Objectives =


=== [https://docs.google.com/spreadsheets/d/1P1d7rzA27t3OHljraxapNgDl-1Hsc_novoj78w8qog0/edit#gid=584732224 2019/Q3] ===
== [https://docs.google.com/spreadsheets/d/1P1d7rzA27t3OHljraxapNgDl-1Hsc_novoj78w8qog0/edit#gid=584732224 2019/Q3] ==
* KR1.3: Fenix Startup Test Development [rwood] {{notstarted|}} [%]
* KR1.3: Fenix Startup Test Development [rwood] {{notstarted|}} [%]
* KR3.1: Video QoE testing for desktop [marauder] {{ok|}} [90%]
* KR3.1: Video QoE testing for desktop [marauder] {{ok|}} [90%]
Line 50: Line 50:
* KR6.6: Improve perf sheriff efficiency [igoldan] {{ok|}} [70%]
* KR6.6: Improve perf sheriff efficiency [igoldan] {{ok|}} [70%]


== Projects ==
= Projects =


=== Raptor ===
=== Raptor ===
Line 60: Line 60:
* Documentation: https://wiki.mozilla.org/Performance_sheriffing/Raptor
* Documentation: https://wiki.mozilla.org/Performance_sheriffing/Raptor


=== Talos ===
== Talos ==
Talos is a Python performance testing framework that is usable on Windows, Mac and Linux. Talos is our versatile performance testing framework we use at Mozilla. It was created to serve as a test runner for the existing performance tests that Mozilla was running back in 2007 as well as providing an extensible framework for new tests as they were created.
Talos is a Python performance testing framework that is usable on Windows, Mac and Linux. Talos is our versatile performance testing framework we use at Mozilla. It was created to serve as a test runner for the existing performance tests that Mozilla was running back in 2007 as well as providing an extensible framework for new tests as they were created.


Line 68: Line 68:
* Documentation: https://wiki.mozilla.org/Performance_sheriffing/Talos
* Documentation: https://wiki.mozilla.org/Performance_sheriffing/Talos


=== WebPageTest ===
== WebPageTest ==
* Owner: Stephen Donner [:stephend]
* Owner: Stephen Donner [:stephend]
* Source: https://github.com/mozilla/wpt-api
* Source: https://github.com/mozilla/wpt-api
Line 74: Line 74:
* Documentation: https://mozilla-wpt-api-docs.readthedocs.io/en/master/
* Documentation: https://mozilla-wpt-api-docs.readthedocs.io/en/master/


=== Perfherder ===
== Perfherder ==
Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android).
Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android).
Perfherder is part of the Treeherder project.
Perfherder is part of the Treeherder project.
Line 85: Line 85:
* Developer documentation: https://treeherder.readthedocs.io/
* Developer documentation: https://treeherder.readthedocs.io/


== Dashboards ==
= Dashboards =


=== Firefox Are We Fast Yet Dashboard ===
== Firefox Are We Fast Yet Dashboard ==
Shows a variety of benchmarks, run on a variety of platforms.Β  Meant to be a detailed view of performance.
Shows a variety of benchmarks, run on a variety of platforms.Β  Meant to be a detailed view of performance.


Line 94: Line 94:
* Source: https://github.com/mozilla-frontend-infra/firefox-performance-dashboard
* Source: https://github.com/mozilla-frontend-infra/firefox-performance-dashboard


=== Firefox Health Dashboard ===
== Firefox Health Dashboard ==
The health dashboard tracks metrics and statistics important for tracking performance improvements.Β  Meant to be a high level view.
The health dashboard tracks metrics and statistics important for tracking performance improvements.Β  Meant to be a high level view.


Line 102: Line 102:
* Good first bugs: https://github.com/mozilla-frontend-infra/firefox-health-dashboard/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22
* Good first bugs: https://github.com/mozilla-frontend-infra/firefox-health-dashboard/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22


== Resources ==
= Resources =
* [[/Glossary/]]
* [[/Glossary/]]
* [https://docs.google.com/document/d/1SswqYIAm4h8vlwfMc0pfGEJwXpFECyubGDezRZHHPFE/edit Strategies for investigating intermittents]
* [https://docs.google.com/document/d/1SswqYIAm4h8vlwfMc0pfGEJwXpFECyubGDezRZHHPFE/edit Strategies for investigating intermittents]

Revision as of 16:25, 6 August 2019


Who we are

  • Dave Hunt [:davehunt] πŸ‡¬πŸ‡§
  • Stephen Donner [:stephend] πŸ‡ΊπŸ‡Έ
  • Rob Wood [:rwood] πŸ‡¨πŸ‡¦
  • Greg Mierzwinski [:sparky] πŸ‡¨πŸ‡¦
  • IonuΘ› Goldan [:igoldan] πŸ‡·πŸ‡΄
  • Florian Strugariu [:bebe] πŸ‡·πŸ‡΄
  • Marian Raiciof [:marauder] πŸ‡·πŸ‡΄
  • Alex Ionescu [:alexandrui] πŸ‡·πŸ‡΄
  • Alex Irimovici [:air] πŸ‡·πŸ‡΄

Where to find us

  • IRC: #perftest
  • Slack: #perftest

What we do

We provide automated testing support for measuring the performance of Firefox products. Here are a few examples of what you can expect our team to be working on.

  • Provide advice/troubleshooting related to performance testing or any of our tools and harnesses.
  • Develop test plans that involve automation of performance testing.
  • Prototype, build, and maintain test harnesses for performance testing.
  • Monitoring and reporting of performance regressions.

What we don't do

  • Own all performance tests. We work on the test harnesses and tools that are used for performance testing, but the tests themselves are often developed outside of our team. Every test should have an owner, who is responsible for responding to questions related to the test, and may be asked to assist when the test detects a regression.
  • Review all performance tests. Similar to test ownership, we enable others to contribute performance tests. We can provide advice and reviews, but do not impose this as a restriction to landing test changes.
  • Maintain the infrastructure the tests run on
  • Maintain the continuous integration pipeline
  • Maintain the reporting tools

Meetings

Onboarding

Welcome to the team! You are encouraged to improve the onboarding page. If you need to ask questions that are not already covered, please update the page so that the next person has a better onboarding experience.

Task Workflow

Visit our task workflow page for details on how we manage tasks.

Objectives

2019/Q3

  • KR1.3: Fenix Startup Test Development [rwood] [NOT STARTED] [%]
  • KR3.1: Video QoE testing for desktop [marauder] [ON TRACK] [90%]
  • KR3.2: Video QoE testing for Fenix [marauder] [ON TRACK] [90%]
  • KR3.4: Complete MVP for browsertime in CI [rwood] [ON TRACK] [100%]
  • KR4.1: Migrate CI from Chromium to Chrome [sparky] [NOT STARTED] [80%]
  • KR4.2: Power tests on macOS laptops [sparky] [ON TRACK] [100%]
  • KR6.6: Improve perf sheriff efficiency [igoldan] [ON TRACK] [70%]

Projects

Raptor

Raptor is a Python testing framework used to run browser benchmark and browser page-load performance tests. Raptor is cross-browser compatible and is currently running in Mozilla taskcluster production on Firefox Desktop, Firefox Android, and on Google Chromium.

Talos

Talos is a Python performance testing framework that is usable on Windows, Mac and Linux. Talos is our versatile performance testing framework we use at Mozilla. It was created to serve as a test runner for the existing performance tests that Mozilla was running back in 2007 as well as providing an extensible framework for new tests as they were created.

WebPageTest

Perfherder

Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android). Perfherder is part of the Treeherder project.

Dashboards

Firefox Are We Fast Yet Dashboard

Shows a variety of benchmarks, run on a variety of platforms. Meant to be a detailed view of performance.

Firefox Health Dashboard

The health dashboard tracks metrics and statistics important for tracking performance improvements. Meant to be a high level view.

Resources

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Performance&oldid=1216100"