TestEngineering/Performance/Raptor: Difference between revisions

TestEngineering/Performance/Raptor (view source)

Revision as of 14:13, 23 April 2019

1,874 bytes added , 23 April 2019

m

→‎Page-Load Tests updating section (WIP)

Rwood

Confirmed users

372

edits

@@ Line 7: / Line 7: @@
 === Page-Load Tests ===
-Page-load tests basically involve loading a specific web page and measuring the load performance (i.e. [https://wiki.mozilla.org/TestEngineering/Performance/Glossary#First_Non-Blank_Paint_.28fnbpaint.29 time-to-first-non-blank-paint], [https://wiki.mozilla.org/TestEngineering/Performance/Glossary#First_Contentful_Paint_.28fcp.29 first-contentful-paint] , [https://wiki.mozilla.org/TestEngineering/Performance/Glossary#DOM_Content_Flushed_.28dcf.29 dom-content-flushed], [https://wiki.mozilla.org/TestEngineering/Performance/Glossary#Time_To_First_Interactive_.28ttfi.29 ttfi]). The pageload measurements are 'warm load' in that a new tab is opened only at the start of the test for each new page, and each pagecycle is a reload in the same browser tab.
+Page-load tests involve loading a specific web page and measuring the load performance (i.e. [https://wiki.mozilla.org/TestEngineering/Performance/Glossary#First_Non-Blank_Paint_.28fnbpaint.29 time-to-first-non-blank-paint], [https://wiki.mozilla.org/TestEngineering/Performance/Glossary#First_Contentful_Paint_.28fcp.29 first-contentful-paint] , [https://wiki.mozilla.org/TestEngineering/Performance/Glossary#DOM_Content_Flushed_.28dcf.29 dom-content-flushed], [https://wiki.mozilla.org/TestEngineering/Performance/Glossary#Time_To_First_Interactive_.28ttfi.29 ttfi]).
-For page-load tests, instead of using live web pages for performance testing, Raptor uses a tool called [[https://wiki.mozilla.org/Performance_sheriffing/Raptor/Mitmproxy Mitmproxy]]. Mitmproxy allows us to record and playback test pages via a local Firefox proxy. The Mitmproxy recordings are stored on [https://github.com/mozilla/build-tooltool tooltool] and are automatically downloaded by Raptor when they are required for a test.
+For page-load tests by default, instead of using live web pages for performance testing, Raptor uses a tool called [[https://wiki.mozilla.org/Performance_sheriffing/Raptor/Mitmproxy Mitmproxy]]. Mitmproxy allows us to record and playback test pages via a local Firefox proxy. The Mitmproxy recordings are stored on [https://github.com/mozilla/build-tooltool tooltool] and are automatically downloaded by Raptor when they are required for a test. Raptor uses mitmproxy via the [https://searchfox.org/mozilla-central/source/testing/mozbase/mozproxy mozbase mozproxy] package.
+There are two different types of Raptor page-load tests; warm page-load and cold page-load.
+==== Warm Page-Load ====
+For warm page-load tests, the desktop browser (or android browser app) is just started up once; so the browser is warm on each page-load.
+'''Raptor warm page-load test process when running on Firefox/Chrome/Chromium desktop:'''
+- The desktop browser (or android browser app) is started up
+- todo finish this
+'''Raptor warm page-load test process when running on Firefox android browser apps:'''
+- The android app data is cleared (via `adb shell pm clear firefox.app.binary.name`)
+- - todo finish this
+==== Cold Page-Load ====
+For cold page-load tests, the desktop browser (or android browser app) is shutdown and re-started between page load cycles; so the browser is cold on each page-load. This is what happens for Raptor cold page-load tests:
+'''Raptor cold page-load test process when running on Firefox/Chrome/Chromium desktop:'''
+- The desktop browser (or android browser app) is started up
+- todo finish this
+'''Raptor cold page-load test process when running on Firefox android browser apps:'''
+- The android app data is cleared (via `adb shell pm clear firefox.app.binary.name`)
+- todo finish this
+==== Using Live Sites ====
+It is possible to use live web pages for the page-load tests instead of using the mitproxy recordings. This option is available when running on Try only; as we don't want to submit data from live pages to Perfherder (since live page content will always be changing).
+To run a particular Raptor tp6 page-load test with live sites, open the raptor-tp6*.ini file ([https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests Raptor tests folder]), and for the test default (or under a single page/subtest) just add this attribute:
+  use_live_sites = true
+And push that change to Try (./mach try fuzzy --full) and run the Raptor page-load test.
 === Benchmark Tests ===