Necko/MobileCache/MicroBenchmarks: Difference between revisions

Necko/MobileCache/MicroBenchmarks (view source)

Revision as of 20:58, 28 September 2011

660 bytes added , 28 September 2011

→‎Telemetry vs microbenchmarks

BjarneG

97

edits

@@ Line 14: / Line 14: @@
 == Telemetry vs microbenchmarks ==
-There has been some discussion about using xpcshell-based microbenchmarks instead of telemetry. IMO these approaches are complementary but there '''may''' be a point in using the same code as telemetry does to capture data from xpcshell-tests (as opposed to use JS time-functions in the test). In other words, we may try to use the same mechanisms to harvest data for microbenchmarks as telemetry-code uses to harvest data.
+There has been some discussion about using xpcshell-based microbenchmarks instead of telemetry. IMO these approaches are complementary: Telemetry is real-life browsing patterns on real-life platforms and environments, whereas microbenchmarks are artificial browsing patterns running on a machine in a lab-environment. These are complementary because IMO it is impractical to experiment with code-changes using telemetry to measure the effect - a benchmark in the lab is much more practical for this. On the other hand, telemetry is the (only?) way to ensure that improvements also are valid in real-life.
-The major benefit of this is to get real-life verification by telemetry '''after''' using synthetic, isolated and focused benchmarks in the lab. I.e. we can use synthetic test-patterns implemented by xpcshell-tests (the microbenchmarks) in the lab to identify and qualify code-changes, then after landing code-changes we should be able to verify the effect of these changes on real-life usage-patterns via telemetry. If we measure differently in microbenchmarks and telemetry we may quickly end up "comparing apples and oranges".
+Moving on, IMO performance in necko on a given platform is the product of two factors: The browsing pattern (i.e. which urls are loaded in which sequence), and what exactly is measured. As discussed in the previous paragraph, microbenchmarks and telemetry are inherently different with respect to the browsing pattern, but we can align telemetry and microbenchmarks wrt the second factor. Put in a different way: We should try to use the same code in telemetry and microbenchmarks to capture data used to measure performance, and we should ensure we interpret this data in the same way.
-Below is a pro/con list for using telemetry-code vs JS time-functions to harvest data for microbenchmarks - feel free to add and comment.
+The major benefit of this is to have telemetry give us real-life verification '''after''' using synthetic, isolated and focused benchmarks in the lab. I.e. we can use synthetic test-patterns implemented by xpcshell-tests (the microbenchmarks) in the lab to identify and qualify code-changes, then after landing code-changes we should be able to see predictable effects of these changes on real-life usage-patterns via telemetry. If we measure performance differently in microbenchmarks and telemetry we may quickly end up "comparing apples and oranges".
-'''Note that lab-experiments in both approaches will use synthetic test-patterns - the difference is in the way we harvest data.'''
+Below is a pro/con list for using telemetry-code vs JS time-functions to capture data for microbenchmarks - feel free to add and comment.
 {| border="1" cellpadding="5" cellspacing="0" align="center"