SDK Performance Testing
Testing SDK start-up impact
As a first step, measuring Firefox start-up impact of the SDK is simple and wel established using this technique:
We want two data points from this:
- Immediate validation of the loader introduced in 1.4. The start-up impact of the SDK is already low by AMO's standards but we anticipate that this will improve with the new loader. This should be measurable.
- Ongoing regression detection. If we run these tests automatically every time we commit code to master, regressions that impact Firefox start-up time will be immediately apparent.
Start-up Testing methodology
There is existing support in Talos for measuring start-up impact:
Using this method we need to extract 2 sets of numbers:
- start-up impact of the current SDK master branch
- start-up impact of the current SDK stabilization branch
We should also capture and retain historical snapshots of start-up impact for each release of the SDK.
We should have some single graph or set of graphs that track these results over time.
We should create additional tests to evaluate the performance of SDK code. As well, we should be able to curate a collection of add-ons that implement typical or pathological patterns, and evaluate the impact of Firefox performance when these add-ons are enabled.