SDK Performance Testing

Testing SDK start-up impact

As a first step, measuring Firefox start-up impact of the SDK is simple and wel established using this technique:

https://developer.mozilla.org/en/Measuring_Add-on_Startup_Performance

We want two data points from this:

1. Immediate validation of the loader introduced in 1.4. The start-up impact of the SDK is already low by AMO's standards but we anticipate that this will improve with the new loader. This should be measurable.
2. Ongoing regression detection. If we run these tests automatically every time we commit code to master, regressions that impact Firefox start-up time will be immediately apparent.

Start-up Testing methodology

There is existing support in Talos for measuring start-up impact:

https://developer.mozilla.org/en/Measuring_Add-on_Startup_Performance

Using this method we need to extract 2 sets of numbers:

1. start-up impact of the current SDK master branch
2. start-up impact of the current SDK stabilization branch

We should also capture and retain historical snapshots of start-up impact for each release of the SDK.

We should have some single graph or set of graphs that track these results over time.

Other tests

We should create additional tests to evaluate the performance of SDK code. As well, we should be able to curate a collection of add-ons that implement typical or pathological patterns, and evaluate the impact of Firefox performance when these add-ons are enabled.