User:Mconnor/Pending/WeaveBenchmarks

Basic goals

• Test full upload speed for all engines (easy)
• Test full download speed for all engines (easy)
• Test merge speed for all engines (hard)

Test outlines

• Full upload
  • Create a test profile with arbitrary data, including random IDN domains, for all engines shipped by default
  • wipe the server, then force a full sync to the server, timing the sync from start to finish.
  • repeat this 20 times, drop high/low and average.

• Full download
  • After the last run of the upload benchmark, do not wipe the server data.
  • Wipe the local data and import everything from the server.
  • Repeat 20x, as before, and generate a score.

• Merge speed
  • <smart ideas go here>

First production test setup

• Each test machine will need an account on the test server.
• The test profile will need to be generated at the start of the run, to avoid hitting expiration issues. Some sort of format, probably JSON, to store the data and offsets will need to be generated that we can read in and add everything via the appropriate APIs.
• Once we have a local profile built up:
  • Run the upload benchmarks, leaving the data on the server after the last run.
  • Run the download benchmarks from the uploaded dataset, wiping the server after the last run.
  • Nuke the local profile.