Project Eideticker/Profiling with Eideticker
Scenario 1: Measuring with video capture
- Gives the most comprehensive and reliable results.
- Slow. Each capture/analysis combination takes about 5-10 minutes. Multiplied by the number of tests that you would want to run and the number of runs you'd need to get a representative result and you're easily talking several hours.
- Requires a dedicated machine with an HDMI-capable phone connected to it.
ssh into an Eideticker machine (as of this writing, we have several in Mountain View and a few on the way in Toronto, ask William Lachance for details) and run the "get-metric-for-build" script:
cd src/eideticker source bin/activate wget <apk of build> ./bin/get-metric-for-build.py --num-runs 5 <apk of build> <path to test>
Tests that are good to run for a representative sample of performance include CNN.com ('src/tests/cnn/cnn.com/index.html'), nytimes-zoom ('src/tests/ep1/nytimes/nytimes.com/nytimes-zoom.html') and nytimes-scroll ('src/tests/ep1/nytimes/nytimes.com/nytimes-scroll.html').
You will get a bunch of output regarding amount of checkerboarding and number of unique frames. This may be compared to the existing Eideticker dashboard.
Scenario 2: Measuring with telemetry
- Fast. Can do a test in under a minute.
- Work on any suitably configured linux machine.
- Not technically a black box test, relies on internal measurements that may be inaccurate
- Only measures checkerboarding, not the smoothness of animations, etc.
- Results are only relevant to Fennec, and are not comparable with other browsers.
The Eideticker README contains instructions on how to set up / run this type of test.