Project Eideticker/Meetings/2012-01-11

Agenda

• Overview of the hardware, the two machines
• Define the automated video tests and their priority for video capture
• Define how often these should be run, and on what hardware
• Define how to capture video of phones that do not have HDMI output.
• Define how automated any system with a video camera involved can be (i.e. we need humans to put the right phone in front of the camera).
• Define how you want to access this remotely to do analysis and what the priority of that is. (Is MV VPN OK?)
• Define the B2G usecases so that we can solve any blockers there while we iron out native fennec details
• Define our deadlines
• Get any phones we need ordered ordered.
• Roundtable

Meeting

Overview of the hardware, the two machines

• Eideticker idea is simple to instead of tweaking internals of app to determine performance, we watch external indicators to see what the performance characteristics of the software are.
• Can also use the same generic infastructure on all the browsers since we're observing external behavior.
• Will has been experimenting with the last three months - using a standard dell workstation with a decklink video capture card in it, and a LG G2X phone which has HDMI video output.
• The test framework works by loading an arbitrary page on the phone, pull the video of the page load/interaction through the decklink card, and then analyze it on the machine.

Define the automated video tests and their priority for video capture

Speed test demos

• tried using the sample HTML5 demos that MSoft put up to see if the framerate that the test reported was accurate with the actual framerate
• Wasn't that useful or interesting due to limitations in mobile Fennec (most demos run at < 3 fps)
• Need to try on other browsers -- good to find out what we do well on against competitors and then use it for ongoing regression testing to ensure we keep progressing

Checkerboarding

• It's incredibly valuable to measure it at the moment.
• We want to measure this across browsers
• This doesn't allow us to work through telemetry, it's more useful for comparing against other browsers as well.
• There's also some benefit to this in that it will help us check the instrumentation code as well.

Cross Browser Startup

• Currently we have a canon 5d video camera that points at the phone (see below for discussion of what to do on hardware without HDMI output)
• they send a command line to start the test on the phone and tapes the different browsers starting up on the phone
• This has been pretty manually intensive to do so we only do it once a week.
• This might be fairly automated at this point.

Define how often these should be run, and on what hardware

• Set up an arewefastyet.com for it - have something that is set up on a representative phone and then track results over time
  • Metrics for starutp, checkerboarding etc across browsers on one or two phone platforms that we control heavily (require HDMI out for instance)
  • Checkerboarding is first target to get setup.
  • Getting results from benchmarks on other browsers would be useful as well for a second item.

• Choose the benchmark and checkerboarding test
  • start with what we use manually - but we only have that data for startup
  • checkerboarding test would be good to get results for on a daily basis
  • we only need to run competitor browsers once and then once again when they update - so concentrate more on running fennec native and keep a run of stock browser, opera and dolphin to compare against.

• what is a good test for checkerboarding?
  • used real world sites for checkerboarding - timecube.com
  • googlereader - interesting mix of content/ frames etc.
  • html5tests.com
  • planet.mozilla.org - need a fix version
  • testing checkerboarding with zooming is also interesting, never had a way to do that, always did checkerboarding with panning
  • TODO: follow up with frontend folks on this, no current automated testing on this right now, they are planning to use telemetry. Could use the telemetry reported sites, and make an autoamted suite with those.
  • Ensure that we fix the version of hte page we test. Just like we do with talos tp5 etc

Define how to capture video of phones that do not have HDMI output

• Should be possible by using raw video capture data
• Might need a small fuzz factor that needs to apply because of camera capture and positioning etc.
• Caution that some of the analysis will depend on picture perfect capture

Define how automated any system with a video camera involved can be (i.e. we need humans to put the right phone in front of the camera)

(skipped: more interest in HDMI capture for now)

Define how you want to access this remotely to do analysis and what the priority of that is. (Is MV VPN OK?)

This is probably pretty low priority right now, better to worry abotu the automation first and then get the debugging stuff setup later.

Define the B2G usecases so that we can solve any blockers there while we iron out native fennec details

• B2g not quite at the stage where it would need this yet
• B2G wants same kind of automated regression testing but the usecases are small enough that we can check them manually
• They will want to use it, but later.

Define our deadlines

• Checkerboarding automation would be up and running first - compare XUL with Native first, hen cross browser checkerboarding with other browsers
  • Gives us a "user metric" to complement the numbers that talos will give us.
• 2nd would be cross browser startup so that we could run that > 1 time a week

Get any phones we need ordered ordered

• LG G2X is on order.