Platform/Games/GameFocusedBenchmarking/Redesign

Device joins the test framework:

• 'perfy-client start worker'
• start daemon on test device; put manager address in config
• worker: send <ANNOUNCE> message to manager
• manager: receive <ANNOUNCE> message; add new device to device pool (collection of same devices)
• manager: if there are jobs associated with this pool, claim a task and send a <DISPATCH> message

Add jobs to the queue:

• `perfy-client run --device="nexus4" --browser="firefox" --channel="release" --benchmark="octane-2.0" --replicates=30`
• client: send <COMMAND:queue> message to manager
• manager: assign an ID to the job
• manager: create tasks for each element of the job and assign each task an ID
• manager: associate the job with a device pool
• manager: for each device in the associated device pool, send <DISPATCH> message
• manager: when a <RETURN> message is received, return the worker from to the pool and possibly assign it another task

Client

• zmq.REQ sends <COMMAND> messages to the manager or worker and gets replies
• Sends a single <COMMAND> then exits

Manager

• zmq.ROUTER listens for <ANNOUNCE> messages from workers
• zmq.PUSH sends <DISPATCH> messages to workers
• zmq.PULL receives <RETURN> and <ERROR> messages from workers
• Results are pushed via pyes library to ES cluster
• zmq.RES listens for <COMMAND> messages from the command-line client and sends replies
• Tasks are dispatched to workers based on their device name

 * Tests results should be consistent across workers with the same device name

• When a job is received from a client, create a job ID and a set of tasks for that job

Worker

• zmq.REQ sends <ANNOUNCE> messages to the manager
• zmq.PULL receives <DISPATCH> Messages from the manager
• Small web server receives POST XHR messages from tests to record results
• zmq.PUSH sends <RETURN> and <ERROR> messages to the manager
• Should manage installed browser versions

 * Download/install new versions
 * Remove old versions
 * Keep track of installed browsers by vendor/channel

• zmq.RES listens for <COMMAND> messages from the command-line client and sends replies

Messages

• <ANNOUNCE>

 * zmq.ROUTER records the worker ID
 * {'device': <string>}

• <COMMAND>

 * {'name': 'queue',
    'args': {
             'device': <string>,
             'channel': <string>,
             'browser': <string:'firefox','chrome'>,
             'benchmark': <string:'octane-2.0','sunspider-1.0','kraken','canvasmark'>
    }}
   * Queue a job for the manager
 * {'name': 'list'}
   * Return a list of queued jobs along with status
 * {'name': 'restart'}
   * Restart the target daemon
 * {'name': 'stop'}
   * Stop the daemon

• <DISPATCH>

 * {'id': <unsigned>,
    'browser-installer': <string:url>,
    'benchmark-url': <string:url>
   }

• <RETURN>

 * {'id': <unsigned>,
    'value': <unsigned>
   }

• <ERROR>

 * {'id': <unsigned>,
    'name': <string:'timeout'>,
    'message': <string>
   }

Problems

• How to make sure that all machines are running the right version?
  • Send installer URL with dispatch message
  • Worker should verify based on the URL that it has the correct version installed
• How to determine which release channel a browser belongs to?
  • Match release channel with version number when results are collected
  • Manager knows release channel, worker knows version number
• How to group machines into pools so that tasks are run on the right hardware?
  • Store device name in config file
• Firefox OS devices need to be reflashed
  • Each device will need a USB connection and a desktop-based client to automate the updates
  • Is there a way to do OTA updates for nexus 4?