User:Ctalbert/OrodruinAndBuildbot

So, we have gotten to a point now where the Orodruin code can instantiate and run tests on devices. We are moving out to more devices. Here is an overview of the way this works.

TestServer

• Accumulates build test "jobs" i.e. mochitest, reftest etc
• Accumulates devices from the device registration step
• Waits to be contacted by a Test Agent

Test AgentMgr (instantiates testAgents as needed)

• Polls server for a device until its device queue is full (this is a configurable number of devices, corresponds to the number of testAgent threads we want to spawn).
• When it gets a device:
  • Instantiates a testAgent with the given device
  • The testAgent will then poll the server for a job to run

TestAgent

• Spawned with a device and will own that device for its lifetime
• Polls server for jobs
• Downloads builds, downloads test packages, extracts and runs them on the device.

That's what we've got, and it currently runs mochitest. We're working on xpchsell and reftest.

Buildbot

I spent a while looking at how to hook this into buildbot today and I see some differences in methodology that are going to be difficult to overcome. What we have above is essentially a pull methodology and buildbot is a push methodology. That's at a high level of course, but I think this gap can be bridged.

To hook this system into buildbot, we are proposing a couple of setups:

Setup 1

TestServer (registration only)

• Registers devices and keeps track of the pool of devices
• Waits for test Agents to request devices

Buildbot

• Downloads builds, test packages, handles getting the changeset ID for the build.
• Instantiates testAgent with the above information and a test type. For example: python testAgent.py <binary> <tests> <changeset> <testtype>

• TestAgent
  • This would be instantiated from buildbot and upon instantiation would ask the testserver for a device of the desired platform.
  • On obtaining the device, it would carry out the test that was asked of it.
  • It would put the logs in some place that buildbot would be looking for them.

Issues:

• The main issue with this is that we really wanted to do a one to many relationship between PCs and devices. This approach implemented as described is a 1:1 relationship. However, we think that if we could put in several slaves that all point at the same box: i.e: winmo-test-1, winmo-test-2 that all point at winmo-test.

Then by adding a bit of smarts in the test Factory, we could instantiate several testAgent's on one machine by using different directories for each:

• winmo-test1 --> build, tests, and testAgent would be unpacked to build/winmo-test1 directory. Then tests would be run from there.
• winmo-tes2 --> build, tests, and testAgent unpacked to build/winmo-test2 and so on.

Setup 2

This method would eschew all the testAgent/Device management stuff in favor of a more direct buildbot approach.

Buildbot

• Would have all mobile devices under test as part of its slave pool.
• Would have to talk to some box that is a winmo-test-slave.
• On that slave box, it would unpack builds/test packages
• From that slave box, a set of steps would be created that would do the following:
  • unpack the tests properly
  • move what is required for the remote tests to the devices calling devicemanager APIs (device manager is the python wrapper to the SUT agent running on the device, device manager is on the PC, talking through a socket to the SUTAgent)
  • run the tests by adding steps for each process (starting remote web server, sending the test command line etc, collect results off the device)

Issues:

• We'd have to import the devicemanager.py into the buildsteps that would be on the client, or refactor it to allow a "single shot" type of command interface. The former is a question for the build team whether or not that is acceptable, but the latter would be pretty easy to do.
• This is still a 1:1 PC to device ratio, unless you do the same trick that I described above, but I'm not clear on whether or not a slave can run multiple steps in different processes.

Is there a better solution that we're missing?