QA/TDAI/Mobile Test Framework Meeting

This is a quick agenda for the meeting today. The idea is that we want to agree on a solid approach for WinMO and WinCE test automation going forward, and while doing that we want to lay the necessary groundwork to make it very easy to move to platform X in the next few months as our mobile ambitions scale ever upward toward new devices and platforms.

Where We Are

Unit Tests on WinMo/WinCE Platforms

• Serving tests remotely has been a big win
• Active Sync is not reliable
• The XDA APIs have been generally usable, though not with active sync
• PythonCE cannot solve all the issues here, but we can generally get around it

Talos on WinMO/WinCE

• Reducing the amount of tests run on the Talos boxes
  • reduce test set to tp4/ts
  • serve pages from local file rather than running a local http server
  • do not attempt to collect memory usage
• Active Sync is not reliable
• PythonCE cannot solve all the issues here, but so much of Talos is in Python, it's unclear how to get around it.

What Does the Next Generation Test Framework Look Like?

Idea Number 1

• Run the tests over a network, use a small, platform specific agent to communicate with the server to arrange which tests to run, start processes etc.
  • ADVANTAGES
    • 90% cross platform, will need to rewrite the agent code for each platform
    • Already validated that unit tests can run from remote server
    • Can use agent 'heartbeat' to identify when and which devices go down
    • Less burden on system under test
  • DISADVANTAGES
    • Talos timing calculations become more complicated
  • QUESTIONS
    • Can we depend on the device always having a network connection?

Idea Number 2

• Get Python running on each device OS, run tests normally except for various types of chunking
  • ADVANTAGES
    • We know how to do this - this is what we do with maemo
  • DISADVANTAGES
    • This puts us into a situation of trying to support/improve python on each platform
    • This runs a lot of code on each device, contributing to the overall instability on devices.

Idea Number 3

• Use existing tools that control via tethering to hosts
  • ADVANTAGES
    • Most platforms seem to have some sort of "tethered" development mode that provides the support we need
    • Verified this approach on WinMO/WinCE mochitests
    • Gets around issues with wifi in the building
  • DISADVANTAGES
    • At least in WinMo/WinCE case activesync tethering has not proved to be reliable enough for large scale automation
    • Host to device mapping might constrain us unnecessarily (especially if it requires a 1:1 mapping rather than a 1:many)

Other Ideas?