View source for QA/TDAI/Mobile Test Framework Meeting

This is a quick agenda for the meeting today.  The idea is that we want to agree on a solid approach for WinMO and WinCE test automation going forward, and while doing that we want to lay the necessary groundwork to make it very easy to move to platform X in the next few months as our mobile ambitions scale ever upward toward new devices and platforms.

= Where We Are =

== Unit Tests on WinMo/WinCE Platforms ==
* Serving tests remotely has been a big win
* Active Sync is not reliable
* The XDA APIs have been generally usable, though not with active sync
* PythonCE cannot solve all the issues here, but we can generally get around it

== Talos on WinMO/WinCE ==
* Reducing the amount of tests run on the Talos boxes
** reduce test set to tp4/ts
** serve pages from local file rather than running a local http server
** do not attempt to collect memory usage 
* Active Sync is not reliable
* PythonCE cannot solve all the issues here, but so much of Talos is in Python, it's unclear how to get around it.

= What Does the Next Generation Test Framework Look Like? =
== Idea Number 1 ==
* Run the tests over a network, use a small, platform specific agent to communicate with the server to arrange which tests to run, start processes etc.
** '''ADVANTAGES'''
*** 90% cross platform, will need to rewrite the agent code for each platform
*** Already validated that unit tests can run from remote server
*** Can use agent 'heartbeat' to identify when and which devices go down
*** Less burden on system under test
** '''DISADVANTAGES'''
*** Talos timing calculations become more complicated
** '''QUESTIONS'''
*** Can we depend on the device always having a network connection?

== Idea Number 2 ==
* Get Python running on each device OS, run tests normally except for various types of chunking
** '''ADVANTAGES'''
*** We know how to do this - this is what we do with maemo
** '''DISADVANTAGES'''
*** This puts us into a situation of trying to support/improve python on each platform
*** This runs a lot of code on each device, contributing to the overall instability on devices.

== Idea Number 3 ==
* Use existing tools that control via tethering to hosts
** '''ADVANTAGES'''
*** Most platforms seem to have some sort of "tethered" development mode that provides the support we need
*** Verified this approach on WinMO/WinCE mochitests
*** Gets around issues with wifi in the building
** '''DISADVANTAGES'''
*** At least in WinMo/WinCE case activesync tethering has not proved to be reliable enough for large scale automation
*** Host to device mapping might constrain us unnecessarily (especially if it requires a 1:1 mapping rather than a 1:many)

''Other Ideas?''

= Notes =
== What do we need the numbers for? ==
* Insulation against crashbugs
* Do need performance numbers for a/b testing on winmo
* Catch regressions (crash and perf)
* People are using the current numbers even though it is painful

== Things mobile team needs ==
* browser-chrome tests - to stop regressing the front end
* getting talos up and running has been on the list forever
** launching the browser, running it through some pages (topsite tests)
** getting numbers so that we undestand where we are and can see how far we are from where we want to be
** need a framework we need for future perf testing
** have unittests on maemo but they are slow and have lots of failures.  Really want that to go green, even if it is on the desktop then that will be a bigger win than getting unit tests up and running on another platform.
** CE is lower priority than everything else but a lot of this work should work on both

= oddities =
* if you do not have 8.3 names you can have only 999 files in one folder.

= python =
* we spent a lot of time working on this and it seems like a fight we're going to lose.  Probably should write that bit off
* Tethering seems like windowsce/mo solution, theoretically you can support 1:many with windows 7 but that does't help with new platforms.

* seems like the easiest way forward is to go with the native agent approach since we only need a small set of APIs on each device system.
* The agent is not dependent on any langage implementation - the agent could be in python or whatever.  It's more about the way we go about building these things so that we can get as cross platform as possible.
* Also makes it easier to run the tests since most of the logic lives on the servers
* If we do everything over the network, it is most likely going to be considerably slower than if the tests were on the file system
** There are definitely latency costs there.
** One thing the tests being remote buys us is that we don't have a remote web server.
** If we move the rest of testing on this, then we might need to consider the speed implications here doing it over the network.
** If we do it then there is a question about what is the maintenance question here, are we maintaining talos A and Talos B or do we use the new thing everywhere?  And what does that mean in terms of rebasing the old performance numbers etc.
** Talos was done on the local host specifically to avoid the latency issues with networks to remove the problem with measuring response times across the network.
** is a more real world type of scenario - both with talos and the unit tests

== maintenance burden == 
* continuously upgrading and fixing issues in talos and would have to port those issues over - already hit that with the nokia code
* the difference between talos and unit tests is that talos is 90% python, all the interesting stuff in talos is python.
* The python Talos that runs on the device must live on the client side, it can't be moved to the server
* This would have to be added to the agent anyway, though.  
* There is a lot of stuff to be done on python to get talos fully supported.  For unit tests it is much easier, less support.
* We have a lot of experience on developing the C++ agent stuff, makes more sense to work on that rather than having us develop for python.
* Developers will have to take a lot of control on working keeping the system up and running across the different platforms.  But that is something we can do.
* If we do this, then we might want to move toward making this THE talos system in the future.  The definite aim should be to end up with one framework in the end.
* maybe even giving it a different name for the new framework would help, so that it is clear that it is different.

= Assignment =
* bmoss's team will drive this, working with brad lassey to get this working.
* First step is prove out that this will actually work and that we have reasonable answers to the questions that have been raised.
* If it is acceptable then we can look at rolling it out moving forward.

== mechanical questions ==
* what platform try it on first?
** Is there any reason it can't run on one and not the other? No.  If it works on winmo then it should be fine on CE.  So, go for WinMO first
* Do we stop at the other attempt on talos in the meantime?
** Yes.  It would make sense for releng to scale up windows mobile devices for infrastructure/powered/network etc.
** Let all the reporting and software pieces be hammered out by the infrastructure guys.
* Alice and Aki/John should let us know about things that they hit when scaling out talos so taht we can avoid those roadblocks.  Most of the strange errors were with buildbot.
** There may not be any need to talk buildbot at all.
** Talos does not generate the builds it tests.  It feels like you could still use buildbot on the server side so that it knows when the builds are available. and then the server could take care of moving the builds down to the devices under test.
** Queueing the buildbot notifications on what builds to run what tests/talos on will be something that the server will understand.  Ability to resend and retest builds is something that we want to keep up and running, so the new server will integrate with the existing buildbot infrastructure.
* Might be a good idea to get this stuff up and running on desktop as fast as you can, so that we know what a success is.  Build it on both the main platforms to verify that there is consistency between the mobile platform and the desktop platforms.  We definitely want to know what that level of consistency is.

= timeline =
* what is our timeline?  do we have one?
** we don't have one.
** The next step is to put together a road map.
** We will use #mobile as a backchannel and ctalbert will keep up a mailing list for it.