Mobile/Fennec WinCE Testing
WinMobile / WinCE Fennec Testing Notes
This page is a brain dump of all that I learned in the past few months trying to build a support architecture for doing Fennec automated testing on WinCE / WinMobile devices.
Automated testing requires:
1. One time testing device setup with everything required for automated testing
2. Method of automatically copying just-built bits onto devices
3. Method of running tests and collecting output from those tests
4. Method of parsing that output for success or failure information
5. Method of storing all the testing information for later review
6. Method of resetting each device to a known good state
Requirements 2. through 6. need to happen without user intervention.
In an ideal world, requirement 1. would be accomplished via a single large installation CAB for setting up devices.
Copying of just-built software onto devices
Within the mobile-browser project's object directory, issuing a make installer should create both an installation CAB file, as well as a staging directory which has all the bits you need to run Fennec on a WinCE / WinMobile device.
This staging directory is usually found at $OBJDIR/mobile/dist/fennec
Creating a ZIP file of this directory, then copying the resulting file to each device, and then unzipping the file on device results in at least one dialog box which must be acknowledged by user interaction.
Using the created CAB file to install bits on-device results in at least two dialog boxes which must be acknowledged by user interaction.
So I began looking into methods for copying files onto device via linux-like methods. Ideally, the resulting method should be useful for running testing scripts as well.
First I explored TELNETD, which was sample code from Microsoft included in the Platform Builder for WinCE and Platform Builder for Windows Mobile Yes, these are two different products from Microsoft. The TELNETD sample code was written in 2003, and not updated since.
After working for a while to bring that sample code into 2009, and compiling that sample code from within Visual Studio 2008 (VS9) - I found that the code uses the Microsoft-supplied shell CMD.EXE The CMD.EXE shell is very limited, and offered no scripting ability.
My next effort was directed at a code base named Open SSH for Windows CE, or CESSH. This code was intended to be compiled and run from within a Platform Builder Board Support Package, or BSP. This package was intended to be built into a WinCE or Windows Mobile device's ROM - not as an add-on to an already existing device.
After working to get the project building in Visual Studio 2008 (VS9), I found that it, also, depended upon the simple Microsoft CMD.EXE command shell.
This lead me to Voxware and Craig Vanderborgh. Their shish solution (SH-ish, or SH-like) command shell is a production quality piece of their business offerings. Their micro-shell, or barely functional shell, has a lot of unix-like qualities.
I believe that the shish shell is the best quality solution for communication with WinCE / Windows Mobile devices that is currently available to use in automated Fennec testing.
Running Tests And Collecting Output Of Tests
Automated Firefox and Fennec tests are run via shell scripts running python scripts which actually call each test in turn. Each test outputs their results to STDOUT / STDERR in the manner of LINUX applications. This is not a problem for Windows desktop computers.
WinCE / Windows Mobile suffers from a lack of pipes. That is, this operating system does not have native support for a simple way of redirecting stdin / stdout / stderr streams from console applications.
Because of this lack, the PythonCE port of a Python interpreter is not able to run child applications via the Popen() function. There are a couple of solutions to this issue:
A. Create support for sending printf output to both STDOUT/STDERR and a text file. This would allow output capture/analysis via file parsing, and could be the simplest route toward getting some sort of working Popen() for use with testing.
B. Use the PipeLib.dll / PipeDev.dll implementation of Pipes using a device driver. This implementation seems to be fine for small executables and small tasks, but seems to crash on larger executables such as xulrunner.exe - the basis for Fennec.
I have been working to port the Python version 2.5 SUBPROCESS.PY script to WinCE / Windows Mobile using the PipeLib.dll functions CreatePipe(), GetPipeName(), and SetPipeTag().
I have a subprocess.py script that has a simple test to run if called like this: python.exe \Complete\path\to\subprocess.py
However, when I run the command as listed, the PipeDev.dll creates a crash within the device.exe application running under Windows Mobile 6.
In attempting to debug this crash, I grabbed sources for the PipeLib.dll and PipeDev.dll from the cegcc project.
I created a simple Visual Studio 2008 (VS9) solution for building PipeLib.dll and PipeDev.dll. A TAR of this code can be downloaded from here.