Auto-tools/Projects/Loderunner

We have a ton of duplicated code in our test harness infrastructure. We have several components that ought to be easy to reuse and aren't. Some that were created to be easy to reuse and still aren't. We need a better set of APIs for doing low-level harness tasks that all our code can reuse and share and that shared code should provide an API that is useful and simple to use so that other people (and us) can build on top of it without reinventing the wheel or further duplicating then tailoring the existing code. This project aims to provide this low level API.

Let's start a discussion on the API and so in that vien, let's list out the requirements.

Some Requirements

I've broken down requirements into specific actions to make it easier to grapple with one problem space at a time.

Profile Management

• Purpose: Used by the harness to manage the profile for the application under test
• Create an empty, blank profile
• Create a profile with a set of default data
  • Able to specify a set of extensions for the profile
  • Able to specify a set of preferences for the profile
  • Able to specify a set of bookmark data for the profile (bookmarks, folders, tags, livemarks, annotations, etc)
  • Able to specify a set of stored password data for the profile
  • Able to specify a set of download data for the profile
  • Able to specify a set of history data for the profile
  • Able to specify a set of "sessionrestore" data for the profile
• Extension point - allow people to easily create their own profiles with their own unique data

Application Management

• Purpose: Used by the harness to handle the specific details of the application under test
• Starts an application
  • able to specify any command line arguments to the application under test
• Determine if the application is hung
• Determine if the application crashes
• Provide the application's path
• Provide the application's name
  • Able to log the crash stack if such a crash occurs
• Kill the application and all child processes
• Able to start the application with a given profile
• Extension point - allow an application to use any command line attributes easily

Log Management

• Purpose: Provides ability to log information from both within the application and from the outside harness
• Allow levels of logging
  • Example: DEBUG, INFO, ERROR, WARNING etc
• Provides log to stdout in a common format for all test harnesses, regardless if the logging is occuring from the harness logic (Python) or from application logic (JavaScript)
• Handles writing to log files
• Handles writing to std out
• Extension Point - provide both in test logging as well as debug logging for testing during development.
• Extension Point - should be able to reuse this logging functionality in new programs without bringing along unreleated pieces (should be stand alone)

Dependcy Management

• Purpose: Allow test harnesses to create dependencies that they need in order to run, for example, a webserver
• Provide a way to start dependent application
• Provide a way to stop dependent application
• Provide a way to send command line arguments to the dependent applcation
• Extensiblity point - this has to be very extensible by nature. Each new dependent application will need its own "Dependency Manager" api which can then be used by the test harness in conjunction with the Application manager to ensure that the application has all needed dependencies when it starts up.

Command Line Manager

• Purpose: Maintains a common set of command lines across all the harnesses.
• Handles parsing
• Handles defaults
• Handles help messages
• Handles validation of command lines (which can be overrided by the harness)
• Extensibilty point - test harnesses will need to be able to add their own specific command line options

API Discussion

So perhaps a harness that uses the above abstractions could look like this:

mylogger = LogHandler(self.myCmdLineMgr.logFileOption) myPrefs = [{"pref": "user.pref.foo", "value": "5"}] profile = ProfileManager(myPrefs)

1. Create a dependency manager object for Httpd

mywebsrv = HttpdWebServer()

1. Do some set up

profile.writePref([{"pref":"some.new.pref","value":"some new value"}])

1. Start our webserver

mywebserv.start()

myApplication = ApplicationManager(profile, myCmdLineMgr.getAllOptions(), mylogger) myApplication.start()

Some Goals

• Each piece of functionality should have a clear interface built into it. Each component should only rely on the published interfaces of the other components. That way, we can easily quanitfy the impact of a given change by knowing what depends on what.
  • In the example above, I assert that the Application Manager doesn't know a thing about the commandline manager, it simply uses a list returned by getAllOptions(), for example.
• Each interface definition for each component will be required and codified into a set of unit tests for the component. That way we can easily know if we break them. Furthermore the tests will become their own documentation for how the component is used.

Next Steps

• Did I define the components properly?
  • Are any missing?
  • Are there smaller building blocks that these can be broken into that makes sense?

• What are the interfaces on each of these components?
  • How are the components extended?
  • How are those interfaces consumed?
  • How will those interfaces be denoted in Python (which I assume most of this will be implemented in) - since python has no concept of "public" versus "private" what code convention will we use to differentiate the two?

• How will the components be packaged?
  • Should each one be callable from the command line as its own tool in as much as it makes sense (for instance the profilemanager might be useful this way. The command line manager might not be).
  • Should we require packaging? Can co-location in directories work?

Plan

1. Identify the set of components to generalize
2. Identify the interfaces on the components (review the existing harnesses and see what basic functions need to be supported in these)
3. Identify the API usage mechanism for extensibility and for harness use (review existing harnesses and see what specialized needs each will have and ensure those can be provided given the interface definitions in step 2 and the API usage mechanism of this step).
4. Code the API use into a set of tests
5. Code the interfaces and the modules.
6. Integrate said modules into the existing harnesses, refactoring specialized data as needed (and moving basic functionality into these components where appropriate)