Grinder

Grinder is a distributed tool for load testing a resource. A master program, known the console, marshals a set of agents. The agents, in turn, marshal processes and threads that exercise the resource repeatedly. The agents track statistics and report them back to the console. The console can then save the aggregated statistics for further analysis.

I've set up a tar file with the complete self contained grinder environment with the appropriate versions of java and jython. Untar this file and set the GRINDER_PROJECT_BASE_PATH environment variable to the base path of the install path. Within the Grinder project tree, there is a file called .../etc/grinder.properties. This file controls how many processes and threads the agents use. The file .../project/bouncer/bouncergrind.py is the actual jython code for the test. Within that file are the hard coded URLs to be tested. The code is setup to allow for a function to do discovery of the URLs, but it is unimplemented.

The Bouncer Tests

Morgamic and I ran some preliminary tests using two agent machines (download02.sj.mozilla.com and download03.sj.mozilla.com). We setup the Grinder agents to test three URLs in sequence repeatedly. In Grinder terms, each URL was considered to be a “test”. We ran the tests in three sessions with each session having more tests per second than the previous. The number of tests per second is controlled by adjusting the number of threads and processes used by each agent.

When submitting a URL to bouncer, we wait for the http result, but we do not follow through with the download. We abandon the request as soon as we've received the http result.

Results

Session 1

In this first session, we used two agents for five minutes. Each agent used one process with fifty threads. This means that each of the fifty threads was hitting each of the three URLs in turn repeatedly over the five minutes. With two agents running, this simulated approximately 100 simultaneous connections constantly throughout the session time.

Session 2

In this session we doubled the number of test by telling each agent to use to a second process of fifty threads. This means that 200 simultaneous requests constantly throughout the session.

Session 3

In this session, we doubled again, but increasing the number of threads to 100 for each agent process. This is means 400 simultaneous active requests constantly throughout the session. This session resulted in very poor performance – we theorize that we're hitting some limit on the machines running the agents. Monitoring the statistics on the server side did not show any undo load.

Aborted Session 4

For this session we left the total number of tests the same, but changed to four processes of fifty threads instead of Session 3's two processes of one hundred threads. We tried this to make sure that they perform equivalently. Both 3 and 4 performed about the same, so we aborted test 4.

Interpretation

It is clear that thee results are incomplete. It appears that we ran into limitations on testing side rather than the server side.