TestEngineering/Services/LoopServerLoadTesting: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 157: Line 157:
  git clone https://github.com/mozilla-services/loop-server.git
  git clone https://github.com/mozilla-services/loop-server.git
  cd loop-server
  cd loop-server
  make build
  npm install
  make test *
  npm test *
  cd loadtests
  cd loadtests
  make build
  make build

Revision as of 20:17, 19 June 2014

Summary for Loop Server, Loop Client, Mock Server, MSISDN Gateway

Quick Verification Of Stage Deployments

  • This is a quick sanity test of the environment before getting started on load tests.
  • Loop Server
For now, just run a quick loadtest 'make test'
cd loop-server
cd loadtests
make test SERVER_URL=https://loop.stage.mozaws.net
  • Loop Client
Check https://call.stage.mozaws.net/config.js
Should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.stage.mozaws.net'};
  • MSISDN Gateway
    • TBD
  • Mock Server
    • N/A

Load Test Tool Client/Host

Creating a RHEL AWS instance

  • Pick a Region then Create Instance > Launch Instance
  • Follow the prompts to create a basic, RHEL-flavored instance
  • Use of the QA/Dev key pairs that have been set up for this:
    • US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
    • US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
  • Once the instance is running, log in as "ec2-user"
  • The following apps, tools, and libs will need to be installed for use with various Services applications:
    • gcc, gcc-c++
    • hg
    • git
    • python-devel
    • automake, autoconf, and libtool (required for libzmq, for easy_install)
    • pip
    • virtualenv
    • node/npm
    • zeromq 3.X
    • gmp, gmp-devel
  • Also, general rhel updates:
$ sudo yum -y update
and/or
$ sudo yum -y upgrade
  • Now, the instance should be ready for installing and using the Loads tool.

Creating an Ubuntu AWS instance

  • Pick a Region then Create Instance > Launch Instance
  • Follow the prompts to create a basic, Ubuntu-flavored instance
  • Use of the QA/Dev key pairs that have been set up for this:
    • US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
    • US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
  • Once the instance is running, log in as "ubuntu"
  • The following apps, tools, and libs will need to be installed for use with various Services applications:
    • gcc, g++
    • mercurial
    • git
    • python-setuptools, python-virtualenv, and python-dev
    • automake, autoconf, libtool
    • m4
    • node/npm
    • libzmq and zeromq 3.X
    • gmp-5.1.3 or newer
  • Also, general rhel updates:
$ sudo apt-get update
and/or
$ sudo apt-get upgrade
  • Now, the instance should be ready for installing and using the Loads tool.

Installing Loop-Server and the Loads tool on the AWS instance

  • Installation:
git clone https://github.com/mozilla-services/loop-server.git
cd loop-server
npm install
npm test *
cd loadtests
make build
make test

* This step requires the redis server to be installed and running:
Mac:
brew install redis
redis-server /usr/local/etc/redis.conf

Ubuntu Linux:
sudo apt-get install redis-server
sudo /usr/bin/redis-server /etc/redis/redis.conf
sudo tail -f /var/log/redis/redis-server.log

RHEL Linux:
Install redis from here: http://download.redis.io/releases
then
/usr/local/bin/redis-server /home/ec2-user/redis-2.8.9/redis.conf
or similar
  • Note: This will install a local copy of the Loads tool for use with the Loop-Server.

Running the load test against the Loop-Server in Stage

  • Stage environment:
$ make test
or
$ make test SERVER_URL=https://loop.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://loop.stage.mozaws.net
  • To hit the partner test servers, the following configuration file will need to be updated by OPs:
    • /data/loop-server/config/settings.json
  • Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

Using the Loads Services Cluster for the Loop-Server in Stage

  • By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
  • Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
  • Stage environment:
$ make megabench SERVER_URL=https://loop.stage.mozaws.net
  • To hit the partner test servers, the following configuration file will need to be updated by OPs:
    • /data/loop-server/config/settings.json
  • Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

Installing MSISDN-Gateway and the Loads tool on the AWS instance

  • Installation:
TBD
  • Note: This will install a local copy of the Loads tool for use with MSISDN-Gateway.

Running the load test against MSISDN-Gateway in Stage

  • Stage environment:
$ make test SERVER_URL=BLAH
$ make bench SERVER_URL=BLAH
  • This environment also contains its own mock server: TBD

Using the Loads Services Cluster for the MSISDN-Gateway

  • By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
  • Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
  • Stage environment:
$ make megabench SERVER_URL=BLAH
  • This environment also contains its own mock server: TBD

Configuring The Load Tests

  • For make test (Loop-Server and MSISDN-Gateway):
    • Number of hits
    • Number of concurrent users
    • See the loadtest Makefile and the config/test.ini file
  • For make bench (Loop-Server and MSISDN-Gateway):
    • Number of concurrent users
    • Duration of test
    • See the loadtest Makefile and the config/bench.ini file
  • For make megabench (Loop-Server and MSISDN-Gateway):
    • Number of concurrent users
    • Duration of test
    • Include file (this is code dependent)
    • Python dependencies (this is code dependent)
    • Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
    • Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
    • Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
    • Observer (this can be email or irc - the default is irc #services-dev channel)
    • See the loadtest Makefile and the config/megabench.ini file

Test Coverage and Stats

  • Basic tweakable values for all load tests
    • users = number of concurrent users/agent
    • agents = number of agents out of the cluster, otherwise errors out
    • duration = in seconds
    • hits = 1 or X number of rounds/hits/iterations
  • Loop-Server
    • TBD
  • MSISDN-Gateway
    • TBD

Analyzing the Results

  • There are several methods and tools for analyzing the load test results.

Debugging the Issues

  • There are several methods and tools for debugging the load test errors and other issues.
  • 1. Important logs for Loop-Server (per server)
    • /var/log/circus.log
    • /var/log/loop_err.log
    • /var/log/loop_out.log
    • /var/log/hekad/loop.stdout.log
    • /var/log/hekad/loop.stderr.log
    • /var/log/nginx/access.log
    • /var/log/nginx/error.log
  • 2. Important logs for MSISDN-Gateway (per server)
    • TBD
  • Acceptable/Unacceptable Loop-Server errors:
nginx logs:
Some percentage of 200s, 204s, and 404s is acceptable.
Any percentage of 405s, 502s, or 503s is not acceptable.

/var/log/loop_err.log
The following are acceptable: connect: res.on("header"): use on-headers module directly

In the Loads Cluster dashboard, watch for the following errors/failures:
string indices must be integers
OR
No JSON object could be decoded
OR
'hawk-session-token'
  • Acceptable/Unacceptable MSISDN-Gateway errors:
TBD

Monitoring Loop Stage

Agents statuses
Launch a health check on all agents

Performance Testing Information

  • TBD

Details on the Load Test tool

Known Bugs, Issues, and Tasks

  • Loop Client: none at this time
  • Mock Server: none at this time

References

  • OPs pages for stats collection, logging, monitoring
    • TBD
Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/LoopServerLoadTesting&oldid=990721"