	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

Last updated: 2015/05/21

Overview

Summary

Loadtesting Loop is currently performed with loadsv1 tool. Loadsv2 is under development.

Loadtesting should be performed against

loop-server
msisdn-gateway

It's not necessary to loadtest against loop-client

Guidelines for loadtesting with loadsv1 tool are provided below.

Loadsv1

Installing Loop-Server and the Loads tool (on Localhost or AWS)

refer to github README

Installation

loop-server

$ git clone https://github.com/mozilla-services/loop-server
$ cd loop-server/loadtests </pred>
$ make install

msisdn-gateway

$ git clone https://github.com/mozilla-services/loop-server
$ cd msisdn-gateway/loadtests </pred>
$ make install

Pre-requisite

edit loadtest config file
enter desired values for current test

$ vim /loadtests/config/megabench.ini

Run Test There are 3 commands available for testing with loadsv1:

$ make test
$ make bench
$ make megabench

#toc

MSISDN - loadtesting

There are 2 staging environments for msisdn-gateway

msisdn.stage.mozaws.net - for end-2-end testing (NOT for loadtesting!)
msisdn-loadtest.stage.mozaws.net - for loadtesting
- This environment also contains its own mock server: http://omxen.dev.mozaws.net

see:

configuration file on the Stage server: /data/msisdn-gateway/config/production.json

#toc

Test Execution

make test

Summary

Running $ make test allows you to kick off a loadtest locally against a remote endpoint, but then receive test output locally.

Optional: if you want to specify a test endpoint other than the default (see: Makefile), you can do it like so:

$ make test SERVER_URL=https://loop.stage.mozaws.net
 - or -
$ make test SERVER_URL=https://msisdn.stage.mozaws.net

WARNING: Do not launch loadtests against any PRODUCTION server, ever!

Example Output

$ make test SERVER_URL=https://loop.stage.mozaws.net
SIMPLE_PUSH_URL=https://call.stage.mozaws.net/ ./venv/bin/loads-runner --config=./config/test.ini --server-url=https://loop.stage.mozaws.net loadtest.TestLoop.test_http_calls
[                                                                                                                                                                                                                                 ]   0%/Users/rpappalardo/git/services-test/build/loop-server/loadtests/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
[=================================================================================================================================================================================================================================] 100%
Duration: 4.99 seconds
Hits: 17
Started: 2015-05-21 22:26:10.930852
Approximate Average RPS: 3
Average request time: 0.28s
Opened web sockets: 0
Bytes received via web sockets : 0

Success: 1
Errors: 0
Failures: 0


Slowest URL: https://loop.stage.mozaws.net/     Average Request Time: 1.121348

Stats by URLs:
- https://loop.stage.mozaws.net/                        Average request time: 1.121348  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/Dko0_Uvdks8       Average request time: 0.689918  Hits success rate: 1.0
- https://loop.stage.mozaws.net/rooms                   Average request time: 0.429905  Hits success rate: 1.0
- https://loop.stage.mozaws.net/rooms/C4lqDXc6W3M       Average request time: 0.189937375       Hits success rate: 1.0
- https://loop.stage.mozaws.net/registration            Average request time: 0.16524675        Hits success rate: 1.0
- https://loop.stage.mozaws.net/call-url                Average request time: 0.163699  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls?version=200       Average request time: 0.163517  Hits success rate: 1.0

Custom metrics:
- delete-room : 1
- register : 4
- generate-call-url : 1
- refresh-room-presence : 1
- initiate-call : 1
- leave-room : 3
- room-with-context : 1
- join-room : 3
- create-room : 1
- list-pending-calls : 1
- num-participants-3 : 1
SIMPLE_PUSH_URL=https://call.stage.mozaws.net/ ./venv/bin/loads-runner --config=./config/test.ini --server-url=https://loop.stage.mozaws.net loadtest.TestLoop.test_websockets
[                                                                                                                                                                                                                                 ]   0%/Users/rpappalardo/git/services-test/build/loop-server/loadtests/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
[                                                                                                                                                                                                                                 ]   0%e
[=================================================================================================================================================================================================================================] 100%
Duration: 81.80 seconds
Hits: 17
Started: 2015-05-21 22:26:16.133047
Approximate Average RPS: 0
Average request time: 0.24s
Opened web sockets: 7
Bytes received via web sockets : 1350

Success: 1
Errors: 0
Failures: 0


Slowest URL: https://loop.stage.mozaws.net/     Average Request Time: 0.713177

Stats by URLs:
- https://loop.stage.mozaws.net/                        Average request time: 0.713177  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/KR414mzrf-s       Average request time: 0.359527  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/bYhXDs86FTQ       Average request time: 0.358428  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/vqp9-BYyYBQ       Average request time: 0.357578  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/Nbi4t9_pLAc       Average request time: 0.356943  Hits success rate: 1.0
- https://loop.stage.mozaws.net/registration            Average request time: 0.1638505 Hits success rate: 1.0
- https://loop.stage.mozaws.net/call-url                Average request time: 0.16342975        Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls?version=200       Average request time: 0.16315475        Hits success rate: 1.0

Custom metrics:
- websocket-basic-callee-messages : 5
- websocket-supervisory-caller-messages : 2
- list-pending-calls : 4
- generate-call-url : 4
- initiate-call : 4
- websocket-ringing-callee-messages : 3
- websocket-basic-caller-messages : 5
- websocket-ringing-caller-messages : 3
- register : 4

#toc

make bench

Use megabench instead

#toc

make megabench

In Terminal $ make megabench

./venv/bin/loads-runner --config=./config/megabench.ini --user-id=bonzogonzo --server-url=https://loop.stage.mozaws.net:443 loadtest.TestLoop.test_all
[2015-05-21 16:00:21,868][11285] Detached. run --attach to reattach

'Open the following URL in browser:' https://loads.mozilla.services.com

Login with credentials provided by Loop dev/QA team
You will see real-time output from your test run in your browser

#toc

Loadtest Monitoring

Monitoring Loop Stage

Loads dashboard:
- http://loads.services.mozilla.com
Cluster status
- Check directly from the Loads Cluster dashboard:

Agents statuses
Launch a health check on all agents

For all other monitoring, see the following section:
- https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Stage_Environment

OPs and Infra
- https://github.com/mozilla-services/puppet-config/issues
- https://github.com/mozilla-services/svcops/issues

Loads Tool and Cluster

The QA Test Environments

#toc

Reference

Load Test Tool Client/Host

It is always best to configure an AWS instance as the host for all load testing.
All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
- https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
- https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

Installation:

git clone https://github.com/mozilla-services/loop-server.git
cd loop-server
Note: You may want to install a specific branch for testing vs defaulting to Master
npm install
ulimit -S -n 2048
npm test *
cd loadtests
make build
make test

#toc

OLD Notes

Coverage report can be found here:
/loop-server/coverage/lcov-report/index.html

* This step requires the redis server to be installed and running:
Mac:
brew install redis
redis-server /usr/local/etc/redis.conf

Ubuntu Linux:
sudo apt-get install redis-server
sudo /usr/bin/redis-server /etc/redis/redis.conf
sudo tail -f /var/log/redis/redis-server.log

RHEL Linux:
Install redis from here: http://download.redis.io/releases
then
/usr/local/bin/redis-server /home/ec2-user/redis-2.8.9/redis.conf
or similar

Note:
- This will install a local copy of the Loads tool for use with the Loop-Server.

- To hit the partner test servers, the following configuration file will need to be updated by OPs:
- /data/loop-server/config/settings.json
- Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

REFs:

#toc

Using the Loads V1 Services Cluster for the Loop-Server in Stage

By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
Stage environment:

$ make megabench SERVER_URL=https://loop.stage.mozaws.net

By default, the Loop-Server in Stage is configured to talk to our mock server:
- https://loop-delayed-response.stage.mozaws.net/

To hit the partner test servers, the following configuration file will need to be updated by OPs:
- /data/loop-server/config/settings.json

Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

REFs:

Using the Loads V1 Services Cluster for the MSISDN-Gateway

By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
Stage environment:

$ make megabench SERVER_URL=https://msisdn.stage.mozaws.net

This environment also contains its own mock server: http://omxen.dev.mozaws.net
The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

REFs

Configuring The Load Tests

Makefile
- The SERVER_URL constant can be changed.

Config files
- For make test (Loop-Server and MSISDN-Gateway):
  - Number of hits
  - Number of concurrent users

- For make bench (Loop-Server and MSISDN-Gateway):
  - Number of concurrent users
  - Duration of test

- For make megabench (Loop-Server and MSISDN-Gateway):
  - Number of concurrent users
  - Duration of test
  - Include file (this is code dependent)
  - Python dependencies (this is code dependent)
  - Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
  - Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
  - Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
  - Observer (this can be email or irc - the default is irc #services-dev channel)

Loop-Server load test code
- The Loop-Server load test can not currently be configured in the code

MSISDN-Gateway load test code
- The MSISDN-Gateway load test can be configured in the code - see the following lines:
- Test server URL: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L15
- Error percentages: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22

General REFs:
- https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
- https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py

Test Coverage and Stats

Basic tweakable values for all load tests
- users = number of concurrent users/agent
- agents = number of agents out of the cluster, otherwise errors out
- duration = in seconds
- hits = 1 or X number of rounds/hits/iterations

Analyzing the Results

There are several methods and tools for analyzing the load test results.

1. Using the Loads Services Cluster dashboard
- All loads tests using this cluster generate a live report and a run report available on this site:
  - http://loads.services.mozilla.com

Debugging the Issues

There are several methods and tools for debugging the load test errors and other issues.

1. Important logs for Loop-Server (per server)
- /var/log/circus.log
- /var/log/loop_err.log
- /var/log/loop_out.log
- /var/log/hekad/loop.stdout.log
- /var/log/hekad/loop.stderr.log
- /var/log/nginx/access.log
- /var/log/nginx/error.log

2. Important logs for MSISDN-Gateway (per server)
- TBD

Acceptable/Unacceptable Loop-Server errors:

hekad loop.stderr.log
The following are acceptable:
Decoder 'LoopServer-LoopServerDecoder' error: Failed parsing
Plugin 'AggregatorOutput' error: writing to heka.shared....

nginx logs:
The 200s (good stuff) and 101s (websockets) are acceptable. 
The 499s are an artifact of the current load testing tool (V1).
You should only see them at the end of the load test.
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
Any percentage of 405s, 502s, or 503s is not acceptable.

Application logs:
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
See the loop_server.out.log file.

/var/log/loop_err.log
The following are acceptable: connect: res.on("header"): use on-headers module directly

In the Loads Cluster dashboard, watch for the following errors/failures:
string indices must be integers
OR
No JSON object could be decoded
OR
'hawk-session-token'

Acceptable/Unacceptable MSISDN-Gateway errors:

The updated load test does generate a certain percentage of errors:
https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
So, expect to see a predefined percentage of 204s and 400s, along with the usual 200s in the nginx access logs.
The msisdn-gateway app logs should be clean with just msisdn and test data.

Loop/Test/Loadtesting

Contents

Overview

Summary

Loadsv1

MSISDN - loadtesting

Test Execution

make test

Summary

make bench

make megabench

Loadtest Monitoring

Monitoring Loop Stage

Reference

Load Test Tool Client/Host

OLD Notes

Using the Loads V1 Services Cluster for the Loop-Server in Stage

Using the Loads V1 Services Cluster for the MSISDN-Gateway

Configuring The Load Tests

Test Coverage and Stats

Analyzing the Results

Debugging the Issues