View source for Loop/Test/Loadtesting


{{Draft}}
{{LastUpdated}}

= Overview =
== Summary ==
Loadtesting Loop is currently performed with loadsv1 tool.
Loadsv2 is under development.

Loadtesting should be performed against
* loop-server
* msisdn-gateway

It's not necessary to loadtest against loop-client

Guidelines for loadtesting with loadsv1 tool are provided below.


== Loadsv1 ==

'''Installing Loop-Server and the Loads tool (on Localhost or AWS)'''
* refer to github README


'''Installation'''

loop-server

<pre>
$ git clone https://github.com/mozilla-services/loop-server
$ cd loop-server/loadtests </pred>
$ make install
</pre>

msisdn-gateway

<pre>
$ git clone https://github.com/mozilla-services/loop-server
$ cd msisdn-gateway/loadtests </pred>
$ make install
</pre>


'''Pre-requisite'''
# edit loadtest config file
# enter desired values for current test

<pre>
$ vim /loadtests/config/megabench.ini
</pre>



'''Run Test'''
There are 3 commands available for testing with loadsv1:
* [[#make test|<code>$ make test</code>]]
* [[#make bench|<code>$ make bench</code>]]
* [[#make megabench|<code>$ make megabench</code>]]


[[#toc]]


== MSISDN - loadtesting ==
There are 2 staging environments for msisdn-gateway
* msisdn.stage.mozaws.net - for end-2-end testing (NOT for loadtesting!)
* msisdn-loadtest.stage.mozaws.net - for loadtesting
** This environment also contains its own mock server: http://omxen.dev.mozaws.net

see:

configuration file on the Stage server: /data/msisdn-gateway/config/production.json

[[#toc]]


= Test Execution =
== make test ==
=== Summary ===

Running <code>$ make test</code> allows you to kick off a loadtest locally against a remote endpoint, but then receive test output locally.

Optional: if you want to specify a test endpoint other than the default (see: Makefile), you can do it like so:
<pre>
$ make test SERVER_URL=https://loop.stage.mozaws.net
 - or -
$ make test SERVER_URL=https://msisdn.stage.mozaws.net
</pre>

WARNING:
Do not launch loadtests against any PRODUCTION server, ever!


'''Example Output'''
<pre>
$ make test SERVER_URL=https://loop.stage.mozaws.net
SIMPLE_PUSH_URL=https://call.stage.mozaws.net/ ./venv/bin/loads-runner --config=./config/test.ini --server-url=https://loop.stage.mozaws.net loadtest.TestLoop.test_http_calls
[                                                                                                                                                                                                                                 ]   0%/Users/rpappalardo/git/services-test/build/loop-server/loadtests/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
[=================================================================================================================================================================================================================================] 100%
Duration: 4.99 seconds
Hits: 17
Started: 2015-05-21 22:26:10.930852
Approximate Average RPS: 3
Average request time: 0.28s
Opened web sockets: 0
Bytes received via web sockets : 0

Success: 1
Errors: 0
Failures: 0


Slowest URL: https://loop.stage.mozaws.net/     Average Request Time: 1.121348

Stats by URLs:
- https://loop.stage.mozaws.net/                        Average request time: 1.121348  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/Dko0_Uvdks8       Average request time: 0.689918  Hits success rate: 1.0
- https://loop.stage.mozaws.net/rooms                   Average request time: 0.429905  Hits success rate: 1.0
- https://loop.stage.mozaws.net/rooms/C4lqDXc6W3M       Average request time: 0.189937375       Hits success rate: 1.0
- https://loop.stage.mozaws.net/registration            Average request time: 0.16524675        Hits success rate: 1.0
- https://loop.stage.mozaws.net/call-url                Average request time: 0.163699  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls?version=200       Average request time: 0.163517  Hits success rate: 1.0

Custom metrics:
- delete-room : 1
- register : 4
- generate-call-url : 1
- refresh-room-presence : 1
- initiate-call : 1
- leave-room : 3
- room-with-context : 1
- join-room : 3
- create-room : 1
- list-pending-calls : 1
- num-participants-3 : 1
SIMPLE_PUSH_URL=https://call.stage.mozaws.net/ ./venv/bin/loads-runner --config=./config/test.ini --server-url=https://loop.stage.mozaws.net loadtest.TestLoop.test_websockets
[                                                                                                                                                                                                                                 ]   0%/Users/rpappalardo/git/services-test/build/loop-server/loadtests/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
[                                                                                                                                                                                                                                 ]   0%e
[=================================================================================================================================================================================================================================] 100%
Duration: 81.80 seconds
Hits: 17
Started: 2015-05-21 22:26:16.133047
Approximate Average RPS: 0
Average request time: 0.24s
Opened web sockets: 7
Bytes received via web sockets : 1350

Success: 1
Errors: 0
Failures: 0


Slowest URL: https://loop.stage.mozaws.net/     Average Request Time: 0.713177

Stats by URLs:
- https://loop.stage.mozaws.net/                        Average request time: 0.713177  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/KR414mzrf-s       Average request time: 0.359527  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/bYhXDs86FTQ       Average request time: 0.358428  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/vqp9-BYyYBQ       Average request time: 0.357578  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/Nbi4t9_pLAc       Average request time: 0.356943  Hits success rate: 1.0
- https://loop.stage.mozaws.net/registration            Average request time: 0.1638505 Hits success rate: 1.0
- https://loop.stage.mozaws.net/call-url                Average request time: 0.16342975        Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls?version=200       Average request time: 0.16315475        Hits success rate: 1.0

Custom metrics:
- websocket-basic-callee-messages : 5
- websocket-supervisory-caller-messages : 2
- list-pending-calls : 4
- generate-call-url : 4
- initiate-call : 4
- websocket-ringing-callee-messages : 3
- websocket-basic-caller-messages : 5
- websocket-ringing-caller-messages : 3
- register : 4

</pre>


[[#toc]]


== make bench ==
* Use megabench instead


[[#toc]]


== make megabench ==

'''In Terminal'''
$ make megabench

<pre>
./venv/bin/loads-runner --config=./config/megabench.ini --user-id=bonzogonzo --server-url=https://loop.stage.mozaws.net:443 loadtest.TestLoop.test_all
[2015-05-21 16:00:21,868][11285] Detached. run --attach to reattach
</pre>


''''Open the following URL in browser:''''
https://loads.mozilla.services.com

* Login with credentials provided by Loop dev/QA team
* You will see real-time output from your test run in your browser

[[#toc]]


= Loadtest Monitoring = 
== Monitoring Loop Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard: 
 Agents statuses
 Launch a health check on all agents

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Stage_Environment


* OPs and Infra
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** http://loads.services.mozilla.com
** https://loads.readthedocs.org/en/latest
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues

* The QA Test Environments
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments


[[#toc]]



= Reference =
== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information: 
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

* Installation:
 git clone https://github.com/mozilla-services/loop-server.git
 cd loop-server
 Note: You may want to install a specific branch for testing vs defaulting to Master
 npm install
 ulimit -S -n 2048
 npm test *
 cd loadtests
 make build
 make test


[[#toc]]




= OLD Notes =
 
 Coverage report can be found here:
 /loop-server/coverage/lcov-report/index.html
 
 * This step requires the redis server to be installed and running:
 Mac:
 brew install redis
 redis-server /usr/local/etc/redis.conf
 
 Ubuntu Linux:
 sudo apt-get install redis-server
 sudo /usr/bin/redis-server /etc/redis/redis.conf
 sudo tail -f /var/log/redis/redis-server.log
 
 RHEL Linux:
 Install redis from here: http://download.redis.io/releases
 then
 /usr/local/bin/redis-server /home/ec2-user/redis-2.8.9/redis.conf
 or similar
 
* Note: 
** This will install a local copy of the Loads tool for use with the Loop-Server.


** To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details


[[#toc]]


== Using the Loads V1 Services Cluster for the Loop-Server in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
 $ make megabench SERVER_URL=https://loop.stage.mozaws.net

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the MSISDN-Gateway ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
 $ make megabench SERVER_URL=https://msisdn.stage.mozaws.net

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json 

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (Loop-Server and MSISDN-Gateway):
*** Number of hits
*** Number of concurrent users 

** For make bench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test 

** For make megabench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)

* Loop-Server load test code
** The Loop-Server load test can not currently be configured in the code

* MSISDN-Gateway load test code
** The MSISDN-Gateway load test can be configured in the code - see the following lines: 
** Test server URL: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L15
** Error percentages: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22

* General REFs: 
** https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations


== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com


== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for Loop-Server (per server)
** /var/log/circus.log
** /var/log/loop_err.log
** /var/log/loop_out.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /var/log/nginx/access.log
** /var/log/nginx/error.log 

* 2. Important logs for MSISDN-Gateway (per server)
** TBD

* Acceptable/Unacceptable Loop-Server errors:
 hekad loop.stderr.log
 The following are acceptable:
 Decoder 'LoopServer-LoopServerDecoder' error: Failed parsing
 Plugin 'AggregatorOutput' error: writing to heka.shared....
 
 nginx logs:
 The 200s (good stuff) and 101s (websockets) are acceptable. 
 The 499s are an artifact of the current load testing tool (V1).
 You should only see them at the end of the load test.
 Right now, we are getting a lot of 404s and 307s.
 They all appear to be caused by bots. There is a bug open about this.
 Any percentage of 405s, 502s, or 503s is not acceptable.
 
 Application logs:
 Right now, we are getting a lot of 404s and 307s.
 They all appear to be caused by bots. There is a bug open about this.
 See the loop_server.out.log file.
 
 /var/log/loop_err.log
 The following are acceptable: connect: res.on("header"): use on-headers module directly
 
 In the Loads Cluster dashboard, watch for the following errors/failures:
 string indices must be integers
 OR
 No JSON object could be decoded
 OR
 'hawk-session-token'

* Acceptable/Unacceptable MSISDN-Gateway errors:
 The updated load test does generate a certain percentage of errors:
 https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
 So, expect to see a predefined percentage of 204s and 400s, along with the usual 200s in the nginx access logs.
 The msisdn-gateway app logs should be clean with just msisdn and test data.