Loop/Test/Loadtesting: Difference between revisions

From MozillaWiki
< Loop‎ | Test
Jump to navigation Jump to search
(initial page)
 
m (formatting)
 
(12 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{Draft}}
{{LastUpdated}}
= Overview =
= Overview =
== Summary ==
== Summary ==
Loadtesting Loop is currently performed with loadsv1 tool.
Loadtesting Loop is currently performed with loadsv1 tool.
Loadsv2 is under development.
Loadsv2 is under development (stay tuned for more info...)


Loadtesting should be performed against
Loadtesting should be performed against
Line 54: Line 50:
'''Run Test'''
'''Run Test'''
There are 3 commands available for testing with loadsv1:
There are 3 commands available for testing with loadsv1:
* [[#make megabench|<code>$ make megabench</code>]]
* [[#make test|<code>$ make test</code>]]
* [[#make test|<code>$ make test</code>]]
* [[#make bench|<code>$ make bench</code>]]
* <code>$ make bench</code> (use megabench instead)
* [[#make megabench|<code>$ make megabench</code>]]
 




Line 76: Line 73:


= Test Execution =
= Test Execution =
== Summary ==
Of the three tests available, <code>make megabench</code> will probably be the one you use most.
<code>make test</code> is largely for a quick smoke test and <code>make bench</code> doesn't do anything you can't do with the other two.
For information about test monitoring:
[[Loop/Test/Monitoring]]
[[#toc]]
== make megabench ==
'''In Terminal'''
$ make megabench
<pre>
./venv/bin/loads-runner --config=./config/megabench.ini --user-id=bonzogonzo --server-url=https://loop.stage.mozaws.net:443 loadtest.TestLoop.test_all
[2015-05-21 16:00:21,868][11285] Detached. run --attach to reattach
</pre>
''''Open the following URL in browser:''''
https://loads.mozilla.services.com
* Login with credentials provided by Loop dev/QA team
* You will see real-time output from your test run in your browser
[[#toc]]
== make test ==
== make test ==
=== Summary ===
'''Description'''


Running <code>$ make test</code> allows you to kick off a loadtest locally against a remote endpoint, but then receive test output locally.
Running <code>$ make test</code> allows you to kick off a loadtest locally against a remote endpoint, but then receive test output locally.
Line 182: Line 213:




== make bench ==
= Loadtest - Tweaking =
* Use megabench instead
loadtests can be configured in the code...
 
* Test server URL
 
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L15
[[#toc]]
* Error percentages
 
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
 
== make megabench ==
 
'''In Terminal'''
$ make megabench
 
<pre>
./venv/bin/loads-runner --config=./config/megabench.ini --user-id=bonzogonzo --server-url=https://loop.stage.mozaws.net:443 loadtest.TestLoop.test_all
[2015-05-21 16:00:21,868][11285] Detached. run --attach to reattach
</pre>
 
 
''''Open the following URL in browser:''''
https://loads.mozilla.services.com
 
* Login with credentials provided by Loop dev/QA team
* You will see real-time output from your test run in your browser
 
[[#toc]]
 
 
= Loadtest Monitoring =
== Monitoring Loop Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
 
* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Stage_Environment
 
 
* OPs and Infra
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues
 
* Loads Tool and Cluster
** http://loads.services.mozilla.com
** https://loads.readthedocs.org/en/latest
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues
 
* The QA Test Environments
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments




[[#toc]]
[[#toc]]




= Reference =
= Reference =
== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
* Installation:
git clone https://github.com/mozilla-services/loop-server.git
cd loop-server
Note: You may want to install a specific branch for testing vs defaulting to Master
npm install
ulimit -S -n 2048
npm test *
cd loadtests
make build
make test
[[#toc]]


== Monitoring ==
* [[Loop/Test/Monitoring]]


= OLD Notes =
Coverage report can be found here:
/loop-server/coverage/lcov-report/index.html
* This step requires the redis server to be installed and running:
Mac:
brew install redis
redis-server /usr/local/etc/redis.conf
Ubuntu Linux:
sudo apt-get install redis-server
sudo /usr/bin/redis-server /etc/redis/redis.conf
sudo tail -f /var/log/redis/redis-server.log
RHEL Linux:
Install redis from here: http://download.redis.io/releases
then
/usr/local/bin/redis-server /home/ec2-user/redis-2.8.9/redis.conf
or similar
* Note:
** This will install a local copy of the Loads tool for use with the Loop-Server.


== Ops / Infra ==
* https://github.com/mozilla-services/puppet-config/issues
* https://github.com/mozilla-services/svcops/issues


** To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.


* REFs:
== Loads Tool and Cluster ==
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
* http://loads.services.mozilla.com
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
* https://loads.readthedocs.org/
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
* https://github.com/mozilla-services/loads/issues
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
* https://github.com/mozilla-services/loads-web/issues
* https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
* https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1




[[#toc]]
[[#toc]]
== Using the Loads V1 Services Cluster for the Loop-Server in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://loop.stage.mozaws.net
* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/
* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json
* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details
== Using the Loads V1 Services Cluster for the MSISDN-Gateway ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://msisdn.stage.mozaws.net
* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json
* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest
== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.
* Config files
** For make test (Loop-Server and MSISDN-Gateway):
*** Number of hits
*** Number of concurrent users
** For make bench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test
** For make megabench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)
* Loop-Server load test code
** The Loop-Server load test can not currently be configured in the code
* MSISDN-Gateway load test code
** The MSISDN-Gateway load test can be configured in the code - see the following lines:
** Test server URL: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L15
** Error percentages: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
* General REFs:
** https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py
== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations
== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.
* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.
* 1. Important logs for Loop-Server (per server)
** /var/log/circus.log
** /var/log/loop_err.log
** /var/log/loop_out.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /var/log/nginx/access.log
** /var/log/nginx/error.log
* 2. Important logs for MSISDN-Gateway (per server)
** TBD
* Acceptable/Unacceptable Loop-Server errors:
hekad loop.stderr.log
The following are acceptable:
Decoder 'LoopServer-LoopServerDecoder' error: Failed parsing
Plugin 'AggregatorOutput' error: writing to heka.shared....
nginx logs:
The 200s (good stuff) and 101s (websockets) are acceptable.
The 499s are an artifact of the current load testing tool (V1).
You should only see them at the end of the load test.
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
Any percentage of 405s, 502s, or 503s is not acceptable.
Application logs:
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
See the loop_server.out.log file.
/var/log/loop_err.log
The following are acceptable: connect: res.on("header"): use on-headers module directly
In the Loads Cluster dashboard, watch for the following errors/failures:
string indices must be integers
OR
No JSON object could be decoded
OR
'hawk-session-token'
* Acceptable/Unacceptable MSISDN-Gateway errors:
The updated load test does generate a certain percentage of errors:
https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
So, expect to see a predefined percentage of 204s and 400s, along with the usual 200s in the nginx access logs.
The msisdn-gateway app logs should be clean with just msisdn and test data.

Latest revision as of 16:15, 22 May 2015

Overview

Summary

Loadtesting Loop is currently performed with loadsv1 tool. Loadsv2 is under development (stay tuned for more info...)

Loadtesting should be performed against

  • loop-server
  • msisdn-gateway

It's not necessary to loadtest against loop-client

Guidelines for loadtesting with loadsv1 tool are provided below.


Loadsv1

Installing Loop-Server and the Loads tool (on Localhost or AWS)

  • refer to github README


Installation

loop-server 

$ git clone https://github.com/mozilla-services/loop-server
$ cd loop-server/loadtests </pred>
$ make install

msisdn-gateway 

$ git clone https://github.com/mozilla-services/loop-server
$ cd msisdn-gateway/loadtests </pred>
$ make install


Pre-requisite

  1. edit loadtest config file
  2. enter desired values for current test
$ vim /loadtests/config/megabench.ini


Run Test There are 3 commands available for testing with loadsv1:


#toc


MSISDN - loadtesting

There are 2 staging environments for msisdn-gateway

  • msisdn.stage.mozaws.net - for end-2-end testing (NOT for loadtesting!)
  • msisdn-loadtest.stage.mozaws.net - for loadtesting

see:

configuration file on the Stage server: /data/msisdn-gateway/config/production.json

#toc


Test Execution

Summary

Of the three tests available, make megabench will probably be the one you use most. make test is largely for a quick smoke test and make bench doesn't do anything you can't do with the other two.

For information about test monitoring:

Loop/Test/Monitoring


#toc


make megabench

In Terminal $ make megabench 

./venv/bin/loads-runner --config=./config/megabench.ini --user-id=bonzogonzo --server-url=https://loop.stage.mozaws.net:443 loadtest.TestLoop.test_all
[2015-05-21 16:00:21,868][11285] Detached. run --attach to reattach


'Open the following URL in browser:' https://loads.mozilla.services.com

  • Login with credentials provided by Loop dev/QA team
  • You will see real-time output from your test run in your browser

#toc


make test

Description

Running $ make test allows you to kick off a loadtest locally against a remote endpoint, but then receive test output locally.

Optional: if you want to specify a test endpoint other than the default (see: Makefile), you can do it like so: 

$ make test SERVER_URL=https://loop.stage.mozaws.net
 - or -
$ make test SERVER_URL=https://msisdn.stage.mozaws.net

WARNING: Do not launch loadtests against any PRODUCTION server, ever!


Example Output 

$ make test SERVER_URL=https://loop.stage.mozaws.net
SIMPLE_PUSH_URL=https://call.stage.mozaws.net/ ./venv/bin/loads-runner --config=./config/test.ini --server-url=https://loop.stage.mozaws.net loadtest.TestLoop.test_http_calls
[                                                                                                                                                                                                                                 ]   0%/Users/rpappalardo/git/services-test/build/loop-server/loadtests/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
[=================================================================================================================================================================================================================================] 100%
Duration: 4.99 seconds
Hits: 17
Started: 2015-05-21 22:26:10.930852
Approximate Average RPS: 3
Average request time: 0.28s
Opened web sockets: 0
Bytes received via web sockets : 0

Success: 1
Errors: 0
Failures: 0


Slowest URL: https://loop.stage.mozaws.net/     Average Request Time: 1.121348

Stats by URLs:
- https://loop.stage.mozaws.net/                        Average request time: 1.121348  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/Dko0_Uvdks8       Average request time: 0.689918  Hits success rate: 1.0
- https://loop.stage.mozaws.net/rooms                   Average request time: 0.429905  Hits success rate: 1.0
- https://loop.stage.mozaws.net/rooms/C4lqDXc6W3M       Average request time: 0.189937375       Hits success rate: 1.0
- https://loop.stage.mozaws.net/registration            Average request time: 0.16524675        Hits success rate: 1.0
- https://loop.stage.mozaws.net/call-url                Average request time: 0.163699  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls?version=200       Average request time: 0.163517  Hits success rate: 1.0

Custom metrics:
- delete-room : 1
- register : 4
- generate-call-url : 1
- refresh-room-presence : 1
- initiate-call : 1
- leave-room : 3
- room-with-context : 1
- join-room : 3
- create-room : 1
- list-pending-calls : 1
- num-participants-3 : 1
SIMPLE_PUSH_URL=https://call.stage.mozaws.net/ ./venv/bin/loads-runner --config=./config/test.ini --server-url=https://loop.stage.mozaws.net loadtest.TestLoop.test_websockets
[                                                                                                                                                                                                                                 ]   0%/Users/rpappalardo/git/services-test/build/loop-server/loadtests/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
[                                                                                                                                                                                                                                 ]   0%e
[=================================================================================================================================================================================================================================] 100%
Duration: 81.80 seconds
Hits: 17
Started: 2015-05-21 22:26:16.133047
Approximate Average RPS: 0
Average request time: 0.24s
Opened web sockets: 7
Bytes received via web sockets : 1350

Success: 1
Errors: 0
Failures: 0


Slowest URL: https://loop.stage.mozaws.net/     Average Request Time: 0.713177

Stats by URLs:
- https://loop.stage.mozaws.net/                        Average request time: 0.713177  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/KR414mzrf-s       Average request time: 0.359527  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/bYhXDs86FTQ       Average request time: 0.358428  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/vqp9-BYyYBQ       Average request time: 0.357578  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/Nbi4t9_pLAc       Average request time: 0.356943  Hits success rate: 1.0
- https://loop.stage.mozaws.net/registration            Average request time: 0.1638505 Hits success rate: 1.0
- https://loop.stage.mozaws.net/call-url                Average request time: 0.16342975        Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls?version=200       Average request time: 0.16315475        Hits success rate: 1.0

Custom metrics:
- websocket-basic-callee-messages : 5
- websocket-supervisory-caller-messages : 2
- list-pending-calls : 4
- generate-call-url : 4
- initiate-call : 4
- websocket-ringing-callee-messages : 3
- websocket-basic-caller-messages : 5
- websocket-ringing-caller-messages : 3
- register : 4


#toc


Loadtest - Tweaking

loadtests can be configured in the code...


#toc


Reference

Monitoring


Ops / Infra


Loads Tool and Cluster


#toc

Retrieved from "https://wiki.mozilla.org/index.php?title=Loop/Test/Loadtesting&oldid=1076037"