View source for Loop/Test/Loadtesting

= Overview =
== Summary ==
Loadtesting Loop is currently performed with loadsv1 tool.
Loadsv2 is under development (stay tuned for more info...)

Loadtesting should be performed against
* loop-server
* msisdn-gateway

It's not necessary to loadtest against loop-client

Guidelines for loadtesting with loadsv1 tool are provided below.


== Loadsv1 ==

'''Installing Loop-Server and the Loads tool (on Localhost or AWS)'''
* refer to github README


'''Installation'''

loop-server

<pre>
$ git clone https://github.com/mozilla-services/loop-server
$ cd loop-server/loadtests </pred>
$ make install
</pre>

msisdn-gateway

<pre>
$ git clone https://github.com/mozilla-services/loop-server
$ cd msisdn-gateway/loadtests </pred>
$ make install
</pre>


'''Pre-requisite'''
# edit loadtest config file
# enter desired values for current test

<pre>
$ vim /loadtests/config/megabench.ini
</pre>



'''Run Test'''
There are 3 commands available for testing with loadsv1:
* [[#make megabench|<code>$ make megabench</code>]]
* [[#make test|<code>$ make test</code>]]
* <code>$ make bench</code> (use megabench instead)



[[#toc]]


== MSISDN - loadtesting ==
There are 2 staging environments for msisdn-gateway
* msisdn.stage.mozaws.net - for end-2-end testing (NOT for loadtesting!)
* msisdn-loadtest.stage.mozaws.net - for loadtesting
** This environment also contains its own mock server: http://omxen.dev.mozaws.net

see:

configuration file on the Stage server: /data/msisdn-gateway/config/production.json

[[#toc]]


= Test Execution =
== Summary ==

Of the three tests available, <code>make megabench</code> will probably be the one you use most.
<code>make test</code> is largely for a quick smoke test and <code>make bench</code> doesn't do anything you can't do with the other two.


[[#toc]]


== make megabench ==


'''In Terminal'''
$ make megabench

<pre>
./venv/bin/loads-runner --config=./config/megabench.ini --user-id=bonzogonzo --server-url=https://loop.stage.mozaws.net:443 loadtest.TestLoop.test_all
[2015-05-21 16:00:21,868][11285] Detached. run --attach to reattach
</pre>


''''Open the following URL in browser:''''
https://loads.mozilla.services.com

* Login with credentials provided by Loop dev/QA team
* You will see real-time output from your test run in your browser

[[#toc]]


== make test ==
'''Description'''

Running <code>$ make test</code> allows you to kick off a loadtest locally against a remote endpoint, but then receive test output locally.

Optional: if you want to specify a test endpoint other than the default (see: Makefile), you can do it like so:
<pre>
$ make test SERVER_URL=https://loop.stage.mozaws.net
 - or -
$ make test SERVER_URL=https://msisdn.stage.mozaws.net
</pre>

WARNING:
Do not launch loadtests against any PRODUCTION server, ever!


'''Example Output'''
<pre>
$ make test SERVER_URL=https://loop.stage.mozaws.net
SIMPLE_PUSH_URL=https://call.stage.mozaws.net/ ./venv/bin/loads-runner --config=./config/test.ini --server-url=https://loop.stage.mozaws.net loadtest.TestLoop.test_http_calls
[                                                                                                                                                                                                                                 ]   0%/Users/rpappalardo/git/services-test/build/loop-server/loadtests/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
[=================================================================================================================================================================================================================================] 100%
Duration: 4.99 seconds
Hits: 17
Started: 2015-05-21 22:26:10.930852
Approximate Average RPS: 3
Average request time: 0.28s
Opened web sockets: 0
Bytes received via web sockets : 0

Success: 1
Errors: 0
Failures: 0


Slowest URL: https://loop.stage.mozaws.net/     Average Request Time: 1.121348

Stats by URLs:
- https://loop.stage.mozaws.net/                        Average request time: 1.121348  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/Dko0_Uvdks8       Average request time: 0.689918  Hits success rate: 1.0
- https://loop.stage.mozaws.net/rooms                   Average request time: 0.429905  Hits success rate: 1.0
- https://loop.stage.mozaws.net/rooms/C4lqDXc6W3M       Average request time: 0.189937375       Hits success rate: 1.0
- https://loop.stage.mozaws.net/registration            Average request time: 0.16524675        Hits success rate: 1.0
- https://loop.stage.mozaws.net/call-url                Average request time: 0.163699  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls?version=200       Average request time: 0.163517  Hits success rate: 1.0

Custom metrics:
- delete-room : 1
- register : 4
- generate-call-url : 1
- refresh-room-presence : 1
- initiate-call : 1
- leave-room : 3
- room-with-context : 1
- join-room : 3
- create-room : 1
- list-pending-calls : 1
- num-participants-3 : 1
SIMPLE_PUSH_URL=https://call.stage.mozaws.net/ ./venv/bin/loads-runner --config=./config/test.ini --server-url=https://loop.stage.mozaws.net loadtest.TestLoop.test_websockets
[                                                                                                                                                                                                                                 ]   0%/Users/rpappalardo/git/services-test/build/loop-server/loadtests/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
[                                                                                                                                                                                                                                 ]   0%e
[=================================================================================================================================================================================================================================] 100%
Duration: 81.80 seconds
Hits: 17
Started: 2015-05-21 22:26:16.133047
Approximate Average RPS: 0
Average request time: 0.24s
Opened web sockets: 7
Bytes received via web sockets : 1350

Success: 1
Errors: 0
Failures: 0


Slowest URL: https://loop.stage.mozaws.net/     Average Request Time: 0.713177

Stats by URLs:
- https://loop.stage.mozaws.net/                        Average request time: 0.713177  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/KR414mzrf-s       Average request time: 0.359527  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/bYhXDs86FTQ       Average request time: 0.358428  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/vqp9-BYyYBQ       Average request time: 0.357578  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/Nbi4t9_pLAc       Average request time: 0.356943  Hits success rate: 1.0
- https://loop.stage.mozaws.net/registration            Average request time: 0.1638505 Hits success rate: 1.0
- https://loop.stage.mozaws.net/call-url                Average request time: 0.16342975        Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls?version=200       Average request time: 0.16315475        Hits success rate: 1.0

Custom metrics:
- websocket-basic-callee-messages : 5
- websocket-supervisory-caller-messages : 2
- list-pending-calls : 4
- generate-call-url : 4
- initiate-call : 4
- websocket-ringing-callee-messages : 3
- websocket-basic-caller-messages : 5
- websocket-ringing-caller-messages : 3
- register : 4

</pre>


[[#toc]]


= Loadtest Monitoring = 
== Monitoring Loop Stage ==
* Use the loadsv1 dashboard:
** http://loads.services.mozilla.com


[[#toc]]



== Monitoring Tools / Dashboards ==
== Monitoring ==
=== STAGE Environment ===

'''Sentry:'''<br />
* https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry STAGE - talk to the OPs team


'''Stackdriver:'''<br />
* https://app.stackdriver.com/
* https://app.stackdriver.com/groups/11042/loop-server/stage-loop


'''Graphite:'''<br />
* https://graphite.shared.us-east-1.stage.mozaws.net
** Graphite > stats > loop-activated-users
** Graphite > stats > loop-call-urls
** User Graphs > not currently defined


'''Kibana:'''<br />
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD


'''Heka'''<br />
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

=== PRODUCTION Environment ===
'''Sentry:'''<br />
* http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/


'''Stackdriver:'''<br />
* https://app.stackdriver.com/groups/11044/loop-server/prod-loop


'''Kibana:'''<br />
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw


'''Heka:'''<br />
* Main: https://heka.shared.us-west-2.prod.mozaws.net/#
* https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
* https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
* https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus


'''loop-server Dashboard'''<br />
* https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard


'''Graphite:'''<br />
* https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)


'''Firefox Input Dashboard:'''<br />
* https://input.mozilla.org/en-US/?product=Loop


'''Pingdom:'''<br />
* loop-client: http://stats.pingdom.com/20dar76w4hmv/1304575
* loop-server: http://stats.pingdom.com/20dar76w4hmv/1304565

[[#top]]


== Log Monitoring ==
Important logs for monitoring.

For example, for loop-server:<br />
* /var/log/circus.log
* /var/log/loop_err.log
* /var/log/loop_out.log
* /var/log/hekad/loop.stdout.log
* /var/log/hekad/loop.stderr.log
* /var/log/nginx/access.log
* /var/log/nginx/error.log 


[[#top]]


== HTTP Access Log - Parsing ==

'''unique REST activity with counts'''
<pre>
$ cat /media/ephemeral0/nginx/logs/loop_server.access.log | grep "HTTP/" | awk '{print $6" "$3" "}' | sort | uniq -c
</pre>


'''unique REST activity with counts (from a .gz file)'''
<pre>
$ zgrep -a "HTTP/"  /media/ephemeral0/nginx/logs/loop_server.access.log-20150114.gz |  awk '{print $6" "$3" "}' | sort | uniq -c
</pre>


'''return only the interesting stuff'''
<pre>
$ cat loop_server.access.log | grep -v '200 ' | grep -v '101 ' | grep -v '499 ' | grep -v '201 ' | grep -v '204 '
</pre>


'''Tail multiple logs'''
<pre>
$ cd /media/ephemeral0/nginx/logs
$ tail -f loop_server.access.log loop_server.error.log  default.access.log default.error.log
</pre>


[[#toc]]


= Loadtest - Tweaking =
loadtests can be configured in the code...
* Test server URL
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L15
* Error percentages
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22


[[#toc]]


= Reference =


== Ops / Infra ==
* https://github.com/mozilla-services/puppet-config/issues
* https://github.com/mozilla-services/svcops/issues


== Loads Tool and Cluster ==
** http://loads.services.mozilla.com
** https://loads.readthedocs.org/en/latest
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues
* https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
* https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1


[[#toc]]