Loop/Test/Loadtesting: Difference between revisions

From MozillaWiki
< Loop‎ | Test
Jump to navigation Jump to search
m (remove unnecessary content)
(Adding dashboards)
Line 225: Line 225:


[[#toc]]
[[#toc]]
== Monitoring Tools / Dashboards ==
== Monitoring ==
=== STAGE Environment ===
'''Sentry:'''<br />
* https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry STAGE - talk to the OPs team
'''Stackdriver:'''<br />
* https://app.stackdriver.com/
* https://app.stackdriver.com/groups/11042/loop-server/stage-loop
'''Graphite:'''<br />
* https://graphite.shared.us-east-1.stage.mozaws.net
** Graphite > stats > loop-activated-users
** Graphite > stats > loop-call-urls
** User Graphs > not currently defined
'''Kibana:'''<br />
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD
'''Heka'''<br />
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf
=== PRODUCTION Environment ===
'''Sentry:'''<br />
* http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/
'''Stackdriver:'''<br />
* https://app.stackdriver.com/groups/11044/loop-server/prod-loop
'''Kibana:'''<br />
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw
'''Heka:'''<br />
* Main: https://heka.shared.us-west-2.prod.mozaws.net/#
* https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
* https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
* https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus
'''loop-server Dashboard'''<br />
* https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard
'''Graphite:'''<br />
* https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)
'''Firefox Input Dashboard:'''<br />
* https://input.mozilla.org/en-US/?product=Loop
'''Pingdom:'''<br />
* loop-client: http://stats.pingdom.com/20dar76w4hmv/1304575
* loop-server: http://stats.pingdom.com/20dar76w4hmv/1304565
[[#top]]




Revision as of 04:21, 22 May 2015

Overview

Summary

Loadtesting Loop is currently performed with loadsv1 tool. Loadsv2 is under development (stay tuned for more info...)

Loadtesting should be performed against

  • loop-server
  • msisdn-gateway

It's not necessary to loadtest against loop-client

Guidelines for loadtesting with loadsv1 tool are provided below.


Loadsv1

Installing Loop-Server and the Loads tool (on Localhost or AWS)

  • refer to github README


Installation

loop-server 

$ git clone https://github.com/mozilla-services/loop-server
$ cd loop-server/loadtests </pred>
$ make install

msisdn-gateway 

$ git clone https://github.com/mozilla-services/loop-server
$ cd msisdn-gateway/loadtests </pred>
$ make install


Pre-requisite

  1. edit loadtest config file
  2. enter desired values for current test
$ vim /loadtests/config/megabench.ini


Run Test There are 3 commands available for testing with loadsv1:


#toc


MSISDN - loadtesting

There are 2 staging environments for msisdn-gateway

  • msisdn.stage.mozaws.net - for end-2-end testing (NOT for loadtesting!)
  • msisdn-loadtest.stage.mozaws.net - for loadtesting

see:

configuration file on the Stage server: /data/msisdn-gateway/config/production.json

#toc


Test Execution

Summary

Of the three tests available, make megabench will probably be the one you use most. make test is largely for a quick smoke test and make bench doesn't do anything you can't do with the other two.


#toc


make megabench

In Terminal $ make megabench 

./venv/bin/loads-runner --config=./config/megabench.ini --user-id=bonzogonzo --server-url=https://loop.stage.mozaws.net:443 loadtest.TestLoop.test_all
[2015-05-21 16:00:21,868][11285] Detached. run --attach to reattach


'Open the following URL in browser:' https://loads.mozilla.services.com

  • Login with credentials provided by Loop dev/QA team
  • You will see real-time output from your test run in your browser

#toc


make test

Description

Running $ make test allows you to kick off a loadtest locally against a remote endpoint, but then receive test output locally.

Optional: if you want to specify a test endpoint other than the default (see: Makefile), you can do it like so: 

$ make test SERVER_URL=https://loop.stage.mozaws.net
 - or -
$ make test SERVER_URL=https://msisdn.stage.mozaws.net

WARNING: Do not launch loadtests against any PRODUCTION server, ever!


Example Output 

$ make test SERVER_URL=https://loop.stage.mozaws.net
SIMPLE_PUSH_URL=https://call.stage.mozaws.net/ ./venv/bin/loads-runner --config=./config/test.ini --server-url=https://loop.stage.mozaws.net loadtest.TestLoop.test_http_calls
[                                                                                                                                                                                                                                 ]   0%/Users/rpappalardo/git/services-test/build/loop-server/loadtests/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
[=================================================================================================================================================================================================================================] 100%
Duration: 4.99 seconds
Hits: 17
Started: 2015-05-21 22:26:10.930852
Approximate Average RPS: 3
Average request time: 0.28s
Opened web sockets: 0
Bytes received via web sockets : 0

Success: 1
Errors: 0
Failures: 0


Slowest URL: https://loop.stage.mozaws.net/     Average Request Time: 1.121348

Stats by URLs:
- https://loop.stage.mozaws.net/                        Average request time: 1.121348  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/Dko0_Uvdks8       Average request time: 0.689918  Hits success rate: 1.0
- https://loop.stage.mozaws.net/rooms                   Average request time: 0.429905  Hits success rate: 1.0
- https://loop.stage.mozaws.net/rooms/C4lqDXc6W3M       Average request time: 0.189937375       Hits success rate: 1.0
- https://loop.stage.mozaws.net/registration            Average request time: 0.16524675        Hits success rate: 1.0
- https://loop.stage.mozaws.net/call-url                Average request time: 0.163699  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls?version=200       Average request time: 0.163517  Hits success rate: 1.0

Custom metrics:
- delete-room : 1
- register : 4
- generate-call-url : 1
- refresh-room-presence : 1
- initiate-call : 1
- leave-room : 3
- room-with-context : 1
- join-room : 3
- create-room : 1
- list-pending-calls : 1
- num-participants-3 : 1
SIMPLE_PUSH_URL=https://call.stage.mozaws.net/ ./venv/bin/loads-runner --config=./config/test.ini --server-url=https://loop.stage.mozaws.net loadtest.TestLoop.test_websockets
[                                                                                                                                                                                                                                 ]   0%/Users/rpappalardo/git/services-test/build/loop-server/loadtests/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
[                                                                                                                                                                                                                                 ]   0%e
[=================================================================================================================================================================================================================================] 100%
Duration: 81.80 seconds
Hits: 17
Started: 2015-05-21 22:26:16.133047
Approximate Average RPS: 0
Average request time: 0.24s
Opened web sockets: 7
Bytes received via web sockets : 1350

Success: 1
Errors: 0
Failures: 0


Slowest URL: https://loop.stage.mozaws.net/     Average Request Time: 0.713177

Stats by URLs:
- https://loop.stage.mozaws.net/                        Average request time: 0.713177  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/KR414mzrf-s       Average request time: 0.359527  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/bYhXDs86FTQ       Average request time: 0.358428  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/vqp9-BYyYBQ       Average request time: 0.357578  Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls/Nbi4t9_pLAc       Average request time: 0.356943  Hits success rate: 1.0
- https://loop.stage.mozaws.net/registration            Average request time: 0.1638505 Hits success rate: 1.0
- https://loop.stage.mozaws.net/call-url                Average request time: 0.16342975        Hits success rate: 1.0
- https://loop.stage.mozaws.net/calls?version=200       Average request time: 0.16315475        Hits success rate: 1.0

Custom metrics:
- websocket-basic-callee-messages : 5
- websocket-supervisory-caller-messages : 2
- list-pending-calls : 4
- generate-call-url : 4
- initiate-call : 4
- websocket-ringing-callee-messages : 3
- websocket-basic-caller-messages : 5
- websocket-ringing-caller-messages : 3
- register : 4


#toc


make bench

  • Use megabench instead


#toc



Loadtest Monitoring

Monitoring Loop Stage


#toc


Monitoring Tools / Dashboards

Monitoring

STAGE Environment

Sentry:


Stackdriver:


Graphite:


Kibana:


Heka

PRODUCTION Environment

Sentry:


Stackdriver:


Kibana:


Heka:


loop-server Dashboard


Graphite:


Firefox Input Dashboard:


Pingdom:

#top


Log Monitoring

Important logs for monitoring.

For example, for loop-server:

  • /var/log/circus.log
  • /var/log/loop_err.log
  • /var/log/loop_out.log
  • /var/log/hekad/loop.stdout.log
  • /var/log/hekad/loop.stderr.log
  • /var/log/nginx/access.log
  • /var/log/nginx/error.log


#top


HTTP Access Log - Parsing

unique REST activity with counts 

$ cat /media/ephemeral0/nginx/logs/loop_server.access.log | grep "HTTP/" | awk '{print $6" "$3" "}' | sort | uniq -c


unique REST activity with counts (from a .gz file) 

$ zgrep -a "HTTP/"  /media/ephemeral0/nginx/logs/loop_server.access.log-20150114.gz |  awk '{print $6" "$3" "}' | sort | uniq -c


return only the interesting stuff 

$ cat loop_server.access.log | grep -v '200 ' | grep -v '101 ' | grep -v '499 ' | grep -v '201 ' | grep -v '204 '


Tail multiple logs 

$ cd /media/ephemeral0/nginx/logs
$ tail -f loop_server.access.log loop_server.error.log  default.access.log default.error.log


#toc


Loadtest - Tweaking

loadtests can be configured in the code...


#toc


Reference

Ops / Infra


Loads Tool and Cluster


#toc

Retrieved from "https://wiki.mozilla.org/index.php?title=Loop/Test/Loadtesting&oldid=1075916"