Loop/Test/Loadtesting: Difference between revisions
Jump to navigation
Jump to search
Rpappalardo (talk | contribs) m (remove unnecessary content) |
Rpappalardo (talk | contribs) (monitoring section) |
||
| Line 209: | Line 209: | ||
= | = Monitoring Tools (STAGE) = | ||
== | |||
== Loadtest (loadv1) Test Dashboard == | |||
* http://loads.services.mozilla.com | |||
| Line 218: | Line 219: | ||
== Sentry: == | |||
* https://sentry.shared.us-east-1.stage.mozaws.net | * https://sentry.shared.us-east-1.stage.mozaws.net | ||
** Getting access to Sentry STAGE - talk to the OPs team | ** Getting access to Sentry STAGE - talk to the OPs team | ||
[[#toc]] | |||
== Stackdriver: == | |||
* https://app.stackdriver.com/ | * https://app.stackdriver.com/ | ||
* https://app.stackdriver.com/groups/11042/loop-server/stage-loop | * https://app.stackdriver.com/groups/11042/loop-server/stage-loop | ||
[[#toc]] | |||
== Graphite == | |||
* https://graphite.shared.us-east-1.stage.mozaws.net | * https://graphite.shared.us-east-1.stage.mozaws.net | ||
** Graphite > stats > loop-activated-users | ** Graphite > stats > loop-activated-users | ||
| Line 240: | Line 247: | ||
[[#toc]] | |||
== Kibana == | |||
* https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json | |||
* https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json | |||
[[#toc]] | |||
== Heka == | |||
* https://heka.shared.us-east-1.stage.mozaws.net/ | * https://heka.shared.us-east-1.stage.mozaws.net/ | ||
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health) | * (or https://heka.shared.us-east-1.stage.mozaws.net/#health) | ||
| Line 286: | Line 302: | ||
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf | *** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf | ||
=== | |||
[[#toc]] | |||
= Monitoring Tools (PRODUCTION) = | |||
== Sentry == | |||
* http://sentry.shared.us-west-2.prod.mozaws.net | * http://sentry.shared.us-west-2.prod.mozaws.net | ||
** Getting access to Sentry Prod - talk to the OPs team | ** Getting access to Sentry Prod - talk to the OPs team | ||
| Line 294: | Line 315: | ||
[[#top]] | |||
== Stackdriver == | |||
* https://app.stackdriver.com/groups/11044/loop-server/prod-loop | * https://app.stackdriver.com/groups/11044/loop-server/prod-loop | ||
[[#top]] | |||
== Kibana == | |||
** Main: https://kibana.shared.us-west-2.prod.mozaws.net | ** Main: https://kibana.shared.us-west-2.prod.mozaws.net | ||
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json | ** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json | ||
| Line 304: | Line 333: | ||
[[#top]] | |||
== Heka == | |||
* Main: https://heka.shared.us-west-2.prod.mozaws.net/# | * Main: https://heka.shared.us-west-2.prod.mozaws.net/# | ||
* https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers | * https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers | ||
| Line 311: | Line 346: | ||
[[#top]] | |||
== loop-server Dashboard == | |||
* https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard | * https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard | ||
[[#top]] | |||
== Graphite: == | |||
* https://graphite.shared.us-west-2.prod.mozaws.net (statsd data) | * https://graphite.shared.us-west-2.prod.mozaws.net (statsd data) | ||
[[#top]] | |||
== Firefox Input Dashboard: == | |||
* https://input.mozilla.org/en-US/?product=Loop | * https://input.mozilla.org/en-US/?product=Loop | ||
[[#top]] | |||
== Pingdom: == | |||
* loop-client: http://stats.pingdom.com/20dar76w4hmv/1304575 | * loop-client: http://stats.pingdom.com/20dar76w4hmv/1304575 | ||
* loop-server: http://stats.pingdom.com/20dar76w4hmv/1304565 | * loop-server: http://stats.pingdom.com/20dar76w4hmv/1304565 | ||
[[#top]] | [[#top]] | ||
= Log Monitoring = | |||
'''Tail multiple logs''' | |||
<pre> | |||
$ cd /media/ephemeral0/nginx/logs | |||
$ tail -f loop_server.access.log loop_server.error.log default.access.log default.error.log | |||
</pre> | |||
'''Other logs of interest:'''<br /> | |||
For example, for loop-server:<br /> | For example, for loop-server:<br /> | ||
* /var/log/circus.log | * /var/log/circus.log | ||
| Line 346: | Line 413: | ||
= HTTP Access Log - Parsing = | |||
'''unique REST activity with counts''' | '''unique REST activity with counts''' | ||
| Line 363: | Line 430: | ||
<pre> | <pre> | ||
$ cat loop_server.access.log | grep -v '200 ' | grep -v '101 ' | grep -v '499 ' | grep -v '201 ' | grep -v '204 ' | $ cat loop_server.access.log | grep -v '200 ' | grep -v '101 ' | grep -v '499 ' | grep -v '201 ' | grep -v '204 ' | ||
</pre> | </pre> | ||
Revision as of 04:37, 22 May 2015
Overview
Summary
Loadtesting Loop is currently performed with loadsv1 tool. Loadsv2 is under development (stay tuned for more info...)
Loadtesting should be performed against
- loop-server
- msisdn-gateway
It's not necessary to loadtest against loop-client
Guidelines for loadtesting with loadsv1 tool are provided below.
Loadsv1
Installing Loop-Server and the Loads tool (on Localhost or AWS)
- refer to github README
Installation
loop-server
$ git clone https://github.com/mozilla-services/loop-server $ cd loop-server/loadtests </pred> $ make install
msisdn-gateway
$ git clone https://github.com/mozilla-services/loop-server $ cd msisdn-gateway/loadtests </pred> $ make install
Pre-requisite
- edit loadtest config file
- enter desired values for current test
$ vim /loadtests/config/megabench.ini
Run Test There are 3 commands available for testing with loadsv1:
$ make megabench$ make test$ make bench(use megabench instead)
MSISDN - loadtesting
There are 2 staging environments for msisdn-gateway
- msisdn.stage.mozaws.net - for end-2-end testing (NOT for loadtesting!)
- msisdn-loadtest.stage.mozaws.net - for loadtesting
- This environment also contains its own mock server: http://omxen.dev.mozaws.net
see:
configuration file on the Stage server: /data/msisdn-gateway/config/production.json
Test Execution
Summary
Of the three tests available, make megabench will probably be the one you use most.
make test is largely for a quick smoke test and make bench doesn't do anything you can't do with the other two.
make megabench
In Terminal $ make megabench
./venv/bin/loads-runner --config=./config/megabench.ini --user-id=bonzogonzo --server-url=https://loop.stage.mozaws.net:443 loadtest.TestLoop.test_all [2015-05-21 16:00:21,868][11285] Detached. run --attach to reattach
'Open the following URL in browser:'
https://loads.mozilla.services.com
- Login with credentials provided by Loop dev/QA team
- You will see real-time output from your test run in your browser
make test
Description
Running $ make test allows you to kick off a loadtest locally against a remote endpoint, but then receive test output locally.
Optional: if you want to specify a test endpoint other than the default (see: Makefile), you can do it like so:
$ make test SERVER_URL=https://loop.stage.mozaws.net - or - $ make test SERVER_URL=https://msisdn.stage.mozaws.net
WARNING: Do not launch loadtests against any PRODUCTION server, ever!
Example Output
$ make test SERVER_URL=https://loop.stage.mozaws.net SIMPLE_PUSH_URL=https://call.stage.mozaws.net/ ./venv/bin/loads-runner --config=./config/test.ini --server-url=https://loop.stage.mozaws.net loadtest.TestLoop.test_http_calls [ ] 0%/Users/rpappalardo/git/services-test/build/loop-server/loadtests/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning [=================================================================================================================================================================================================================================] 100% Duration: 4.99 seconds Hits: 17 Started: 2015-05-21 22:26:10.930852 Approximate Average RPS: 3 Average request time: 0.28s Opened web sockets: 0 Bytes received via web sockets : 0 Success: 1 Errors: 0 Failures: 0 Slowest URL: https://loop.stage.mozaws.net/ Average Request Time: 1.121348 Stats by URLs: - https://loop.stage.mozaws.net/ Average request time: 1.121348 Hits success rate: 1.0 - https://loop.stage.mozaws.net/calls/Dko0_Uvdks8 Average request time: 0.689918 Hits success rate: 1.0 - https://loop.stage.mozaws.net/rooms Average request time: 0.429905 Hits success rate: 1.0 - https://loop.stage.mozaws.net/rooms/C4lqDXc6W3M Average request time: 0.189937375 Hits success rate: 1.0 - https://loop.stage.mozaws.net/registration Average request time: 0.16524675 Hits success rate: 1.0 - https://loop.stage.mozaws.net/call-url Average request time: 0.163699 Hits success rate: 1.0 - https://loop.stage.mozaws.net/calls?version=200 Average request time: 0.163517 Hits success rate: 1.0 Custom metrics: - delete-room : 1 - register : 4 - generate-call-url : 1 - refresh-room-presence : 1 - initiate-call : 1 - leave-room : 3 - room-with-context : 1 - join-room : 3 - create-room : 1 - list-pending-calls : 1 - num-participants-3 : 1 SIMPLE_PUSH_URL=https://call.stage.mozaws.net/ ./venv/bin/loads-runner --config=./config/test.ini --server-url=https://loop.stage.mozaws.net loadtest.TestLoop.test_websockets [ ] 0%/Users/rpappalardo/git/services-test/build/loop-server/loadtests/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning [ ] 0%e [=================================================================================================================================================================================================================================] 100% Duration: 81.80 seconds Hits: 17 Started: 2015-05-21 22:26:16.133047 Approximate Average RPS: 0 Average request time: 0.24s Opened web sockets: 7 Bytes received via web sockets : 1350 Success: 1 Errors: 0 Failures: 0 Slowest URL: https://loop.stage.mozaws.net/ Average Request Time: 0.713177 Stats by URLs: - https://loop.stage.mozaws.net/ Average request time: 0.713177 Hits success rate: 1.0 - https://loop.stage.mozaws.net/calls/KR414mzrf-s Average request time: 0.359527 Hits success rate: 1.0 - https://loop.stage.mozaws.net/calls/bYhXDs86FTQ Average request time: 0.358428 Hits success rate: 1.0 - https://loop.stage.mozaws.net/calls/vqp9-BYyYBQ Average request time: 0.357578 Hits success rate: 1.0 - https://loop.stage.mozaws.net/calls/Nbi4t9_pLAc Average request time: 0.356943 Hits success rate: 1.0 - https://loop.stage.mozaws.net/registration Average request time: 0.1638505 Hits success rate: 1.0 - https://loop.stage.mozaws.net/call-url Average request time: 0.16342975 Hits success rate: 1.0 - https://loop.stage.mozaws.net/calls?version=200 Average request time: 0.16315475 Hits success rate: 1.0 Custom metrics: - websocket-basic-callee-messages : 5 - websocket-supervisory-caller-messages : 2 - list-pending-calls : 4 - generate-call-url : 4 - initiate-call : 4 - websocket-ringing-callee-messages : 3 - websocket-basic-caller-messages : 5 - websocket-ringing-caller-messages : 3 - register : 4
Monitoring Tools (STAGE)
Loadtest (loadv1) Test Dashboard
Sentry:
- https://sentry.shared.us-east-1.stage.mozaws.net
- Getting access to Sentry STAGE - talk to the OPs team
Stackdriver:
Graphite
- https://graphite.shared.us-east-1.stage.mozaws.net
- Graphite > stats > loop-activated-users
- Graphite > stats > loop-call-urls
- User Graphs > not currently defined
Kibana
- https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
- https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
Heka
- https://heka.shared.us-east-1.stage.mozaws.net/
- (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
- PlugIns - Filters
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
- PlugIns - Outputs
- PlugIns - Encoders
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
- Sandboxes
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf
- PlugIns - Filters
Monitoring Tools (PRODUCTION)
Sentry
- http://sentry.shared.us-west-2.prod.mozaws.net
- Getting access to Sentry Prod - talk to the OPs team
- http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
- http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/
Stackdriver
Kibana
Heka
- Main: https://heka.shared.us-west-2.prod.mozaws.net/#
- https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
- https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
- https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus
loop-server Dashboard
Graphite:
- https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)
Firefox Input Dashboard:
Pingdom:
- loop-client: http://stats.pingdom.com/20dar76w4hmv/1304575
- loop-server: http://stats.pingdom.com/20dar76w4hmv/1304565
Log Monitoring
Tail multiple logs
$ cd /media/ephemeral0/nginx/logs $ tail -f loop_server.access.log loop_server.error.log default.access.log default.error.log
Other logs of interest:
For example, for loop-server:
- /var/log/circus.log
- /var/log/loop_err.log
- /var/log/loop_out.log
- /var/log/hekad/loop.stdout.log
- /var/log/hekad/loop.stderr.log
- /var/log/nginx/access.log
- /var/log/nginx/error.log
HTTP Access Log - Parsing
unique REST activity with counts
$ cat /media/ephemeral0/nginx/logs/loop_server.access.log | grep "HTTP/" | awk '{print $6" "$3" "}' | sort | uniq -c
unique REST activity with counts (from a .gz file)
$ zgrep -a "HTTP/" /media/ephemeral0/nginx/logs/loop_server.access.log-20150114.gz | awk '{print $6" "$3" "}' | sort | uniq -c
return only the interesting stuff
$ cat loop_server.access.log | grep -v '200 ' | grep -v '101 ' | grep -v '499 ' | grep -v '201 ' | grep -v '204 '
Loadtest - Tweaking
loadtests can be configured in the code...
- Test server URL
- Error percentages
Reference
Ops / Infra
- https://github.com/mozilla-services/puppet-config/issues
- https://github.com/mozilla-services/svcops/issues