Loop/Test/Loadtesting
From MozillaWiki
Contents
Overview
Summary
Loadtesting Loop is currently performed with loadsv1 tool. Loadsv2 is under development (stay tuned for more info...)
Loadtesting should be performed against
- loop-server
- msisdn-gateway
It's not necessary to loadtest against loop-client
Guidelines for loadtesting with loadsv1 tool are provided below.
Loadsv1
Installing Loop-Server and the Loads tool (on Localhost or AWS)
- refer to github README
Installation
loop-server
$ git clone https://github.com/mozilla-services/loop-server $ cd loop-server/loadtests </pred> $ make install
msisdn-gateway
$ git clone https://github.com/mozilla-services/loop-server $ cd msisdn-gateway/loadtests </pred> $ make install
Pre-requisite
- edit loadtest config file
- enter desired values for current test
$ vim /loadtests/config/megabench.ini
Run Test There are 3 commands available for testing with loadsv1:
-
$ make megabench -
$ make test -
$ make bench(use megabench instead)
MSISDN - loadtesting
There are 2 staging environments for msisdn-gateway
- msisdn.stage.mozaws.net - for end-2-end testing (NOT for loadtesting!)
- msisdn-loadtest.stage.mozaws.net - for loadtesting
- This environment also contains its own mock server: http://omxen.dev.mozaws.net
see:
configuration file on the Stage server: /data/msisdn-gateway/config/production.json
Test Execution
Summary
Of the three tests available, make megabench will probably be the one you use most.
make test is largely for a quick smoke test and make bench doesn't do anything you can't do with the other two.
For information about test monitoring:
make megabench
In Terminal $ make megabench
./venv/bin/loads-runner --config=./config/megabench.ini --user-id=bonzogonzo --server-url=https://loop.stage.mozaws.net:443 loadtest.TestLoop.test_all [2015-05-21 16:00:21,868][11285] Detached. run --attach to reattach
'Open the following URL in browser:'
https://loads.mozilla.services.com
- Login with credentials provided by Loop dev/QA team
- You will see real-time output from your test run in your browser
make test
Description
Running $ make test allows you to kick off a loadtest locally against a remote endpoint, but then receive test output locally.
Optional: if you want to specify a test endpoint other than the default (see: Makefile), you can do it like so:
$ make test SERVER_URL=https://loop.stage.mozaws.net - or - $ make test SERVER_URL=https://msisdn.stage.mozaws.net
WARNING: Do not launch loadtests against any PRODUCTION server, ever!
Example Output
$ make test SERVER_URL=https://loop.stage.mozaws.net SIMPLE_PUSH_URL=https://call.stage.mozaws.net/ ./venv/bin/loads-runner --config=./config/test.ini --server-url=https://loop.stage.mozaws.net loadtest.TestLoop.test_http_calls [ ] 0%/Users/rpappalardo/git/services-test/build/loop-server/loadtests/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning [=================================================================================================================================================================================================================================] 100% Duration: 4.99 seconds Hits: 17 Started: 2015-05-21 22:26:10.930852 Approximate Average RPS: 3 Average request time: 0.28s Opened web sockets: 0 Bytes received via web sockets : 0 Success: 1 Errors: 0 Failures: 0 Slowest URL: https://loop.stage.mozaws.net/ Average Request Time: 1.121348 Stats by URLs: - https://loop.stage.mozaws.net/ Average request time: 1.121348 Hits success rate: 1.0 - https://loop.stage.mozaws.net/calls/Dko0_Uvdks8 Average request time: 0.689918 Hits success rate: 1.0 - https://loop.stage.mozaws.net/rooms Average request time: 0.429905 Hits success rate: 1.0 - https://loop.stage.mozaws.net/rooms/C4lqDXc6W3M Average request time: 0.189937375 Hits success rate: 1.0 - https://loop.stage.mozaws.net/registration Average request time: 0.16524675 Hits success rate: 1.0 - https://loop.stage.mozaws.net/call-url Average request time: 0.163699 Hits success rate: 1.0 - https://loop.stage.mozaws.net/calls?version=200 Average request time: 0.163517 Hits success rate: 1.0 Custom metrics: - delete-room : 1 - register : 4 - generate-call-url : 1 - refresh-room-presence : 1 - initiate-call : 1 - leave-room : 3 - room-with-context : 1 - join-room : 3 - create-room : 1 - list-pending-calls : 1 - num-participants-3 : 1 SIMPLE_PUSH_URL=https://call.stage.mozaws.net/ ./venv/bin/loads-runner --config=./config/test.ini --server-url=https://loop.stage.mozaws.net loadtest.TestLoop.test_websockets [ ] 0%/Users/rpappalardo/git/services-test/build/loop-server/loadtests/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning [ ] 0%e [=================================================================================================================================================================================================================================] 100% Duration: 81.80 seconds Hits: 17 Started: 2015-05-21 22:26:16.133047 Approximate Average RPS: 0 Average request time: 0.24s Opened web sockets: 7 Bytes received via web sockets : 1350 Success: 1 Errors: 0 Failures: 0 Slowest URL: https://loop.stage.mozaws.net/ Average Request Time: 0.713177 Stats by URLs: - https://loop.stage.mozaws.net/ Average request time: 0.713177 Hits success rate: 1.0 - https://loop.stage.mozaws.net/calls/KR414mzrf-s Average request time: 0.359527 Hits success rate: 1.0 - https://loop.stage.mozaws.net/calls/bYhXDs86FTQ Average request time: 0.358428 Hits success rate: 1.0 - https://loop.stage.mozaws.net/calls/vqp9-BYyYBQ Average request time: 0.357578 Hits success rate: 1.0 - https://loop.stage.mozaws.net/calls/Nbi4t9_pLAc Average request time: 0.356943 Hits success rate: 1.0 - https://loop.stage.mozaws.net/registration Average request time: 0.1638505 Hits success rate: 1.0 - https://loop.stage.mozaws.net/call-url Average request time: 0.16342975 Hits success rate: 1.0 - https://loop.stage.mozaws.net/calls?version=200 Average request time: 0.16315475 Hits success rate: 1.0 Custom metrics: - websocket-basic-callee-messages : 5 - websocket-supervisory-caller-messages : 2 - list-pending-calls : 4 - generate-call-url : 4 - initiate-call : 4 - websocket-ringing-callee-messages : 3 - websocket-basic-caller-messages : 5 - websocket-ringing-caller-messages : 3 - register : 4
Loadtest - Tweaking
loadtests can be configured in the code...
- Test server URL
- Error percentages
Reference
Monitoring
Ops / Infra
- https://github.com/mozilla-services/puppet-config/issues
- https://github.com/mozilla-services/svcops/issues
