TestEngineering/Services/LoopServerLoadTesting: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(137 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== Summary for Loop Server, Loop Client, Mock Server, MSISDN Gateway ==
* Latest Results
** Link to loads cluster: https://loads.services.mozilla.com/
*** Note: this now requires login privileges and a password
** Snapshots from StackDriver - TBD
** Snapshots from Kibana - TBD
** Snapshots from Sentry - TBD
* Latest Deployments
** TBD
* In Progress
** Build out of Stage environments
** Ongoing testing of Loop releases
** Bug review and issue debug - there are a lot of issues to work on (see the long list near the bottom of the wiki)
* Bugs To Verify:
** https://bugzilla.mozilla.org/show_bug.cgi?id=984325
** https://bugzilla.mozilla.org/show_bug.cgi?id=984325
** https://bugzilla.mozilla.org/show_bug.cgi?id=987086
** https://bugzilla.mozilla.org/show_bug.cgi?id=999457
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019929
** https://bugzilla.mozilla.org/show_bug.cgi?id=1007451
** https://bugzilla.mozilla.org/show_bug.cgi?id=1014962
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019416
*** https://github.com/mozilla-services/loop-server/pull/87
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019504
*** https://github.com/mozilla-services/loop-server/pull/90
*** https://github.com/mozilla-services/loop-server/pull/92
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019878
** https://bugzilla.mozilla.org/show_bug.cgi?id=1021955
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019733
** https://bugzilla.mozilla.org/show_bug.cgi?id=1021884
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019053
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019733
** https://bugzilla.mozilla.org/show_bug.cgi?id=1021884
** https://bugzilla.mozilla.org/show_bug.cgi?id=1021119
** https://bugzilla.mozilla.org/show_bug.cgi?id=1022688
** https://bugzilla.mozilla.org/show_bug.cgi?id=1021855
** https://bugzilla.mozilla.org/show_bug.cgi?id=1023281
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019487
** https://bugzilla.mozilla.org/show_bug.cgi?id=1023296
** https://bugzilla.mozilla.org/show_bug.cgi?id=1020572
** https://bugzilla.mozilla.org/show_bug.cgi?id=1016886
** https://bugzilla.mozilla.org/show_bug.cgi?id=1024383
** https://bugzilla.mozilla.org/show_bug.cgi?id=1021726
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025869
** or check here: http://mzl.la/1lzHC88
* Planned
** Load Testing
** Scaling for production traffic after release of Loop on Fx SOME VERSION
** Focused testing on Loop Client for desktop
** Focused testing on Loop Client for FxOS
* Blockers
** none at this time
* Completed
** None
* Performance
** TBD


== Quick Verification Of Stage Deployments ==
<font size=6><span style="color:#FF0000">This wiki has moved!</span><br /><br />------> [[Loop/Test/Loadtesting]]</font>
* This is a quick sanity test of the environment before getting started on load tests.
* Loop Server
** TBD
 
* Loop Client
** TBD
 
* MSISDN Gateway
** TBD
 
* Mock Server
** N/A
 
== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following link for more information: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loads_Services_Cluster_Environment
 
=== Creating a RHEL AWS instance ===
* Pick a Region then Create Instance > Launch Instance
* Follow the prompts to create a basic, RHEL-flavored instance
* Use of the QA/Dev key pairs that have been set up for this:
** US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
** US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
* Once the instance is running, log in as "ec2-user"
 
* The following apps, tools, and libs will need to be installed for use with various Services applications:
** gcc, gcc-c++
** hg
** git
** python-devel
** automake, autoconf, and libtool  (required for libzmq, for easy_install)
** pip
** virtualenv
** node/npm
** zeromq 3.X
** gmp, gmp-devel
 
* Also, general rhel updates:
$ sudo yum -y update
and/or
$ sudo yum -y upgrade
 
* Now, the instance should be ready for installing and using the Loads tool.
 
=== Creating an Ubuntu AWS instance ===
* Pick a Region then Create Instance > Launch Instance
* Follow the prompts to create a basic, Ubuntu-flavored instance
* Use of the QA/Dev key pairs that have been set up for this:
** US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
** US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
* Once the instance is running, log in as "ubuntu"
 
* The following apps, tools, and libs will need to be installed for use with various Services applications:
** gcc, g++
** mercurial
** git
** python-setuptools, python-virtualenv, and python-dev
** automake, autoconf, libtool
** m4
** node/npm
** libzmq and zeromq 3.X
** gmp-5.1.3 or newer
 
* Also, general rhel updates:
$ sudo apt-get update
and/or
$ sudo apt-get upgrade
 
* Now, the instance should be ready for installing and using the Loads tool.
 
== Installing Loop-Server and the Loads tool on the AWS instance==
* Installation:
TBD
 
* Note: This will install a local copy of the Loads tool for use with the Loop-Server.
 
== Running the load test against the Loop-Server in Stage ==
* Stage environment:
$ make test
or
$ make test SERVER_URL=BLAH
$ make bench
or
$ make bench SERVER_URL=BLAH
 
* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/
 
* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json
 
* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
 
* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
 
== Using the Loads Services Cluster for the Loop-Server in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=BLAH
 
* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/
 
* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json
 
* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
 
* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loads_Services_Cluster_Environment
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest
 
== Installing MSISDN-Gateway and the Loads tool on the AWS instance ==
* Installation:
TBD
 
* Note: This will install a local copy of the Loads tool for use with MSISDN-Gateway.
 
== Running the load test against MSISDN-Gateway in Stage ==
* Stage environment:
$ make test SERVER_URL=BLAH
$ make bench SERVER_URL=BLAH
 
* This environment also contains its own mock server: TBD
 
== Using the Loads Services Cluster for the MSISDN-Gateway ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=BLAH
 
* This environment also contains its own mock server: TBD
 
* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loads_Services_Cluster_Environment
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest
 
== Configuring The Load Tests ==
* Loop-Server
** TBD
** REF: https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
 
* MSISDN-Gateway
** TBD
** REF: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py
 
== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations
 
* Loop-Server
** TBD
 
* MSISDN-Gateway
** TBD
 
== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.
 
* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.
** If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads
 
== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.
 
* 1. Important logs for Loop-Server (per server)
** TBD
 
* 2. Important logs for MSISDN-Gateway (per server)
** TBD
 
* Acceptable Loop-Server errors:
TBD
 
* Acceptable MSISDN-Gateway errors:
TBD
 
== Monitoring Loop Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check from any loadtest folder or loads install:
./bin/loads-runner --ping-broker --ssh=ubuntu@loads.services.mozilla.com
./bin/loads-runner --check-cluster --ssh=ubuntu@loads.services.mozilla.com
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster
 
* Monitoring TS/Verifier/Sync Stage:
** Stackdriver
*** TBD: https://app.stackdriver.com/
** Kibana
*** TBD: https://kibana.shared.us-east-1.stage.mozaws.net/
/Token%20App%20Logs%20POC
** Heka - TBD
*** https://heka.shared.us-east-1.stage.mozaws.net/
*** https://heka.shared.us-east-1.stage.mozaws.net/#health
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
 
== Performance Testing Information ==
* TBD
 
== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com
 
== Known Bugs, Issues, and Tasks ==
* Loop Server:
** https://github.com/mozilla-services/loop-server/pull/88
** https://github.com/mozilla-services/loop-server/pull/98
** https://github.com/mozilla-services/loop-server/pull/89
** https://github.com/mozilla-services/loop-server/pull/91
** https://github.com/mozilla-services/loop-server/pull/112
 
** https://github.com/mozilla-services/docs/pull/14
** https://github.com/mozilla-services/docs (various)
 
** https://bugzilla.mozilla.org/show_bug.cgi?id=985388
** https://bugzilla.mozilla.org/show_bug.cgi?id=998332
** https://bugzilla.mozilla.org/show_bug.cgi?id=985387
** https://bugzilla.mozilla.org/show_bug.cgi?id=999715
** https://bugzilla.mozilla.org/show_bug.cgi?id=1021855
** https://bugzilla.mozilla.org/show_bug.cgi?id=1022680
** https://bugzilla.mozilla.org/show_bug.cgi?id=1022688
** https://bugzilla.mozilla.org/show_bug.cgi?id=1022947
** https://bugzilla.mozilla.org/show_bug.cgi?id=1023436
** https://bugzilla.mozilla.org/show_bug.cgi?id=1023768
** https://bugzilla.mozilla.org/show_bug.cgi?id=1023518
*** https://github.com/mozilla-services/loop-server/pull/100
** https://bugzilla.mozilla.org/show_bug.cgi?id=1024408
** https://bugzilla.mozilla.org/show_bug.cgi?id=1024409
** https://bugzilla.mozilla.org/show_bug.cgi?id=1024490
** https://bugzilla.mozilla.org/show_bug.cgi?id=1024548
** https://bugzilla.mozilla.org/show_bug.cgi?id=1024920
** https://bugzilla.mozilla.org/show_bug.cgi?id=1024927
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025779
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025872
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025876
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025881
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025883
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025894
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019564
** https://bugzilla.mozilla.org/show_bug.cgi?id=1020596
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019878
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019564
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019504
** https://bugzilla.mozilla.org/show_bug.cgi?id=1021855
** https://bugzilla.mozilla.org/show_bug.cgi?id=1021726
*** https://github.com/mozilla-services/loop-server/pull/109
** https://bugzilla.mozilla.org/show_bug.cgi?id=1023217
*** https://github.com/mozilla-services/loop-server/pull/99
** https://bugzilla.mozilla.org/show_bug.cgi?id=1023792
** https://bugzilla.mozilla.org/show_bug.cgi?id=1015087
** https://bugzilla.mozilla.org/show_bug.cgi?id=1013382
** https://bugzilla.mozilla.org/show_bug.cgi?id=1015085
** https://bugzilla.mozilla.org/show_bug.cgi?id=1003170
** https://bugzilla.mozilla.org/show_bug.cgi?id=1003174
** https://bugzilla.mozilla.org/show_bug.cgi?id=1022678
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025895
** https://bugzilla.mozilla.org/show_bug.cgi?id=1024462
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025792
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019504
 
* Loop Client: none at this time
 
* MSISDN Gateway:
** https://github.com/mozilla-services/msisdn-gateway/pull/48
** https://github.com/mozilla-services/msisdn-gateway/issues/40
** https://github.com/mozilla-services/msisdn-gateway/issues/72
** https://github.com/mozilla-services/msisdn-gateway/issues/83
 
* Mock Server: none at this time
 
* Or check here: http://mzl.la/1lzHC88
 
* OPs and Infrastructure
** https://github.com/mozilla-services/puppet-config/issues/598
** https://github.com/mozilla-services/puppet-config/issues/558
** https://github.com/mozilla-services/puppet-config/issues/563
** https://github.com/mozilla-services/svcops/pull/158
** https://github.com/mozilla-services/puppet-config/pull/575
 
* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues/222
** https://github.com/mozilla-services/loads/issues/234
** https://github.com/mozilla-services/loads/issues/235
** https://github.com/mozilla-services/loads/issues/251
** https://github.com/mozilla-services/loads/issues/257
** https://github.com/mozilla-services/loads/issues/259
** https://github.com/mozilla-services/loads/issues/265
** https://github.com/mozilla-services/loads/issues/266
** https://github.com/mozilla-services/loads-web/issues/24
 
== References ==
* Repositories
 
* Documentation
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP
 
* The QA Test Environments:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
 
* OPs pages for stats collection, logging, monitoring
** TBD

Latest revision as of 20:04, 26 August 2016

This wiki has moved!

------> Loop/Test/Loadtesting

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/LoopServerLoadTesting&oldid=1145591"