TestEngineering/Services/LoopServerLoadTesting: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 261: Line 261:


== Known Bugs, Issues, and Tasks ==
== Known Bugs, Issues, and Tasks ==
** Loop Server: TBD
* Loop Server:
** https://github.com/mozilla-services/loop-server/pull/88
** https://github.com/mozilla-services/loop-server/pull/98
** https://github.com/mozilla-services/loop-server/pull/89
** https://github.com/mozilla-services/loop-server/pull/91
** https://github.com/mozilla-services/loop-server/pull/112


** Loop Client: TBD
** https://github.com/mozilla-services/docs/pull/14
** https://github.com/mozilla-services/docs (various)


** MSISDN Gateway: TBD
** https://bugzilla.mozilla.org/show_bug.cgi?id=985388
** https://bugzilla.mozilla.org/show_bug.cgi?id=998332
** https://bugzilla.mozilla.org/show_bug.cgi?id=985387
** https://bugzilla.mozilla.org/show_bug.cgi?id=999715
** https://bugzilla.mozilla.org/show_bug.cgi?id=1021855
** https://bugzilla.mozilla.org/show_bug.cgi?id=1022680
** https://bugzilla.mozilla.org/show_bug.cgi?id=1022688
** https://bugzilla.mozilla.org/show_bug.cgi?id=1022947
** https://bugzilla.mozilla.org/show_bug.cgi?id=1023436
** https://bugzilla.mozilla.org/show_bug.cgi?id=1023768
** https://bugzilla.mozilla.org/show_bug.cgi?id=1023518
*** https://github.com/mozilla-services/loop-server/pull/100
** https://bugzilla.mozilla.org/show_bug.cgi?id=1024408
** https://bugzilla.mozilla.org/show_bug.cgi?id=1024409
** https://bugzilla.mozilla.org/show_bug.cgi?id=1024490
** https://bugzilla.mozilla.org/show_bug.cgi?id=1024548
** https://bugzilla.mozilla.org/show_bug.cgi?id=1024920
** https://bugzilla.mozilla.org/show_bug.cgi?id=1024927
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025779
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025872
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025876
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025881
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025883
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025894
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019564
** https://bugzilla.mozilla.org/show_bug.cgi?id=1020596
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019878
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019564
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019504
** https://bugzilla.mozilla.org/show_bug.cgi?id=1021855
** https://bugzilla.mozilla.org/show_bug.cgi?id=1021726
*** https://github.com/mozilla-services/loop-server/pull/109
** https://bugzilla.mozilla.org/show_bug.cgi?id=1023217
*** https://github.com/mozilla-services/loop-server/pull/99
** https://bugzilla.mozilla.org/show_bug.cgi?id=1023792
** https://bugzilla.mozilla.org/show_bug.cgi?id=1015087
** https://bugzilla.mozilla.org/show_bug.cgi?id=1013382
** https://bugzilla.mozilla.org/show_bug.cgi?id=1015085
** https://bugzilla.mozilla.org/show_bug.cgi?id=1003170
** https://bugzilla.mozilla.org/show_bug.cgi?id=1003174
** https://bugzilla.mozilla.org/show_bug.cgi?id=1022678
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025895
** https://bugzilla.mozilla.org/show_bug.cgi?id=1024462
** https://bugzilla.mozilla.org/show_bug.cgi?id=1025792
** https://bugzilla.mozilla.org/show_bug.cgi?id=1019504


** Mock Server: TBD
* Loop Client: none at this time
 
* MSISDN Gateway:
** https://github.com/mozilla-services/msisdn-gateway/pull/48
** https://github.com/mozilla-services/msisdn-gateway/issues/40
** https://github.com/mozilla-services/msisdn-gateway/issues/72
** https://github.com/mozilla-services/msisdn-gateway/issues/83
 
* Mock Server: none at this time


* OPs and Infrastructure
* OPs and Infrastructure
** TBD
** https://github.com/mozilla-services/puppet-config/issues/598
** https://github.com/mozilla-services/puppet-config/issues/558
** https://github.com/mozilla-services/puppet-config/issues/563
** https://github.com/mozilla-services/svcops/pull/158
** https://github.com/mozilla-services/puppet-config/pull/575


* Loads Tool and Cluster
* Loads Tool and Cluster

Revision as of 23:49, 16 June 2014

Summary for Loop Server, Loop Client, Mock Server, MSISDN Gateway

Quick Verification Of Stage Deployments

  • This is a quick sanity test of the environment before getting started on load tests.
  • Loop Server
    • TBD
  • Loop Client
    • TBD
  • MSISDN Gateway
    • TBD
  • Mock Server
    • N/A

Load Test Tool Client/Host

Creating a RHEL AWS instance

  • Pick a Region then Create Instance > Launch Instance
  • Follow the prompts to create a basic, RHEL-flavored instance
  • Use of the QA/Dev key pairs that have been set up for this:
    • US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
    • US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
  • Once the instance is running, log in as "ec2-user"
  • The following apps, tools, and libs will need to be installed for use with various Services applications:
    • gcc, gcc-c++
    • hg
    • git
    • python-devel
    • automake, autoconf, and libtool (required for libzmq, for easy_install)
    • pip
    • virtualenv
    • node/npm
    • zeromq 3.X
    • gmp, gmp-devel
  • Also, general rhel updates:
$ sudo yum -y update
and/or
$ sudo yum -y upgrade
  • Now, the instance should be ready for installing and using the Loads tool.

Creating an Ubuntu AWS instance

  • Pick a Region then Create Instance > Launch Instance
  • Follow the prompts to create a basic, Ubuntu-flavored instance
  • Use of the QA/Dev key pairs that have been set up for this:
    • US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
    • US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
  • Once the instance is running, log in as "ubuntu"
  • The following apps, tools, and libs will need to be installed for use with various Services applications:
    • gcc, g++
    • mercurial
    • git
    • python-setuptools, python-virtualenv, and python-dev
    • automake, autoconf, libtool
    • m4
    • node/npm
    • libzmq and zeromq 3.X
    • gmp-5.1.3 or newer
  • Also, general rhel updates:
$ sudo apt-get update
and/or
$ sudo apt-get upgrade
  • Now, the instance should be ready for installing and using the Loads tool.

Installing Loop-Server and the Loads tool on the AWS instance

  • Installation:
TBD
  • Note: This will install a local copy of the Loads tool for use with the Loop-Server.

Running the load test against the Loop-Server in Stage

  • Stage environment:
$ make test
or
$ make test SERVER_URL=BLAH
$ make bench
or
$ make bench SERVER_URL=BLAH

Using the Loads Services Cluster for the Loop-Server in Stage

  • By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
  • Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
  • Stage environment:
$ make megabench SERVER_URL=BLAH
  • Dev environment: TBD
  • Production environment: TBD

Installing MSISDN-Gateway and the Loads tool on the AWS instance

  • Installation:
TBD
  • Note: This will install a local copy of the Loads tool for use with MSISDN-Gateway.

Running the load test against MSISDN-Gateway in Stage

  • Stage environment:
$ make test SERVER_URL=BLAH
$ make bench SERVER_URL=BLAH

Using the Loads Services Cluster for the MSISDN-Gateway

  • By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
  • Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
  • Stage environment:
$ make megabench SERVER_URL=BLAH
  • Dev environment: TBD
  • Production environment: TBD

Configuring The Load Tests

Test Coverage and Stats

  • Basic tweakable values for all load tests
    • users = number of concurrent users/agent
    • agents = number of agents out of the cluster, otherwise errors out
    • duration = in seconds
    • hits = 1 or X number of rounds/hits/iterations
  • Loop-Server
    • TBD
  • MSISDN-Gateway
    • TBD

Analyzing the Results

  • There are several methods and tools for analyzing the load test results.

Debugging the Issues

  • There are several methods and tools for debugging the load test errors and other issues.
  • 1. Important logs for Loop-Server (per server)
    • TBD
  • 2. Important logs for MSISDN-Gateway (per server)
    • TBD
  • Acceptable Loop-Server errors:
TBD
  • Acceptable MSISDN-Gateway errors:
TBD

Monitoring Loop Stage

./bin/loads-runner --ping-broker --ssh=ubuntu@loads.services.mozilla.com
./bin/loads-runner --check-cluster --ssh=ubuntu@loads.services.mozilla.com

/Token%20App%20Logs%20POC

Performance Testing Information

  • TBD

Details on the Load Test tool

Known Bugs, Issues, and Tasks

  • Loop Client: none at this time
  • Mock Server: none at this time

References

  • Repositories
  • Documentation
  • OPs pages for stats collection, logging, monitoring
    • TBD
Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/LoopServerLoadTesting&oldid=989465"