TestEngineering/Services/TokenServerAndSyncLoadTesting

From MozillaWiki
Jump to navigation Jump to search

Load Test Tool Client/Host

Creating a RHEL AWS instance

  • Pick a Region then Create Instance > Launch Instance
  • Follow the prompts to create a basic, RHEL-flavored instance
  • Use of the QA/Dev key pairs that have been set up for this:
    • US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
    • US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
  • Once the instance is running, log in as "ec2-user"
  • The following apps, tools, and libs will need to be installed for use with various Services applications:
    • gcc, gcc-c++
    • hg
    • git
    • python-devel
    • automake, autoconf, and libtool (required for libzmq, for easy_install)
    • pip
    • virtualenv
    • node/npm
    • zeromq 3.X
    • gmp, gmp-devel
  • Also, general rhel updates:
$ sudo yum -y update
and/or
$ sudo yum -y upgrade
  • Now, the instance should be ready for installing and using the Loads tool.

Creating an Ubuntu AWS instance

  • Pick a Region then Create Instance > Launch Instance
  • Follow the prompts to create a basic, Ubuntu-flavored instance
  • Use of the QA/Dev key pairs that have been set up for this:
    • US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
    • US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
  • Once the instance is running, log in as "ubuntu"
  • The following apps, tools, and libs will need to be installed for use with various Services applications:
    • gcc, g++
    • mercurial
    • git
    • python-setuptools, python-virtualenv, and python-dev
    • automake, autoconf, libtool
    • m4
    • node/npm
    • libzmq and zeromq 3.X
    • gmp-5.1.3 or newer
  • Also, general rhel updates:
$ sudo apt-get update
and/or
$ sudo apt-get upgrade
  • Now, the instance should be ready for installing and using the Loads tool.

Installing the Loads tool on the AWS instance for TokenServer

  • Installation:
$ git clone https://github.com/mozilla-services/tokenserver
$ cd tokenserver
$ make build
$ make test
    Note: This is for local testing only
$ cd loadtest
$ make build
    Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com
  • Note: This will install a local copy of the Loads tool for use with TokenServer.

Running the Loads tool against TokenServer in Stage

  • Stage environment:
$ make test SERVER_URL=https://token-stage3.stage.mozaws.net
$ make bench SERVER_URL=https://token-stage3.stage.mozaws.net		
NOTE: The URL for Stage environment will most likely change on a frequent basis.
  • And while we are at it...
  • Dev environment:
$ make test SERVER_URL=https://token.dev.lcip.org
$ make bench SERVER_URL=https://token.dev.lcip.org
  • Production environment:
$ make test SERVER_URL=https://token.services.mozilla.com
$ make bench SERVER_URL=https://token.services.mozilla.com
  • NOTE: only one load test can be run at a time against Stage since we are making use of the Loads environment broker and agents. Allow a load test to finish before starting a new load test. See below for more information...
  • NOTE: Also, we can configure the load test (make bench) to run in detached mode with an appropriate observer. See below for more information...

Installing the Loads tool on the AWS instance for Sync 1.5

Installation:
$ git clone https://github.com/mozilla-services/server-syncstorage/
$ cd server-syncstorage
$ make build
$ make test
$ cd loadtest
$ make build
  • Note: This will install a local copy of the Loads tool for use with Sync 1.5.

Running the Loads tool against Sync 1.5 in Stage

  • Loads against specific Sync nodes in Stage
$ make test SERVER_URL=https://your.storagenode.here#SECRET
$ make bench SERVER_URL=https://your.storagenode.here#SECRET
Sync Stage nodes:
    https://sync-1-us-east-1.stage.mozaws.net
    https://sync-2-us-east-1.stage.mozaws.net
    https://sync-3-us-east-1.stage.mozaws.net
NOTE: The Stage sync nodes are likely to change frequently, so verify the URLs.
    See https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Sync_1.5_Stage_Environment
NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.
  • NOTE: only one load test can be run at a time against Stage since we are making use of the Loads environment broker and agents. Allow a load test to finish before starting a new load test. See below for more information...
  • NOTE: Also, we can configure the load test (make bench) to run in detached mode with an appropriate observer. See below for more information...

Running the Loads tool against TokenServer and Sync 1.5 in Stage

  • A combined loads test against TokenServer and Sync 1.5 in Stage
  • This is done via the server-syncstorage directory that was cloned and built above
$ cd server-syncstorage
$ cd loadtest
$ make test SERVER_URL=https://your.tokenserver.here
$ make bench SERVER_URL=https://your.tokenserver.here
Examples for Stage:
$ make test SERVER_URL=https://token-stage3.stage.mozaws.net
$ make bench SERVER_URL=https://token-stage3.stage.mozaws.net
See https://wiki.mozilla.org/QA/Services/FxATestEnvironments#TokenServer_Stage_Environment
  • And while we are at it...
Dev environment:
Examples:
$ make test SERVER_URL=https://token.dev.lcip.org
$ make bench SERVER_URL=https://token.dev.lcip.org
Prod environment:
Examples:
$ make test SERVER_URL=https://token.services.mozilla.com
$ make bench SERVER_URL=https://token.services.mozilla.com
See https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA.2C_TokenServer.2C_and_Sync_Production_Environments
  • And, finally, with the new "megabench" tool that makes use of the FxA Loads environment broker and agents...
$ make megabench SERVER_URL=https://token.dev.lcip.org
$ make megabench SERVER_URL=https://token-stage2.stage.mozaws.net
$ make megabench SERVER_URL=https://token.services.mozilla.com
  • NOTE: only one load test can be run at a time against Stage since we are making use of the Loads environment broker and agents. Allow a load test to finish before starting a new load test. See below for more information...
  • NOTE: Also, we can configure the load test (make bench) to run in detached mode with an appropriate observer. See below for more information...

Using the Loads Test environment

  • For the broker and agents - will probably have Dev create a temporary addition to the Make file and tools to allow for a "make megabench" that will run a "make bench" but use the Loads Test environment broker and agent set up.
  • Note: normally, we can only run one load test at a time with this environment, since it was not designed to share the broker and all the agents. If you attempt to run two loads tests with this environment, you will probably get a failure that looks something like this:
ExecutionError: Not enough agents running on that broker
  • We can, with the proper permissions, SSH into the environment and restart the broker and agent processes.

Using the Loads Services cluster

  • To get around the issues with using the loads tool on a local host
  • And as an alternative to Detached Mode and Observers
  • TBD

Test Coverage and Stats

  • Basic tweakable values for all load tests
    • users = number of concurrent users/agent
    • agents = number of agents out of the cluster, otherwise errors out
    • duration = in seconds
    • hits = 1 or X number of rounds/hits/iterations
  • TokenServer
    • File location: tokenserver/loadtest/loadtest.py
    • Inside NoteAssignmentTest, test_realistic is the main load test; the others are for specific behaviors
    • The test runs as following:
95% ask for assertions on existing users (on a DB filled by test_single_token_exchange)
4% ask for assertion on a new use
1% ask for a bad assertion
    • A bug has been filed to get the following additional coverage for the load test:
      • generation numbers in assertion
      • client state string
    • A bug has been filed to get some integration tests written:
      • to cover the edge/error cases not in the load test
      • to be pointed at a remote server
  • Sync
    • File location: server-syncstorage/loadtest/stress.py
    • This is the Sync 2.0 load test that has been back-ported for Sync 1.5.
    • The stress.py file is fully configurable for the following:
      • client probability
      • client distribution
      • collections
    • A bug has been filed to add support for load testing tabs
      • The tab collection it uses memcache; we need to figure out a way to test it without overloading the server
    • There are currently no constants to define how to select percentages per collection type
    • Right now, we need to manually configure the collections list in stress.py:
      • collections = ['bookmarks', 'forms', 'passwords', 'history', 'prefs']
      • Basically, you can add more entries of each type, since the load test (per user/again/hit/pass) picks randomly from the list for any given request...

Monitoring FxA Stage

Details on the Load Test tool

Known Issues

  • Because of the issues listed above, the Loads tool team is working on some fixes for better use of the tool in detached mode with a given observer that will allow us to send the test output to a specific email or a given IRC channel.
  • The team is also looking at performing all FxA loads testing through the Services cluster.

References

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/TokenServerAndSyncLoadTesting&oldid=925247"