TestEngineering/Services/FxALoadTesting

From MozillaWiki
Jump to navigation Jump to search

Load Test Tool Client/Host

Creating a RHEL AWS instance

  • Pick a Region then Create Instance > Launch Instance
  • Follow the prompts to create a basic, RHEL-flavored instance
  • Use of the QA/Dev key pairs that have been set up for this:
    • US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
    • US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
  • Once the instance is running, log in as "ec2-user"
  • The following apps, tools, and libs will need to be installed for use with various Services applications:
    • gcc, gcc-c++
    • hg
    • git
    • python-devel
    • automake, autoconf, and libtool (required for libzmq, for easy_install)
    • pip
    • virtualenv
    • node/npm
    • zeromq 3.X
    • gmp, gmp-devel
  • Also, general rhel updates:
$ sudo yum -y update
and/or
$ sudo yum -y upgrade
  • Now, the instance should be ready for installing and using the Loads tool.

Creating an Ubuntu AWS instance

  • Pick a Region then Create Instance > Launch Instance
  • Follow the prompts to create a basic, Ubuntu-flavored instance
  • Use of the QA/Dev key pairs that have been set up for this:
    • US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
    • US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
  • Once the instance is running, log in as "ubuntu"
  • The following apps, tools, and libs will need to be installed for use with various Services applications:
    • gcc, g++
    • mercurial
    • git
    • python-setuptools, python-virtualenv, and python-dev
    • automake, autoconf, libtool
    • m4
    • node/npm
    • libzmq and zeromq 3.X
    • gmp-5.1.3 or newer
  • Also, general rhel updates:
$ sudo apt-get update
and/or
$ sudo apt-get upgrade
  • Now, the instance should be ready for installing and using the Loads tool.

Installing the Loads tool on the AWS instance via FxA-Auth-Server

Installation:
$ git clone https://github.com/mozilla/fxa-auth-server.git
$ cd ./fxa-auth-server
$ npm install
$ npm test
$ cd ./loadtest
$ make build
  • Note: This will install a local copy of the Loads tool for use with FxA-Auth-Server.

Running the Loads tool against FxA Stage

  • Note: the following value can stay fixed in the file loadsrunner.sh: BROKER=broker.loads.lcip.org. This will allow the current loads tool install to make use of the broker and agents in the Loads environment.
  • The basic load test can be run as follows
$ make test SERVER_URL=https://api-accounts.stage.mozaws.net
  • Note: With an install of fxa-auth-server, the load test tool is installed in the following location:
    • ../fxa-auth-server/loadtest/bin
  • The full, default load test can be run as follows
$ make bench SERVER_URL=https://api-accounts.stage.mozaws.net
  • Configuring the load test - makefile:
    • The default "make bench" command looks like this:
./loadsrunner.sh --users=20 --duration=1800 --agents=5 --include-file=./loadtests.py --python-dep=hawkauthlib loadtests.LoadTest.test_auth_server

You can change the following:
    users = number of concurrent users
    duration = length of the load test
    agents = the number of agents the broker will start up/use for the load test

Start with the defaults, then tweak the duration. Users and Agents are optional tweaks/changes.
  • Configuring the load test - loadsrunner.sh:
    • Note: the defaults should not be changed unless you need a very customized load test
Note the following ENV:
    BROKER=broker.loads.lcip.org
This allows us to use the broker already defined in the loads environment.

Note the key loads-runner line:
    ./bin/loads-runner --broker=tcp://localhost:7780 --zmq-publisher=tcp://localhost:7776 $@
This information is useful if you will be running the load test in detached mode.
  • NOTE: only one load test can be run at a time against FxA Stage since we are making use of the Loads environment broker and agents. Allow a load test to finish before starting a new load test. See below for more information...
  • NOTE: Also, we can configure the load test (make bench) to run in detached mode with an appropriate observer. See below for more information...

Running the Loads tool against FxA Development or Production

  • This can be done if we are comparing Stage vs. some other environment and have access to the AWS logs in Dev or Production:
  • Dev:
$ make test SERVER_URL=https://accounts.dev.lcip.org
$ make bench SERVER_URL=https://accounts.dev.lcip.org
  • Prod:
$ make test SERVER_URL=https://api.accounts.firefox.com
$ make bench SERVER_URL=https://api.accounts.firefox.com

Using the Loads Services cluster

  • To get around the issues with using the loads tool on a local host
  • And as an alternative to Detached Mode and Observers
  • TBD

Test Coverage and Stats

  • Basic tweakable values for all load tests
    • users = number of concurrent users/agent
    • agents = number of agents out of the cluster, otherwise errors out
    • duration = in seconds
    • hits = 1 or X number of rounds/hits/iterations
  • Location fxa-auth-server/loadtest/loadtests.py
  • The following items are covered in the load test
    • test_auth_server is the main entry point in the loadtests.py file
      • account creation
      • session creation
    • account deletion
    • session deletion
  • Integration tests
    • These are designed to cover the edge/error cases that are not applicable to the load test
    • The tests can be run against a remote server

Monitoring FxA Stage

Details on the Load Test tool

Known Issues

Success: 6125
Errors: 0
Failures: 524099
with a high number of 500s and 502s
  • Because of the issues listed above, the Loads tool team is working on some fixes for better use of the tool in detached mode with a given observer that will allow us to send the test output to a specific email or a given IRC channel.
  • The team is also looking at performing all FxA loads testing through the Services cluster.

References

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/FxALoadTesting&oldid=932754"