TestEngineering/Services/FxALoadTesting

From MozillaWiki
Jump to navigation Jump to search

Load Test Tool Client/Host

Creating a RHEL AWS instance

  • Pick a Region then Create Instance > Launch Instance
  • Follow the prompts to create a basic, RHEL-flavored instance
  • Use of the QA/Dev key pairs that have been set up for this:
    • US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
    • US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
  • Once the instance is running, log in as "ec2-user"
  • The following apps, tools, and libs will need to be installed for use with various Services applications:
    • gcc, gcc-c++
    • hg
    • git
    • python-devel
    • automake, autoconf, and libtool (required for libzmq, for easy_install)
    • pip
    • virtualenv
    • node/npm
    • zeromq 3.X
    • gmp, gmp-devel
  • Also, general rhel updates:
$ sudo yum -y update
and/or
$ sudo yum -y upgrade
  • Now, the instance should be ready for installing and using the Loads tool.

Creating an Ubuntu AWS instance

  • Pick a Region then Create Instance > Launch Instance
  • Follow the prompts to create a basic, Ubuntu-flavored instance
  • Use of the QA/Dev key pairs that have been set up for this:
    • US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
    • US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
  • Once the instance is running, log in as "ubuntu"
  • The following apps, tools, and libs will need to be installed for use with various Services applications:
    • gcc, g++
    • mercurial
    • git
    • python-setuptools, python-virtualenv, and python-dev
    • automake, autoconf, libtool
    • m4
    • node/npm
    • libzmq and zeromq 3.X
    • gmp-5.1.3 or newer
  • Also, general rhel updates:
$ sudo apt-get update
and/or
$ sudo apt-get upgrade
  • Now, the instance should be ready for installing and using the Loads tool.

Installing the Loads tool on the AWS instance via FxA-Auth-Server

Installation:
$ git clone https://github.com/mozilla/fxa-auth-server.git
$ cd ./fxa-auth-server
$ npm install
$ node ./scripts/gen_keys.js
$ npm test
$ cd ./loadtest
$ make build
  • Note: This will install a local copy of the Loads tool for use with FxA-Auth-Server.

Running the Loads tool against FxA Stage

  • Note: the following value can stay fixed in the file loadsrunner.sh: BROKER=broker.loads.lcip.org. This will allow the current loads tool install to make use of the broker and agents in the Loads environment.
  • The basic load test can be run as follows
$ make test SERVER_URL=https://api-accounts.stage.mozaws.net
  • Note: With an install of fxa-auth-server, the load test tool is installed in the following location:
    • ../fxa-auth-server/loadtest/bin
  • The full, default load test can be run as follows
$ make bench SERVER_URL=https://api-accounts.stage.mozaws.net
  • Configuring the load test - makefile:
    • The default "make bench" command looks like this:
./loadsrunner.sh --users=20 --duration=1800 --agents=5 --include-file=./loadtests.py --python-dep=hawkauthlib loadtests.LoadTest.test_auth_server

You can change the following:
    users = number of concurrent users
    duration = length of the load test
    agents = the number of agents the broker will start up/use for the load test

Start with the defaults, then tweak the duration. Users and Agents are optional tweaks/changes.
  • Configuring the load test - loadsrunner.sh:
    • Note: the defaults should not be changed unless you need a very customized load test
Note the following ENV:
    BROKER=broker.loads.lcip.org
This allows us to use the broker already defined in the loads environment.

Note the key loads-runner line:
    ./bin/loads-runner --broker=tcp://localhost:7780 --zmq-publisher=tcp://localhost:7776 $@
This information is useful if you will be running the load test in detached mode.
  • NOTE: only one load test can be run at a time against FxA Stage since we are making use of the Loads environment broker and agents. Allow a load test to finish before starting a new load test. See below for more information...
  • NOTE: Also, we can configure the load test (make bench) to run in detached mode with an appropriate observer. See below for more information...

Running the Loads tool against FxA Development or Production

  • This can be done if we are comparing Stage vs. some other environment and have access to the AWS logs in Dev or Production:
  • Dev:
$ make test SERVER_URL=https://accounts.dev.lcip.org
$ make bench SERVER_URL=https://accounts.dev.lcip.org
  • Prod:
$ make test SERVER_URL=https://api.accounts.firefox.com
$ make bench SERVER_URL=https://api.accounts.firefox.com

Using the Loads Test environment

  • For the broker and agents - will probably have Dev create a temporary addition to the Make file and tools to allow for a "make megabench" that will run a "make bench" but use the Loads Test environment broker and agent set up.
  • Note: normally, we can only run one load test at a time with this environment, since it was not designed to share the broker and all the agents. If you attempt to run two loads tests with this environment, you will probably get a failure that looks something like this:
ExecutionError: Not enough agents running on that broker
  • We can, with the proper permissions, SSH into the environment and restart the broker and agent processes.

Using the Loads Services cluster

  • To get around the issues with using the loads tool on a local host
  • And as an alternative to Detached Mode and Observers
  • TBD

Test Coverage and Stats

  • The following items are covered in the load test
    • test_auth_server is the main entry point in the loadtests.py file
      • account creation
      • session creation
  • The following items are not covered in the load test
    • account deletion
    • session deletion
      • NOTE: a bug has been filed to get these added to the load test
    • constants to control the number of account/sessions creations and deletions
      • NOTE: a bug has been filed to get these added to the load test
  • Integration tests
    • These are designed to cover the edge/error cases that are not applicable to the load test
    • The tests can be run against a remote server

Monitoring FxA Stage

Details on the Load Test tool

Known Issues

Success: 6125
Errors: 0
Failures: 524099
with a high number of 500s and 502s
  • Because of the issues listed above, the Loads tool team is working on some fixes for better use of the tool in detached mode with a given observer that will allow us to send the test output to a specific email or a given IRC channel.
  • The team is also looking at performing all FxA loads testing through the Services cluster.

References

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/FxALoadTesting&oldid=923261"