TestEngineering/Services/FxALoadTesting

Load Test Tool Client

• It is always best to configure an AWS instance as the load test tool client/host.
• The actual Load Test broker and agents run in the Load Test environment set up by rfkelly. See the following wiki:
  • https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA_Load_Test_Environment

Creating a RHEL AWS instance

• Pick a Region then Create Instance > Launch Instance
• Follow the prompts to create a basic, RHEL-flavored instance
• Use of the QA/Dev key pairs that have been set up for this:
  • US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
  • US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
• Once the instance is running, log in as "ec2-user"

• The following apps, tools, and libs will need to be installed for use with various Services applications:
  • gcc, gcc-c++
  • hg
  • git
  • python-devel
  • automake, autoconf, and libtool (required for libzmq, for easy_install)
  • pip
  • virtualenv
• Install node/npm
• Install zeromq 3.X
• Install gmp, gmp-devel

• Also, general rhel updates:

$ sudo yum -y update
and/or
$ sudo yum -y upgrade

• Now, the instance should be ready for installing and using the Loads tool.

Creating an Ubuntu AWS instance

• Pick a Region then Create Instance > Launch Instance
• Follow the prompts to create a basic, Ubuntu-flavored instance
• Use of the QA/Dev key pairs that have been set up for this:
  • US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
  • US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
• Once the instance is running, log in as "ubuntu"

• The following apps, tools, and libs will need to be installed for use with various Services applications:
  • gcc, g++
  • mercurial
  • git
  • python-setuptools, python-virtualenv, and python-dev
  • automake, autoconf, libtool
  • m4
• Install node/npm
• Install libzmq and zeromq 3.X
• Install gmp-5.1.3 or newer

• Also, general rhel updates:

$ sudo apt-get update
and/or
$ sudo apt-get upgrade

• Now, the instance should be ready for installing and using the Loads tool.

Installing the Loads tool on the AWS instance via FxA-Auth-Server

Installation:
$ git clone https://github.com/mozilla/fxa-auth-server.git
$ cd ./fxa-auth-server
$ npm install
$ node ./scripts/gen_keys.js
$ npm test
$ cd ./loadtest
$ make build

• Note: This will install a local copy of the Loads tool for use with FxA-Auth-Server.

Running the Loads tool against FxA Stage

• Note: the following value can stay fixed in loadsrunner.sh.
  • BROKER=broker.loads.lcip.org
• This will allow the current loads tool install to make use of the broker and agents in the Loads environment.

• The basic load test can be run as follows

$ make test SERVER_URL=https://api-accounts.stage.mozaws.net

• • Note: With an install of fxa-auth-server, the load test tool is installed in the following location:
    • ../fxa-auth-server/loadtest/bin

• The full, default load test can be run as follows

$ make bench SERVER_URL=https://api-accounts.stage.mozaws.net

• Configuring the load test - makefile:
  • The default "make bench" command looks like this:

./loadsrunner.sh --users=20 --duration=1800 --agents=5 --include-file=./loadtests.py --python-dep=hawkauthlib loadtests.LoadTest.test_auth_server

You can change the following:
    users = number of concurrent users
    duration = length of the load test
    agents = the number of agents the broker will start up/use for the load test

Start with the defaults, then tweak the duration. Users and Agents are optional tweaks/changes.

• Configuring the load test - loadsrunner.sh:
  • Note: the defaults should not be changed unless you need a very customized load test

Note the following ENV:
    BROKER=broker.loads.lcip.org
This allows us to use the broker already defined in the loads environment.

Note the key loads-runner line:
    ./bin/loads-runner --broker=tcp://localhost:7780 --zmq-publisher=tcp://localhost:7776 $@
This information is useful if you will be running the load test in detached mode.

• NOTE: only one load test can be run at a time against FxA Stage since we are making use of the Loads environment broker and agents. Allow a load test to finish before starting a new load test.

Monitoring FxA Stage

• OPs has set up two dashboards for monitoring Stage:
  • https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json
  • https://heka.shared.us-east-1.stage.mozaws.net/#health
• NOTE: There is currently a bug that makes it difficult to access these pages in Fx without adding an exception. The use of the Mozilla cert "should" get around this, but the bug remains...

Details on the Load Test tool

• The documentation can be found here:
  • https://loads.readthedocs.org/en/latest/
  • The most useful information here is running in detached mode and using an observer
    • https://loads.readthedocs.org/en/latest/distributed/#detach-mode
    • https://loads.readthedocs.org/en/latest/commands/
    • and here: https://loads.readthedocs.org/en/latest/internals/?highlight=observer
• The repo is here:
  • https://github.com/mozilla-services/loads

Known Issues

• https://github.com/mozilla-services/puppet-config/issues/64
• https://github.com/mozilla-services/loads/issues/181
• https://github.com/mozilla-services/loads/issues/198
• https://github.com/mozilla-services/loads/issues/199

• Because of the issues listed above, the Loads tool team is working on some fixes for better use of the tool in detached mode with a given observer that will allow us to send the test output to a specific email or a given IRC channel.