TestEngineering/Services/FxALoadTesting

From MozillaWiki
Jump to navigation Jump to search

Summary for FxA-Auth-Server

Quick Verification Of Stage Deployments

  • This is a quick sanity test of the environment before getting started on load tests.
Install FxA-Auth-Server to a local host or an AWS instance (see below)
$ cd fxa-auth-server
Run the integration tests against the remote Stage server (load balancer)
$ PUBLIC_URL=<FxA Stage> npm run test-remote
Current example:
$ PUBLIC_URL=https://api-accounts.stage.mozaws.net npm run test-remote
  • NOTE: Make sure to install and test from the same branch that is deployed to Stage (ie do not use Master for running the tests against Stage or Production).

Load Test Tool Client/Host

Creating a RHEL AWS instance

  • Pick a Region then Create Instance > Launch Instance
  • Follow the prompts to create a basic, RHEL-flavored instance
  • Use of the QA/Dev key pairs that have been set up for this:
    • US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
    • US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
  • Once the instance is running, log in as "ec2-user"
  • The following apps, tools, and libs will need to be installed for use with various Services applications:
    • gcc, gcc-c++
    • hg
    • git
    • python-devel
    • automake, autoconf, and libtool (required for libzmq, for easy_install)
    • pip
    • virtualenv
    • node/npm
    • zeromq 3.X
    • gmp, gmp-devel
  • Also, general rhel updates:
$ sudo yum -y update
and/or
$ sudo yum -y upgrade
  • Now, the instance should be ready for installing and using the Loads tool.

Creating an Ubuntu AWS instance

  • Pick a Region then Create Instance > Launch Instance
  • Follow the prompts to create a basic, Ubuntu-flavored instance
  • Use of the QA/Dev key pairs that have been set up for this:
    • US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
    • US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
  • Once the instance is running, log in as "ubuntu"
  • The following apps, tools, and libs will need to be installed for use with various Services applications:
    • gcc, g++
    • mercurial
    • git
    • python-setuptools, python-virtualenv, and python-dev
    • automake, autoconf, libtool
    • m4
    • node/npm
    • libzmq and zeromq 3.X
    • gmp-5.1.3 or newer
  • Also, general rhel updates:
$ sudo apt-get update
and/or
$ sudo apt-get upgrade
  • Now, the instance should be ready for installing and using the Loads tool.

Installing the Loads tool on the AWS instance via FxA-Auth-Server

Installation:
$ git clone https://github.com/mozilla/fxa-auth-server.git
$ cd ./fxa-auth-server
$ npm install
$ npm test
$ cd ./loadtest
$ make build
  • Note: This will install a local copy of the Loads tool for use with FxA-Auth-Server.

Running the Loads tool against FxA Stage

  • The basic load test can be run as follows
$ make test SERVER_URL=https://api-accounts.stage.mozaws.net
  • The full, default load test can be run as follows
$ make bench SERVER_URL=https://api-accounts.stage.mozaws.net
  • Configuring the bench load test - config folder:
    • The test.ini file (for make test) can be configured for the following:
      • Number of hits
      • Number of concurrent users
    • The bench.ini file (for make bench) can be configured for the following:
      • Number of concurrent users
      • Duration of test
  • For both tests, start with the defaults, then tweak the duration. Users and Agents are optional tweaks/changes. Also, we can configure the bench load test to run in detached mode with an appropriate loads detach and observer settings.

Running the Loads tool against FxA Development or Production

  • This can be done if we are comparing Stage vs. some other environment and have access to the AWS logs in Dev or Production:
  • Dev:
$ make test SERVER_URL=https://accounts.dev.lcip.org
$ make bench SERVER_URL=https://accounts.dev.lcip.org
  • Prod:
$ make test SERVER_URL=https://api.accounts.firefox.com
$ make bench SERVER_URL=https://api.accounts.firefox.com
  • The same optional configuration changes apply here.

Using the Loads Services Cluster

  • By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
  • Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
  • Testing against the Stage environment:
$ make megabench SERVER_URL=https://api-accounts.stage.mozaws.net
  • Testing against the Dev environment:
$ make megabench SERVER_URL=https://api-accounts.dev.lcip.org
  • Testing against the Prod enviornment:
$ make megabench SERVER_URL=https://api.accounts.firefox.com
  • Configuring the megabench load test - config folder:
    • The megabench.ini file (for make megabench) can be configured for the following:
      • Number of concurrent users
      • Duration of test
      • Include file (leave as defined for now)
      • Python dependencies (leave as defined for now)
      • Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
      • Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
      • Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
      • Observer (this can be email or irc - the default is irc #services-dev channel)

Test Coverage and Stats

  • Basic tweakable values for all load tests
    • users = number of concurrent users/agent
    • agents = number of agents out of the cluster, otherwise errors out
    • duration = in seconds
    • hits = 1 or X number of rounds/hits/iterations
  • Location fxa-auth-server/loadtest/loadtests.py
  • The following items are covered in the load test
    • test_auth_server is the main entry point in the loadtests.py file
      • account creation
      • session creation
    • account deletion
    • session deletion
  • Integration tests
    • These are designed to cover the edge/error cases that are not applicable to the load test
    • The tests can be run against a remote server

Analyzing the Results

  • TBD

Debugging the Issues

  • TBD

Monitoring FxA Stage

../bin/loads-runner --ping-broker --broker=tcp://loads.services.mozilla.com:7780
../bin/loads-runner --check-cluster --broker=tcp://loads.services.mozilla.com:7780
(or similar)

Performance Testing Information

  • TBD

Details on the Load Test tool

Known Bugs, Issues, and Tasks

References

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/FxALoadTesting&oldid=957133"