TestEngineering/Services/LoadsToolsAndTesting1

From MozillaWiki
Jump to: navigation, search

Loads V1 and Vaurien

Two tools Loads (V1) and Vaurien

  • Most Stage deployment verification is partially handled through the use of the Loads tool for stress/load (and someday performance) testing.
  • Vaurien is a TCP proxy which will let you simulate chaos between your application and a backend server.
    • One active(?) POC for Vaurien is with GeoLocation (ichnaea).

Loads Cluster Usage Rules

  • Note: There are a number of open bugs and issues (see below) that require Loads use to be focused and specific per project:
    • Do not over do the loads test - start with the default values in the config files.
    • Do not run more than two tests in parallel.
    • Do not use more than 5 agents per load test unless you need to use more.
    • Do not run a load test of more than 8 - 10 hours
    • There are more limitations/rules...

Loads V1 Cluster Environment/Stack

  • Versions
Loads Cluster/Broker/Agents:
$ cd /home/ubuntu/loads/bin
$ ./loads-runner --version
  • AWS in US West
    • loads-master (broker and agent processes)
    • loads-slave-1 (agent processes)
    • loads-slave-2 (agent processes)
    • NOTE: there is no stack or ELB for this cluster
  • Files
    • /home/ubuntu
      • loads
      • loads-aws
      • loads-web
  • Processes
    • Search for processes owned by ubuntu, loads, nginx, circus
  • Logs
    • /var/log/redis
    • /var/log/nginx
  • QA access
    • You need special access to be able to SSH into these devices
    • You need to make some changes to your .ssh/config file

Loads V1 Cluster Monitoring

Agents statuses
Launch a health check on all agents

Loads V1 Cluster Maintenance

  • If things should go wrong...
  • Checking the cluster dashboard
  • TBD
  • Checking the stack
  • TBD
  • Restarting the Master/Broker
  • TBD
  • Restarting the Slaves/Agents
  • TBD

Repos

Bugs

Documentation

Loads Cluster Dashboard

Deployment and AWS Instances

  • Master, two slaves in US West
  • loads-master (broker and agent processes)
  • loads-slave-1 (agent processes)
  • loads-slave-2 (agent processes)
  • Note: there is no CF stack or ELB for this cluster
  • Note: the load cluster state/health can be check directly from the dashboard (see above)

Monitoring the cluster via Stackdriver

Monitoring the Loads Cluster via the Dashboard

Monitoring the Stage environment during Load Tests

  • We have various dashboards created by OPs that capture and display all sorts of data via the Heka/ES/Kibana pipeline
    • Heka dashboard
    • Kibana dashboard
    • Stackdriver

Load Test Results

  • Load test results are always listed in the dashboard.
  • A clean up of the dashboard is a high priority for V2 - we want a much better/accurate representation of the test(s) run with the kind of human-readable results that provide additional meaning/context to the metrics provided by the various OPs dashboards

Reporting (or lack of it)

  • There were plans to create some reporting in the style of what we had with the Funkload tool.
  • There are bugs open about getting some reporting out of Loads.
  • No action taken at this time, but a very good candidate for V2.

QA Wikis

Current projects using Loads

New/Planned projects using Loads

  • SimplePush (probably)
  • Tiles (maybe)

Other projects doing load testing

Vaurien

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/LoadsToolsAndTesting1&oldid=1145579"