• NOTE 1: Original Source: https://etherpad.mozilla.org/Loads-Current-Status-Aug2014
• NOTE 2: This is specifically for Loads V2
• NOTE 3: For Loads V1 information, please see https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

Loads V2

• What is it?
• Changes for V2
• Overview/Slides: http://blog.ziade.org/slides/loadsv2/#/
• Initial Diagram:
• Initial Look: https://etherpad.mozilla.org/Loadsv2
• Ben's design work: https://etherpad.mozilla.org/loadsv2-design

Comparison of Load Test Tools

• Siege: http://www.joedog.org/siege-home/
• And some others in comparison: http://www.appdynamics.com/blog/devops/load-testing-tools-explained-the-server-side/
• https://github.com/newsapps/beeswithmachineguns
• Some of these require large sums of $ in order to run adequate load tests (size/time)
• Straight HTTP vs. smart tests (that we are sending)
• Dumb testing vs. smart testing (what we are doing)
• Some of the off-the-shelf are quite limited - we need to be able to use a programming language to define very specific tests/requirements
• The Grinder, for example, is not really designed to be deployed on AWS, for example.
• Tsunami is good at sending a lot of load on web service, but it requires writing XML

Tasks

• Tarek says:
• So what I was thinking: I can lead Loads v2 development with the help of QA and Ops and Benjamin for SimplePush, and then slowly transition ownership to the QA and Ops team - because at the end of the day that's the two teams that should benefit the most from this tool.

New Repos

• https://github.com/loads
• https://github.com/loads/docs
• https://github.com/loads/loads-broker
• https://github.com/loads/loads-tester
• https://github.com/loads/old-loads-agent
• https://github.com/loads/old-loads-broker
• https://github.com/loads/old-loads-base
• https://github.com/loads/old-loads-web
• Note: naming is a bit strange right now because the architecture is in transition

New Documentation

• TBD: for now see https://github.com/loads/docs

Brown Bag and Info Session

• https://etherpad.mozilla.org/loads-brownbag
• Note: This will not take place in September, but could take place in December (2014)

Brainstorming Loads and V2

• What we need going forward
• What we want going forward
• Some issues (generalized - see the github issues for details):
  • 1- very long runs (>10hours) are not really working. This is a design problem.
  • 2- spinning new slaves to make big tests has not yet been automated. We have 2 slaves boxes that run 10 agents each. This was enough for most of our needs though.
  • 3- The dashboard is scarce. It'll tell you what;s going on, but we don't have any real reporting features yet.
  • 4- running a test using another language than Python is a bit of a pain (you need to do some zmq messaging)

• Stealing from Tarek's slide deck:
  • Day-long runs don't really work
  • Crappy dashboard
  • No direct link to Logs/CPU/Mem usage of stressed servers
  • No automatic slaves deployment yet
  • Python client only really supported
  • High bar to implement clients in Haskell/Go

• Also, we have a lot of open bugs that need to get fixed. Some prevent better use of the tool for newer projects/services.
  • Get Loads "fixed" for Mac 10.9 and XCode 5.1.1: https://bugzilla.mozilla.org/show_bug.cgi?id=1010567

• Figure out how to run loads from personal AWS instances
• Monitoring
  • What we currently have for Stage
  • What do we want/need?
• Reporting
• Loads dashboard
  • What about CPU/memory information (like from atop, tops)
  • Links to some snapshoted graphs
  • code version
  • red/yellow/green states
  • Deployment bug
  • Bugs opened
  • Bugs closed
• Scaling the cluster dynamically (see V2)
• Quarterly results/trending
• Targets
  • PM targets
  • expected targets
  • actual targets
• Wiki design
  • One per service?
  • One per service per deployment?
• Weekly reporting
  • What does the QE team want to see
• Getting the right data/metrics requirements from PMs then extracting that information and displaying on the Loads dashboard and/or in the OPs-built dashboards