QA/Execution/Web Testing/Automation tools

From MozillaWiki
Jump to: navigation, search

List of Automation tools

Evaluations of current Automation tools, languages, tools for continuous delivery.

Big picture questions:

  • When do you get a new framework or language?
  • When is it time to move away from how things have been done?
  • Risk of complexity and instability of going with new tools vs. Risk of irrelevancy and being left out of the process

Continuous Integration

  • Jenkins
    • Open to all contributors
    • Considered to be outside of the project development release process
  • Travis CI

Data collection tools

  • Datadog
  • Google Analytics

Site Monitoring tools

  • New Relic


  • JS
    • More support from developers who prefer the language
    • Async testing framework
    • Scripts vs. Page Objects
  • Python
    • More support from Web QA
    • More options for testing using pytest

Load Testing

Headless testing

  • Are page renderings being tested?
  • What approximation to a "real" browser do these frameworks represent?
  • Are these proper
  • Value of running speed


  • Uses Phantom.js
  • Opinion: Easy to learn and set up, simple API
    • Easy to use CSS selectors
  • Not a good replacement for cross browser testing
  • No page object model in tests, removes a level of abstraction & removes consistency found in page objects.
  • Runs very fast in dev/stage
  • Runs very slow locally as each page must load
  • Can only check basic functionality; example: plugin check page
  • Does not check redirects reliably

? Can it run on Travis?



  • runs using Casper


  • Opinion: Verbose API

Webdriver JS framework

  • Opinion: better than Intern, not as complete as Selenium/Python