QA/External Tests

From MozillaWiki
< QA
Jump to: navigation, search

Problem

As Services play a more significant role in Mozilla products, the need to test against a publicly accessible server is apparent. Currently, tests cannot access the public internet for Gecko and Gaia when run via releng/buildbot. This is a big problem for services such as Loop, FindMyDevice, Sync, and Firefox Accounts.

Goals

Enable the following use cases:

  • Developers can run marionette.js or other tests in their tree against a public server supporting Test-Driven Development.
  • It's much easier to test someone's fork with the core repo's tests.
  • We can leverage existing test frameworks such as mochitest and marionette.js without having to write custom runners or setting up a new environment.
  • Reduce barriers of writing complex mock servers. Many avoid writing mocks in early development as the APIs change, this helps landing tests earlier in cycle. Reduce the need to maintain the mock to match server behavior.
  • Dev teams can write and maintain tests that are beneficial to their development workflow.

Proposal

To solve this, let's create an directory named 'external' define external tests in a manifest file in Gecko and Gaia, specifically for tests with external access that will not be run by the releng/buildbot system or the standard make targets. Implement support for these tests in mach/make with a specific flag for local execution and non buildbot CI.

These tests will:

  • NOT run on the releng/buildbot system. Tests will be run on another CI system.
  • NOT run by default make/mach test commands.
  • Will required a flag to execute
  • Test reports will be sent to TreeHerder
  • Devs can choose to run these tests locally when they are on the internet and are working on code that affects that service.
  • Environment variables will be set to direct tests to point to {PROD,STAGE,DEV} servers.

Details

  • implement support in test manifest and mach/make to enable execution of external tests only (default would be not to run external tests)

FAQ

Pros and cons to having tests in m-c

  • Downloading m-c is big but this seems to be the reality of working with gecko or gaia.
  • Most devs may not maintain these tests as there isn’t visibility in tbpl (or treeherder for now). However, devs who are working on Loop, FindMyDevice, Sync, or Firefox Accounts will benefit from marionette.js tests in the core repo and may maintain them.
  • To execute these external access tests, an option would be passed in such as:
    • mach marionette-test --external-only
    • mach test --external-only
    • EXTERNAL_ONLY=true make test-integration
  • Having some facility to land external dependent tests in m-c doesn’t preclude us from spawning an externally hosted test repo. This really gives us another tool.