Confirmed users
964
edits
QA/External Tests: Difference between revisions
Jump to navigation
Jump to search
no edit summary
No edit summary |
|||
| Line 1: | Line 1: | ||
== Problem == | == Problem == | ||
As technology changes, the need to test against a publicly accessible server is apparent. Currently, tests cannot access the public internet for Gecko and Gaia when run via tinderbox. This is a big problem for services such as Loop, FindMyDevice, Sync, and Firefox Accounts. | As technology changes, the need to test against a publicly accessible server is apparent. Currently, tests cannot access the public internet for Gecko and Gaia when run via tinderbox. This is a big problem for services such as Loop, FindMyDevice, Sync, and Firefox Accounts. | ||
== Goals == | |||
The Use Cases this enables: | |||
* Developers can run marionette.js or other tests in their tree against a public server supporting Test-Driven Development. | |||
* It's much easier to test someone's fork with the core repo's tests. | |||
* We can leverage existing test frameworks such as mochitest and marionette.js without having to write custom runners or setting up a new environment. | |||
* Reduce barriers of writing complex mock servers. Many avoid writing mocks in early development as the APIs change, this helps landing tests earlier in cycle. Reduce the need to maintain the mock to match server behavior. | |||
* Dev teams can write and maintain tests that are beneficial to their development workflow. | |||
== Proposal == | == Proposal == | ||
To solve this, let's create an directory named 'external' specifically for tests with external access that will not be run by the buildbot build system or the standard make targets. This would follow this format: [area]/[framework]/external/test_example.js | To solve this, let's create an directory named 'external' in Gecko and Gaia, specifically for tests with external access that will not be run by the buildbot build system or the standard make targets. This would follow this format: [area]/[framework]/external/test_example.js | ||
These tests will: | These tests will: | ||
| Line 17: | Line 25: | ||
* content/media/test/mochitest/external/test_loop.html | * content/media/test/mochitest/external/test_loop.html | ||
* gaia/apps/findmydevice/test/marionette/external/fmd_locate.py | * gaia/apps/findmydevice/test/marionette/external/fmd_locate.py | ||
=== FAQ === | |||
Pros and cons to having tests in m-c | |||
* Downloading m-c is big but this seems to be the reality of working with gecko or gaia. | |||
* Most devs may not maintain these tests as there isn’t visibility in tbpl (or treeherder for now). However, devs who are working on Loop, FindMyDevice, Sync, or Firefox Accounts will benefit from marionette.js tests in the core repo and may maintain them. Eventually we could have a new make target for these types of tests (e.g. make test-external) | |||
* Having some facility to land external dependent tests in m-c doesn’t preclude us from spawning an externally hosted test repo. This really gives us another tool. | |||