CloudServices/Basic Service Testing
Different levels and frequencies of testing, in the context of web services:
- 1 Unit tests -- run on every commit -- dev responsibility
- 2 Integration tests -- run on every 'build' -- dev/qa shared
- 3 End-to-end testing (also called verification testing or acceptance testing) -- run on every 'deploy' -- qa/ops shared
- 4 Monitoring (also called heartbeat testing) -- run continuously -- dev/ops responsibility
- 5 Glossary of terms
Unit tests -- run on every commit -- dev responsibility
These should only touch a single module, and should be referentially transparent.
They are stored alongside the source code, and should change at the same time.
They can use fixtures, but should generally not use mocks.
These tests should bridge more than one module (exactly two where practical)
They should live in a tests directory in the source repository.
If you have mocks for external components, this is where they should live.
Depending on the framework, these may or may not require a 'deploy' to run. If possible they should not.
In staging environments, qa should run these.
In production environments, ops will run them.
Ideally, no mocks here; staging testing and production testing should factor out all differences between them into parameters. This is important: both sets of tests should use the exact same scripts.
Monitoring (also called heartbeat testing) -- run continuously -- dev/ops responsibility
Custom endpoints are a possibility here (but watch out for security implications)
The tricky thing here is that the devs have to build testing functionality into the server-side, so as to make the ops scripts to access the testing as simple as possible.
Glossary of terms
module -- a minimally testable functional unit of code.
build -- compile but don't necessarily package
deploy -- package, install, make talk to the network.