L10n Talk:Testing

... or

why we shouldn't test search

• Some search engines are coming with revenue generating keys. Pounding those entry points with bots might violate the terms of service, in fact, I'd be surprised if it wouldn't.
• We're testing a volatile implementation (in CVS) against a more volatile specification (wiki). The assumption that someone breaking one wouldn't fix the test by breaking the other isn't justified to me.
• The complexity of search makes a exhaustive testing framework exhaustive.
• The win for that cost is low.

Alternative proposal

How about making the current process

• review
• freeze

more complete and add a well-defined litmus test in between. We'd carry over that test for new plugins only, checking for things like:

• comes up
• seems to search for the words in question (try "mozilla", "firefox", "test")
• doesn't query a different server than the result is on, server is obviously belonging to the provider
• (if applicable) search suggest provides results, check for some inflammatory terms. Record response time and warn if it's near or above 500ms.