L10n Talk:Testing

From MozillaWiki
Jump to: navigation, search

... or

why we shouldn't test search

  • Some search engines are coming with revenue generating keys. Pounding those entry points with bots might violate the terms of service, in fact, I'd be surprised if it wouldn't.
  • We're testing a volatile implementation (in CVS) against a more volatile specification (wiki). The assumption that someone breaking one wouldn't fix the test by breaking the other isn't justified to me.
  • The complexity of search makes a exhaustive testing framework exhaustive.
  • The win for that cost is low.

Alternative proposal

How about making the current process

  • review
  • freeze

more complete and add a well-defined litmus test in between. We'd carry over that test for new plugins only, checking for things like:

  • comes up
  • seems to search for the words in question (try "mozilla", "firefox", "test")
  • doesn't query a different server than the result is on, server is obviously belonging to the provider
  • (if applicable) search suggest provides results, check for some inflammatory terms. Record response time and warn if it's near or above 500ms.