Revision as of 20:12, 18 August 2006

Requirements

Gather build data and determine realm of possible update URIs
Use this data to allow us to define a particular subset of these combinations based on:
- Product
- Version
- Platform
- BuildID
- Locale
- Channel
Generate an easy-to-understand interface that allows us to see whether or not output has changed between AUS versions.
- Simple pass/fail table comparing one version to another (see cf's example)
- Some way to visualize partner channel fallback -- should separate this out into its own table?

What are the differences between davel's tests and simple output tests?
- Hash checking after actual download?

These need to be consolidated into a more complete single testing suite:

cf's output comparison
- Good visualization idea, good coverage
morgamic's output comparison
- Has selection of different options and AUS2 sources (prod, stage, etc.) as well as a dynamic report that is saved and logged
rhelmer/davel's update tests
- Has better coverage per-test -- downloads .mar and compares hashes. Notifies if there is a 404 or hash mismatch.
- l10n tests

@@ Line 11: / Line 11: @@
 ** Simple pass/fail table comparing one version to another (see [http://users.ox.ac.uk/~clar0239/upwatch/staging/ cf's example])
 ** Some way to visualize partner channel fallback -- should separate this out into its own table?
-= Strategy =
-* Build off the stuff rhelmer/davel wrote -- if we can use Perl, fine -- if not, it'd be pretty easy to reimplement in PHP (so we can webify it)
-* Create a simple interface that allows you to choose what test you want to run (PHP), and what limits on the range of possibilities we want to enforce
-* Show results in an easy-to-read format (PHP as well)
 = Questions =