AUS:Tests

Test Cases

Normal Tests

test	release update exists?	matches fallback regex?	fallback update exists?	is nightly channel?	nightly update exists?	release update offered?	fallback update offered?	nightly update offered?
Invalid parameter	No	No	No	No	No	No	No	No
*-cck-* channel, no update	No	Yes	No	No	No	No	No	No
*-cck-* channel, explicit update	Yes	Yes	No	No	No	No	No	Yes
*-cck-* channel, fallback update	No	Yes	Yes	No	No	No	Yes	No
*-cck-* channel, explicit update	Yes	Yes	Yes	No	No	No	No	Yes

Requirements

• Gather build data and determine realm of possible update URIs
• Use this data to allow us to define a particular subset of these combinations based on:
  • Product
  • Version
  • Platform
  • BuildID
  • Locale
  • Channel
• Generate an easy-to-understand interface that allows us to see whether or not output has changed between AUS versions.
  • Simple pass/fail table comparing one version to another (see cf's example)
  • Some way to visualize partner channel fallback -- should separate this out into its own table?

Questions

• What are the differences between davel's tests and simple output tests?
  • Hash checking after actual download?

Existing Tools

These need to be consolidated into a more complete single testing suite:

• cf's output comparison
  • Good visualization idea, good coverage
• morgamic's output comparison
  • Has selection of different options and AUS2 sources (prod, stage, etc.) as well as a dynamic report that is saved and logged
• rhelmer/davel's update tests
  • Has better coverage per-test -- downloads .mar and compares hashes. Notifies if there is a 404 or hash mismatch.
  • l10n tests