AUS:Tests

AUS Flow

To learn a little about what logic determines what XML is displayed, see the AUS flow chart.

Test Assumptions

Sets of rules we can use to make our tests easier:

• For a given product, platform and locale, the output for all possible combinations having buildID more than one-off the latest will be identical
• One-off buildID offerings will have an added partial update that points to the latest build.

The General Plan

• Gather build data from the patcher config to determine realm of possible update URIs and/or test cases
• Use our test rules to define expected behavior for our AUS output
• Generate CLI script to show pass/fail status of new changes pre-release

Scope

• Start small -- start by only testing release channels
• Branch into release and fallback scenarios
• Delve into nightly testing as time allows (bigger job)

Existing Tools

These need to be consolidated into a more complete single testing suite:

• cf's output comparison
  • Good visualization idea, good coverage, but only for nightlies
• morgamic's output comparison
  • Has selection of different options and AUS2 sources (prod, stage, etc.) as well as a dynamic report that is saved and logged
• rhelmer/davel's update tests
  • Has better coverage per-test -- downloads .mar and compares hashes. Notifies if there is a 404 or hash mismatch.
  • l10n tests