AUS:Tests: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
|||
| Line 1: | Line 1: | ||
= Questions = | |||
* Do we start over? | |||
* What can we move some things over? | |||
* Do we just keep things in bash and wrap the new stuff? | |||
* Where should we plug in the existing updates.cfg? | |||
* How would we piggyback the actual update application onto a new set of .py tests? Doable? | |||
= TODO = | |||
* Decide how we will integrate the existing release tests with pyFit-based tests | |||
* Start setting up very simple tests and utilize pyFit to store our 'expected results' | |||
** We would have to generate some mock build data, and use an aus2 test instance to help generate some of the 'expected results' | |||
** Start building off of this to eventually cover all of our possible combinations of inputs/outputs | |||
* Develop a script to finally retrieve AUS XML data based on passed inputs/fixtures then compare to expected results | |||
* Think about ways to display results or provide reporting capabilities down the road? Logging? | |||
= AUS Flow = | = AUS Flow = | ||
To learn a little about what logic determines what XML is displayed, see the [http://people.mozilla.org/~morgamic/aus-partner.jpg AUS flow chart]. | To learn a little about what logic determines what XML is displayed, see the [http://people.mozilla.org/~morgamic/aus-partner.jpg AUS flow chart]. | ||
| Line 6: | Line 21: | ||
* For a given product, platform and locale, the output for all possible combinations having buildID more than one-off the latest will be identical | * For a given product, platform and locale, the output for all possible combinations having buildID more than one-off the latest will be identical | ||
* One-off buildID offerings will have an added partial update that points to the latest build. | * One-off buildID offerings will have an added partial update that points to the latest build. | ||
= Existing Tools = | = Existing Tools = | ||
Revision as of 11:05, 1 September 2006
Questions
- Do we start over?
- What can we move some things over?
- Do we just keep things in bash and wrap the new stuff?
- Where should we plug in the existing updates.cfg?
- How would we piggyback the actual update application onto a new set of .py tests? Doable?
TODO
- Decide how we will integrate the existing release tests with pyFit-based tests
- Start setting up very simple tests and utilize pyFit to store our 'expected results'
- We would have to generate some mock build data, and use an aus2 test instance to help generate some of the 'expected results'
- Start building off of this to eventually cover all of our possible combinations of inputs/outputs
- Develop a script to finally retrieve AUS XML data based on passed inputs/fixtures then compare to expected results
- Think about ways to display results or provide reporting capabilities down the road? Logging?
AUS Flow
To learn a little about what logic determines what XML is displayed, see the AUS flow chart.
Test Assumptions
Sets of rules we can use to make our tests easier:
- For a given product, platform and locale, the output for all possible combinations having buildID more than one-off the latest will be identical
- One-off buildID offerings will have an added partial update that points to the latest build.
Existing Tools
These need to be consolidated into a more complete single testing suite:
- cf's output comparison
- Good visualization idea, good coverage, but only for nightlies
- morgamic's output comparison
- Has selection of different options and AUS2 sources (prod, stage, etc.) as well as a dynamic report that is saved and logged
- rhelmer/davel's update tests
- Has better coverage per-test -- downloads .mar and compares hashes. Notifies if there is a 404 or hash mismatch.
- l10n tests