AUS:Tests: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(32 intermediate revisions by the same user not shown)
Line 1: Line 1:
= Requirements =
[[AUS|« back to AUS main page]]
* Gather build data and determine realm of possible update URIs
* Use this data to allow us to define a particular subset of these combinations based on:
** Product
** Version
** Platform
** BuildID
** Locale
** Channel
* Generate an easy-to-understand interface that allows us to see whether or not output has changed between AUS versions.
** Simple pass/fail table comparing one version to another (see [http://users.ox.ac.uk/~clar0239/upwatch/staging/ cf's example])
** Some way to visualize partner channel fallback -- should separate this out into its own table?


= Strategy =
= Existing Tools =
* Write spider script in Perl or PHP to generate hashes and/or XML of possible combinations
* Create a simple interface that allows you to choose what test you want to run (PHP)
* Show results in an easy-to-read format (PHP as well)
* Since we don't have anybody else to do it, really, morgamic can take this on
 
= Questions =
* What are the differences between davel's tests and simple output tests?
** Hash checking after actual download?
 
= Existing Tests =
These need to be consolidated into a more complete single testing suite:
These need to be consolidated into a more complete single testing suite:
* [http://khan-vm.mozilla.org:8080/VerifyAus?test AUS acceptance tests]
* [http://users.ox.ac.uk/~clar0239/upwatch/staging/ cf's output comparison]
* [http://users.ox.ac.uk/~clar0239/upwatch/staging/ cf's output comparison]
** Good visualization idea, good coverage, but only for nightlies
* [http://lxr.mozilla.org/mozilla/source/webtools/aus/sanity morgamic's output comparison]
* [http://lxr.mozilla.org/mozilla/source/webtools/aus/sanity morgamic's output comparison]
* [http://lxr.mozilla.org/mozilla/source/testing/release/ davel's update tests]
** Has selection of different options and AUS2 sources (prod, stage, etc.) as well as a dynamic report that is saved and logged
* [http://lxr.mozilla.org/mozilla/source/testing/release/ rhelmer/davel's update tests]
** Has better coverage per-test -- downloads .mar and compares hashes.  Notifies if there is a 404 or hash mismatch.
** l10n tests
 
= Inputs =
* uri version
* product
* version
* build
* platform
* locale
* channel
 
= Outputs =
* See [[Software_Update:updates.xml_Format|Software Update XML format definition]]
 
Example:
<?xml version="1.0"?>
<updates>
    <update type="minor" version="1.5.0.7" extensionVersion="1.5.0.7" buildID="2006083105" >
        <patch type="complete" URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2006-08-31-05-mozilla1.8.0/firefox-1.5.0.7.en-US.mac.complete.mar" hashFunction="md5" hashValue="90f58b8836f288d42e2c14ea5ba3c71e" size="15408802"/>
        <patch type="partial" URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2006-08-30-05-mozilla1.8.0/firefox-1.5.0.7.en-US.mac.partial.2006083005-2006083105.mar" hashFunction="MD5" hashValue="712b418e5c300be941515d5f75eb167f" size="6859"/>
    </update>
</updates>
 
= AUS Flow =
To learn a little about what logic determines what XML is displayed, see the [http://people.mozilla.org/~morgamic/aus-partner.jpg AUS flow chart].
 
= Test Assumptions =
Sets of rules we can use to make our tests easier:
* For a given product, platform and locale, the output for all possible combinations having buildID more than one-off the latest will be identical
* One-off buildID offerings will have an added partial update that points to the latest build.

Latest revision as of 07:44, 24 January 2007

« back to AUS main page

Existing Tools

These need to be consolidated into a more complete single testing suite:

Inputs

  • uri version
  • product
  • version
  • build
  • platform
  • locale
  • channel

Outputs

Example: 

<?xml version="1.0"?>
<updates>
    <update type="minor" version="1.5.0.7" extensionVersion="1.5.0.7" buildID="2006083105" >
        <patch type="complete" URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2006-08-31-05-mozilla1.8.0/firefox-1.5.0.7.en-US.mac.complete.mar" hashFunction="md5" hashValue="90f58b8836f288d42e2c14ea5ba3c71e" size="15408802"/>
        <patch type="partial" URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2006-08-30-05-mozilla1.8.0/firefox-1.5.0.7.en-US.mac.partial.2006083005-2006083105.mar" hashFunction="MD5" hashValue="712b418e5c300be941515d5f75eb167f" size="6859"/>
    </update>
</updates>

AUS Flow

To learn a little about what logic determines what XML is displayed, see the AUS flow chart.

Test Assumptions

Sets of rules we can use to make our tests easier:

  • For a given product, platform and locale, the output for all possible combinations having buildID more than one-off the latest will be identical
  • One-off buildID offerings will have an added partial update that points to the latest build.
Retrieved from "https://wiki.mozilla.org/index.php?title=AUS:Tests&oldid=47350"