NOTE: Final content will be here: http://quality.mozilla.org/en/projects/automation/minotaur

Minotaur

Minotaur is a testing framework that can automatically check the preferences, search settings, bookmarks, extensions, and update channels for a Firefox build. These settings are vetted against a previous gold standard of settings and/or verification files. This is used to simplify and reduce the manual testing currently required for Localization and Partner builds of Firefox.

How it Works

There are two basic pieces of the Minotaur tool, the Exporter and the Verifier.

Exporter

The exporter is a small chrome window that is automatically inserted into a given Firefox build. This enables the Minotaur javascript files to use XPCOM calls to collect the information. Specifically, it does the following:

• Search: Gets search engine order, localized name, and base URL through the nsIBrowserSearchService interface
• Preferences: Gets Preference information for any preferences matching: /(app\.distributor)|(app\.partner)|(app\.update\.channel)|(homepage)|(startup)|(browser\.contentHandlers)/
  • Uses nsIPrefBranch2::GetComplexValue to decode locale specific preference substitution strings
• Extensions: Uses nsIExtensionManager to retrieve all extensions, their IDs, names, version, xpiURL, and iconURL.

All of these items are dumped into an output file in XML format. A sample XML file can be found here.

Bookmarks

It also exports bookmarks. It does this by detecting your Gecko runtime version. If 1.8 or earlier, it uses nsIBookmarksService and the BookmarksCommand::doBookmarksCommand to cause an export of bookmarks to bookmarks.html. This is the same mechanism used to perform the export if you were to do it through the UI.

If you are using 1.9 or higher, then it uses the nsIPlacesImportExportService::exportHTMLToFile, which does the same.

Release Channel

The release channel is automatically configured during an AUS ping. So, Minotaur switches on HTTP debug logging, performs the release ping from inside the chrome (by mimicking the same behavior the button uses in the browser).

The debug log is captured, and sent to the verifier.

Verifier

The Verifier needs three files from the Exporter: output.xml, bookmarks.html, and the http debug log. Once it has these files, it needs to get "golden" approved files to use to make a comparison in order to judge whether or not the test passes.

The Verifier checks out the following "golden standard" files from CVS (or in the case of partner builds, these will be supplied).

• Output.xml (from a Minotaur export on a known good build)
• bookmarks.html (from a Minotaur export on a known good build)
• release-channel.txt (simple text file with the name of the release channel in the first line)

These files are compared with their counterparts from the Exporter as follows:

• Output.xml -- Simple diff comparison. If something is different, then the test fails and can generate a log of the key difference that caused the issue.
• Bookmarks.html -- For a variety of reasons, bookmarks.html cannot be diff'd by a simple diff. Instead a Python script is used that can smartly parse the bookmarks.html and diff the relevant information making certain that all folders are present and that each folder contains the same bookmarks with the same name and same URL.
• Release Channel -- The release Channel Verifier is also a Python script that searches the http debug log for the AUS ping GET. It then parses that line and ensures that it finds the release channel as specified in the release-channel.txt file mentioned above.

All of these processes must pass in order for Minotaur to mark a build as a "PASS". If any steps fail, they append the failure to a log, and can return a FAIL status to a calling entity (i.e. buildbot in the L10N case, so that a Tinderbox can be burned).