User:Ctalbert/MinotaurDesign

Changes to Minotaur

Current

We have two sections, one that exports items from the build under test, and the second half that verifies this information against a known verification XML file.

Here are the basic steps that occur:

1. Install the build (via build bot -- integration needed)
2. Run minotaur*.sh
   1. Inserts itself into chrome
   2. Exports extensions, preferences, search settings, bookmarks, content handlers
   3. Generates three files: http debug log (to verify update channel), bookmarks.html, and an output.xml. Output.xml contains the preference, search, extension, and protocol/content handler information.
3. Next, the verification tool takes these three files plus the verification XML file and runs several python scripts to vet that the exported data matches the expected data as specified in the verification XML file
4. Generates two files:
   1. results.html - simple html table that shows which parts of the test passed, which fails (i.e. search passes, preferences fail, extensions pass, etc) If any one item fails
   2. results.log - detailed log about what failed

Issues with this

• Maintainability -- creating these verification scripts is very error prone and tedious. One single mispelling will cause the test to fail.
• Partner vs. L10N -- In order to really do this correctly, we want one test for Partners to include the partner build + all localizations for the partner build. For L10N, each localization is a specific test. Ideally, we'd want separate styles of verification files for each build. Then that has implications on the python verification scripts that need to recognize each of those.

A solution

Do everything above as far as exporting goes. Generate the three files Then...

1. Checkout a copy of the reviewed and checked in output.xml, bookmarks.html and a verification script (that only contains the computed release channel). These will be specific to the build under test and the localization.
2. Diff the exported output.xml and bookmarks.html with the copies from CVS. If there are differences --> test fails, use the diff output as the results.log for debugging/review purposes.
3. Run the python script to parse the http debug log and vet the computed release channel.
4. If the test fails, we turn any associated (L10N) tinderboxes red and we can file a bug containing the diff output.
5. To fix the breakage, we will get the drivers to review the diff and there will be two outcomes:
   1. Outcome 1: We have changed something in the build under test on purpose. In this case, we change the checked-in file(s) to reflect the new reality
   2. Outcome 2: A developer or localizer changed something that we didn't want. In this case, we get the developer/localizer to backout their change.

Advantages

• To change this, we must remove code (the python code that vets extensions/bookmarks/preferences/searches). We keep the python code that vets the channels. This is because an diffing two http debug logs won't be informative
• This eliminates maintaining of the verification files.
• This makes it easier to use the same mechanism for partner and L10N builds.
• It also gives us an easy mechanism to review any changes that are detected and a process for getting them addressed.
• This simplifies adding future functionality - such as protocol handlers. We only have to write the code to export this information and we do not have to worry about writing similar code to vet the exported data. I have not added protocol export because that code is still in development in the product (but it does export feed handler preferences in the current patch that is in bugzilla).