Auto-tools/Projects/W3C CSS Test Mirroring

From MozillaWiki
Jump to: navigation, search

Rationale

The W3C CSS Test working group has a bunch of tests in their hg repo on the W3C site. We need an automated way to pull the tests back from the W3C site into our infrastructure so that these tests can be run in addition to the rest of our automated test suites.

We also need a way to automatically annotate new tests for inclusion into the upstream W3C repo. As a point of process, when a test is submitted to the W3C, it goes into a "submitted" tree, and then once it is accepted it is in an "approved" tree. This is important to remember for the process outlined below.

Proposed Process

  • Developer writes test to be synced to W3C - these have a separate license, so they land in the other-licenses/ directory (fantasai will determine the proper directory for this), for now, let's call it other-licenses/w3c-css/submitted
  • An hg hook watches for checkins to this directory and causes the new test to be pushed into the W3C "submitted" tree.
  • On some time sequence (once a day or something) we pull from the W3C "approved" tree into our other-licenses/w3c-css/received (name TBD) directory.
    • When we pull the tests down, we need to generate the reftest manifests for the newly downloaded tests. (Note: W3C has build scripts that can generate reftest manifests.)
    • Because these newly downloaded tests may exist in our submitted directory we need to ensure that the reftest manifest annotations those files had in the submitted directory are added to the generated manifest for the new file in the received/ directory.
    • Once the manifests are merged, the submitted test files (and their corresponding manifest entries) should be deleted.

Other considerations

We probably want to sync the W3C tests back into some kind of project branch. This way we can run the automated tests on them and let developers work on them to get them running and passing. The tests may need some specific Gecko reftest manifest annotation or something to get our tests to green. Once we get the tests to green, we can then merge them into mozilla-central. This last merge would be a manual step.

Goal

  • Incorporate W3C-hosted CSS reftests into our automated testing framework
  • Have a place and process for devs to land tests that should be submitted to W3C
  • Keep our copy and W3C's copy of the tests in sync

Non-Goal

  • Not to solve every corner case. If the W3C ends up changing the name of the test between the "submitted" and the "accepted" step, we may lose the ability to track the test, and we'll have to deal with that manually.

More Info

Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Projects/W3C_CSS_Test_Mirroring&oldid=362276"