The master copy of each W3C test is kept in a test repository. So the journey of a W3C CSS test begins when the test is checked into the CSSWG repository.

As soon as the test lands in the repository, our test management system, code-named Shepherd, automatically creates a record for it. This system reads in the metadata from the test, allowing people to run searches for specific tests, tests written by a particular contributor, tests for a particular section of the spec, etc. It also tracks the status of each test and allows people to comment on it. This let's us see whether it has been reviewed, what problems have been reported against it, etc.

Then each night, the tests are compiled into a nightly build. Occasionally the CSSWG will save a snapshot build. For example, when a spec goes to REC, we archive the version of the test suite that qualified the spec to exit CR.

The built test suite is hooked into a manual test harness. This allows someone—or a team of people—to run the manual tests and easily record the results. The system can also import results from a tab-separated data file, which is useful for reporting results from automated test runs and for importing results compiled using a different test runner.

The collected test results are saved into a database, which can generate reports. We use these to generate implementation reports for exiting CR, and we also have a script that can annotate each spec with the implementation status as evaluated through the tests in the suite.

If you are interested in using this data, contact Peter Linss with questions; we encourage people to reuse the data and make it more useful to the world.

The next few sections dive deeper into the various systems. Feel free to trim down to fit into time / interest constraints.

Ok, so now that we've taken a tour of what the various systems are, we'll take a quick tour into each system to see what it does and how it operates.

The most critical piece of W3C's test infrastructure, and the one that is common to all of our test suites, is the test repository. This is where the master copy of all the tests are kept. Rather than using just a database or a filesystem, we keep them in a version control system. Originally we used CVS, then migrated to Subversion. Right now we're using Mercurial, though some groups are experimenting with Git.

A version control system has two main features: it records a history of the the changes to the repository, so that you can go back and examine earlier versions or revert changes that turned out to be a bad idea; and it handles the conflicts resulting from simultaneous edits.

In the case of Mercurial—or distributed version control in general, including systems like Git—conflicts are handled by treating each person's edits as a branch off the main history trunk; when two branches are brought together in the master repository, they are merged together. Most often the branches can be automatically merged, but if both of them changed the same files in the same place, this is considered a conflict and the person doing the merging has to examine both sets of changes to figure out what the merged result should look like.

The first step in working with Mercurial is cloning the repository. This basically downloads a copy of it to your local filesystem. Once you've done that, you can just edit files as usual; Mercurial knows which files are part of the repository, and can track those changes. The only difference is that when you add, move, rename, or delete files you have to do it through Mercurial so that it knows about those changes.

When you want to create a savepoint in your work, or share it with the master repository, you "commit" your changes as a "changeset" and give it a comment. Then you can always revert back to this save point, or check in the changeset by "pushing" it to the master repository. Once you've pushed your changeset, your changes will show up in the master repository for others to download.

Of course, while you are doing this other people are probably also making changes and sending them to the master repository! So be sure to periodically "pull" from the master repository. This will find any new changesets and download them. The --update or --rebase flags tell Mercurial to merge them into your working copy.

If someone pushes a changeset after you have committed one, but before you've pushed it to the master repository, Mercurial will consider your work to be a different branch than the current tip of the repository. In this case you'll need to merge. If there are conflicts during the merge, or during an update or rebase operation, you'll need to use the resolve commands to tell Mercurial when you've handled them.

There's an introductory primer for Mercurial on the CSS Working Group wiki; I strongly recommend reading it if you're going to use Mercurial directly.

Showing off parts of a test record in Shepherd...

As you can see, the metadata in the test is extracted and shown here, along with link to the test in the repository, and a preview of the test and its source code. Shepherd's database also stores a status code, which can be modified here, and any comments that have been entered against the testcase in this system. Some information from the Mercurial repository is also extracted: the people who have checked in changes to the file, and a history of all the changes and their checkin comments. This history is interleaved with the comments, so you can see, for example, the original source of the test, any comments were added about problems in the test, and the changes were checked in to address those comments.

Since Shepherd stores all of this information in a database, it makes it easy to search for the set of tests matching some criteria. For example, I can search for all the tests fantasai has added, that belong to the CSS3 Backgrounds and Borders test suite, and that need work. This returns a list of tests I need to work on. If I want to, I can then further filter this search, e.g. looking only at tests for 'background-position', because that's what I want to work on today.

Or, I could search for all the tests for 'background-position' that still need review. Unfortunately, the system still can't modify tests, so if I want to mark a test as reviewed, I have to open it in my copy of the repository, mark it with my name, and check that in.

While showing off the index...

The CSSWG build system takes as input of all the source tests, which are individually either in HTML or XHTML, and reads them into a DOM, and then serializes them back out in both HTML and XHTML; and, in some cases, other formats, too. It also builds a table of contents for the test suite, associating each test with every section for which it is relevant; and generates a reftest manifest file so that reftests can be run in an automated browser harness. The end result is put on the CSSWG server, and also archived into a zip file that testers can download. Note that tests which rely on HTTP headers cannot be run locally; they only work when served off an Apache server.