Here at Test the Web Forward, we're going to handle the submission process for you, just to streamline things so that you don't have to figure out how to use all our tools right away. But I'm going to give you an overview of the systems we use so that if you want to continue contributing, you'll have a good idea of how this all works.
This presentation is geared towards the CSS Working Group's process. The process for other groups is similar, but they'll be running under slightly different systems due to differences in the test formats and the fact that the CSSWG started building out these systems before equivalents existed at W3C.
The master copy of each W3C test is kept in a test repository. So the journey of a W3C CSS test begins when the test is checked into the CSSWG repository.
As soon as the test lands in the repository, our test management system, code-named Shepherd, automatically creates a record for it. This system reads in the metadata from the test, allowing people to run searches for specific tests, tests written by a particular contributor, tests for a particular section of the spec, etc. It also tracks the status of each test and allows people to comment on it. This let's us see whether it has been reviewed, what problems have been reported against it, etc.
Then each night, the tests are compiled into a nightly build. Occasionally the CSSWG will save a snapshot build. For example, when a spec goes to REC, we archive the version of the test suite that qualified the spec to exit CR.
The built test suite is hooked into a manual test harness. This allows someone—or a team of people—to run the manual tests and easily record the results. The system can also import results from a tab-separated data file, which is useful for reporting results from automated test runs and for importing results compiled using a different test runner.
The collected test results are saved into a database, which can generate reports. We use these to generate implementation reports for exiting CR, and we also have a script that can annotate each spec with the implementation status as evaluated through the tests in the suite.
If you are interested in using this data, contact Peter Linss with questions; we encourage people to reuse the data and make it more useful to the world.
The next few sections dive deeper into the various systems. Feel free to trim down to fit into time / interest constraints.
Ok, so now that we've taken a tour of what the various systems are, we'll take a quick tour into each system to see what it does and how it operates.
The most critical piece of W3C's test infrastructure, and the one that is common to all of our test suites, is the test repository. This is where the master copy of all the tests are kept. Rather than using just a database or a filesystem, we keep them in a version control system. Originally we used CVS, then migrated to Subversion. Right now we're using Mercurial, though some groups are experimenting with Git.
A version control system has two main features: it records a history of the the changes to the repository, so that you can go back and examine earlier versions or revert changes that turned out to be a bad idea; and it handles the conflicts resulting from simultaneous edits.
In the case of Mercurial—or distributed version control in general, including systems like Git—conflicts are handled by treating each person's edits as a branch off the main history trunk; when two branches are brought together in the master repository, they are merged together. Most often the branches can be automatically merged, but if both of them changed the same files in the same place, this is considered a conflict and the person doing the merging has to examine both sets of changes to figure out what the merged result should look like.
Read the details at csswg.org/tools/hg
hg pull --updateor
hg pull --rebase
The first step in working with Mercurial is cloning the repository. This basically downloads a copy of it to your local filesystem. Once you've done that, you can just edit files as usual; Mercurial knows which files are part of the repository, and can track those changes. The only difference is that when you add, move, rename, or delete files you have to do it through Mercurial so that it knows about those changes.
When you want to create a savepoint in your work, or share it with the master repository, you "commit" your changes as a "changeset" and give it a comment. Then you can always revert back to this save point, or check in the changeset by "pushing" it to the master repository. Once you've pushed your changeset, your changes will show up in the master repository for others to download.
Of course, while you are doing this other people are probably also making changes and sending them to the master repository! So be sure to periodically "pull" from the master repository. This will find any new changesets and download them. The --update or --rebase flags tell Mercurial to merge them into your working copy.
If someone pushes a changeset after you have committed one, but before you've pushed it to the master repository, Mercurial will consider your work to be a different branch than the current tip of the repository. In this case you'll need to merge. If there are conflicts during the merge, or during an update or rebase operation, you'll need to use the resolve commands to tell Mercurial when you've handled them.
There's an introductory primer for Mercurial on the CSS Working Group wiki; I strongly recommend reading it if you're going to use Mercurial directly.
As soon as a test is checked into the repository, a record is automatically created for it in Shepherd
Showing off parts of a test record in Shepherd...
As you can see, the metadata in the test is extracted and shown here, along with link to the test in the repository, and a preview of the test and its source code. Shepherd's database also stores a status code, which can be modified here, and any comments that have been entered against the testcase in this system. Some information from the Mercurial repository is also extracted: the people who have checked in changes to the file, and a history of all the changes and their checkin comments. This history is interleaved with the comments, so you can see, for example, the original source of the test, any comments were added about problems in the test, and the changes were checked in to address those comments.
Searching for Tests
Since Shepherd stores all of this information in a database, it makes it easy to search for the set of tests matching some criteria. For example, I can search for all the tests fantasai has added, that belong to the CSS3 Backgrounds and Borders test suite, and that need work. This returns a list of tests I need to work on. If I want to, I can then further filter this search, e.g. looking only at tests for 'background-position', because that's what I want to work on today.
Or, I could search for all the tests for 'background-position' that still need review. Unfortunately, the system still can't modify tests, so if I want to mark a test as reviewed, I have to open it in my copy of the repository, mark it with my name, and check that in.
While showing off the index...
The CSSWG build system takes as input of all the source tests, which are individually either in HTML or XHTML, and reads them into a DOM, and then serializes them back out in both HTML and XHTML; and, in some cases, other formats, too. It also builds a table of contents for the test suite, associating each test with every section for which it is relevant; and generates a reftest manifest file so that reftests can be run in an automated browser harness. The end result is put on the CSSWG server, and also archived into a zip file that testers can download. Note that tests which rely on HTTP headers cannot be run locally; they only work when served off an Apache server.
The CSSWG test harness allows for entering and querying results from a CSS test suite.
It is currently optimized for manual tests.
When you choose to enter test results for a particular test suite, it will (by default) serve up the tests for which we are missing results from your particular browser version. As you click through the tests, you can see their titles and some brief metadata at the top, some buttons at the bottom, and the test loaded in an iframe for most of the page. Manual tests include instructions for how to determine their pass or fail results; here's an example. The buttons at the bottom will send results to the server indicating your browser passed, that it failed, that you can't tell the results (often this means the test has a problem). The skip button goes to the next test without entering any results.
Results include information about your UA through the UA string, along with your IP address and some other information.
The results can be queried to generate implementation reports. The CSSWG uses these to prove interoperable implementations when it moves a spec from CR to REC. We are also connecting the results into annotations to the specs, so that you can see the implementation status of each section as determined by test results for that section.
System Administrator: Peter Linss <firstname.lastname@example.org>
Alright! That's it for our tour of the CSSWG test systems. Please use them and help us improve them!