Test harness for WebDriver specification

Mozilla recently added support in wptrunner for running WebDriver specification tests. wptrunner is the cross-platform and cross-browser test runner for the Web Platform Tests curated by the WHATWG and W3C communities.

The harness enables the writing of Python scripts using the pytest test framework to test the wire protocol described by WebDriver. I discovered that pytest lends itself well to writing the sort of tests we need for testing a protocol: It composes test dependencies in an interesting way through so-called “fixtures”, providing a refreshing break from unittest’s insistence on complex class inheritance to achieve the same self-contained effect.

The tests for the WebDriver specification have some unique requirements: In particular they need to test a protocol from an out-of-process program, and secondly they need to assert control over the browser process. They also need to make raw HTTP requests to the WebDriver remote.

For this we have written a specification-compatible WebDriver client that is vendored in WPT under tools/webdriver and picked up by wptrunner at runtime. The tests have a set of fixtures exposed by default, such as a session fixture that is lazily set up to create sessions so that certain parts of the protocol can be tested without an active session. It makes it possible to do non-session requests and send malformed payloads, that is with incorrect JSON structures or wrong type information.

Moving a W3C specification to the Recommendation stage in theory requires at least two interoperable implementations by two separate vendors. As the specification text itself is coming along nicely, there’s an increasing need for conformance tests. WebDriver has a history as an open source library available from the Selenium browser automation project, but because the specification deviates from their wire protocol in a number of unique ways, we need to ensure new implementations are not using Selenium’s body of tests as reference.

The tests are selected by selecting the test type wdspec:

% RUST_LOG=debug wptrunner \
	--metadata . --tests . --log-mach -\
	--product firefox \
	--binary objdir/dist/bin/firefox-bin \
	--certutil-binary objdir/dist/bin/certutil \
	--prefs-root source/testing/profiles \
	--webdriver-binary wires/target/debug/wires \
	--test-type wdspec \
	[TEST…]

Documentation on running the tests is available from the README file in wptrunner. Guidance on writing tests for WPT is found on Test the Web Forward’s website, although I have not yet had the time to add anything specific to WebDriver. Until we have landed a couple of tests, I don’t think we fully know what the structure of a “good” WebDriver specification test ought to look like.

The ability to write and run WebDriver tests should be great news for browser vendors. It paves the way for more interoperable implementations, which is important if we want to use it to automate tests requiring user interaction or get privileged access to browser internals.