Testing the Web Platform With WebDriver

Testing the
Web Platform
With WebDriver

Andreas Tolfsen

Tools- and Automation, Mozilla
SeleniumConf 2014 in Bangalore

Improving the Web

Testing for
a Better Web

The Web Platform

In short: All cross browser technologies that browsers like Firefox, IE, and Chrome build and ship.

Collection of web technology standards developed by WHATWG, W3C, IETF, Ecma International, and Unicode Consortium.

The Web Platform: Core Technologies

Core platform consists of:

HTML, DOM, ECMAScript/JavaScript, URL, XHR, CORS, CSS, Encoding

The Web Platform: Detailed (1/4)

Foundations: HTTP, TLS, cookies, origin, unicode, MIME sniffing

CSS: animation, fonts, positioning, media queries, multi-column, shadows

Graphics and typography: canvas, WebGL, SVG, WOFF, MathML

The Web Platform: Detailed (2/4)

Media: web audio

Events/messaging: notifications, cross-document messaging, channels, fullscreen, geolocation, device orientation, DOM events, input, battery, vibration, beacon, media capture, clipboard

Storage and files: IndexDB, web storage, file API, blob URLs

The Web Platform: Detailed (3/4)

Real-time communication: WebRTC, WebSocket, server-sent events

Web Components: custom elements, shadow DOM, HTML imports, templates

Performance and analysis: web workers, page visibility, user timing, performance timeline, high resolution time

The Web Platform: Detailed (4/4)

Security and privacy: content security policy, crypto, DNT

Other: DOM parsing, XPath, quirks, JSON, service workers, selectors, data URLs, drag and drop, ARIA, defer, async for scripts, dataset, WebIDL, WebDriver

Complexity of the Web

Can We Do Better?

Automated

Vendor Neutral

Run in Continuous Integration

That Anyone Can Contribute To

An Open Source
Project for Tests

http://testthewebforward.org/

Web Platform Tests

What Do You Test In a Specification?

  1. Look for conformance requirements in the spec
  2. Some statements are unambiguously requirements
  3. Some statements are candidate requirements
  4. Requirements are often stated within algorithms
  5. A single requirement usally requires multiple tests
  6. A single test case may test a combination of requirements
  7. In short, it's complicated (-:
Quoted from Mike Smith's Testing Howto

Specs lie

Test Types

  • Script test (testharness.js)
  • WebIDL test
  • Reftest
  • Manual test
  • WebDriver implementation test

Script test

When the result of your test ca be determined by a script.

<!DOCTYPE html>
<meta charset=utf-8>
<title>Document.title</title>
<script src=/resources/testharness.js></script>
<script>
	test(function() {
		assert_equals(document.title, "Document.title")
	}, "Document.title matches the document's title");
</script>

testharness.js: Assertions

assert_true, assert_false, assert_equals, assert_not_equals, assert_approx_equals

assert_in_array, assert_array_equals, assert_regexp_equals

assert_own_property, assert_inherits, assert_idl_attribute, assert_readonly, assert_throws, assert_unreached, assert_any

testharness.js: Asynchronous Testing

Intended for:

  • callback functions: requestAnimationFrame(callback), getCurrentPosition(callback), &c.
  • event handlers: addEventListener("…", listener)

testharnessjs: Async Test

var t = async_test("onload event fires");

window.onload = function(ev) {
	t.step(function() { assert_equals(ev.type, "load") });
	t.done();
};

WebIDL Test

<pre id=idl>
partial interface Document {
	readonly attribute boolean hidden;
	readonly attribute DOMString visibilityState; 
};
</pre>

<script>
var idl = new IdlArray();
idl.add_idls(document.getElementById("idl").textContent);
idl.add_objects({Document: ["window.document"]});
idl.test();
</script>

Reftest

When the result of your test is visual.

bdo-000.html bdo-000-ref.html

WERBEH

<bdo dir=rtl>HEBREW</bdo>

WERBEH

<span>WERBEH</span>

Manual Test (1/2)

When everything else wouldn't work.

  • onvisibilitychange
  • Browser chrome events (onresize, onblur, onfocus ,onfullscreenchange)

Manual Test (2/2)

  • Web APIs: telephony, vibration, SMS, idle, screen orientation, power management, mobile connection, bluetooth, battery status, alarm, push notifications, radio, ambient light, proximity, camera, NFC, spell check
  • Visual (CSS) tests:
 

If the square is still red, please indicate that the test has failed.

Client ←→ Server Test (1/2)

Some things can only be tested by speaking to a server. And some times you need server modifications.

  • XMLHttpRequest, WebSocket
  • Server-side behaviour (redirects, status codes)
  • Manipulating connection data

Client ←→ Server Test (1/2)

wptserve allows you to write a .py file that has complete control over the HTTP response you get back from the server.

def handler(request, response):
	# Don't implicitly add HTTP headers:
	response.add_required_headers = False
	response.writer.write_status(200)
	response.writer.write_header("Content-Type", "text/plain")
	response.writer.end_headers()
	response.writer.write("Some ")
	time.sleep(1)
	response.writer.write("example content")

Implementation Requirements (1/2)

  • Tests must be pulled from upstream
  • Need to store the expected result of each test
  • Need to be super-robust against misbehaving tests
  • Can't require things that only make sense in vendor context on the upstream repos
Quoted from James Graham's wptrunner talk

Implementation Requirements (2/2)

  • Automation easy for browser vendors to use
  • Cross-browser and cross-platform
  • Possible to run tests from W3C's w-p-t repo
  • Run fast
  • Run in “normal” browser
  • Misbehaviour
  • Machine readable test output
Quoted from James Graham's wptrunner talk

Solving the Problems

  • Push test scheduling into harness
  • Machine readable output solved through structured logging
  • Communicate with the browser via some control protocol, such as WebDriver

General Design

  • Python test runner, which schedules tests
  • Typically uses WebDriver to drive the browser
  • For JS tests
    • Open a window at the start of the testrun
    • executeAsyncScript to start a test
    • callback to return the results
  • Reftests are even simpler
    • Use normal WebDriver APIs
Quoted from James Graham's wptrunner talk

wptrunner Architecture

  • TestManager instance owns a Browser and a TestRunner
  • Browser is an abstraction over different products; one subclass for each product
  • TestRunner owns a TestExecutor that actually knows how to run the tests
  • TestExecutor has a subclass per test type and method of executing tests
  • This provides the two axes of extensibility
Quoted from James Graham's wptrunner talk

Current Status

  • Code on https://github.com/w3c/wptrunner
  • Works for Firefox and B2G/Firefox OS (via Marionette)
  • And Servo (using one process per test)
  • And Chrome (via WebDriver)
  • Microsoft are adding IE support

Moving Forward

What about test cases that can't be automated using traditional means? Interaction, low-level device access, visuals

WebDriver can do all of these.

Un-Automatable Test Case

onkeydown, onkeypress, onkeyup events attached to document.

Tests if the default focus is on document.documentElement and that it receives the event.

No way in JavaScript to detect exactly which element has focus and the event will bubble down to in this case.

http://w3c-test.org/html/editing/focus/focus-02-manual.html

//These tests can be automated once we have an uniform way to use webdriver.
var t1 = async_test("The keydown event must be targeted at the body element"),
    t2 = async_test("The keypress event must be targeted at the body element"),
    t3 = async_test("The keyup event must be targeted at the body element");

document.onkeydown = t1.step_func_done(function(evt){
  assert_equals(evt.target, document.body,
                "The keydown events must be targeted at the document's body.");
});

document.onkeypress = t2.step_func_done(function(evt){
  assert_equals(evt.target, document.body,
                "The keypress events must be targeted at the document's body.");
});

document.onkeyup = t3.step_func_done(function(evt){
  assert_equals(evt.target, document.body,
                "The keyup events must be targeted at the document's body.");
});
import os
from selenium import webdriver
from selenium.webdriver.support import wait
from selenium.webdriver.support import expected_conditions

browser = os.environ["BROWSER"].upper()
caps = webdriver.DesiredCapabilities.get(browser)

driver = webdriver.Remote("http://localhost:4444/wd/hub", caps)
driver.get("http://w3c-test.org/html/editing/focus/focus-02-manual.html")
driver.send_keys("a")

ew = wait.WebDriverWait(driver, 3)
ew.until(expected_conditions.presence_of_element_located((By.ID, "summary")))

result_el = driver.find_element_by_css("section#summary p span")
result = result_el.text.upper()

drive.quit()
print >> sys.stderr, result
sys.exit(int(not(result == "PASS")))

The Problem With People

  • They don't want to learn new tools
  • They don't want to learn a new framework
  • They don't want to learn a new language

… and that's alright!

Inception

  • A remote WebDriver is in all simplicity an HTTP server
  • We already have a WebDriver session
  • Use case for WebDriver is often in combination with assertions that can be made using testharness.js
  • What if… we had an interface from the JS test to the WebDriver controlling the browser session?

webdriver.js (proposal)

In-document Javascript controlling its own browser session from an out-of-process driver through a WebDriver remote server.

alert(driver.title);

el = document.querySelector("#foo");
driver.WebElement(el).sendKeys("Hello SeConf!");

for (var win of driver.windowHandles) { … }

Interop Through Testing

Thanks!

Andreas Tolfsen
ato@mozilla.com
@tolfsen