Gaia Testing Tips

Because it is often difficult to remember how to run tests for Gaia, the front-end for Firefox OS, I’ve noted down some of my frequently used commands here. Consider this document a brain dump, made primarily for my own forgetful self.

Changes to Node.js code

The build system targets do not automatically deploy changes to source code. This appears to be the de-facto workflow with Node.js, however odd it seems. The correct workflow is to re-install the changed package each time you make a change.

You can use a tool like Gareth Aye’s npmr that will detect which local packages have changed and install only those. You call it with the install argument every time you make a change.

As far as I understand, the long term plan is to upstream this behaviour to npm itself and make it a default.

Marionette

Most of the tests I run are Marionette related. The Marionette Node.js client comes with unit- and integration tests.

Running the unit tests is fairly straight forward because they do not rely on a B2G Desktop browser binary. Yet, because they depend on the node_modules Makefile target, they still need to download the latest B2G build off Treeherder, although the tests themselves does not use it.

The integration test target encompasses all the integration tests for Gaia, including integration tests for the Marionette client. To limit the tests only to those related to the harness, one can use the HARNESS_ONLY output variable. This is essentially a predefined test list that populates TEST_FILES.

The TEST_FILES variable lets you override the tests that are picked up from the various manifests by specifying the (relative or absolute) paths to individual tests, separated by spaces.

In this case it will run the test files test, cases, separated, by, and spaces.

An important peculiarity with the Gaia test targets is that they run tests marked as ignored in manifests by default. This means you need to set TEST_MANIFEST to the path of the ignore manifest used by Treeherder (TBPL) explicitly.

Reproducible tests

Running make test-integration does not replicate how tests are run by the test slaves. To properly replicate the exact environment under which the tests are run, you should consider running them in Linux containers.

Since Gaia tests are run on Taskcluster, you can access the artifcat link from the job information pane on Treeherder, where Taskcluster gives you the exact instructions for running a test job locally on your system.

Custom Gecko builds

For integration tests or tests that require the use of a browser binary, you can specify RUNTIME with the absolute path of the binary to have the build system pick up on it.

Unfortunately it will still download the latest B2G binary even if this output variable is set.

Note that since a few months ago, you must now specify the b2g-bin binary, and not the b2g shell script that wraps it. If you do the former, and you run your tests non-headless, you will get a popup saying it cannot load your profile. If you run the tests headless, they will simply time out after a very long time.

Gecko log

To get access to the Gecko log like you would with --gecko-log - with mach, you can set the VERBOSE output variable.

Defunct processes

In Gaia, make wraps Node.js which in turn wraps a Python virtualenv, wrapping another Node.js process managing the B2G binary. In other words there are many opportunities to mess up process ownership.

It can sometimes be wise to ensure that you have no defunct/zombie processes lying around, because it happens that the Marionette client connects to old processes which will cause you much pain.