TL;DR: JavaScript Unit Testing with VSTS using real browsers.
- We would like to run JavaScript unit tests;
- And we prefer a real browser for this, so no PhantomJS or equivalent;
- And run our tests from a Visual Studio Team Services build;
- VSTS hosted build agents don’t have Chrome or Firefox installed;
- So things break;
- We fix this by providing a private build agent and our own Selenium standalone server;
- Using Docker of course.
![😊]()
The Setup
So let’s start with a unit test that we will use to demonstrate our problem. There are a few ways to write unit tests (and a couple of frameworks that can help us), but I will not cover any options in detail here. In this case I will work with Karma as a test runner, Jasmine as the test framework, Google Chrome as the browser and write the actual unit test in TypeScript too.
Jasmine: https://jasmine.github.io
Jasmine is our test framework. It allows us to describe our tests and run on any JavaScript enabled platform. Simply add it to our project using npm:
npm install jasmine-core --save-dev
And since we are going to write our tests using TypeScript, we need to install the correct typings:
npm install @types/jasmine --save-dev
Karma: https://karma-runner.github.io
Karma is our test runner and responsible for executing our code and unit tests using real browsers.
npm install karma --save-dev npm install karma-cli --save-dev npm install karma-jasmine --save-dev npm install karma-chrome-launcher --save-dev
Karma-TypeScript: https://www.npmjs.com/package/karma-typescript
This enables us to write unit tests in Typescript with full type checking, seamlessly without extra build steps or scripts. As a bonus we get remapped test coverage with karma-coverage and Istanbul.
npm install karma-typescript --save-dev
The Test
We have a single component based on a “Hello World” service.
import { IHelloService } from "./hello.service.interface";
export class HelloComponent {
constructor(private helloService: IHelloService) {}
public sayHello(): string {
return this.helloService.sayHello();
}
}
And the unit test that looks something like this:
import { HelloComponent } from "./hello.component";
import { IHelloService } from "./hello.service.interface";
class MockHelloService implements IHelloService {
public sayHello(): string {
return "Hello world!";
}
}
describe("HelloComponent", () => {
it("should say 'Hello world!'", () => {
const mockHelloService = new MockHelloService();
const helloComponent = new HelloComponent(mockHelloService);
expect(helloComponent.sayHello()).toEqual("Hello world!");
});
});
The Error
Locally this works fine if you have Google Chrome installed and add the following karma config:
module.exports = function(config) {
config.set({
frameworks: ["jasmine", "karma-typescript"],
files: [
{ pattern: "src/**/*.ts" }
],
preprocessors: {
"**/*.ts": ["karma-typescript"]
},
reporters: ["dots","karma-typescript"],
browsers: ["Chrome"],
singleRun: true
});
};
If we run our Karma test locally, all is well and Chrome launches to run our unit test.
For this code sample, please see this link: https://github.com/yuriburger/karma-chrome-demo
Next step is to try this on VSTS so let’s create a build. Our build is very simple and runs 3 simple “NPM” tasks:
- Npm Install: the usual node modules installer;
- Npm run build: just runs the TypeScript Compiler;
- Npm run test: this will run our Karma test.
If we save and queue this build, we will end up with an error basically telling us: “No Chrome for you”.
The Fix
No Chrome on the Build Agent, so we need to jump a couple of hoops. The idea is, that we launch a Chrome browser on a remote server (i.e. not on the build agent) and let the remote server connect back to our Test Runner to perform the unit tests. There are some challenges with this approach:
- Network connection. We cannot easily allow a remote server to connect back to a hosted build agent for a couple of reasons (being ports/firewalls/etc.);
- We need a server implementing the WebDriver API to drive our browser automation;
- And a way to launch a remote browser from our Karma test.
For the first two challenges we will bring our own infrastructure (a private build agent and a server hosting the WebDriver API). For the remote browser part, we can easily extend Karma to support his to let’s start with that:
Karma-Webdriver-Launcher: https://www.npmjs.com/package/karma-webdriver-launcher
Remotely launch webdriver instance.
Npm install karma-webdriver-launcher –save-dev
This wires up Karma and a remote Webdriver instance, basically an API to drive browser automation. The way this works, is that during the build (1) Karma starts a Test Runner Server (2) and instructs the launcher (3) to connect to a remote Webdriver instance (4) to fire up a browser (5) and connect back to the Karma Test Runner Server (6) instance to perform the tests.
So the two remaining missing pieces are the build agent and the WebDriver API host. For both we will use docker containers to avoid having to install any software manually. Microsoft provides an official image for the VSTS Agent on Docker Hub and as far as a WebDriver API host: Selenium provides a nice implementation, also available on Docker Hub.
If you are new to Docker make sure you have met all the requirements for running containers on your favorite platform. See https://docs.docker.com/engine/docker-overview/ for more information.
Since we need the two machines communicating to each other, I like to create an isolated network first where we add our named images:
docker network create vsts-net
We can then run our Selenium container on this network:
docker run \ -d -p 4444:4444 \ --shm-size=2g --name webdriver \ --net vsts-net selenium/standalone-chrome
And eventually our private build agent:
docker run \ -e VSTS_ACCOUNT=<accountname> \ -e VSTS_TOKEN=<token> \ -it --name agent \ --net vsts-net microsoft/vsts-agent
Important parts:
- Selenium runs on the default port 4444, but you can modify the mapping
- The shm-size=2g switch is to avoid Chrome crashes and uses the hosts memory (https://bugs.chromium.org/p/chromium/issues/detail?id=519952 for more information)
- The VSTS container needs your VSTS accountname and an Access Token which you can create on visualstudio.com
- The Selenium container is accessible using the name “webdriver” and the VSTS container using the name “agent”
The VSTS agent container automatically connects to your VSTS account and downloads the correct agent version. After this it registers itself so we can target our builds to this particular agent. Please note: when used with TFS, make sure you use an image that matches the installed TFS version.
Lastly we need to update the Karma configuration to enable the WebDriver launcher and add the required hostnames and ports. Remember that we declared our hostnames (webdriver and agent) when we started our docker containers.
Karma.conf.js
module.exports = function(config) {
var webdriverConfig = {
hostname: 'webdriver',
port: 4444
}
config.set({
hostname: 'agent',
port: 9876,
config: webdriverConfig,
frameworks: ["jasmine", "karma-typescript"],
files: [
{ pattern: "src/**/*.ts" }
],
preprocessors: {
"**/*.ts": ["karma-typescript"]
},
reporters: ["dots","karma-typescript"],
browsers: ["ChromeSelenium"],
customLaunchers: {
ChromeSelenium: {
base: 'WebDriver',
config: webdriverConfig,
browserName: 'ChromeSelenium',
flags: []
}
},
singleRun: true
});
};
And if we now run our build from VSTS all tasks should complete nicely.
So eventually we had two start a private build agent and a server running Selenium. But by using Docker containters almost no effort was involved and this setup can easily be moved back to the cloud by leveraging Azure Container Services, Docker Cloud, etc.
For the source code used in this blogpost, see the following:
https://github.com/yuriburger/vsts-selenium-demo
https://github.com/yuriburger/karma-chrome-demo
/Y.