found drama

get oblique

headless JavaScript unit testing with Jasmine and PhantomJS

by Rob Friesel

Yesterday, I gave a talk (slides are here, and/or rate my talk here) at the fourth Vermont Code Camp on how to run fast, reliable, headless unit tests for JavaScript using the Jasmine BDD test framework and PhantomJS. What follows here is a summary of “the meat” of the talk: specifically, how to execute your Jasmine-based test suites using PhantomJS as the runtime environment.1

@founddrama giving his talk

Photo credit to Josh Sled, 2012. (source)

Let’s assume that you’re a front-end developer, and let’s further assume that you’ve already looked into this subject at least a little bit. We’ll skip the primer on unit testing, because you already know what that is, and what its benefit are. We’ll further assume that you’ve read Ben Cherry’s blog post, “Writing Testable JavaScript” which, while not necessarily an exhaustive post on the subject, is an easy-to-digest post with just the right amount of information, and just the right tone. So all that being said, our basic assumptions for the rest of this post:

  1. You know what unit testing is;
  2. You’re writing testable JavaScript;
  3. You’re already writing some unit tests for your JavaScript;
  4. You’re frustrated by “playing the browser refresh game”; and
  5. You want to automate your tests, possibly even as part of your CI2 strategy.

With these assumptions in place, let’s ask the obvious question: How would we go about setting up some automated headless unit tests?

Why Jasmine?

Jasmine

To start on this journey of automating your unit tests with PhantomJS, you’ve got to have a suite of tests. Though you could roll your own assertion framework, there are plenty of existing JavaScript testing frameworks out there already.3 In other words: this is a solved problem. The testing frameworks that are out there run the spectrum with respect to features, maturity, and integrations with specific libraries or other frameworks. It’s a blog post in and of itself just to do a side-by-side comparison of even just a handful of them. At some point you need to just pick one that works for you and go with it.

I chose Jasmine.

You may be asking: Why Jasmine?

I first became exposed to Jasmine through Rebecca Murphey’s screencast where she talks about using it to test jQuery-based code. I read through the docs; I read through some of the source code; and I read a number of blog posts, articles, and forum threads, all discussing Jasmine in one capacity or another. From this research, I decided that it was mature enough, that it had the features I was looking for,4 and the BDD style that it used just “clicked” with me. So when it came time for me to “just pick one”, that was the one that I picked.

Jasmine itself has the usual suspects for features: an “expect” test method; various matchers for equivalence, equality, defined (and undefined), negations, and so forth; “before” and “after” style setup and teardown functions; spies; async support; and more. Equally attractive to me, Jasmine seems to have developed a great community of users, including some intrepid folks who have built some marvelous extensions. (More on this later.) Lastly, it was sufficiently “library agnostic” for my tastes.5

It wasn’t that choosing Jasmine was a no-brainer, but it didn’t take me long to prefer it over others. As such, it provides a foundational component for our later discussion around how to automate these tests using PhantomJS.

What is PhantomJS?

PhantomJSPhantomJS is a headless WebKit with a JavaScript API that allows you to work with that browser. If you’re asking “Headless” WebKit? WTF? then try to think of it this way: it’s like Chrome with no window, no browser chrome, and an invisible viewport.6 It’s a command-line utility into which you can feed JavaScript-based programs for performing a variety of tasks that browsers and JavaScript engines are otherwise well-suited to. Observe this totally contrived example:

In that script we:

  1. Create one of PhantomJS’s WebPage objects;
  2. Load the main URL for this blog;
  3. After the landing page loads, we grab all the headlines;
  4. Output the text of those headlines to the console; and
  5. Exit from PhantomJS.

The above demonstrates just a few of the basics of the PhantomJS API. You can let your imagination run wild with this; it’s possible to use it for SVG rendering, site crawling and scraping, network monitoring, and–of course–headless testing.

Headless Testing

Because PhantomJS is “just a browser”, the tests that you already have written can (theoretically) be executed in that environment as-is. You just need to write a test runner to do the following things:

  1. Load the page that contains your tests;
  2. Wait for the tests to finish executing;
  3. Parse the HTML to identify the number of successes and failures;
  4. (Optionally) report those success and failures, writing them to the console (and/or to a file); and
  5. (Ideally) exit the PhantomJS runtime with an exit status of 0 or 1, to make it scriptable like a good UNIX-y citizen.

If you’ve followed along with the example in the Jasmine documentation, you used the jasmine.HtmlReporter (like so) to get an attractive test report in your browser window. While this reporter is much more pleasing to the eye, it arguably means you’re doing a lot more overhead in parsing the report.7 Jasmine core ships with a jasmine.TrivialReporter as well, which has some simpler output, but isn’t as attractive to look at.

Again–theoretically–we have enough to get started without changing a thing about our existing tests. But we’ll probably want more from our test runner than “just” the results. And wouldn’t it be nice to have “stripped” output in the console (i.e., in PhantomJS) and “pretty” output in every other browser?

Fortunately, we don’t need to do all this heavy lifting ourselves.

larrymyers/jasmine-reporters

Larry Myers recognized a lot of these problems opportunities long before I came along dreaming of doing something like this. Luckily, Mr. Myers created a series of Jasmine reporters (and runners!) to solve this problem, and open-sourced them on Github.

Of particular interest to us are the jasmine.ConsoleReporter and the phantomjs-testrunner.js test runner script.

In a nutshell, what Mr. Myers has given us with the jasmine.ConsoleReporter, is a Jasmine reporter which was designed explicitly for console output–not something that we’re jury-rigging along the way. Furthermore, he provides a sister reporter in the jasmine.JUnitXmlReporter, which provides a mechanism for writing JUnit-style XML reports to the filesystem, directly from the PhantomJS runtime. (Can you see where this is headed?)

Putting It All Together

Along with those reporters, Mr. Myers’ project offers us the aforementioned phantomjs-testrunner.js. Somewhat ironically, the test runner depends on the jasmine.TrivialReporter from Jasmine core, but for best results, you’ll want to add the jasmine.ConsoleReporter. Used in combination, the test runner handles loading the target test page into the PhantomJS runtime, and adds the function necessary to write the XML reports to the filesystem; meanwhile, the jasmine.TrivialReporter provides the DOM output that’s parsed for the test results, and the jasmine.ConsoleReporter outputs results to the console. And if we’re feeling adventurous, we can add in the jasmine.JUnitXmlReporter to get a test report written to the filesystem.8

On top of that, we can have the best of both worlds–an easily consumable report in PhantomJS and the console, and the “pretty” report for every other browser. We just need to do a little sniffing around in our runtime and decide which reporters to stand up. (Here’s what I did for the VT Code Camp demo.)

It works well, and it’s a pretty extensible solution, providing a lot of the right hooks to do things like fail builds from Jenkins, block git commits,9 or watch a directory of files and show Growl alerts when tests start failing. I myself have only started to scratch the surface of what’s possible here, but I’m pretty excited about what I’ve seen so far.

The Catch

This is the part where someone out there is ready to shout me down, to tell me that I’m giving everyone bad advice. This is the part where I admit that this is not the “best” solution for this kind of test automation.

But “best” is relative. If by “best” you mean “fast”, then I’m going to say that this has a chance of winning the prize for “best” automated and/or headless testing solution. However, if by “best” you mean “comprehensive”, then no, this is not the best solution to test automation. A comprehensive test automation solution is going to involve:

  • A server (e.g., Selenium) to serve as a remote “driver” for “slaved” clients;
  • A pool of such clients in a “browser lab” with a representative sample of the browsers and devices you’re targeting and/or supporting (virtualized and/or on dedicated hardware); and10
  • Everything else that you’ll need no matter what other strategy you choose (i.e., the actual test suites, the test runners, the CI server, etc.)

When all the apples and oranges are sliced, the solutions aren’t all that different–one just happens to have much broader coverage across browsers and devices. But that “much broader coverage” isn’t without its own costs. For starters, it’s more expensive to stand up a browser lab like this; not to mention the expense of maintaining such an apparatus.11 And then you have to have someone who knows how to set up such a thing and/or be willing to learn all the intricacies involved. Even assuming that you’re prepared to cope with these expenses and challenges, even assuming that you believe that the broader coverage is worth it12–you’re still still likely to be bound by a couple of things: first, that such a browser lab is probably a one-of-a-kind item in your shop, and as such, getting time in the lab is competitive and contentious; and second, if you’ve “gone all the way” and introduced functional/acceptance/end-to-end tests, then you’ll quickly see that the reality with this setup is that those kinds of tests take a long time to run, and when they run, they can tie-up your testing environment for a good long while.

Having breadth of coverage is important, but so is getting immediate feedback about your test results. The solution proposed in this post won’t solve the former problem, but it goes a long way toward solving the latter problem.

Summary and Errata

Perhaps the “tl;dr” belongs at the top, but I felt that it made sense to state “the short version” of all this once more at the end, just to drive it home: If you have a suite of JavaScript unit tests written in Jasmine, you can use a selection of reporters from Larry Myers’ jasmine-reporters, along with his test runner script, to automate the headless execution of those tests using PhantomJS. I don’t know about you, but I’m pretty excited about the possibilities opened up with such an arrangement.

I’d also like to share a couple of interesting items that came out of the discussion I had with the audience at VT Code Camp.

  • On test coverage. When we started talking about Jasmine and BDD, one member of the audience asked: “How much coverage is enough coverage? How do you know if you’re testing enough?” Naturally, I gave the naïve developer’s response’ and said “coverage reports” and went on to say that I hadn’t explored code coverage tools well enough to speak to that. However, the gentleman surprised me and explained that he did not mean code coverage. We went back and forth a little bit, but we agreed that “enough” test coverage isn’t a function of code paths, but is a level of satisfaction that the business owners have. Knowing that every line of code is tested might satisfy a developer, but stakeholders are thinking about how the software should behave and what it should do. In retrospect, his point was an almost perfect illustration as to why I prefer the BDD approach, and why I mentioned that the developers should be writing the test cases/specs with the domain experts; then the question becomes not “are we testing enough things?” but “are we testing the right things?”
  • Crowdsourced test reports. One person in the audience speculated something of a blasphemous idea: (paraphrased) “What about just delivering the test payload to the end-users? executing the tests client-side and posting the results back to you with ajax?” I can think of a bunch of reasons not to do this, but for a moment there, everyone in the room just kind of looked around as if to say: That’s just crazy enough to work! Who needs a browser lab when your users can be your browser lab? (Just kidding.)
  • “It’s still just WebKit.” More/less hand-in-hand with the section above (“The Catch”), was the astute observation by Mike Fowler that PhantomJS is “still just WebKit” and that by and large, that’s not the browser where we encounter the defects in our code. Mike cited versions of Internet Explorer that don’t support indexOf as a “gotcha”; and though I didn’t say so at the time, I immediately thought of trim. This was an excellent point to be raised, and I’m glad that he did. That being said (and if I recall correctly, I did say this), having tests, and being able to run them quickly using a tool like PhantomJS isn’t going to bail you out of every problem. You should still be linting. You should still be doing code reviews. You should still be testing across different browser runtimes. You should still have top-notch QA engineers to keep you honest. (Hopefully the above technique just means that there’s less for those QA engineers to “keep you honest” about.)

All in all, I had a great audience at Vermont Code Camp this year, and I’m grateful for their attention, and even more grateful for their fantastic questions. Thanks out to Julie Lerman and Rob Hale for organizing the event;13 it was a lot of fun, and I hope to do it again next year.

Resources

UPDATE: (July 2014) This talk and blog post became the basis for my first book: The PhantomJS Cookbook!

  1. My talk included a bunch of discussion around testing in general, unit testing in JavaScript specifically, and the landscape of JavaScript testing frameworks. (Spoiler alert: there’s a lot of them.) []
  2. ”CI” of course being “continuous integration”. If I assumed too much, and you don’t know what that is, check out Wikipedia’s entry on CI. (I’ll still be here when you get back.) []
  3. At the time of this writing, the Wikipedia article for “Unit testing frameworks” lists 35 different unit testing frameworks for JavaScript–and that’s not even an exhaustive list. []
  4. Although, to be fair: any of the “mature” testing frameworks pretty much all have the same features. []
  5. A lot of other testing frameworks seem more/less “attached” to some other library or framework. This is not to say that you can only test YUI-based JavaScript with YUI Test, nor that you can only test Dojo-based JavaScript with the Dojo Object Harness–but I really was looking for something that didn’t even have one these associations even by name. (After all: names are important.) []
  6. Faith-based browsing? []
  7. Admittedly: that’s kind of a weak argument. []
  8. Don’t forget to tell the jasmine.JUnitXmlReporter where to write those reports. (I recommend target/test-reports–but that’s just showing by background and biases.) []
  9. The code examples from my VT Code Camp talk include this example pre-commit hook. []
  10. …and if you’re going to all the trouble to set up a browser lab like this, you may as well go all-out and also set up some hard-core functional/acceptance/end-to-end tests with a tool like Geb as well. But let’s not get crazy. []
  11. Not to mention fraught with its own tangled questions: how do you keep browsers up to date? when do you let older versions drop off? how do you keep Chrome from updating itself? what about ? all those Android devices? []
  12. And don’t get me wrong: it’s almost definitely worth it, if you can afford the up-front involved. []
  13. And many thanks to Apprenda for sponsoring the speaker/volunteer party! []

About Rob Friesel

Software engineer by day, science fiction writer by night. Author of The PhantomJS Cookbook and a short story in Please Do Not Remove. View all posts by Rob Friesel →

22 Responses to headless JavaScript unit testing with Jasmine and PhantomJS

Asha12345 says:

When running PhantomJS script on Jenkins, following error is thrown: “_RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL”. Any idea how to resolve this?

found_drama says:

@Asha12345: Sorry, can’t say that I’ve seen that one before. That looks suspiciously like something coming from Jenkins, so my suggestion would be to start searching those forums. (Nothing about that looks specific to PhantomJS to me.)

UPDATE: (3/21/2014) Almost a year later: now I have seen that error message! I encountered it myself when testing a Jenkins arrangement on my OS X laptop. I found this StackOverflow thread that makes reference to it, and also this Apple support discussion thread. It seems that on OS X, the Jenkins user cannot access a UI thread (even though technically it shouldn’t need one) and so we get this error. Interestingly, the error seems to be benign, and the tests themselves should still be fine.

Ayush says:

I am able to run your phantonjs example successfully. I have one scenario, where I have to access headless browser in jasmine test case. I am trying to do this, but I am facing lot of problems.
Suppose If I open browser “blog.founddrama.net” in phantomjs, could it be possible to access :

var list = Array.prototype.slice.call(document.querySelectorAll(‘h2 > a[rel=”bookmark”]’), 0);

in my jasmine test cases . If yes , please let me know how I can do this. Thanks in advance .

found_drama says:

@Ayush: I’m not sure what you’re asking here. The example code is out of context and as far as I can tell, will work just fine — are you saying that you need code from the test cases (running in the browser) to tell PhantomJS to do something? If that’s what you’re asking then you might want to look at the window.callPhantom method; if it’s the other way around (PhantomJS calling the page) then you’ll want to look at evaluate. (I’ve linked to the PhantomJS API above in the post.)

Ayush says:

Thanks for your reply .
I am writing code for acceptance testing for one of my web application . For this, I have been suggested to use phantomjs and jasmine.

my phantomjs is :

phantom.injectJs(“../lib/utils/core.js”)

if ( phantom.args.length !== 2 ) {
console.log(“Usage: phantom_test_runner.js HTML_RUNNER RESULT_DIR”);
phantom.exit();
} else {
var htmlrunner = phantom.args[0],
resultdir = phantom.args[1],
fs = require(“fs”),
page = new WebPage();

// Echo the output of the tests to the Standard Output
page.onConsoleMessage = function(msg, source, linenumber) {
console.log(msg);
};

page.open(“file://” + fs.absolute(htmlrunner), function(status) {
console.log(htmlrunner);
if (status === “success”) {
utils.core.waitfor(function() { // wait for this to be true
return page.evaluate(function() {
return typeof(jasmine.phantomjsXMLReporterPassed) !== “undefined”;
});
}, function() { // once done…
// Retrieve the result of the tests
var suitesResults = page.evaluate(function(){
return jasmine.phantomjsXMLReporterResults;
});

// Save the result of the tests in files
var f = null;
for ( var i = 0, len = suitesResults.length; i Unable to save result of Suite ‘”+ suitesResults[i][“xmlfilename”] +”‘”);
}
}

// Return the correct exit status. ‘0’ only if all the tests passed
phantom.exit(page.evaluate(function(){
return jasmine.phantomjsXMLReporterPassed ? 0 : 1; // Could not load ‘” + htmlrunner + “‘.”);
phantom.exit(1);
}

One of sample jasmine test case is :
describe(
‘Sample Jasmine test cases’,
function() {

it(
“When enter key is pressed , return true.”,
function() {
expect(true).toEqual(true);
});
});

I am able to run this using below command :
phantomjs igo/src/test/jasmine/phantomjs_jasminexml_runner.js igo/src/test/jasmine/test_runner.html xml_output_dir/

This is working fine. But What I have to test in actual, the scenario is :

I have to access DOM elements of running application in my jasmine test cases .
My application is running on local host . I am using jetty server for that . For starting application , I am using mvn jetty:run .
Now my questions are :
1) Can we access DOM elements of running application in jasmine test cases? If yes , please let me know how I can do this .
2) For running test cases, should I need to start jetty separately or is there any process in jasmine/phantomjs to start jetty and stop after passing test cases ?

Thanks in advance .

Ayush says:

Right, But I am not getting the exact solution, which I am looking for.
May be I am not able to put my question properly so that it can be explained to my actual requirement.
It will be great, if you can me help me out on this topic. If you want me to elaborate my question a little more, please let me know.

raghavan says:

Hi, i have tried your code bundle as mentioned in README.md but I could not get the sample test spec to work. It looks like the testrunner.js requires jasmine as well and I am not sure how to make Jasmine available to the testrunner.js. Any help is appreciated.


D:\vt-code-camp-2012>phantomjs lib/jasmine-reporters/test/phantomjs-testrunner.j
s D:\vt-code-camp-2012\test\all-tests-spec.htm
ReferenceError: Can't find variable: jasmine

phantomjs://webpage.evaluate():3
phantomjs://webpage.evaluate():9
phantomjs://webpage.evaluate():9
ReferenceError: Can't find variable: jasmine

phantomjs://webpage.evaluate():3
phantomjs://webpage.evaluate():9
phantomjs://webpage.evaluate():9
ReferenceError: Can't find variable: jasmine

found_drama says:

@raghavan — My guess is that you either forgot to pull in Jasmine as a submodule (a common mistake!) or else the submodule did not update properly. I would try git submodule init && git submodule update again, paying close attention to whether the submodules come in OK or not. If they continue to fail then you could always just got and download Jasmine the old fashioned way and stick it at the paths specified in the .gitmodules file.

found_drama says:

@Kevin– Thanks. I’ve updated it to use the more modern require('webpage') syntax and to drop the encodeURI part. So it should now work in versions 1.7.0 of PhantomJS and above.

Edward says:

Hi, I’ve tried your sample and it works fine for me. Now I want to write jasmine test with phantom functions. For example, my test file is:
var page = require(‘webpage’).create();
page.open(‘http://google.com’, function () {
page.render(‘google.png’);
phantom.exit();
});
Then I try to verify page.open is called in the spec. Run the test I got an error says “Can’t find variable: require”. I notice you mentioned window.callPhantom earlier, not sure if this is the solution to my problem, also not sure how to use this method. Any help is appreciated.

found_drama says:

@Edward — how are you executing this script? The only thing that jumps out at me is that you’re callback will try to execute in the page context and not the PhantomJS context — but that also wouldn’t explain the error you’re getting. (What’s this about a spec? Sounds like there’s more to the story.)

Edward says:

Hi, thanks for the reply. So all I did is under vt-code-camp-2012 folder:
1. Added a tmp.js under src/js contains:
var page = require(‘webpage’).create(); page.open(‘http://www.google.com’,function() {
page.render(‘http://www.google.com’);
phantom.exit();
});
2. Added a tmp-spec.js under test/js
describe(“Tmp test”, function() {
it(“calls the create() function”,function(){
expect(require.create).toHaveBeenCalled();
});
it(“calls the open() function”, function() {
expect(page.open).toHaveBeenCalled();
});
it(“calls the render() function”, function() {
expect(page.render).toHaveBeenCalled();
});
});
3. Then in all-tests-spec.htm added:

4. In the created folder, executed: phantomjs lib/jasmine-reporters/test/phantomjs-testrunner.js $(pwd)/test/all-tests-spec.htm

Please let me know is you need more information.

found_drama says:

@Edward–

I’ve never seen this before, so I can’t say for sure without digging pretty deeply, but here’s my educated guess: tmp-spec.js is loaded by all-tests-spec.htm, which is running in the browser context where require is not exposed (vs. the PhantomJS context where it is).

Hope that helps!

Edward says:

Then how could I run the tests in the PhantomJS context but still be able to group tests in the Jasmine way? I guess that is the aim we want PhantomJS and Jasmine working together.
Could window.callPhantom be a solution? I tried window.callPhantom(require(‘webpage’).create()) but didn’t work out, maybe I’m not using it right.

Edward says:

It seems Jasmine does not work well outside of a browser and we can’t control Phantom with Jasmine.

The answer could be to swap Jasmine for Mocha. Here is a Mocha BDD test (a bit different to Cucumber though):

var User = require(“user”);
describe(‘Creating a new User’,function(){
var user;

before(function(done){
User.create({ username: ‘test’ , function(err,u){
user = u;
done();
});
});

it(‘should have a username’,function(){
user.should.have.property(‘username’,’test’);
});
});

Mocha can work nicely with PhantomJS too – see https://github.com/metaskills/mocha-phantomjs

Andrew M. Farrell says:

This post is a little frustrating because I came to it trying to figure out some way to run jasmine tests headlessly. Unfortunately for me, it doesn’t tell me the answer but gives me more places to look. If you actually did intend to give the answer to “how do I run jasmine tests headlessly”, then you have included not enough actual commands or code.

otherwise, great post.

found_drama says:

@Andrew — Sorry it wasn’t more comprehensive. I just published a book on this subject though, and I have to imagine that it has what you’re looking for.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

*