First, great work on this implementation. I can't imagine all the work that must have gone into this.
To me, this seems to only solve a corner case in the main problem of JS testing - testing the DOM without a browser. The first question that comes to mind is, is a DOM even useful in a headless browser when it comes to JS? For now, grant that we're only testing presence and manipulation of DOM elements and no CSS manipulations or accuracy -- which I'd argue is a decent case for testing. Now, testing presence and manipulation of DOM elements in the abstract isn't very useful, as different browsers might treat DOM manipulations differently. Using a headless browser that no one actually uses doesn't solve this problem.
I'd see a case for this in running something like QUnit[1] test that just test JS completeness, but again, implementations differ so testing in one environment that no one actually is using isn't good enough.
The only thing that I've come up with for a solution for actually testing JavaScript in a robust, real-world way is using QUnit and Selenium[2] to define the tests, with Testswarm[3] actually running the tests. This setup allows unit tests to be run against JS by itself (QUnir), mock interactions to take place to ensure bindings work properly (Selenium), and have these executed against real browsers in a distributed manner (Testswarm).
I concede that this isn't an ideal solution, and I wish there were more people working on solutions like Zombie.js that would combine all of these in one so that JS could finally be apart of our continuous integration testing. The only (sic) thing preventing this solution is that it needs to have headless, identical implementations of all the browser's JS engines and DOM parsers in order to be valid.
Reliable headless DOM implementations are one of major obstacles for JS as a first class language. Kudos and thanks to everyone that is working to solve this!
Does this load external styles, or javascript files? What about the images and does it evaluate any script tags?
I have done a few albeit scattered attempts at headless or semi-headless html/js unit tests. What I found out to be a good middle ground is running is this:
Run selenium server and firefox on a remote server. Trigger tests via a task queue, which at the moment is Gearman, have firefox draw to a virtual frame buffer.
What that does for me is cover pretty much every case that a real browser does, and completeness is key in my point of view. I would like to see how Zombie compares on the completeness side and then speed.
As I understand it, the JS DOM implementation does not have the quirks of the different browsers. So theoretically, you could write a test that passes and works in browser X but does not work in browser Y.
My understanding is that jQuery spends most of its time dealing with these quirks.
But it seems like you'll have to duplicate a lot of functionality and as soon as there are a couple of things you can't test against or that don't produce the exact same result then this becomes unreliable.
Simulated-browser tests are more like smoke tests. It's an approximation of the environment you're running tests in.
Zombie and other simulated-browser systems can be useful because they're faster and convenient, but especially because they're faster. If you're not able to test quickly, like the article says, you lose the flow.
Nevertheless, you'll be burned if you don't also test in the browsers you're supporting.
davnola, posted above, has the right idea: use Selenium with your CI system in addition to Zombie.
Another idea: make real-browser testing for development faster. I'm working on a tool that does just that, if you're curious:
Like it's peers, my project won't work for everyone, yet. But it's awesome to see so many people caring about testing web apps in dev quickly. Hooray! :)
Definitely! But it would then also require a CSS renderer. Automated testing of something like Decal CMS (decalcms.com) where we provide javascript which manipulates the DOM but allows the designer to plug whatever markup/CSS they like into the system means that automated testing is incredibly difficult and does rely on actually checking for the position (and other computed styles) of an element after we've run certain pieces of code.
To me, this seems to only solve a corner case in the main problem of JS testing - testing the DOM without a browser. The first question that comes to mind is, is a DOM even useful in a headless browser when it comes to JS? For now, grant that we're only testing presence and manipulation of DOM elements and no CSS manipulations or accuracy -- which I'd argue is a decent case for testing. Now, testing presence and manipulation of DOM elements in the abstract isn't very useful, as different browsers might treat DOM manipulations differently. Using a headless browser that no one actually uses doesn't solve this problem.
I'd see a case for this in running something like QUnit[1] test that just test JS completeness, but again, implementations differ so testing in one environment that no one actually is using isn't good enough.
The only thing that I've come up with for a solution for actually testing JavaScript in a robust, real-world way is using QUnit and Selenium[2] to define the tests, with Testswarm[3] actually running the tests. This setup allows unit tests to be run against JS by itself (QUnir), mock interactions to take place to ensure bindings work properly (Selenium), and have these executed against real browsers in a distributed manner (Testswarm).
I concede that this isn't an ideal solution, and I wish there were more people working on solutions like Zombie.js that would combine all of these in one so that JS could finally be apart of our continuous integration testing. The only (sic) thing preventing this solution is that it needs to have headless, identical implementations of all the browser's JS engines and DOM parsers in order to be valid.
[1] http://docs.jquery.com/Qunit [2] http://seleniumhq.org/ [3] https://github.com/jeresig/testswarm/wiki