The Quest Continues: Introducing capybara-webkit

We recently blogged about our experiences and ongoing efforts to discover and improve tools for Javascript testing. In closing, we mentioned that we were fairly happy with Akephalos, but we didn’t believe the quest was over. Today, the quest continues.

Akephalos

Akephalos is a well-written driver that integrates soundly with the htmlunit virtual browser. However, as well-integrated as Akephalos is into both Capybara and htmlunit, it can only ever be as strong as htmlunit. As our applications became more sophisticated and large, we began to run into issues with htmlunit.

• Bugs: as previously mentioned, there are bugs in htmlunit, specifically with jQuery’s live. Although all browser implementations have bugs, it’s more useful if tests experience the same bugs as actual browsers.
• Compatibility: htmlunit doesn’t fully implement the feature set that modern browsers do. For example, it doesn’t fully handle DOM ranges or Ajax file uploads.
• Rendering: htmlunit doesn’t actually render the page, so tests that depend on CSS visibility or positioning won’t work.
• Performance: when most of your tests use Javascript, test suites with htmlunit start to crawl. It takes a while to start up a test with Akephalos, and a large test suite can easily take 10 or 15 minutes.

These issues made us realize that virtual browsers like envjs and htmlunit are never going to keep pace with actual browsers. If we want to write applications using features from the latest and greatest browsers, we’re going to need to test with them, too.

Selenium

That brought us back to the old mirage: Selenium. Selenium is well-supported, well-maintained, and works with real browsers. Integration with Ruby tests has improved considerably since the old Webrat days, and speed is better in some cases, as real browsers have been heavily optimized.

However, Selenium still has some ongoing issues that aren’t easy to solve:

• Timing: because Selenium uses a real browser, it doesn’t have as much control as a virtual browser does. This leads to issues with timing, causing erratic test failures.
• Overhead: real browsers come with features we don’t need, like a UI and plugins. This slows things down a bit.
• Gotchas: real browsers come with strange edge cases. Using Selenium and Firefox, sometimes the tests crash because Firefox tries to update the browser. Firefox doesn’t allow more than one instance to run at once. These gotchas are inevitable, because real browsers weren’t designed to be driven programatically.

We were forced to use Selenium because we needed all the capabilities of a real browser, but all these issues kept us looking for something better.

capybara-webkit

What we need was a real rendering engine coupled with a full Javascript and DOM implementation, but without all the cruft of a GUI browser. What we really wanted was a headless implementation of WebKit’s rendering engine that could be driven by Capybara tests. After Tristan “Websockets” Dunn showed me PhantomJs, I realized this might be possible using Qt’s WebKit implementation.

It was possible. I’d like to introduce a headless WebKit driver for capybara: capybara-webkit.

capybara-webkit has the following benefits:

• Compatibility: uses a real rendering engine (WebKit)
• Simplicity: runs headlessly, and designed to be controlled programatically
• Performance: faster than htmlunit - the test suites we’ve tried it on so far yielded 15-25% speed increases

The holy grail?

We designed capybara-webkit to solve the problems we had with other solutions, and so far it fits the bill nicely. We believe the quest is far from over, but we invite you to try out capybara-webkit on your own and see if this latest tool can take you a little further on your quest for the perfect testing solution.

Please keep in mind that this is a new driver and isn’t battle-tested as well as Akephalos or Selenium. If you discover any bugs, please report them in Github Issues. We’d also be happy to accept tested patches.