capybara-webkit doesn’t like #fork

Recently, when helping to convert a feature suite to use capybara-webkit, we ran into a problem where, about halfway through, the tests would start erroring out with Errno::EPIPE.

Errno::EPIPE is a “Broken Pipe” error. A pipe in Unix is a way for data to flow from one process to another, and a broken pipe means that one end of the pipe isn’t connected anymore. In this case, the pipe was broken because the webkit-server process that capybara-webkit uses went away. And since well-behaved processes don’t normally go away unless you ask them to, we assumed it was crashing.

With a little of digging, we found the feature that was causing the problem. Annoyingly, the error never popped up when the feature was run alone, meaning it was the interaction of multiple features or steps. After whittling away the options, we found the right situation to force the error: a feature using capybara-webkit followed by one specific scenario in a different feature (which, oddly, didn’t use any Javascript at all). Being able to reliably reproduce the error is a big step towards finding a solution.

Knowing what scenario we were dealing with, we could find the specific step that was the source of the problem. The step makes a call to fork. The way that capybara-webkit closes the browser when the tests are over is by using an at_exit hook, which is called when Ruby is just about to close. Forking a process clones everything about it, including, in the case of Ruby, its at_exit hooks. So, what happened was that the fork executed what it needed to execute and then exited like normal, but it meant that it took the webkit-server process with it because of the at_exit hooks it inherited, leaving no browser to run the javascript-enabled features.

This was good news, though. The webkit-server process wasn’t crashing! It was behaving exactly like it was expected to, but just a little premature. To get around the at_exit hooks, we call exit! in the forked process, which specifically bypasses at_exit hooks. This lets our fork get on with its work, and it keeps the server running for the rest of the test suite to use.

So, if you’re in the position where you have to fork in a cucumber suite and you’re using capybara-webkit, make sure you exit! from the fork, and you’ll save yourself some headaches. If you can help fix this problem for good, pull requests are welcome.

Use Capybara on any HTML fragment or page

I was upgrading Gemcutter to Cucumber and Capybara 1.0 yesterday from Webrat (a change long overdue!), and I discovered a neat little class within Capybara that is worth sharing. Basically, since I was moving the app from Webrat, matchers like assert_contain and assert_have_selector are no longer available. Capybara’s Node class has a great Matchers mixin with tons of goodies that can be used like so, in RSpec:

page.should have_content("This should be on the page")
page.should have_selector("a[href='http://thoughtbot.com']")

Great, but how does one use that in functional/controller tests?

Enter Capybara::Node::Simple, which I found purely by chance when source diving. This class’ docs proclaim its usefulness:

It is useful in that it does not require a session, an application or a driver, but can still use Capybara’s finders and matchers on any string that contains HTML

Bingo! Now, how to use in our test suite? We’re still on Test::Unit for Gemcutter, so I had to do the following in test/test_helper.rb:

class Test::Unit::TestCase
  def page
    Capybara::Node::Simple.new(@response.body)
  end
end

Now the Gemcutter test suite can do assertions like so:

assert page.has_content?("Rails (3.0.9)")
assert page.has_selector?("a[href='/gems/rails/versions/3.0.9']")

The whole diff is on GitHub if you’d like to see all of the changes of moving our functional tests from Webrat to Capybara.

Gabe also found out that there’s also a shortcut in Capybara for creating a Simple: Capybara.string. The docs for this show that it’s basically sugar on top of the Simple initializer:

node = Capybara.string <<-HTML
  <ul>
    <li id="home">Home</li>
    <li id="projects">Projects</li>
  </ul>
HTML

node.find('#projects').text # => 'Projects'

I think this pattern is really useful not just for upgrading suites from Webrat, but really anywhere you have an HTML fragment or string that you’d like to use Capybara’s matchers on.

The Quest Continues: Introducing capybara-webkit

We recently blogged about our experiences and ongoing efforts to discover and improve tools for Javascript testing. In closing, we mentioned that we were fairly happy with Akephalos, but we didn’t believe the quest was over. Today, the quest continues.

Akephalos

Akephalos is a well-written driver that integrates soundly with the htmlunit virtual browser. However, as well-integrated as Akephalos is into both Capybara and htmlunit, it can only ever be as strong as htmlunit. As our applications became more sophisticated and large, we began to run into issues with htmlunit.

• Bugs: as previously mentioned, there are bugs in htmlunit, specifically with jQuery’s live. Although all browser implementations have bugs, it’s more useful if tests experience the same bugs as actual browsers.
• Compatibility: htmlunit doesn’t fully implement the feature set that modern browsers do. For example, it doesn’t fully handle DOM ranges or Ajax file uploads.
• Rendering: htmlunit doesn’t actually render the page, so tests that depend on CSS visibility or positioning won’t work.
• Performance: when most of your tests use Javascript, test suites with htmlunit start to crawl. It takes a while to start up a test with Akephalos, and a large test suite can easily take 10 or 15 minutes.

These issues made us realize that virtual browsers like envjs and htmlunit are never going to keep pace with actual browsers. If we want to write applications using features from the latest and greatest browsers, we’re going to need to test with them, too.

Selenium

That brought us back to the old mirage: Selenium. Selenium is well-supported, well-maintained, and works with real browsers. Integration with Ruby tests has improved considerably since the old Webrat days, and speed is better in some cases, as real browsers have been heavily optimized.

However, Selenium still has some ongoing issues that aren’t easy to solve:

• Timing: because Selenium uses a real browser, it doesn’t have as much control as a virtual browser does. This leads to issues with timing, causing erratic test failures.
• Overhead: real browsers come with features we don’t need, like a UI and plugins. This slows things down a bit.
• Gotchas: real browsers come with strange edge cases. Using Selenium and Firefox, sometimes the tests crash because Firefox tries to update the browser. Firefox doesn’t allow more than one instance to run at once. These gotchas are inevitable, because real browsers weren’t designed to be driven programatically.

We were forced to use Selenium because we needed all the capabilities of a real browser, but all these issues kept us looking for something better.

capybara-webkit

What we need was a real rendering engine coupled with a full Javascript and DOM implementation, but without all the cruft of a GUI browser. What we really wanted was a headless implementation of WebKit’s rendering engine that could be driven by Capybara tests. After Tristan “Websockets” Dunn showed me PhantomJs, I realized this might be possible using Qt’s WebKit implementation.

It was possible. I’d like to introduce a headless WebKit driver for capybara: capybara-webkit.

capybara-webkit has the following benefits:

• Compatibility: uses a real rendering engine (WebKit)
• Simplicity: runs headlessly, and designed to be controlled programatically
• Performance: faster than htmlunit - the test suites we’ve tried it on so far yielded 15-25% speed increases

The holy grail?

We designed capybara-webkit to solve the problems we had with other solutions, and so far it fits the bill nicely. We believe the quest is far from over, but we invite you to try out capybara-webkit on your own and see if this latest tool can take you a little further on your quest for the perfect testing solution.

Please keep in mind that this is a new driver and isn’t battle-tested as well as Akephalos or Selenium. If you discover any bugs, please report them in Github Issues. We’d also be happy to accept tested patches.

Next Steps & Related Reading

End-to-end testing with RSpec integration tests and Capybara

Unit and Functional Tests are as Useful as 100% Code Coverage

Detect emerging problems in your codebase with. We’ll deliver solutions for fixing them, and demonstrate techniques for building a Ruby on Rails application that will be fun to work on for years to come.

Grab a free sample of Ruby Science today!