End-to-end testing with RSpec integration tests and Capybara

Stability can become an issue as web applications evolve and grow — integration tests provide a great way to perform end-to-end tests that validate the application is performing as expected.

Integration test with rspec and capybara

When writing integration tests, try to model the test around an actor (user of the system) and the action they are performing.

# spec/features/visitor_signs_up_spec.rb
require 'spec_helper'

feature 'Visitor signs up' do
  scenario 'with valid email and password' do
    sign_up_with 'valid@example.com', 'password'

    expect(page).to have_content('Sign out')
  end

  scenario 'with invalid email' do
    sign_up_with 'invalid_email', 'password'

    expect(page).to have_content('Sign in')
  end

  scenario 'with blank password' do
    sign_up_with 'valid@example.com', ''

    expect(page).to have_content('Sign in')
  end 

  def sign_up_with(email, password)
    visit sign_up_path
    fill_in 'Email', with: email
    fill_in 'Password', with: password
    click_button 'Sign up'
  end
end

Extracting common test functionality

To share code between features move common capybara steps into a Ruby module in the rspec support directory.

# spec/support/features/session_helpers.rb
module Features
  module SessionHelpers
    def sign_up_with(email, password)
      visit sign_up_path
      fill_in 'Email', with: email
      fill_in 'Password', with: password
      click_button 'Sign up'
    end

    def sign_in
      user = create(:user)
      visit sign_in_path
      fill_in 'Email', with: user.email
      fill_in 'Password', with: user.password
      click_button 'Sign in'
    end
  end
end

Modules must be explicitly included to share the common code between integration tests.

# spec/support/features.rb
RSpec.configure do |config|
  config.include Features::SessionHelpers, type: :feature
end

Running the integration tests

$ rspec -fd
Visitor signs up
  with valid email and password
  with invalid email
  with blank password

Finished in 0.35837 seconds
3 examples, 0 failures

Takeaways

• Maintain application stability with end-to-end test coverage
• Use RSpec to run your integration tests
• Modules allow shared Capybara steps between specs

By Harlow Ward

Next Steps & Related Reading

Describe the User’s Perspective: DDD, acceptance testing, and you

The Quest Continues: Introducing capybara-webkit

Detect emerging problems in your codebase with Ruby Science. We’ll deliver solutions for fixing them, and demonstrate techniques for building a Ruby on Rails application that will be fun to work on for years to come.

Grab a free sample of Ruby Science today!

Turnip: a tasty Cucumber alternative?

We’ve used Cucumber heavily and successfully on client work, internal projects, and open source. We also love RSpec, so when we heard that Turnip would give the ability to run Gherkin based integration tests in our RSpec suite it was a no-brainer for us to try it out on a project.

Highlights of Turnip

• Integrates directly into your RSpec test suite
• Features and Step definitions live in the spec directory
• No need to maintain two configuration files
• Uses Ruby style symbols instead of regular expressions in step definitions
• No need to write Given/When/Then in the step definitions file
• Speed boost when running unit tests and integration tests together

We use Cucumber integration tests in thoughtbot’s clearance gem. In the example below I remove Cucumber and replace it with Turnip.

Steps used to change from Cucumber to Turnip

• Set up your features directory inside the spec directory
• Remove the Given/When/Then’s from the step definitions
• Replace regular expressions in the step definitions with Ruby style symbols

Cucumber and Turnip use the same Gherkin syntax:

 Scenario: Visitor signs up with valid data
   When I sign up with "email@example.com" and "password"
   Then I should be signed in 

Change the step definitions from Cucumber to Turnip style:

 # Cucumber step definition
 When /^I sign up (?:with|as) "(.*)" and "(.*)"$/ do |email, password| 
   visit sign_up_path 
   page.should have_css("input[type='email']") 
   fill_in "Email", :with => email 
   fill_in "Password", :with => password 
   click_button "Sign up" 
 end 

 # Turnip step definition
 step "I sign in with/as :email and :password" do |email, password|
   visit sign_in_path 
   page.should have_css("input[type='email']")
   fill_in "Email", :with => email 
   fill_in "Password", :with => password 
   click_button "Sign in"
 end 

Run the test suite

An advantage having everything running through RSpec is we get an immediate boost in speed when running the whole test suite. With Cucumber running Rake will run the RSpec and Cucumber tests (the Rails environment will be loaded twice). With Turnip all tests run through directly through RSpec (the Rails environment is only loaded once).

We save around 12 seconds when running the entire suite:

 # Rake running RSpec and Cucumber
 ~/Development/clearance_cucumber(master) $ time rake
 /Users/training/.rvm/rubies/ruby-1.9.2-p290/bin/ruby -S rspec ./spec/controllers/pages_controller_spec.rb ./spec/helpers/application_helper_spec.rb
 .

 Finished in 0.10696 seconds
 1 example, 0 failures
 /Users/training/.rvm/rubies/ruby-1.9.2-p290/bin/ruby -S bundle exec cucumber --profile default
 Using the default profile...
 .................................................

 13 scenarios (13 passed)
 49 steps (49 passed)
 0m1.729s
 rake 19.97s user 2.71s system 98% cpu 22.960 total

 # RSpec with Turnip
 ~/Development/clearance_turnip(master?) $ time rspec
 ..............

 Finished in 1.84 seconds
 14 examples, 0 failures, 0 pending
 rspec 8.85s user 1.06s system 95% cpu 10.362 total

Takeaways

• Great light-weight solution for anyone already using RSpec
• Step definitions easier to read than their Cucumber counterparts
• Low barrier to entry for developers new to integration testing

By Harlow Ward

Related Reading

End-to-end testing with RSpec integration tests and Capybara

Use Deadweight and your integration suite to automatically find unused CSS rules

We’re refactoring the CSS in Hoptoad - the app is a few years old, and has been through several rounds of design improvement. We’re looking to combine duplicated rules, make selectors more intention-revealing, and want to reorganize the stylesheets to reflect our current best practices. We’re also switching to SASS, which we’ve found to be a tremendous help. Together, these improvements should cut developer and designer time during maintenance and future design improvements.

Before diving in, we wanted to know what CSS is used or unused.

Deadweight is an excellent tool for identifying unused CSS selectors. It’s flexible, too: you can identify paths in your application for deadweight to scan, or use it as an HTTP proxy and click around in your site, or just pass it static HTML files to analyze. There are a few caveats (for example, selectors inserted by Javascript aren’t tracked) but it’s generally a solid approach for finding unused CSS rules.

I didn’t want to spend a lot of time clicking through the site, and wanted to eliminate the possibility that I would forget a page or two to include. I decided to run the Hoptoad integration tests (Cucumber features, in this case) and capture the response bodies.

Rack middleware to the rescue

I wrote the following piece of Rack middleware:

class ResponseLoggerMiddleware
  RESPONSE_LOG_DIR = Rails.root.join('log', 'responses')
  Dir.mkdir(RESPONSE_LOG_DIR) unless Dir.exist?(RESPONSE_LOG_DIR)

  def initialize(app)
    @app = app
  end

  def call(env)
    response = @app.call(env)
    log(response)
    response
  end

  def log(rack_response)
    code, headers, response = rack_response

    if response.respond_to?(:body)
      html = response.body
      filename = "response_#{Time.now.to_f}.html"
      File.open(RESPONSE_LOG_DIR.join(filename), 'wb') do |file|
        file.puts(html)
      end
    end
  end
end

and included it:

# For Rails 2, include in config/environment.rb
# For Rails 3, include in config/application.rb
# Why Rack::Lock? http://guides.rubyonrails.org/rails_on_rack.html#internal-middleware-stack
config.middleware.insert_after 'Rack::Lock', 'ResponseLoggerMiddleware'

After the tests ran, I processed the output with deadweight:

$ gem install deadweight
$ cat public/stylesheets/*.css | deadweight log/responses/*

Ta-da! If you’re curious what Deadweight output looks like, here’s what I got: https://gist.github.com/cf34709887bf63b1ec15

Further reading

• Ryan Bates’ Railscast on Deadweight will get you familiar with the tool.
• The Helium tool lets you drop a piece of JavaScript on your site and paste in a list of URLs, and will compute unused CSS selectors.
• SitePoint’s Dust-Me Selectors is a Firefox plugin that identify unused CSS selectors. It can spider through your site with a sitemap, too.

Extra credit

You could bundle this all up into a rake task, run it on your CI server, and fail the build whenever there are new unused CSS rules.

thoughtbot and the Holy Grail

Update: Our quest for full-stack testing continued with writing the capybara-webkit driver. We also wrote about capybara-webkit more recently.

Stop! Who would cross the Bridge of Death must answer me these questions three, ‘ere the other side he see.

Ask me the questions, bridge-keeper. I’m not afraid.

What is your name?

My name is cpytel of thoughtbot.

What is your favorite color?

Red.

What is your quest?

To seek a reliable, robust way of integration testing our entire application to prevent against regressions and to build better systems.

With apologies to Monty Pyton, we’ve been on this quest for several years now. After many false starts and tribulations I feel we’ve reached an important plateau in this quest, and I feel its prudent to take a step back, look at where we’ve been, where we are now, and pass on some of this information for the benefit of all.

Selenium

Throughout this quest, Selenium has been the siren song that continually calls out to us. Unfortunately, in practice we’ve been unable to get Selenium to run reliably for real applications, on both developers machines and on the continuous integration server.

This failure with Selenium has caused us to search for alternative solutions.

Holy Grail

The first promising solution was the aptly named, Holy Grail. This library uses Harmony, which in turn wraps Johnson, env.js and Envjs to execute browser-less, console-based, javascript and DOM code right from within your Rails test suite. Our own Jason Morrison then wrote cucumber-holygrail to allow Cucumber to drive the Holy Grail integration tests.

Unfortunately, this solution wasn’t really designed to accomplish what we wanted; Holy Grail lets you run javascript on a single page and routes xhr requests to a controller in a functional test, and therefore it didn’t work like we expected: we wanted to drive a virtual browser session from action to action that supported javascript and ui interactions.

Envjs

From there, we discovered capybara-envjs. Capybara is a replacement for Webrat (the driver underneath Cucumber) which has a more flexible driver subsystem. capybara env-js provides more of what we needed, it had the goal of being a virtual browser, like we wanted. In order to make it work like we wanted, we needed to provide a number of fixes which we encapsulated into the capybara-envjs-fixes gem (as a holding place while we waited for pull requests to be merged in).

Capybara and capybara-envjs, and capybara-envjs-fixes were (and mostly still are) a deadly combination. We’re actively using it on projects still. However, the more we used it, the more we started to discover cracks in the foundation.

• It doesn’t work with all javascript (for example, we had issues on sites that still use Prototype)
• It doesn’t support jQuery live
• It is sometimes very inconsistent with how real-world browsers behave, or how things should properly work, forcing us to write our code like we wouldn’t normally write it

The three above points, particularly the lack of support for jQuery live (which is used by more and more of the internal of jQuery, jQuery mobile, and even the new Rails unobtrusive javascript) caused us to continue to search for a better solution.

Akephalos

Thankfully, we then found Akephalos. Akephalos provides a Capybara driver that allows you to run your cucumber integration tests in the headless browser HtmlUnit. HtmlUnit is a “GUI-Less browser for Java programs”. It models HTML documents and provides an API that allows you to invoke pages, fill out forms, click links, etc… just like you do in your “normal” browser. With our fork of Akephalos to resolve a couple of issues that we ran into along the way, we were up and running with very reliable, headless browser tests.

HtmlUnit is written in Java, and Akephalos uses jruby-jars to start up and interact with the HtmlUnit browser. It has fairly good JavaScript support (it was able to deal with everything we were able to throw at it, including jQuery 1.4.2 and 1.4.3, jQuery Mobile, and jQuery live).

Note: We also tried culerity which is a cucumber driver that works with Celerity, which also is underpinned by HtmlUnit, but we couldn’t get it to really work well.

One of the great things about HtmlUnit and Akephalos is that it uses technology which has been existence in a while, is actively developed, and well documented. We were able to look at the code and understand what was going on and contributate changes if necessary much more easily than we were able to do with any of the previous attempts.

If you want to get started with Akephalos, I recommend that you use our fork on github until the changes are pulled in. We’ve used Akephalos on both Rails 2 and 3.

The relevant part of our Gemfile looks like this:

group :test, :cucumber do
  gem 'akephalos', :git => 'git://github.com/thoughtbot/akephalos.git'
  gem "cucumber-rails"
  gem "capybara"
  gem "database_cleaner"
  gem "treetop"
  gem "launchy"
end

Then, to a features/support/akephalos.rb add:

require 'akephalos'

You can then tag any scenarios you want to run with Akephalos with the @akephalos tag. If you want to be able to tag your scenarios with @javascript and have it execute in Akephalos (the default javascript driver in Capybara is Selenium) you’ll want to add the following additional line to your features/support/akephalos.rb file.

Capybara.javascript_driver = :Akephalos

There are a couple of important gotchas right now that will effect you, particularly if you have an existing application.

There is a known bug in HtmlUnit where any jQuery live events bound to the click event stop any jQuery live events bound to submit defined after them. In particular, this means that if you’re using Rails 3 unobtrusive jQuery form helpers, you’ll want to edit your rails.js file and move the definition of the follow event:

$('form[data-remote]').live('submit'

Up above any jQuery live click bindings.

Additionally, ajax requests execute asynchronously, just like in a real browser. In order for the tests to not get out of sequence and fail, ajax requests need to be executed synchronously instead. You can do this by adding the following code after your inclusion of jQuery in only your test or cucumber environments:

    $.ajaxSetup({ async: false });  

Akephalos doesn’t have the ability to set cookies, or access the session. These are capabilities in other drivers, and so you may find that your cucumber steps are doing this, and they’ll need to be rewritten. For example, the built in clearance step definitions read the cookies in order to expose a current_user helper for step definitions to use, and deleted cookies in order to force a sign out. There is newly committed support for reading cookies in Akephalos, but I couldn’t get it to work, and I figured as a matter of practice it’d be better not to rely on it anyway.

Have we reached the end of our quest?

We’re not the only ones on this quest, I’m sure, and I encourage those of you who are to take a look at Akephalos and give it a try, I don’t think you’ll be disappointed.

Unfortunately, I don’t think the quest is over.  But Akephalos represents a significant step forward in a reliable headless browser that can allow us to fully integration test our applications. There are probably otther options available to us to try, and I expect there will be more as time goes on. For example, we’ve recently started to use evergreen to do javascript unit testing right alongside the normal tests.

We’ll continue to report back on what we find and consider the current state-of-the-art. Let us know if you’ve found and love something we’ve missed here.