Unit and Functional Tests are as Useful as 100% Code Coverage

Note: code samples updated October 25, 2012

I’m going to try and keep this short: if you’re writing Rails applications and are not writing integration tests, you need to start. Today. At Thoughtbot, we write integration tests with RSpec and Capybara although you could use Cucumber, or any other tool.

Most developers know that achieving 100% C0 code coverage isn’t a worthwhile goal in the grand scheme of things. You can use coverage reports to help guide you in what you may want to write tests for, but the law of diminishing returns becomes a factor in attempting to achieve 100% coverage.

Without integration tests, an awesome test suite doesn’t really mean much. I could test each individual method on every model I write, cover every response for every controller, and even go so far as to testing all my views, but in the end, something is missing. I’m not testing how my application behaves. So, just as 100% coverage doesn’t really mean much, the same goes for an app that’s tested well but lacks integration tests.

Any amount of testing is better than nothing; I’m not advocating that everyone stops writing unit or functional tests.

If you’re writing a Rails app, you’re not writing just an API to your models. You’re providing users an experience, and that experience should be tested! Unit and functional tests do not have the scope to test your full application (they were never meant to.) That’s where acceptance tests enter the picture.

For example, let’s say I have these controller tests from the Clearance gem:

# spec/controllers/sessions_controller_spec.rb
describe Clearance::SessionsController do
  describe 'on GET to /sessions/new' do
    before { get :new }

    it { should respond_with(:success) }
    it { should render_template(:new) }
    it { should_not set_the_flash }
  end

  describe 'on POST to #create with good credentials' do
    before do
      @user = create(:user)
      @user.update_attribute :remember_token, 'old-token'
      post :create, :session => { :email => @user.email, :password => @user.password }
    end

    it { should redirect_to_url_after_create }

    it 'sets the user in the clearance session' do
      controller.current_user.should == @user
    end

    it 'should not change the remember token' do
      @user.reload.remember_token.should == 'old-token'
    end
  end
  # ...
end

By themselves, what do these test? They test that I have a controller that responds to the HTTP verb POST and will set the flash and sign me into the application, and that I can call #authenticate on a User with an email and password and get a boolean response.

Great. That doesn’t mean I can sign into the application from a browser. I could throw an assert_select or two in there, but where’s the overall behavior of going to the homepage, clicking “Login”, filling in the form, and hitting the “Sign In” button?

Let’s use Capybara to drive a headless web browser in an extended example from:

# spec/integration/visitor_signs_in_spec.rb    
feature 'Visitor signs in' do
  scenario 'with valid email and password' do
    create_user 'user@example.com', 'password'
    sign_in_with 'user@example.com', 'password'

    user_should_be_signed_in
  end

  scenario 'tries with invalid password' do
    create_user 'user@example.com', 'password'
    sign_in_with 'user@example.com', 'wrong_password'

    page_should_display_sign_in_error
    user_should_be_signed_out
  end

  private

  def create_user(email, password)
    FactoryGirl.create(:user, :email => email, :password => password)
  end

  def sign_in_with(email, password)
    visit sign_in_path
    fill_in 'session_email', :with => email
    fill_in 'session_password', :with => password
    click_button "Sign in"
  end

  def user_should_be_signed_in
    visit root_path
    page.should have_content('Sign out')
  end

  def user_should_be_signed_out
    page.should have_content('Sign in')
  end

  def page_should_display_sign_in_error
    page.should have_css('div.error', 'Incorrect email or password')
  end
end

Between both of these, which do you think describes the ability for a user to sign in successfully?

Related Reading

End-to-end testing with RSpec integration tests and Capybara

The Quest Continues: Introducing capybara-webkit

Five Ridiculously Awesome Cucumber (and Webrat) Features

Cucumber, if you haven’t heard, is the Next Big Thing™. It affords developers the ability to write human-readable integration tests. Even though it’s only at version 0.3.101, it’s full-featured with a handful of hidden gems. Here are five of my favorites.

Running Targeted Features with Cucumber Profiles

In a Rails application, if a RAILS_ROOT/cucumber.yml file is present, you can store named Cucumber profiles. This becomes handy if you’re running your features in different environments (e.g. Selenium or a simulated browser).

Here’s an example cucumber.yml file:

default: --format pretty -r features/support/env.rb -r features/support/plain.rb -r features/step_definitions features
selenium: -r features/support/env.rb -r features/support/selenium.rb -r features/step_definitions features
html_output: --format html -r features/support/env.rb -r features/support/plain.rb -r features/step_definitions features

Within your terminal, you can run:

cucumber -p html_output

Tagging Scenarios

Another handy Cucumber feature is the ability to tag scenarios. Tagging a scenario is achieved by simply adding @tagname above the scenario definition, as such:

@tagname
Scenario: User is not signed up
  Given no user exists with an email of "email@person.com"
  When I go to the sign in page
  And I sign in as "email@person.com/password"
  Then I should see "Bad email or password"
  And I should not be signed in

To run scenarios with a specific tag:

cucumber --tags @tagname

To exclude scenarios with a certain tag, prefix the tag with a tilde:

cucumber --tags ~@wip

A common tag name that Cucumber supports out of the box is @wip (work in progress) and provides rake tasks to accommodate using this pattern:

rake cucumber:wip # runs only scenarios with a wip tag
rake cucumber:ok # runs all scenarios without a wip tag

One thing to note here is that Cucumber will exit with a status of 1 if your @wip-tagged scenarios pass (it’s a reminder that they’re not works in progress anymore since they pass).

Assigning Data with Tables

Cucumber supports the use of tables, which can be handy for any number of things; my favorite uses are populating fields in a form and instantiating an ActiveRecord object with values defined.

Webrat gives us a step definition for filling in form fields (within features/step_definitions/webrat_steps.rb) and can be used as such:

When I fill in the following:
  | First Name            | John                 |
  | Last Name             | Doe                  |
  | Email                 | john.doe@example.com |
  | Password              | password             |
  | Password Confirmation | password             |

With an ActiveRecord object and FactoryGirl:

# step
Given a customer exists with the following attributes:
  | Name      | Huge Local Company |
  | Code      | HLC                |

# step definition
Factory.factories.each do |name, factory|
  Given /^an? #{name} exists with the following attributes:$/ do |attrs_table|
    attrs = {}
    attrs_table.raw.each do |(attr, value)|
      sanitized_attr = attr.gsub(/\s+/, "-").underscore
      attrs[sanitized_attr.to_sym] = value
    end
    Factory(name, attrs)
  end
end

Segregating Selenium and Simulated Browser Steps

If you’ve been ambitious enough to get your features running with Selenium, you already know that doing so can add a significant amount of time to your tests. Webrat is kind enough to provide you execution blocks that run in specific modes, which makes for some snappier tests when you do run features with Webrat.

For example, within a step that outlines signing in:

webrat.automate do
  When %{I sign in as "#{email}/#{password}"}
end

webrat.simulate do
  visit "/session",
        :post,
        :session => {:email => email, :password => password}
end

Webrat will determine if you’re simulating a browser and execute webrat.simulate if that’s the case; otherwise, it’ll run the code within the webrat.automate block. This can be really helpful if you’re testing the ability to delete an item (especially if there’s a confirmation dialog).

Displaying Page Responses within a Scenario

If you’re having trouble getting a feature to pass and want to view the page, you can add in a step to have it save the result HTML and open the page in your default browser.

Scenario: User is not signed up
  Given no user exists with an email of "email@person.com"
  When I go to the sign in page
  Then show me the page # saves the HTML of that page and opens the file
  And I sign in as "email@person.com/password"
  Then I should see "Bad email or password"
  And I should be signed out
  Then show me the page # more HTML output

This tends to be much more convenient than adding this into your step definition:

puts response.body.inspect

There are, as always, a couple of caveats. The HTML it generates won’t be relative to your public folder so Javascript and CSS probably won’t load correctly. It also doesn’t persist form values, so if you run it after a step like ‘I fill in “field” with “value”’, it won’t keep that data. It also creates a timestamped html file within your RAILS_ROOT folder. A simple fix to ensure this doesn’t get committed with git would be to add the following line to your .gitignore file in the project:

webrat-*.html

Finally…

Don’t forget to read the docs! They cover everything you’d ever want to know about Cucumber and how to use it effectively. I’ve listed only a handful of awesome features Cucumber and Webrat have to offer, but each has helped me in one way or another over the course of the past year. Hopefully they’ll be as much use to you as they were me.

Next Steps & Related Reading

Detect emerging problems in your codebase with. We’ll deliver solutions for fixing them, and demonstrate techniques for building a Ruby on Rails application that will be fun to work on for years to come.

Grab a free sample of Ruby Science today!

Unpopular Developer 5: Stop Unit Testing Your Scaffold Controllers

This goes out to all the veteran Ruby on Rails developers: those who have the RESTful scaffold memorized down to their fingertips; those who can write the tests for it using Cucumber with Webrat, Rails integration tests with Webrat, Rails controller tests with RSpec, Test::Unit, or Shoulda, using mocking, stubbing, test spies, or old-school assertion style:

Stop unit testing the RESTful scaffold

Do write an integration test with Webrat (and Cucumber or Rails integration tests or some crazy script using Hubris), but abstract that out as much as you can. It’s the scaffold. You know it works.

You shouldwrite a unit test for your RESTful controller as soon as it deviates, and you should test-drive that deviation both in the integration test and the unit test.

Here, some code. First the #index action, non-tested:

def index
  @posts = Post.all
end

It works. No unit test will uncover a bug in this. The integration test may - for example, Post may have no database table yet, or it may not derive from ActiveRecord::Base, or may redefine Post.all, or anything else - but the unit test for the controller will fail due to a regression.

So now we’ll make a change: posts should be listed alphabetically.

Feature: Posts
  # ... existing scenarios that test everything go here ...
  Scenario: Viewing posts alphabetically
    Given a post exists with a title of "Abe rents storage space"
    And a post exists with a title of "Aaron opens his garage to friends"
    When I go to the list of posts
    Then I should see the posts sorted alphabetically

Make sure that fails. Then add the unit test for the controller; the very first unit test for this controller. Here I’ll use shoulda and jferris-mocha:

require 'test_helper'

class PostsControllerTest < ActionController::TestCase
  should "alphabeticalize the blog posts on GET to index" do
    Post.stubs(:alphabetical).returns([])
    get :index
    assert_received(Post, :alphabetical) {|expects| expects.with()}
  end
end

That’s all you need because that’s all that has deviated from the RESTful scaffold.

Benefits

• See what deviates from the RESTful scaffold.
• Refactor with ease.
• Spend less time implementing controller unit tests.

The Opposite of Benefits

• Requires your team to know the RESTful scaffold well.

So in summary, do write integration tests all the time, but don’t TATFT. Relax those unit tests.