Productivity and happiness. That’s why Ruby and Rails were written, why developers were drawn to them, and why we keeping using them. They continue to be the stated goals on ruby-lang.org and rubyonrails.org.
Over time, both cultures have developed a very strong testing culture, and more recently a strong integration testing culture, but the search is still on for more productive, happier approaches.
This is by no means comprehensive, and is completely skewed by my personal opinions and biases, but here’s a brief overview of Ruby integration test history:
Rails 1.1 introduces baked-in integration tests. Many developers are still getting used to testing, however, and the announcement post gets less than 10 comments.
It exercises the full Rails stack with methods like get, post, follow_redirect!, xml_http_request, status, path, and open_session.
The intention is that developers use normal Test::Unit methods such as assert_equal to verify expectations on the resulting HTTP response headers and body.
Rails developers can now write:
class ExampleTest < ActionController::IntegrationTest
def test_login
get "/login"
assert_equal 200, status
post "/login",
:username => people(:jamis).username,
:password => people(:jamis).password
follow_redirect!
assert_equal 200, status
assert_equal "/home", path
end
end
RSpec introduces the Story Runner. The idea of customers writing and/or reading story input and/or output can now be tested in practice instead of talked about theoretically.
Rails developers can now write:
Scenario "Root user" do
Given "a user named", "admin"
And "a company named", "Company1"
And "a company named", "Company2"
And "the user has the role", "root" do |role|
@user.update_attribute :role, role
end
And "logged in as", "admin"
When "visiting", "/"
Then "viewer should see", "main/root_home"
And "page should show company named", "Company1" do |name|
response.should have_text(/#{name}/)
end
And "page should show company named", "Company2"
end
Bryan Helmkamp introduces Webrat. It simulates the browser similarly to Rails integration tests but has a better-looking DSL and adds new, fine-grained interaction such as click_link, fill_in, and click_button.
Over time, Webrat will add very strong leverage of Nokogiri to verify HTML responses using XPath and CSS selectors. Developers rejoice.
Rails developers can now write:
def test_sign_up
visits "/"
clicks_link "Sign up"
fills_in "Email", :with => "good@example.com"
select "Free account"
clicks_button "Register"
...
end
Merb recommends using the request method for integration tests. This preserves state across requests, is easier than the older dispatch_to, and comes with some nice RSpec matchers such as be_successful and redirect_to.
Cucumber replaces Story Runner. Every Ruby conference hereafter includes a talk on Cucumber and it maintains a top spot on Github.
Rubyists can now write:
Feature: Addition
In order to avoid silly mistakes
As a math idiot
I want to be told the sum of two numbers
Scenario: Add two numbers
Given I have entered 2 into the calculator
And I have entered 2 into the calculator
When I press add
Then the result should be 4 on the screen
When used in combination with a Rails app, this style heavily uses pattern-matching to map executable English (or one of many supported languages) to code.
The English:
Given I signed up with "email@person.com/password"
The pattern match:
Given /^I signed up with "(.*)\/(.*)"$/ do |email, password|
user = Factory :user,
:email => email,
:password => password,
:password_confirmation => password
end
Integration testing is firmly mainstream.
Bryan Helmkamp releases Rack::Test. Like Rack itself, it aims to provide a common interface for test frameworks to interact with Rack apps.
Rack::Test pays homage to the frameworks before it, using methods that match the HTTP verbs: get, post, put, and delete, head) and adding request (Merb) and follow_redirect! (Rails). It also handles HTTP authentication and stores requests and responses in last_request and last_response.
Rack developers can now write:
class HelloWorldTest < Test::Unit::TestCase
include Rack::Test::Methods
def app
Sinatra::Application
end
def test_it_says_hello_world
get '/'
assert last_response.ok?
assert_equal 'Hello World', last_response.body
end
def test_it_says_hello_to_a_person
get '/', :name => 'Simon'
assert last_response.body.include?('Simon')
end
end
Cucumber and Webrat are increasingly used widely by Rails developers for integration tests. Its costs and benefits are debated based on actual experience with it, and alternatives begin to emerge for common reasons.
Gibbon is created. It brings Cucumber’s widely appreciated Given/When/Then DSL to regular RSpec. It is somewhat of a throwback to the Story Runner. It depends on Webrat to simulate the browser. It looks like this:
Feature "A Tomatoist does a pomodoro" do
Story <<-eos
In order to perform a focused unit of work
As a Tomatoist
I want to start a pomodoro
eos
Scenario "Starting a pomodoro" do
When "I go to the home page" do
executes { visit '/' }
Then "I should be sent to a new session" do
current_url.should =~ /\/\w{3,}/
end
And "I should see an unstarted timer" do
response.should have_tag('#timer .countdown_row','00:00')
end
When "I click the pomodoro button" do
executes do
@session_url = current_url
click_button 'Pomodoro'
end
Then "I should be on my session's page" do
current_url.should == @session_url
end
And "my timer history should show the current pomdoro" do
response.should have_tag('#history ul li', /Pomodoro/, 1)
end
end
end
end
end
Hashrocket converts the codenamed Gibbon into a library called Unencumbered.
Evan Light releases Coulda. It brings the Given/When/Then DSL to Test::Unit. It depends on Webrat or another library to simulate the browser. It looks like this:
Feature "Painfully obvious" do
in_order_to "demonstrate a simple test"
as_a "coulda developer"
i_want_to "provide a straight-forward scenario"
Scenario "Describing something obvious" do
Given "something without a value" do
@no_value = nil
end
When "I give it a value" do
@no_value = true
end
Then "it should have a value" do
assert(@no_value)
end
end
end
Jim Weireich experiments with Given. It brings the Given/When/Then DSL to Test::Unit. It depends on Webrat or another library to simulate the browser. It introduces the concept of an Invariant. It looks like this:
class StackBehavior < Given::TestCase
Invariant { expect(@stack.depth) >= 0 }
Invariant { expect(@stack.empty?) == (@stack.depth == 0) }
def empty_stack
@stack = Stack.new
end
Given(:empty_stack) do
Then { expect(@stack.depth) == 0 }
When { @stack.push(:an_item) }
Then { expect(@stack.depth) == 1 }
Then { expect(@stack.top) == :an_item }
When { @stack.pop }
FailsWith(Stack::UsageError)
Then { expect(exception.message) =~ /empty/ }
end
end
I write about 50 lines of horrible hacks. It brings the Given/When/Then DSL to Test::Unit. It relies on Webrat and Rack::Test to simulate the browser and theoretically work for all Rack apps (only tested on Sinatra). I keep the nesting totally flat, as is my style. It looks like this:
Feature 'Shorten URL' do
Given 'I am on the homepage' do
visit '/'
end
When 'I submit http://example.com' do
fill_in 'url', :with => 'http://example.com'
click_button 'shorten'
end
Then 'I should see a short link' do
assert_have_selector 'a#short'
end
When 'I follow the short link' do
click_link 'short'
end
Then 'I should be on http://example.com' do
assert_equal 'http://example.com', current_url
end
end
That’s quite a history. Looking back, we understand what makes programmers happy and unhappy about integration testing.
Dependencies. Cucumber 0.4.4 has runtime dependencies of term-ansicolor, treetop, polyglot, builder, and diff-lcs. There’s inevitably small headaches and incompatibilities as dependencies increase.
Pattern-matching. Cucumber’s goal of “executable documentation” (good goal, no criticism there) lead it to an advanced pattern-matching style of programming. This is the number one complaint I’ve heard about it, leading to related complaints of:
This last point is most damning but also most anecdotal and personal.
Integration testing in general has two flaws that are inherent in integration testing itself and unrelated to any tool:
That’s just part of the deal, though.
Normal “method programming”. Many programmers are placing lower value on “executable documentation” compared to using familiar tools, seeing method names directly adjacent to the implementation, and relying on familiar programming techniques such as intention revealing names and normal method extraction.
Small DSLs that pay homage to Given/When/Then. Everyone is looking to re-use this in other environments, usually layering a syntax of some kind on top of their test framework of choice. Variations include the super-flat (what I did), the super-RSpec’y (Gibbon), and “Cucumber for Test::Unit” (Coulda, the one actual library of the group).
Re-phrased: syntax matters to programmers.
Favorite old tools. We may be “agile”, but if we like something, we’re loathe to change.
CSS selectors. Parsing the response body with CSS selectors in Rails’ assert_select and Webrat’s assert_have_selector breeds familiarity with old skills. I’ve seen this pattern crop up again in Effigy.
Single responsibility. I have to hand it to Bryan Helmkamp. Looking back over the history of these integration testing tools, his name really sticks out as the one leading innovation in the subset concerns of:
He also clearly took into account the history of integration testing in various Ruby web frameworks.
What about Ruby integration testing is currently making you happy and productive?
