Turnip: a tasty Cucumber alternative?

We’ve used Cucumber heavily and successfully on client work, internal projects, and open source. We also love RSpec, so when we heard that Turnip would give the ability to run Gherkin based integration tests in our RSpec suite it was a no-brainer for us to try it out on a project.

Highlights of Turnip

• Integrates directly into your RSpec test suite
• Features and Step definitions live in the spec directory
• No need to maintain two configuration files
• Uses Ruby style symbols instead of regular expressions in step definitions
• No need to write Given/When/Then in the step definitions file
• Speed boost when running unit tests and integration tests together

We use Cucumber integration tests in thoughtbot’s clearance gem. In the example below I remove Cucumber and replace it with Turnip.

Steps used to change from Cucumber to Turnip

• Set up your features directory inside the spec directory
• Remove the Given/When/Then’s from the step definitions
• Replace regular expressions in the step definitions with Ruby style symbols

Cucumber and Turnip use the same Gherkin syntax:

 Scenario: Visitor signs up with valid data
   When I sign up with "email@example.com" and "password"
   Then I should be signed in 

Change the step definitions from Cucumber to Turnip style:

 # Cucumber step definition
 When /^I sign up (?:with|as) "(.*)" and "(.*)"$/ do |email, password| 
   visit sign_up_path 
   page.should have_css("input[type='email']") 
   fill_in "Email", :with => email 
   fill_in "Password", :with => password 
   click_button "Sign up" 
 end 

 # Turnip step definition
 step "I sign in with/as :email and :password" do |email, password|
   visit sign_in_path 
   page.should have_css("input[type='email']")
   fill_in "Email", :with => email 
   fill_in "Password", :with => password 
   click_button "Sign in"
 end 

Run the test suite

An advantage having everything running through RSpec is we get an immediate boost in speed when running the whole test suite. With Cucumber running Rake will run the RSpec and Cucumber tests (the Rails environment will be loaded twice). With Turnip all tests run through directly through RSpec (the Rails environment is only loaded once).

We save around 12 seconds when running the entire suite:

 # Rake running RSpec and Cucumber
 ~/Development/clearance_cucumber(master) $ time rake
 /Users/training/.rvm/rubies/ruby-1.9.2-p290/bin/ruby -S rspec ./spec/controllers/pages_controller_spec.rb ./spec/helpers/application_helper_spec.rb
 .

 Finished in 0.10696 seconds
 1 example, 0 failures
 /Users/training/.rvm/rubies/ruby-1.9.2-p290/bin/ruby -S bundle exec cucumber --profile default
 Using the default profile...
 .................................................

 13 scenarios (13 passed)
 49 steps (49 passed)
 0m1.729s
 rake 19.97s user 2.71s system 98% cpu 22.960 total

 # RSpec with Turnip
 ~/Development/clearance_turnip(master?) $ time rspec
 ..............

 Finished in 1.84 seconds
 14 examples, 0 failures, 0 pending
 rspec 8.85s user 1.06s system 95% cpu 10.362 total

Takeaways

• Great light-weight solution for anyone already using RSpec
• Step definitions easier to read than their Cucumber counterparts
• Low barrier to entry for developers new to integration testing

By Harlow Ward

Related Reading

End-to-end testing with RSpec integration tests and Capybara

Decoupling Data from Presentation

I’m really happy to see a resurgence in an understanding that writing integration tests that use classes and DOM structure to perform assertions within these tests is typically a bad idea. The DOM structure constantly changes but the data displayed (the stuff we want to assert is present on a page) typically doesn’t.

Why write tests against some markup that a designer may change, sometimes often if he’s iterating over a design? It’ll end up causing everyone headaches and frustration.

How can we mark up our documents without performing assertions against a deep nesting of classes and elements while ensuring our data is displayed properly?

Data attributes.

The browsers I care about support HTML5, so why not start using some of its capabilities to clean up some integration tests?

Imagine this scenario:

Scenario: See the site description on the homepage
  When I am on the homepage
  Then I should see the site description

Simple, straightforward, and each step is a similar level of abstraction.

Now imagine this markup:

<section class="primary-content">
  <header>
    <h1>Welcome to the site!</h1>
    <h2>Insert a witty tagline here</h2>
  </header>
  <p class="description">This product will make your life 100 times
  better.</p>
</section>

I could write a step like this:

Then "I should see the site description" do
  page.should have_selector("section.primary-content p.description",
                            text: "This product will make your life 100 times better.")
end

This step is asserting that there’s a section element with a class of primary-content who has a child paragraph tag with a class of description that contains the correct text. Classes change. Structure changes. This is begging to be rewritten.

<section data-role="primary-content">
  <header>
    <h1>Welcome to the site!</h1>
    <h2>Insert a witty tagline here</h2>
  </header>
  <p data-role="description">This product will make your life 100 times better.</p>
</section>

Then "I should see the site description" do
  page.should have_selector("[data-role='primary-content'] [data-role='description']",
                            text: "This product will make your life 100 times better.")
end

Now, the step is asserting that there’s an element with a role of description within an element with a role of primary-content that contains the correct text.

XHTML introduced a role attribute, but it’s not present currently in HTML5. Luckily, data attributes are even more flexible and we can use any conventions we like. data-role and data-state are two of my favorites.

Interested in how to write awesome Cucumber steps? Head to my Intro to Test-Driven Rails workshop on March 26th and March 27th where we’ll cover this and other awesome topics on writing great tests!

The one-method silent killer in your Cucumber suite

There’s a single method in your Cucumber suite, right now, that could silently allow regressions and ambiguity to slip in without you ever knowing. Guess which one! Actually, bonus points to you if you scroll down to the comments and make a guess before finishing this post.

Did you guess? If not, here are a few more hints.

It uses regular expressions.

It’s a large cascading conditional statement.

It’s well-intentioned, and serves as a central lookup and naming convention for what would otherwise be magic constants.

“I’ll take ‘deprecated methods’ for a thousand, Alex.”

Ready for the reveal? It’s HtmlSelectorsHelpers#selector_for. Yikes! Now that I’ve named names, let’s play everyone’s favorite game, spot-the-regex-bug:

module HtmlSelectorsHelpers
  def selector_for(locator)
    case locator

    # snip a few lines

    when /the "([^"]+)" story/
      story = Story.find_by_title!($1)
      "##{dom_id(story)}"

    # snipped another 100+ lines

    when /^the related link for the "([^"]+)" story$/
      story = Story.find_by_title!($1)
      "a#some_selector_for_story_#{story.id}"
  end
end

Spot it? This one is fairly easy to find - a step definition like:

When I click on the related link for the "Make some bread" story

would match both regular expressions, because the first regular expression doesn’t match against the beginning ^ or end $ of the line. Since the case statement just evaluates from the top down, the first match wins.

How did I find that? I was adding a new piece of functionality to click on the related link, wrote that new selector, ran the test, and it failed. I wrote my passing implementation. Still failed. I tried it out in my browser, where things worked normally. It took some sleuthing to figure out that the wrong selector was being used, and where it was coming from.

Digging deeper

Some other issues are actually lurking somewhere in the 100+ snipped lines of regexes. Let’s say you add another selector like this:

when /^the comment button for the "([^"]+)" story$/
  # ...

If that is listed after the /the "([^"]+)" story/ matcher, it will never be matched. If you’re not careful, even if you are disciplined about reaching the red state in TDD while adding the new feature, this can lead to subtle regressions later down the line. Consider a link or piece of content that gets moved around on the page: if the broad regexp still matches it, you could end up with a false positive test.

So, what’s the general issue here? When I think about the set of selectors I implement, I believe it’s a design drawback to allow developers to add ambiguously matching selectors. It would be better if the selector_for method were to give you early notice of multiple colliding matches.

What might such an implementation look like? Well, if we used a construct other than the case statement, then selector_for could evaluate all possible matches, and raise an exception if there are multiple matches. Consider this example:

def selector_for(locator)
  selectors = {
    "the page" => "html > body",

    /the project dropdown/ => '#project-selection ul',

    /the "([^"]+)" story/ => lambda {
      story = Story.find_by_title!($1)
      "##{dom_id(story)}"
    },

    /^the related link for the "([^"]+)" story$/ => lambda {
      story = Story.find_by_title!($1)
      "a#some_selector_for_story_#{story.id}"
    }
  }

  matches = selectors.map do |selector, result| 
    if locator =~ selector
      if result.respond_to?(:call)
        result.call
      else
        result
      end
    end
  end.flatten

  raise "Ambiguous matches!" if matches.size > 1

  matches.first
end

(This is just an illustration. I’ve no idea what the performance impact is here, or if the above code works. You could smooth out the interface a bit, too.)

So - problem solved! We can identify ambiguous matches. Party hats all around.

But wait, there’s more

But is this the best we can do? The root of ambiguity here is regular expressions, and you, dear astute reader, have recognized that these selectors are being used in Cucumber steps, and not just within step definition implementations. Hopefully, you’re climbing into the commenting cockpit, ready to take aim at the notion of this coupling of front-end implementation to the big-A Acceptance test suite - a notion that has gotten so much blog heat lately.

“Take that, blogger! Feel the wrath of my disagreement!”

In fact, selector_for was removed in cucumber-rails 1.1.0. For new apps, it’s a good change. And we should upgrade our existing apps, and generally move away from imperative steps if we’re using them. But is there still something to selector_for that’s worth keeping around?

Myself, I’ve liked using selector_for in the definitions of acceptance test steps because it provides a central and well-named lookup for front-end implementation details. But I don’t like-like it. Really, we’re trading one set of magic constants (CSS selectors) for another (named selector strings), and the presence of regexp-based dispatch in the land of plain old Ruby methods starts to feel a bit out of place. Unnecessary. Even bulky:

When /^I participate in a discussion related to "([^"]+)"$/ do |story_title|
  click_link(selector_for("the related discussion link for #{story_title}"))
  fill_in 'Comment', with: "What fun discourse"
  click_button 'Create Discussion'
end

Wouldn’t it be nice to just extract these to plain old Ruby methods?

module HtmlSelectorsHelpers
  def related_discussion_link(story_title)
    story = Story.find_by_title!(story_title)
    "a#some_selector_for_story_#{story.id}"
  end
end

World(HtmlSelectorsHelpers)

# ...

When /^I participate in a discussion related to "([^"]+)"$/ do |story_title|
  click_link(related_discussion_link(story_title))
  fill_in 'Comment', with: "What fun discourse"
  click_button 'Create Discussion'
end

Now, we have less clunky dispatch and more cohesive step definitions which operate at a single level of abstraction.

It’s also good because this moves away from encouraging the coupling of Cucumber steps to implementation details. It encourages less string matching, and more plain Ruby method-calling.

But what about our original issue, ambiguity? It’s still possible to overwrite a method in the HtmlSelectorsHelpers module, although I suppose it’s less likely - the real reason I got tripped up at the beginning was that too-general regular expression. I think we’ve done well to mitigate that source of bugs.

Bringing it on home

The one trouble with this approach is that it’s not incremental. I can’t easily convert all my existing selector_for code to use just methods in one fell swoop. This is the same pain point felt when talking about removing web_steps from an existing suite - it’s messy and time-intensive to change all that string-matching lookup into method dispatches. But, worth it? I’d argue that it is.

If you’re writing Cucumber, do you use selector_for, spurn it despite its availability, or use a new version without the method? Do you place your selectors directly in step definitions? Do you have some other lookup facility?

Getting Sequential: A look into a Factory Girl Anti-pattern

I’ve noticed a handful of what I’d consider Factory Girl anti-patterns, either through pull requests, GitHub issues, or questions spread across Stack Overflow, Google Groups, etc. The more common they are, the more I realize that it will benefit plenty of other developers down the road if a list of anti-patterns can be identified and what to do instead.

The one I’d like to outline today is based off of a request to make Factory Girl sequences reset after every test run. The idea of resetting sequences after every run is that then you can start to program against these names in step definitions. For example, imagine the factory:

FactoryGirl.define do
  factory :category do
    sequence(:name) {|n| "Category #{n}" }
  end

  # ... more factories
end

And the scenario:

Scenario: Create a post under a category
  Given a category exists
  And I am signed in as an admin
  When I go to create a new post
  And I select "Category 1" from "Categories"
  And I press "Create"
  And I go to view all posts
  Then I should see a post with the category "Category 1"

That first step is given to us by Factory Girl (check out the getting started docs in Factory Girl to find out more) and will create a category without overriding any of the defaults provided by that factory.

If all sequences were reset after every run, it’d be safe to assume that the category created as the first step of the scenario will create a category named “Category 1”.

There are two problems with this approach. Firstly, we’re now tightly coupling the scenarios to naming conventions outlined in the factories. If the sequence logic changes, all the scenarios that rely on that naming convention will break. Not fun. Secondly, it’s not at all obvious where that “Category 1” record was created. If I’m a developer who’s looking at this code with a fresh set of eyes, I’d assume that this category was created from a global setup or seed file instead of being assigned in the first step of the scenario.

The solution is to name the category explicitly.

Scenario: Create a post under a category
  Given a category exists with a name of "Category 1"
  And I am signed in as an admin
  When I go to create a new post
  And I select "Category 1" from "Categories"
  And I press "Create"
  And I go to view all posts
  Then I should see a post with the category "Category 1"

The first step is, again, given by Factory Girl and allows us to set the name attribute. It’s now immediately obvious that the category we’re creating is the one we’re using throughout the scenario, we’re not reliant on a naming convention outlined by Factory Girl, and any other developer looking at the code won’t question where this category was created.

Factory Girl’s sequences are a great way to ensure that models with uniqueness don’t run into any naming collisions, and it’s handy to have recognizable data (instead of having every bit of data in tests runs be virtually the same). That said, when writing tests against data where the values matter, don’t leave those values to chance; assign them outright. Give them weight. The intent is much more obvious, less error prone, and requires no patches to Factory Girl.

Interacting with a Javascript confirmation from Selenium

Ever needed to interact with a Javascript confirm() dialog? Here’s how to do it in Selenium from Cucumber:

When /^I confirm the dialog$/ do
  page.driver.browser.switch_to.alert.accept
end

When /^I cancel the dialog$/ do
  page.driver.browser.switch_to.alert.dismiss
end