Writing matchers for shoulda

Shoulda provides assertions that allow developers to quickly test common Rails functionality, such as validations, associations, and controller responses. These assertions have traditionally been packaged as “macros” - class methods that generate test methods. For a while now, we’ve been moving towards matchers internally to provide that functionality. This allowed RSpec and Test::Unit users to access the same test assertions with only a thin wrapper. This past Spring, we cut out most of the wrapper code by eliminating the “macro” concept and instead having Test::Unit users access the matchers directly.

We believe it’s a waste for users to write the same test assertions twice (once for RSpec and once for Test::Unit), and now that shoulda supports them directly, matchers can be the common building blocks for test assertions in both frameworks.

One question we’ve frequently seen since switching to matchers is, “how do I write my own custom macros now?” The short answer is, “you don’t: you write a matcher instead.” The long answer is this blog post.

Old school: macros

If you were going to write a should_be_instance_of macro in the old style, you might do it like this:

def self.should_be_instance_of(class_name)
  should "be an instance of #{class_name}" do
    assert_equal class_name, subject.class.name
  end
end

This macro could be used like:

should_be_instance_of 'User'

This is pretty simple and legible, but it has a few flaws:

• The assertion logic is stuck inside the should block, so it isn’t easy to reuse in RSpec or in other assertions. Creating macro methods like this divides the Test::Unit landscape by making some assertions instance-level assertions and others class-level “macros.”
• The test name is also stuck in the macro, so you can’t write your own more meaningful test name. This leads to test names with a “generated” feeling that don’t adequately describe the intent of the test.
• You’d have to write another macro that’s essentially the same if you want the negative case (should_not_be_instance_of)
• Although this is a simple case, more complex assertions can be more than a hundred lines long. In these cases, you’d want to break them up into multiple methods, which is difficult to do in one-off test method generators.
• Macros are difficult to test, because they generate test methods and can’t be invoked separately

What’s a matcher?

The common matcher API shared between RSpec and shoulda involves creating a Ruby object that responds to four methods:

• matches?: returns true if the subject matches the expectations for this matcher
• failure message: should uses this message if matches? returns false
• negative_failure_message: should_not uses this message if matches? returns true
• description: used if a test name is generated from this matcher

The matcher API itself is not a dependency - it’s just a specification. All you need to write a matcher is Ruby.

To write the same macro as a matcher, you might use this code:

class InstanceOfMatcher
  def initialize(class_name)
    @expected_class_name = class_name
  end

  def matches?(subject)
    @actual_class_name = subject.class.name
    @actual_class_name == @expected_class_name
  end

  def failure_message
    "Expected an instance of #{@expected_class_name}, \
     but got an instance of #{@actual_class_name}"
  end

  def negative_failure_message
    "Didn't expect an instance of #{@expected_class_name}, \
     but got one anyway"
  end

  def description
    "should be an instance of #{@expected_class_name}"
  end
end

def be_instance_of(class_name)
  InstanceOfMatcher.new(class_name)
end

This matcher could be used like:

# Test::Unit with Shoulda
should be_instance_of('User')
should_not be_instance_of('Post')

# RSpec
it { should be_instance_of('User') }
it { should_not be_instance_of('Post') }

The matcher has the following benefits over the macro:

• It’s usable from Test::Unit both at the class and instance level
• It’s usable from RSpec
• It’s a separate class, so breaking out into separate methods (or even more classes) is clean and cheap.
• Although this is a simple case, it allows you to expand into private methods and other classes if necessary
• You get the negative case without as much duplication
• Because the matcher doesn’t generate a test, you can write your own test name if you’d like
• Matchers are easier to test, because they’re just Ruby objects

Why the long face?

One thing you’ll immediately notice is that the matcher for this example is undeniably longer. Part of that is because an RSpec matcher has more infrastructure, so a simple matcher is always longer than a macro. However, I find that most cases are not so simple - otherwise, I wouldn’t need to write a reusable matcher for them.

If you’re willing to introduce a dependency on rspec-expectations, you can use the matcher DSL, which is considerably less verbose for simple cases: https://github.com/dchelimsky/rspec/wiki/Custom-Matchers

Using the matcher DSL, the above example could be written as:

RSpec::Matchers.define :be_instance_of do |expected_class_name|
  match do |subject|
    actual_class_name = subject.class.name
    actual_class_name.should == expected_class_name
  end
end

However, if you want to write matchers that are compatible with both RSpec and shoulda without adding any dependencies, you’ll have to write out the full class. In order to have as few dependencies as possible, all of shoulda’s built-in matchers are written as concrete classes and methods.

Another thing that should be said is that, although matchers tend to use more lines than an equivalent assertion or macro, they’re frequently less terse, so they can be easier to read and write despite their length.

Learn by example

You can find more examples by looking at shoulda itself: https://github.com/thoughtbot/shoulda

The shoulda matchers exercise the matcher concept in just about every way possible, and there are dozens of examples available. If you have specific questions about writing or using matchers, you can always ask on the shoulda mailing list: http://groups.google.com/group/shoulda

If you have existing macros that generate test methods using “should,” they’ll continue to work using the shoulda context framework. Nobody will stop you from writing more, but we highly recommend giving matchers a shot - they’re more flexible, more reusable, and will open your assertions to a wider audience.

The future of shoulda

Shoulda has long been one of our most useful and popular open source projects, and it continues to serve us well as we use it daily. However, there have been some changes over the past year in the way that we use Shoulda, and these changes have led to some decisions about its future. In our post about the Rails 3 roadmap, we briefly covered the changes we made in Shoulda 2.11 as well as our intentions for Shoulda 3. However, we wanted to go over these changes in depth and explain some of our motivation.

Putting things in context

Shoulda has two main components: a test context framework, and a set of helpers for testing Rails applications. The context framework was written because we found the method-based style of Test::Unit tedious, but certain objections were raised to the way RSpec was implemented, and we had existing test suites that could not easily be converted to RSpec. The Rails helpers were written in the form of Shoulda-based “macros” for convenience.

While this approach worked great for those of us using Test::Unit, this meant that the Rails helpers were only accessible in Shoulda-based test suites despite relying on very little in Test::Unit or the Shoulda context framework. In order to bring Shoulda’s Rails helpers to a wider community, they were rewritten as a set of RSpec-compatible matchers, and the existing “macros” were rewritten to use them.

A foot in the other camp

As things in the Cucumber and RSpec community started to pick up, we found more and more that Test::Unit users weren’t getting the latest and greatest. Most of the new assertion libraries we found were written as RSpec expectations, and several improvements to RSpec’s context and matcher DSL made it look too appealing to ignore. We tried out RSpec on a few projects, and found that we really didn’t have to give anything up - Shoulda’s macros were fully supported as matchers - and we found ourselves enjoying RSpec more than Test::Unit.

It was at this point that we decided that Shoulda’s future would focus on RSpec. However, we weren’t going to ditch Test::Unit. We had many existing projects that used Test::Unit and Shoulda, and we certainly didn’t want to abandon the fantastic Shoulda community. After several experiments, we decided not to convert our existing Shoulda test suites to RSpec, largely because they didn’t take advantage of all that RSpec had to offer and didn’t come out any better in the end. This meant that we wanted to keep the context framework functioning. However, we also found ourselves reluctant to spend time and effort on a context framework that we didn’t want to use. Therefore, we needed to come up with a way to keep the matchers working with Test::Unit and make sure the context framework continued to function reasonably, but prevent the overhead of maintaining compatibility with two frameworks.

Rails 3

Rails 3 presented another challenge. Although some of the existing matchers just worked, the general strategy for testing them had to change with Rails 3, and some matchers required conditional forks in the implementations and tests. This was more complicated in the existing macro test suite, and maintaining two test Rails roots for backwards compatibility could potentially slow us down and make it harder to add new features. We preferred the unit-style tests in place for the matchers, and wanted to avoid the rails_root-style test going forward if possible.

Embracing matchers

After refactoring all of the macros to use matchers, we found that the macros all followed a basic pattern: convert a number of hash options into RSpec’s “fluid” syntax, and then create a test that performed one assertion based on a matcher. Based on this discovery, we changed Shoulda’s should method so that it could accept a matcher instead of a test name:

# macro style
should_have_many :users
should_ensure_length_at_least :name, 3
should_not_allow_values_for :isbn, "bad"
should_set_the_flash_to /thank you/i
should_not_assign_to :user

# matcher style
should have_many(:users)
should ensure_length_of(:name).is_at_least(3)
should_not allow_value("bad").for(:isbn)
should set_the_flash.to(/thank you/i)
should_not assign_to(:user)

Converting to this syntax would mean that we no longer needed to support both a set of macros and a set of matchers - all matchers essentially function as macros. In fact, this includes most non-Shoulda macros. We currently support both syntaxes, but you’ll receive a deprecation warning for the older macro style. We’ll also be removing the macros entirely starting with version 3, so now is definitely the time to start embracing matchers.

Separating Shoulda’s contexts

We’re committed to using RSpec, so we don’t want to spend time adding features to Shoulda’s context framework. However, we need it to continue to function for our older test suites, and we understand that some users still prefer the lighter weight Test::Unit framework.

Therefore, we plan on separating Shoulda’s context framework into a separate library starting with version 3. The contexts will still function as always, but we have no plans to further improve this aspect of Shoulda.

Putting it all together

There are a lot of changes here, but they’re not too hard to understand:

• Shoulda is now Rails 3 compatible!
• Shoulda is focusing on the RSpec/Shoulda combination and will primarily support that combination of tools, moving away from Test::Unit
• Shoulda’s “macro” methods are deprecated in favor of using the new should/matcher syntax
• Shoulda’s context framework is moving into a separate library

We’re looking forward to Rails 3 and RSpec 2, and we hope you’re looking forward to Shoulda 3.

The Road to Rails 3

Being at RailsConf 5 has given us the opportunity to finalize a lot of the work we’ve done to prepare our plugins and gems for Rails 3. Thankfully, for many of the most popular gems, we’ve been able to maintain both Rails 3 and Rails 2.3.x compatibility in one gem.  However, we’re taking this opportunity to say goodbye to some of our less widely used plugins, and some we plan on dropping Rails 2 support for altogether.

Obviously, Rails 3 isn’t actually out yet, so what we’re talking about here is Rails 3 beta 4.  We’ll continue to keep things up to date and tested as we all move toward the release of Rails 3.  Your help and patches are more than welcome.

So here is a comprehensive overview of the current status of the projects for both Rails 3 beta 4 and Rails 2.

Paperclip

We released Paperclip 2.3.3 a few days ago. This new version of Paperclip will work with Rails 3. Thanks to the investigation of nragaz and help from isaac and joeljunstrom on github, we worked out the kinks and it should be working with the Rails 2.3.x line, and Rails 3-beta 4.  For the latest version of Paperclip, we’re no longer officially supporting Rails 2.0.x. The earliest version that will work is Rails 2.1.0.  If you need support for an older version of Rails than that, you can use Paperclip 2.3.1.1.

hoptoad_notifier

A few days ago we released hoptoad_notifier 2.2.6 with includes support for Rails 3-beta 4 as well as all versions of Rails 2.x and Rails 1.2.6. 

shoulda

We just released shoulda 2.11.  Along with Rails 3 support, we’re maintaining support for Rails 2.3.x in this latest release.  However, the latest version of shoulda will not support versions of Rails less than 2.3.  If you need support for a version of Rails older than that, you can use a previously released version.

In addition to the Rails 3 support, shoulda 2.11 introduces some dramatic changes to shoulda, including a new way of interacting with all shoulda macros.  The previous way has been deprecated and will be removed in shoulda 3.0.  We’ll make a separate blog post detailing many of the very cool changes to shoulda and more details about the future of shoulda soon, but for now, take a look at the README for the latest information on setting up and using shoulda.

Factory Girl

We just pushed factory_girl 1.3 and factory_girl_rails 1.0.  This new version adds Rails 3 support.  Because of the way that Rails 3 loading has changed, we’ve decided to make a separate factory_girl_rails gem that will be used for when you want to use factory_girl with Rails.  The existing factory_girl gem is used by factory_girl_rails and would be used if you’re using factory_girl outside of Rails.  If you want to use factory_girl with Rails 2 you can continue to use the base factory_girl gem.

Clearance

We just released Clearance 0.9.0.rc1.  This is a release candidate for Clearance 0.9.0.  This new version adds support for Rails 3 but drops support for Rails 2.  Don’t fret, if you won’t be upgrading to Rails 3, you can use a previously released version of the gem (0.8.8).  We’re doing this one as a release candidate because of the dropping of backwards compatibility and the fact that we haven’t had a chance to test the new version in a variety of Rails 3 apps using clearance.

Please flex this release candidate with your Rails 3 apps and let us know how it goes.

Suspenders

Suspenders is currently at 2.3.5 (we haven’t been able to upgrade to 2.3.8 because of bugs we’ve seen with mongrel, webrat, and rack).  We anticipate that Suspenders will be upgraded to Rails 3 a little after Rails 3 final comes out.  But to be honest, we’re actually not sure yet what the upgrade path will look like for applications that are currently tracking Suspenders.  It may be impossible to do without so many conflicts that its not worthwhile.  We’re going to have to work on this more and keep you posted.  Additionally, we’re in the process of making some fairly dramatic changes to Suspenders.  Watch it on github and stay tuned here for more.

High Voltage

Fire in the Disco! We’ve also released High Voltage 0.9.0 which supports Rails 3 and is now a gem (it was previously just a plugin).  The new version also drops support for Rails 2.  If you need the previous, Rails 2 plugin there is a rails2 branch you can retrieve it from.

Pacecar

We also just released Pacecar 1.3 which supports Rails 3 and drops support for Rails 2.  As in the other cases where we’ve done this, you can use the previous version of the gem, version 1.2.0 with Rails 2, or track the rails2 branch.

Squirrel

Squirrel was born out of a desire to make a new query syntax that was dynamic while being clean and simple.  With Rails 3’s introduction of the New Active Record chainable query language, that goal has now been achieved in Rails.  As a result, we’ll no longer be maintaining Squirrel.  It was a fun ride.

Mile Marker

Over time, our workflow slightly changed for how we built applications and we haven’t used Mile Marker ourselves for some time now.  As a result, we’re taking this opportunity to cease maintenance of this plugin and bid it farewell.

Moving on down the road

We’ve gotten more and more familiar with Rails 3 during moving all these gems to it.  Many of the new features it offers are great, and existing features have been improved and cleaned up.  We’re looking forward to Rails 3 finally being released in the coming weeks.  Now that our plugins are up and running it should help us all to transition smoothly and quickly.

Thanks to the core team and various other railsconf attendees for spending time with us this week working on some of this - we’re looking forward to the final version of rails3!

Testing Paperclip on S3 with Cucumber & Factory Girl

We’ve been using Heroku as a staging environment for our latest project. One constraint is a read-only filesystem.

The most apparent effect of this is we cannot allow users to upload files to the filesystem.

Fine. Paperclip has an S3 storage option.

Testing

Webrat has a very nice existing convention for interacting with file fields:

When /^I attach the file at "([^\"]*)" to "([^\"]*)"$/ do |path, field|
  attach_file(field, path)
end

Unfortunately, if we have an S3-backed model like this…

has_attached_file :logo,
 :path           => ":attachment/:id/:style.:extension",
 :storage        => :s3,
 :s3_credentials => {
   :access_key_id     => ENV['S3_KEY'],
   :secret_access_key => ENV['S3_SECRET']
 },
 :bucket         => ENV['S3_BUCKET']

… then we’re going to be doing a RESTful PUT to S3 during each test run.

Incidentally, those ENV variables are Heroku’s config vars. The idea is that you keep that configuration separated from your source control.

Total integration?

You could argue that these PUTs to S3 are a good thing because your Cucumber feature will represent total integration.

While that’s true, I’d rather not pay bandwidth costs and I’m comfortable as long as the correct interface to S3 was called.

So what we’ve landed on is something like this:

Given I am on the new band page
When I attach a "demo_tape" "mp3" file to a "band" on S3
And I press "Upload demo tape"
Then I should see "Band was successfully created"

The only non-standard Webrat step is our new S3 step. Let’s take a look at it:

# features/step_definitions/paperclip_steps.rb

When /^I attach an? "([^\"]*)" "([^\"]*)" file to an? "([^\"]*)" on S3$/ do |attachment, extension, model|
  stub_paperclip_s3(model, attachment, extension)
  attach_file attachment,
              "features/support/paperclip/#{model.gsub(" ", "_").underscore}/#{attachment}.#{extension}"
end

The stub_paperclip_s3 method is coming from a custom Shoulda Macro:

# test/shoulda_macros/paperclip.rb

module Paperclip
  module Shoulda
    def stub_paperclip_s3(model, attachment, extension)
      definition = model.gsub(" ", "_").classify.constantize.
                         attachment_definitions[attachment.to_sym]

      path = "http://s3.amazonaws.com/:id/#{definition[:path]}"
      path.gsub!(/:([^\/\.]+)/) do |match|
        "([^\/\.]+)"
      end

      FakeWeb.register_uri(:put, Regexp.new(path), :body => "OK")
    end

    def paperclip_fixture(model, attachment, extension)
      stub_paperclip_s3(model, attachment, extension)
      base_path = File.join(File.dirname(__FILE__), "..", "..",
                            "features", "support", "paperclip")
      File.new(File.join(base_path, model, "#{attachment}.#{extension}"))
    end
  end
end

class ActionController::Integration::Session
  include Paperclip::Shoulda
end

class Factory
  include Paperclip::Shoulda
end

Fakeweb and more conventions

We’re using the Fakeweb gem like we normally use mocking: expect that something happened, and stop it from actually happening.

We’re also leaning on conventions similar to the actor directory convention we’re also trying.

In this case, we’re expecting our features directory to look like this:

features/support/paperclip/band/demo_tape.mp3
features/support/paperclip/band/demo_tape.aac
features/support/paperclip/band/demo_tape.ogg
features/support/paperclip/band/demo_tape.wav
features/support/paperclip/user/avatar.png
features/support/paperclip/user/avatar.jpg
features/support/paperclip/user/avatar.gif

This allows us to test the expected and unexpected formats by changing this line:

When I attach a "demo_tape" "mp3" file to a "band" on S3

Factories

The reason the stub_paperclip_s3 and paperclip_fixture methods are set up as a custom shoulda macro is so that you can use them in your factory code:

Factory.define :band_with_demo_tape, :parent => :band do |band|
  band.demo_tape { band.paperclip_fixture("band", "demo_tape", "png") }
end

Go or no go?

It’s been quite useful so far for us. How are you testing Paperclip uploads to S3? Do you see any way to improve this step definition or the convention?