How to test Sinatra based Web Services

Sinatra is a fantastic lightweight framework for building web services. We’ll use it as the application framework for the HTTP endpoints in our Service Oriented Architecture.

The testing approach will be in-process, which means that the test suite is running in the same Ruby process as the web service. This eliminates the need to run an external HTTP web server.

Application structure

Unlike Rails, Sinatra isn’t all that opinionated on how you set up your application (it has a few sensible defaults), which can lead to a lot of open questions on how to structure the application.

Here’s the directory structure we’ll use for our example application.

app/
app/models
app/my_service.rb
client/
client/lib/my_client.rb
client/my_client.gemspec
config/
spec/

Outside-in development with a Client Gem

For internal services we’ll build a client gem directly into the project. With the client embedded in our codebase we can follow outside-in development cycles — Features start with request specs from the client side, followed by the addition of end points to the service, and then unit tests within the application.

This allows the us to dogfood our client gem and bring it in as the first step of our development process.

To achieve this we need to configure a few things.

1. Set up the project gemfile to use a local copy of the client in test mode

Include the local client gem in test suite.

# Gemfile
group :test do
  gem 'my_client', path: 'client'
  gem 'webmock'
  # ...
end

2. Use webmock to send all http requests to the service

Instead of booting up a web server every time the test suite is run we’ll mount the Sinatra service as a rack application with webmock.

This allows the client to talk directly to the mounted rack application without going through HTTP or a web server.

# spec/spec_helper.rb
RSpec.configure do |config|
  config.include WebMock::API

  config.before(:each) do
    MyClient.base_url = 'http://www.example.com'
    stub_request(:any, /www.example.com/).to_rack(MyService)
  end
end

3. Use the client in our request specs

Once MyService is mounted as a rack application we can use the client gem directly in our test suite.

# spec/requests/widget_management_spec.rb
require 'spec_helper'

describe "Widget management" do
  it "creates a Widget" do
    # set up fixture data if needed

    response = MyClient::Widget.create(widget_params)

    # assert expectations on the response
  end
end

Private gem hosting

To use the client gem in other projects we can use a private gem hosting service like Gemfury. This allows us to include the client via gemfile in our other projects.

# Gemfile
source 'https://452f6E403CDph10714e41@gem.fury.io/me/'
gem 'my_client'

source 'https://rubygems.org
# ...

Takeaways

• Sinatra is great framework for creating lightweight web services.
• Webmock allows us to test the client in-process against a rack application.
• Use private gem hosting to distribute the shared client.

Written by Harlow Ward

Using Capybara to test JavaScript that makes HTTP requests

Testing an application that integrates with an HTTP service can be tricky:

• Making a connection to another server is slow.
• Your tests can become dependent on data that lives outside the test suite.
• Going offline means you can’t run the tests.
• The service going down causes continuous integration to fail.

It may seem like a world of pain, but you’re not going to let a few HTTP requests get between you and your TDD, are you?

Story Time

Let’s say you’re making an internal dashboard for your site, which allows you to view key health metrics. Among other things, you want to display the current status of the build, so that you know whether or not it’s safe to deploy. Your build runs on a third party service, so you need to query their API.

From the Top

You start with an acceptance test:

feature 'health dashboard' do
  scenario 'view health dashboard' do
    create_passing_build
    sign_in_as_admin

    view_health_dashboard

    page.should have_passing_build
  end

  def create_passing_build
    FakeContinuousIntegration.stub_build_message(passing_build_message)
  end

  def view_health_dashboard
    visit '/admin/health_dashboard'
  end

  def have_passing_build
    have_content(passing_build_message)
  end

  def passing_build_message
    'All 2,024 tests passed.'
  end
end

The test immediately fails because of your missing fake, and you TDD your way into this simple class:

class FakeContinousIntegration
  def self.stub_build_message(message)
    @@build_message = message
  end
end

Your testing loop leads to this controller action:

def show
  @latest_build_message = ContinousIntegration.latest_build_message
end

Details Emerge

At this point, it’s time to drop down into a unit test. After a few cycles, you end up with this test:

describe ContinousIntegration, '.latest_build_message' do
  it 'parses the build message from the CI server' do
    message = 'Great success'
    response = { 'message' => message }.to_json
    Net::HTTP.stubs(get: response)

    result = ContinousIntegration.latest_build_message

    Net::HTTP.should have_received(:get).with('buildserver.com', '/latest')
    result.should == message
  end
end

And the implementation emerges:

class ContinousIntegration
  HOST = 'buildserver.com'
  LATEST_BUILD_PATH = '/latest'

  def self.latest_build_message
    new(LATEST_BUILD_PATH).build_message
  end

  def initialize(path)
    @path = path
  end

  def build_message
    data['message']
  end

  private

  def data
    @data ||= JSON.parse(download_build)
  end

  def download_build
    Net::HTTP.get(HOST, @path)
  end
end

Connecting the Dots

With your unit test passing, you return to the integration test. At this point, you no longer receive any errors about missing constants or undefined methods. Instead, everything runs as you expect, but you’re getting a different build message: “All 126 tests passed.” Where did that come from? As the gears start turning, you realize that your test is fetching the actual build status.

There’s no reason to make an actual HTTP request in the test, so you reach for WebMock.

# in spec/support/fake_continuous_integration.rb
stub_request(:any, /buildserver.com/).to_rack(FakeContinuousIntegration)

Now any Net::HTTP requests to “buildserver.com” will route directly to your fake, rather than actually opening a request. All that’s left is to flesh out our fake a little more:

require 'sinatra/base'

class FakeContinousIntegration < Sinatra::Base
  def self.stub_build_message(message)
    @@build_message = message
  end

  get '/latest' do
    content_type :json
    { 'message' => @@build_message }.to_json
  end
end

Tests pass, page looks good. Time to ship.

Two Words: Java. Script.

It doesn’t take long before somebody decides that it’s not a good idea to query your build server in the middle of a request. Luckily, you realize that your build server comes fully equipped with a JSONP API, so you can offload that request to the browser:

// in app/assets/javascripts
function fetchBuildMessage(target) {
  $.ajax({
    url: 'http://buildserver.com/latest',
    dataType: 'jsonp',
    success: function(response) {
      $(target).text(response.message);
    }
  });
}

// in your .erb view
fetchBuildMessage('#buildMessage');

Of course, your fake doesn’t implement this JSON endpoint, so you have to fix that:

get '/latest' do
  callback = params[:callback]
  data = { 'message' => @@build_message }.to_json
  "#{callback}(#{data})"
end

You tag the scenario as javascript and let capybara do its magic, but even after fixing your fake, it’s regressed back to hitting the actual build server over HTTP. Testing this HTTP service was bad enough, and many developers shy away from testing their JavaScript, but the combination of the two is a formidable opponent. After coming this far, though, you’re ready to do what it takes.

What The World Needs Now Is Threads, More Threads

Tools like WebMock are great, but when testing JavaScript, it’s a seperate browser process that loads the page, and not your Ruby test process. That means that the request to your build server isn’t going through Net::HTTP; the requests are coming from Firefox or capybara-webkit, and those tools are gleefully unaware of your feeble attempts to reroute HTTP traffic. Fortunately, there are only two steps remaining towards the testing Holy Grail:

• The JavaScript is going to make an actual HTTP connection, so we need to have an actual HTTP server running somewhere with our fake.
• The JavaScript is talking to “buildserver.com,” which we don’t control, so we need to get it to use a configurable host.

We can use Capybara to solve the first issue. Instead of mounting the application using WebMock, we run it using Capybara::Server:

class FakeContinousIntegration < Sinatra::Base
  def self.boot
    instance = new
    Capybara::Server.new(instance).tap { |server| server.boot }
  end
  # ...
end

Next, we can put the CI host name in a constant. In most environments, this will be “buildserver.com”, but in the test environment, we can get the URL from the server we just spun up:

# config/environments/{development,staging,production}.rb
CI_HOST = 'buildserver.com'

# in spec/support/fake_continuous_integration.rb
server = FakeContinuousIntegration.boot
CI_HOST = [server.host, server.port].join(':')

Now we just need a parameter in our JavaScript function:

// in app/assets/javascripts
function fetchBuildMessage(host, target) {
  $.ajax({
    url: 'http://' + host + '/latest',
    dataType: 'jsonp',
    success: function(response) {
      $(target).text(response.message);
    }
  });
}

// in your .erb view
fetchBuildMessage('<%= CI_HOST %>', '#buildMessage');

Made it, ma! Top of the world!

Do-it-yourself @font-face web service

Phil recently designed Developers Developers Developers Developers, an event we’re running for Boston-area high school and college students:

It looks awesome in part because of liberal use of @font-face.

These days, it’s simple to use @font-face. Just use a stylesheet referencing a web service like Google Font Directory:

 <link href='http://fonts.googleapis.com/css?family=Lobster&subset=latin' rel='stylesheet' type='text/css'>

… then use the font as part of a normal CSS font stack:

#schedule .event-time {
  font-family: 'Lobster', arial, sans-serif;
}

What if the font isn’t on an @font-face web service?

There are plenty of commercial @font-face web services, which Kyle has covered in detail.

However, there are cases where the font you want to use isn’t on an existing service. Or, the font you want to use requires explicit permission from the font’s creator before you can use it.

We ran into this problem on Developers Developers Developers Developers.

First attempt: serve fonts from the app

To solve it, we included the font files in our git repository. This worked, but wasn’t ideal:

• Unnecessarily adds large files to the repo.
• No HTTP caching by default, downloading font files on subsequent requests.
• Repetition. Soon after, I wanted to use League Gothic again on another app.

Second attempt: serve fonts directly from Amazon S3

So our next attempt was to upload the @font-face stylesheet and font files to S3 and serve them directly from there like:

<link href='http://our-bucket.s3.amazonaws.com' type='text/css'>

At first, this seemed to work well. We could set some HTTP headers to handle caching and everything seemed like it was in the right place:

• application code in version control
• assets in an asset host (S3)

However, Firefox (and probably some versions of IE), balks at this technique because of the same origin policy.

On S3, you’re not allowed to tweak the Access-Control-Allow-Origin HTTP header to allow Firefox to serve your font from S3. There’s a long Amazon thread where S3 customers are asking for this ability.

However, even if you could set the Access-Control-Allow-Origin header on S3, you also want the Content-Type, Cache-Control, and Expires to all also be set in a standard way. It’s a pain to have to do that manually.

Solution: serve fonts from a DIY web service

Our final solution was a small Sinatra app called Fist Face. It is now open source under the MIT License and it solves all the problems we experienced.

Fist Face

It works exactly the same way as Google Font Directory, Typekit, or any other @font-face web service, except that you have full control over it:

<link href='http://your-font-face-service.com/league-gothic.css' rel='stylesheet' type='text/css'>

To use Fist Face, this is all you have to write:

# Gemfile
source 'http://rubygems.org'
gem 'sinatra',  '~> 1.1'
gem 'fistface', '~> 1.0'

# config.ru
require 'rubygems'
require 'bundler'
Bundler.require
run FistFace

Then deploy it. Follow a few conventions in the README regarding your asset host (ex: S3) and a few minutes later, you’ll be serving fonts via your own @font-face web service.

Why would I want to do it myself?

If you’ve run into the same issues that we did, have you solved this problem differently?

Among the weaknesses of this approach are:

• No CDN. (patches welcome!)
• Up to you to write a decent cross-browser CSS file.
• The font files you use might be old-school (not “hinted” for computer screens).
• Requires deeper knowledge of typefaces.

With an open mind, the last weakness is actually a strength. Typography is pretty interesting. You’ve got all these independent type foundries doing beautiful work and some release their fonts for free under permissive licensing.

In that way, hunting great typefaces for your @font-face web service is like building your own art collection. It can differentiate your work.

So, when your friends ask you, “whoa, what font is that?”, you can tell them, “you’ve probably never heard of it” … which I’ve heard is hip.

Written by Dan Croak.

Video: Sinatra at Boston.rb, part 3

This is the third in a series of short videos. They feature Blake Mizerany discussing Sinatra and Heroku in great technical detail at September’s Boston.rb. Watch Part 1 and Part 2 if you’d like.

legacy APIs

Blake uses “legacy APIs” as a common use case for Sinatra. The reason params[:splat] and params[:matches] and super-flexible routing in Sinatra exists is because of work done on an existing non-RESTful, ugly API that sent all the data back in the URL itself (not in JSON or some other format).

This seems to be consistent with the philosophy of “Don’t fear the URLs”.

return values must respond to “each”

In order to be Rack-compliant, your return values of Sinatra routes should respond to each. Blake mentions the Ruby 1.9 gotcha that the Rack spec also mentions:

The Body must respond to each and must only yield String values. The Body should not be an instance of String, as this will break in Ruby 1.9. If the Body responds to close, it will be called after iteration. If the Body responds to to_path, it must return a String identifying the location of a file whose contents are identical to that produced by calling each. The Body commonly is an Array of Strings, the application instance itself, or a File-like object.

templates

get '/' do
  erb(:index)
end

The file would be named “index.erb”. Sintra doesn’t use the same “name.format.template” convention because the routes are supposed to be “less magical”. You should know what your response format is based on which route you’re already in.

begin/rescue vs. throw :halt

I hadn’t seen a discussion of these two control structures before. begin/rescue is meant for exceptions and throw :halt is meant for returning a value and returning to another section of code.

The promise is that this will come in very handy as we dive deeper into Sinatra.

get '/' do
  halt(404) unless session[:user]
  # ...
end

As Sinatra processes the route, it’s listening for halts and passes.

when Sinatra, when Rails?

I cut a section out for space reasons where Blake talks about thanking David Heinemeier Hansson for writing Rails. Blake had been trying to get Ruby into companies for years without much success, then Rails came along and made it acceptable.

Rails:

• get bigger apps going quickly
• less pain to get a CRUD app going

Sinatra:

• more control
• choose your own conventions
• legacy APIs
• small apps going quickly
• take full advantage of Rack, no special magic to be compatible

shotgun

At the end of this video, I included a section where Blake shows, in response to an audience question, how to append to an existing body via response.body.

I thought his shotgun command was more interesting, though.

Shotgun is “an automatic reloading version of the rackup command that’s shipped with Rack.” It gets you “application-wide reloading of all source files and templates on each request” by forking into a child process, processing, then exiting the child process.

Note that this is not part of Sinatra at all. This is obviously great for a development environment, but keeps Sinatra clean by being a third party.

next

• Moving from “classic Sinatra” to Sinatra::Base
• Why throw :halt is so cool in practice.