Get Your C On

This year, whyday happened to fall on the first day of Capeco.de. I’d been interested in playing with C for a while, so I decided to sit down and start learning. The K&R helped a lot, but digging through both Redis and Potion were a lot more insightful of how C could actually be used.

Since then, I’ve been trying to get some C in my Ruby by writing gems with C extensions. This isn’t hard to do, per se, but it was difficult to find a comprehensive list of requirements (as well as guidelines for organizing the code). I ended up looking at one of the most-used Ruby gems with C - Nokogiri.

Where to Start

You’ll want a way to test the code, as well as the rake-compiler gem. I don’t test my C explicitly - think of it as a handful of private methods. You’ll want to test the public methods on whatever classes and modules you write, so a C testing framework is overkill. Finally, you’ll want to become acquainted with ruby.h.

I decided to start simple and write a Sieve of Eratosthenes. I’d written one in pure Ruby and wanted a basic translation to C. I was also interested in benchmarking the code since I knew C would be a lot faster in this instance.

The sieve gem can be found here and its source here.

Directory Structure

A C extension’s directory structure is very similar to other Ruby gems; the only addition is an ext directory that will store files necessary for generating a Makefile and compiling the code. This is where your C files and their headers will go. In my case, these files are located in ext/sieve.

extconf.rb

extconf.rb is what will generate your Makefile. You’ll need to require "mkmf" and then call create_makefile("your_gem/your_gem"). The mkmf documentation is an excellent resource if you want to include other libraries or customize anything. My gem is straightforward so all I did was ensure the Makefile was created.

sieve.h

My sieve.h is very straightforward and doesn’t really need any explanation.

sieve.c

This is the meat and potatoes of the gem.

At the top of the file, you’ll need to #include <ruby.h> as well as any other headers you need.

You’ll also need an Init_your_gem() function that will be called similarly to main(). This is where I create the structure of my classes and modules for my sieve. I create a Sieve module, add a sieve instance method, and then have the Numeric class include Sieve.

Finally, there’s the sieve function itself. It returns a Ruby object which is of type VALUE. It also accepts a Ruby object (self), which is also of type VALUE. An important reminder is to make sure you free any memory you allocate or your gem will leak memory, just as you would in C.

The /lib directory

Although most of the actual work is done in C, we’ll want to have a bit of Ruby in the /lib directory. I’ve written a scaffold of the module at lib/sieve.rb, which has a require "sieve/sieve" at the top of the file (remember in extconf.rb when we passed a string to create_makefile? That’s it.).

The Rakefile

Being able to compile the gem and run the tests is important, which is why I mentioned the rake-compiler gem earlier. After requiring rubygems, rake, and your library, you’ll want to require "rake/extensiontask". That’ll give you a couple of handy rake tasks, namely clean and compile.

I like to set my default task to run tests, but you’ll want a couple prerequisites to that task: clean and compile. This will ensure that you’re rebuilding your gem and running with the latest compiled version.

Since I’m using Cucumber, it looks like this:

require "cucumber/rake/task"
Cucumber::Rake::Task.new(:cucumber => [:clean, :compile]) do |t|
  t.rcov = true
end

task :default => :cucumber

I also have my benchmark task here so I can find out how much more performant this library is compared to a pure Ruby implementation.

Testing

You’ll want to test your C extension just like any other Ruby gem. I prefer Cucumber but anything will do. I used a scenario outline for some of the basic primes and then found a file of the first one million primes for some heavy-duty lifting. I also tested that if enough memory couldn’t be allocated, it would raise a Ruby NoMemoryError exception.

Since you’re going to want to run the features against the latest changes of your gem (and not a version of the gem that’s installed), you’ll want to modify the load path within your test helper (test/test_helper.rb, spec/spec_helper.rb, or features/support/env.rb). My env.rb looks like this:

$LOAD_PATH.unshift(File.dirname(__FILE__) + '/../../lib')
require "sieve"
require "spec/expectations"

Building the gem

The gemspec for a Ruby C extension is fairly straightforward. The only thing you’ll need to add is to set the spec’s extensions attribute to the path to the extconf.rb file.

Gem::Specification.new do |s|
  s.require_paths = ["lib"]
  s.extensions = ["ext/sieve/extconf.rb"]
  # ... rest of the gemspec
end

As with any gemspec, you’ll want to make sure that you list the .c and .h files within files.

Results

Armed with this, you should be able to go and write Ruby C extensions to your hearts content. As for my Sieve experiment, here’s the pure-Ruby implementation of the sieve:

# usage:
#   >> sieve 100
#   => [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]
def sieve(n)
  numbers = (0..n).map {|i| i }
  numbers[0] = numbers[1] = nil
  numbers.each do |num|
    next unless num
    break if num**2 > n
    (num**2).step(n, num) {|idx| numbers[idx] = nil }
  end
  numbers.compact
end

My benchmarks were running the sieve on numbers from zero to one million in steps of 100,000. No memoization is used for either form.

On Ruby 1.8.7, here are the results from my benchmark:

                   user     system      total        real
sieve method   4.460000   0.060000   4.520000 (  4.522069)
Numeric#sieve  0.040000   0.000000   0.040000 (  0.046349)

Ruby 1.9.2 is significantly faster, but still doesn’t hold a candle to the C extension:

                   user     system      total        real
sieve method   2.410000   0.060000   2.470000 (  2.468430)
Numeric#sieve  0.050000   0.000000   0.050000 (  0.049053)

What I Learned

Writing C is both fun and can enhance performance of number-crunching and other fun things. It has it’s place and is a great addition to any Rubyist’s toolbox. Have you written any C extensions purely for performance gains? If you’re open to sharing the context, I’d love to hear about it!

Hoptoad and Javascript, Sitting in a Tree, S-E-N-D-I-N-G

With an open API, Hoptoad, the app error app, can track errors from any sort of app you want: Rails, Sinatra, Rack, PHP, C#… the list goes on. We didn’t want to leave front-end developers in the dark when their Javascript throws errors, so we’ve added a Hoptoad notifier for Javascript!

I’m sure you’re asking, “How do I participate in this awesomeness?”. It’s easy.

If you’re using the hoptoad_notifier gem, upgrade to the latest version (2.3.7 at the time of this update) and update your config/initializers/hoptoad.rb:

HoptoadNotifier.configure do |config|
  config.api_key = 'YOUR-PROJECT-API-KEY'
  config.js_notifier = true # ADD ONE LINE FOR JAVASCRIPT ERROR TRACKING!
end

Yes, that’s it.

The gem injects Javascript right after your <head> tag. Done!

Not using hoptoad_notifier gem?

If you’re not using the hoptoad_notifier gem, you’ll just need to include a couple of lines (at the top of your <head>):

<script type="text/javascript" src="http://hoptoadapp.com/javascripts/notifier.js"></script>
<script type="text/javascript">
  Hoptoad.setKey('YOUR-PROJECT-API-KEY');
</script>

The details

You may be proclaiming, “Oh no, Javascript at the top of <head>! What are you thinking?” I’ll let you in on a little secret: we want our Javascript loaded before any other Javascript. That way, if one of those libraries throws an exception, we’re there to let you know!

The nitty-gritty details: we’re currently only supporting browsers that support the onerror event (Firefox 2+, IE6+). You’ll also be including your project API key on the page. This is different from your user API key, so there’s no need to worry about someone deleting your projects or users. We’ll continue to expand this feature and let you know as we progress. If you have any suggestions or spot a bug, let us know!

So, what’re you waiting for? It takes 30 seconds to sign up; go grab an account and start tracking your Javascript errors now!

Capturing Errors on a Global Scale

Has your database gone down recently? What about timeouts from third-party services? Exceptions can be raised anywhere in your app, and Hoptoad’s here to track them for you. Problem is, these exceptions can manifest themselves in your controllers or models, and that’s not helpful when you have to sort through all of them because of Hoptoad’s grouping logic.

To help, we’ve introduced global error classes for all paid accounts. Per project, you can list exception classes that you’d like to be grouped together in Hoptoad. We handle the nitty-gritty so your life is easier when you’re trying to resolve these issues that come up. Since we’re not ones to leave you high and dry, we’ve started everyone off with a list of common global exceptions from Rails projects:

• Mysql::Error
• MemCache::MemCacheError
• Mongrel::TimeoutError
• ThinkingSphinx::ConnectionError
• Net::HTTPFatalError
• OpenURI::HTTPError
• ActiveResource::ResourceNotFound
• ActiveResource::TimeoutError
• ActiveResource::ServerError
• Net::SMTPServerBusy
• Net::SMTPFatalError
• Net::SMTPSyntaxError

To change the list of global errors, go edit a project.

Under the name of the project, you’ll see a list of Global Error Classes (again, prepopulated with a handful of common errors). We separated by new lines, but spaces or commas will work as well.

Once those are set up, you’re ready to roll! Here’s a snapshot of how a global error shows up after adding RuntimeError to the list.

That’s about all there is to it! Note that this isn’t retroactive, so your old errors will still be in the same groups as they were before. Another reminder: if you’re on our Egg plan, be sure to upgrade; your first month is free!

This should_change your mind

There’s been a bunch of hullabaloo about our recent deprecation of the Shoulda macros should_change and should_not_change. A lot of people have asked, “why remove it?” Short answer: we don’t use it anymore. Long answer: it’s confusing and may encourage bad programming practices.

Consider this example of should_change:

context "updating a post" do
  setup do
    @post = Post.create(:title => "old")
    put :update, :post => {:title => "new"}, :id => @post.to_param
  end

  should_change("the post title", :from => "old", :to => "new") { @post.title }
end

This reads well and seems to be fairly straightforward. Sadly, this doesn’t work because of how should_change works internally (hence the confusing part). It would actually need to be written like this:

context "given a post" do
  setup do
    @post = Post.create(:title => "old")
  end

  context "updating" do
    setup do
      put :update, :post => {:title => "new"}, :id => @post.to_param
    end

    should_change("the post title", :from => "old", :to => "new") { @post.title }
  end
end

The @post instance variable needs to be assigned a context above the context containing should_change. I told you it was confusing.

Now, onto the bad practice aspect. This tests that the post was saved! Come on, we don’t want to do that here. That’s why we write Cucumber scenarios! We’re testing the controller, right? If that’s the case, then we shouldn’t care whatsoever if update_attributes worked with specific data passed to it, just that it was called and that we handle the result accordingly.

context "updating a post" do
  setup do
    @post = Factory :post
    Post.stubs(:find => @post)
  end

  context "when the update is successful" do
    setup do
      @post.stubs(:update_attributes => true)
      put :update, :post => {:title => "new"}, :id => @post.to_param
    end

    should set_the_flash.to(/updated/)
    should redirect_to(posts_path)
  end

  context "when the update is unsuccessful" do
    setup do
      @post.stubs(:update_attributes => false)
      put :update, :post => {:title => "new"}, :id => @post.to_param
    end

    should render_template(:edit)
    should assign_to(:post).with(@post)
  end
end

This is more inline with what we really want to test; that the controller handled update_attributes correctly when the update worked and when it didn’t. Leave the model logic and whether it should save or not in the model (and its tests), and test the controller to ensure it handles whether the model was updated.

I mentioned before that Cucumber scenarios should cover overall integration. Something like these would work:

Scenario: Update a post successfully
  Given I am signed in
  And I authored a post titled "Great Scott"
  When I go to edit the "Great Scott" post
  And I fill in "Title" with "Willard Scott"
  And I press "Update"
  Then I should see the flash message "Your post was updated successfully"
  And I should have authored "Willard Scott"

Scenario Example: Update a post unsuccessfully
  Given I am signed in
  And I authored a post titled "Great Scott"
  When I go to edit the "Great Scott" post
  And I fill in "<required field>" with ""
  And I press "Update"
  Then I should see error messages
  When I fill in "<required field>" with "<valid value>"
  And I press "Update"
  Then I should see "Your post was updated successfully"
  Examples:
    | required field | valid value       |
    | Title          | Awesome Post Name |
    | Body           | Awesome story     |

I’m sure this isn’t the only way people were using should_change, but hopefully this clarifies how deprecating the methods will encourage better tests.

Do you use should_change and think it’s by far the best way to test certain pieces of code? I’d like to see some people who’ll sorely miss it post some real code (Pastebin or Gist).

factory_girl 1.3: integrating effectively with Rails 3

factory_girl has always had little Rails-specific code. It only depends on Ruby, and in order to use it in your application, it asks three things of your models:

• They can be instantiated without parameters
• All properties have writer methods
• They respond to #save! (this is only necessary for the “create” strategy)

However, despite this simple API, there’s one feature in factory_girl that has traditionally caused some issues with Rails: automatic factory definition loading.

The definition loader has simple rules: if it finds any of test/factories.rb, spec/factories.rb, test/factories/**/*.rb, or spec/factories/**/*.rb, it will load those files in no particular order. For most applications, this just makes things a little nicer.

Hacking around Rails dependency management

In early versions of factory_girl, it looked for definitions as soon as you loaded factory_girl.rb. Unfortunately, this broke down when Rails 2.1 introduced gem dependency management. Loading factory_girl during the application initialization phase meant that the factory definitions might reference your models before all the gems and plugins they depended on were fully loaded. Contributer technicalpickles fixed that by detecting Rails and deferring definition loading until Rails was initialized. This solution worked effectively for the entire Rails 2.x series of releases.

Bundler: effective but strict dependency management

The upcoming Rails 3 release uses a new library for managing application dependencies: bundler. Bundler’s behavior is more predictable and understandable than Rails 2’s dependency manager ever was. However, with the introduction of Bundler, two important changes were introduced that impacted many libraries that integrate silently with each other:

• Rails is no longer initialized by the time your gem is loaded
• Load order is no longer configurable, so you can’t depend on another gem’s constants being defined when you integrate with it

Yehuda Katz, the author of Bundler, discussed this topic in depth on his blog. In his discussion, he suggests a solution that could be added to Rubygems:

s.integrates_with "rails", "~> 3.0.0.beta2", "haml/rails" 

However, this functionality doesn’t exist yet, so we’ve decided to create a separate factory_girl_rails gem. This way, we can declare an explicit dependency on Rails, including the version, and automatic definition loading still takes place for Rails apps without any additional application code.

What this means for you

This doesn’t change much for an application that uses factory_girl. It boils down to this:

• If you’re using Rails 2, just depend on the factory_girl gem, and Rails 2 will pick up the existing rails/init.rb and load your definitions.
• If you’re using Rails 3, just add factory_girl_rails to your Gemfile. This gem depends on factory_girl and will set up definition loading for you.

A lesson on “easy to add” features

I originally added this feature to factory_girl because it was convenient, and seemed easy to add. However, this trivial feature has turned out to be the most difficult to maintain part of factory_girl, and is the single feature that is most likely to outright break an application. I plan on maintaining this feature because we’re used to having it and we’ve worked out most of the kinks at this point, but next time you consider adding an “easy” feature to your own library, also consider whether or not it’s essential to your library’s core mission.