GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS

Mar 12

Redis: Data Cheeseburgers

Since we’re on a NoSQL craze in Boston this week, it’s about time we started talking about Redis. We’ll be using it on Hoptoad extensively soon, and it’s a truly useful project.

Explaining Redis is tough, it’s easy to say “a data structures server” or “memcached on steroids” or something more jargon filled. It’s not exactly a key value store, it’s definitely not a relational or document-oriented database. The biggest selling point of Redis is that usually as programmers we have to bend our data into a table or document to save it, but with Redis we can persist data as we conceptually visualize it. Tasty!

You may have heard that Redis is super fast, is being used for everything from analytics to job queues, and even on large sites such as Craigslist. Let’s take a deeper look into what Redis can do for us.

Install

We’re Rubyists, so let’s use the Ruby driver courtesy of Ezra:
git clone git://github.com/ezmobius/redis-rb.git
cd redis-rb
rake redis:install # Clones, compiles, and installs Redis
rake dtach:install # Downloads, compiles and installs Dtach
rake install       # Builds and installs the Redis RubyGem
rake redis:start   # Fires up redis-server in dtach so you can take back control of your terminal
Boom, now we should be able to connect to Redis and start messing around with it. Redis writes to disk asynchronously, so you don’t need to worry about potentially losing all of your data like other in-memory stores. There’s also a VM system in place to swap out values that aren’t being accessed which is explained more in depth here. Check out /etc/redis.conf for the async writing settings along with other configuration values you can tweak. Each of the following examples do the following first to connect:
$ irb -rubygems -rredis
irb(main):001:0> r = Redis.new
=> #>Redis:0xb7b4bb3c @password=nil, @db=0, @thread_safe=nil, @host="127.0.0.1", @sock=nil, @timeout=5, @port=6379, @logger=nil>
The “Redis” object is what redis-rb provides for us to talk to Redis. You could telnet directly to the redis-server, but let’s not get too crazy here.

Keys

Redis can be considered a key/value store in this respect. Conceptually you could think of the Redis object as a giant Ruby Hash, but there’s so much more to Redis than that. (Also, word on the street is there will be native support for Hash datatypes soon).
irb(main):002:0> r["fries"] = "are done"
=> "are done"

irb(main):003:0> r["fries"]
=> "are done"

Expiration

Just like memcached you can set a EXPIRE time on a given key, and use TTL to find out how much longer it has left to live. The annoying part is that unlike memcached, if a given key’s value is changed, the expire time is reset. Also, basing functionality around EXPIRE times is sometimes necessary, but testing it is hard. We’ll cover that in a future post.
irb(main):04:0> r.expire("fries", 60)
=> true

irb(main):05:0> r.ttl("fries")
=> 59

irb(main):06:0> sleep(60)
=> 60

irb(main):07:0> r["fries"]
=> nil

Counters

Redis also offers atomic increments and decrements of numbers, which makes it a perfect fit for download/hit/visitor counters. Atomic modifications means that you don’t have to worry about locking or conflict issues with multiple clients connecting to Redis and performing the same operation on the same keys.
irb(main):08:0> r.incr("orders")
=> 1

irb(main):09:0> r.incr("orders", 5)
=> 6

irb(main):10:0> r.decr("orders")  
=> 5

Lists

This is the first “data structure” like operation that Redis supports, and what really sets it apart from any other NoSQL project that I know of. With Redis you can set up simple lists and use them like a queue, a stack, or however you want. With these commands, the first argument is the name of the key, and the second is the value we’re giving to the list.
irb(main):011:0> r.push_tail "fiveguys", "milkshake"
=> "OK"

irb(main):012:0> r.push_tail "fiveguys", "fries"    
=> "OK"

irb(main):013:0> r.push_tail "fiveguys", "burgers"
=> "OK"

irb(main):014:0> r.list_range "fiveguys", 0, -1
=> ["milkshake", "fries", "burgers"]

irb(main):015:0> r.pop_tail "fiveguys"
=> "burgers"

irb(main):016:0> r.pop_head "fiveguys"
=> "milkshake"
There’s a lot of cool features with lists that I’ll explain in future articles, but this is really just the tip of the iceberg.

Sets

You guessed it: lists with guaranteed unique elements. The nice things about sets is that you can natively check if a given element is in a set, and also do a union, intersection, or diff between sets if need be.
irb(main):017:0> r.set_add "nick_orders", "hamburger"
=> true

irb(main):018:0> r.set_add "nick_orders", "hamburger"
=> false

irb(main):019:0> r.set_add "nick_orders", "hotdog"   
=> true

irb(main):020:0> r.set_add "nick_orders", "cheeseburger"
=> true

irb(main):021:0> r.set_members "nick_orders"
=> ["cheeseburger", "hotdog", "hamburger"]

irb(main):022:0> r.set_member? "nick_orders", "hamburger"
=> true

irb(main):023:0> r.set_add "ralph_orders", "hamburger"
=> true

irb(main):024:0> r.set_add "ralph_orders", "fries"
=> true

irb(main):025:0> r.set_intersect "nick_orders", "ralph_orders"
=> ["hamburger"]

Sorted Sets

Finally, there’s sorted sets. I really didn’t get this at first because of the name. This data type stores unique elements that each have an associated score. You can atomically increment the score, remove multiple elements based on score, and so on.
irb(main):026:0> r.zset_add "menu", 4.99, "cheeseburger"
=> true

irb(main):027:0> r.zscore "menu", "cheeseburger"
=> "4.9900000000000002" # not sure where this rounding error came from!

irb(main):028:0> r.zset_add "menu", 2.99, "fries"
=> true

irb(main):029:0> r.zset_add "menu", 1.99, "coke"
=> true

irb(main):030:0> r.zrange "menu", 0, -1
=> ["coke", "fries", "cheeseburger"]

irb(main):031:0> r.zincrby "menu", 5, "fries"
=> "7.9900000000000002"

irb(main):032:0> r.zrange "menu", 0, -1
=> ["coke", "cheeseburger", "fries"]

irb(main):033:0> r.zrangebyscore "menu", 2, 10
=> ["cheeseburger", "fries"]

Get started!

Here’s some other material to get you started:

If you know of other great starting resources for those new to Redis, feel free to let us know in the comments.

Mar 05

Write Fewer Regular Expressions

Oh man Cucumber is awesome but why do I have to write regular expressions? It’s always like:

Then /^show me the emails$/ do
  puts ActionMailer::Base.deliveries.map do |email|
    email.subject
  end.join("\n")
end

Then /^I should see the "([^"]+)" link once$/ do |link_title|
  assert_select '.deal_title', :text => link_title, :count => 1
end

Then /^I should see "([^"]+)" for "([^"]+)" (.*) field in the response XML$/ do |value, xpath, field|
  @parsed_response.xpath("#{xpath}[@#{field}='#{value}']").should_not be_empty
end

Ugh that’s so ugly and it feels like I’m writing awk or Perl, and while I love me a good awk script it really does not belong near my beautiful Ruby. (They’re so ugly that they broke the syntax highlighter!)

Well you probably know how to replace the first one with a string:

Then 'show me the emails' do
  puts ActionMailer::Base.deliveries.map do |email|
    email.subject
  end.join("\n")
end

But what of our parameterized steps?

I had a vision, a vision of a string with printf-like escapes inside a string. I was pumped and ready to make a patch so I cloned the source off github and started looking through the classes (this was about 11PM). That’s when I discovered that this was done for me by the wonderful scientists at the Cucumber Research Institute! We can re-write those steps like this:

Then 'I should see the "$link_title" link once' do |link_title|
  assert_select '.deal_title', :text => link_title, :count => 1
end

Then 'I should see "$value" for "$xpath" $field field in the response XML' do |value, xpath, field|
  @parsed_response.xpath("#{xpath}[@#{field}='#{value}']").should_not be_empty
end

Here’s how it works: the Cucumber step parser for Ruby turns every string into a regular expression, to reduce it to a previously-solved problem. On the way it translates $foo into (.*). That’s it!

if String === regexp
  p = Regexp.escape(regexp)
  p = p.gsub(/\\\$\w+/, '(.*)') # Replace $var with (.*)
  regexp = Regexp.new("^#{p}$") 
end

This, obviously, is not going to rid the world of regular expressions overnight. Take these two for examples:

Given /^(?:|I )am on (.+)$/ do |page_name|
  visit path_to(page_name)
end

Given /^the user with email "([^\"]*)" is (not )?an admin$/ do |email, status|
  user = User.find_by_email!(email)
  user.admin = status.blank?
  user.save!
end

But these steps are few and far between.

Only you can reduce your regexps. Replace them with strings, today!

Mar 02

Where to stay when you visit us

Hey! You!

You’re a hip Rails dev looking to expand your knowledge and hang out in Boston!

So come take our Rails training class April 7th, 8th, and 9th!

Map of hotels around thoughtbot

And while you’re here you could stay in these sweet hotels:

They do not come with totally deck sneakers and wild haircuts but they are near places that can help.

If you’re looking for something more casual may I suggest:

These two actually has a hard time filling up so you can often get some last-minute less expensive deals*. We recommend checking them first (especially if you’re looking last-minute!).

* This is a rumor we’ve heard and passed along from someone who heard it and passed it along.

Transportation

Cars are for the suburbs; we have the first perfectly hateable public transit system here in the city! When using Google Maps to find your way around you can select “by public transit” to see that much of the time the walking directions are more fun and faster than the T.

To get downtown from Logan Airport you have a few options: taxi, subway, or Silver Line. The taxi is undoubtly the most expensive and fastest option (about 20 minutes).

The Silver Line is a bus that runs from the airport terminal to South Station, which is a few blocks away from our office. You can take the Red Line (subway) from South Station to Park St Station (our office) if you want, or you can walk and enjoy the pun that Summer St turns into Winter St. Funny guys.

You can also catch the 22 Logan Shuttle (bus) to Airport Station, and take the Blue Line (subway) to Government Center; from there you can walk down Tremont St or jump on the Green Line (subway) to Park St Station.

You can’t miss us.

Feb 18

Hoptoad Notifier: Rack and automatic Metal support

If you’re not sending your application errors to our hosted service Hoptoad, take a few minutes and try it out for free today!

The newest version of the official Hoptoad Notifier, 2.2, was released to Gemcutter today, and brings with it support for Rack, automatic notification of exceptions occurring in Rails Metal endpoints, and some Rails 1.2.6 fixes.

Rack support

From the README:

In order to use hoptoad_notifier in a non-Rails rack app, just load the hoptoad_notifier, configure your API key, and use the HoptoadNotifier::Rack middleware:

require 'rack'
require 'hoptoad_notifier'

HoptoadNotifier.configure do |config|
  config.api_key = 'my_api_key'
end

app = Rack::Builder.app do
  use HoptoadNotifier::Rack
  run lambda { |env| raise "Rack down" }
end

This makes the notifier usable in any Rack-based application, including Merb or Sinatra. The README also includes an example for using the Rack notifier in Sinatra.

Atticus mans a very fashionable wine rack

Automatic metal notification

The Hoptoad::Rails module also uses this Rack support to provide automatic Hoptoad notification from within your Rails Metal applications:

if defined?(::Rails.configuration) && ::Rails.configuration.respond_to?(:middleware)
  ::Rails.configuration.middleware.insert_after 'ActionController::Failsafe',
                                                HoptoadNotifier::Rack
end

This means that any exceptions during requests handled by metal endpoints in your Rails application will automatically be sent to Hoptoad, just like requests that are handled by ActionController.

Atticus does metal

You can read about the other changes in 2.2 (including fixes for Rails 1.2.6) in the CHANGELOG.