Redis partial word match — you (auto)complete me

We can use partial word matching to rapidly search strings of text such as Names, Cities, States, etc.

We can do this by indexing strings into Redis sets based on partial matches of the string. The indexing process takes a string and breaks it into left-bound substrings which are placed into the appropriate Redis sets.

Partial word matching movie titles

In this example we’ll enable partial word matching queries of movie titles.

The movie titles will be stored in plain text. The keys are simple key-value pairs that are stored with the key structure movies:id, where id is incremented:

$ redis-cli get movies:1
"Bad Santa"
$ redis-cli get movies:2
"Batman"
$ redis-cli get movies:3
"Bad Company"

The movie Batman would be decomposed into the following sets (assuming that the indexing started at two characters):

ba
bat
batm
batma
batman

We’ll index the movie titles into sets based on partial match of their titles. To prevent any key collisions in Redis we’ll create keys with the following structue index:abc:movies.

View the indexed ids using Redis smembers command:

$ redis-cli smembers index:ba:movies
1) "1"
2) "2"
3) "3"

$ redis-cli smembers index:bat:movies
1) "2"

Note: The keys for the movie titles have been parameterized (lowercase and spaces replaced with dashes).

We’ll use a Ruby class Autocomplete to wrap the Redis calls:

# autocomplete.rb
require 'redis'

class Autocomplete
  def initialize(partial_word)
    @partial_word = partial_word
  end

  def movies
    redis.sort "index:#{partial_word}:movies", by: :nosort, get: 'movies:*'
  end

  private

  def redis
    Redis.current
  end

  attr_reader :partial_word
end

movies = Autocomplete.new('bat').movies

We’ll use the sort command with the by: :nosort parameter to query the partial word match index and return the matched movie names.

Written by Harlow Ward

Related Reading

Redis Set Intersection - Using sets to filter data

Redis Pub/Sub…how does it work?

irb & script/console tips

Let’s get interactive. You can learn a lot about your application from irb & by extension, script/console.

Wirble: colors

Color matters. You’re picky about your text editor syntax highlighting and maybe you use the excellent redgreen gem for colors in your test backtraces. Gotta have it in irb, too.

First, install Wirble:

sudo gem install wirble

Then, in your ~/.irbrc:

require 'rubygems'
require 'wirble'
Wirble.init
Wirble.colorize

Wirble: history

Wirble has an added bonus: history.

irb(main):001:0> history = "History?"
=> "History?"
irb(main):002:0> exit

Without Wirble, if you drop back into irb, you can’t arrow up to your previous commands. With Wirble, you can.

Wirble: auto-completion

As if that’s not enough, Wirble gives you auto-completion, too. In irb:

>> un_momento = "Spanish for like, 'hold the phone!'"
=> "Spanish for like, 'hold the phone!'"

Type, “un”, then tab:

>> un
un_momento   undef        unless       untaint      untrace_var  until

Local methods

Stick this in your ~/.irbrc:

# Easily print methods local to an object's class
class Object
  def local_methods
    (methods - Object.instance_methods).sort
  end
end

Use the #local_methods method like this:

>> class BasketballPlayer
>>   attr_accessor :name
>> 
?>   def champion?
>>     name == "Kevin Garnett"
>>   end
>> end
=> nil
>> kevin_garnett = BasketballPlayer.new
=> #<BasketballPlayer:0x11988f8>
>> kevin_garnett.name = "Kevin Garnett"
=> "Kevin Garnett"
>> kevin_garnett.champion?
=> true
>> kevin_garnett.local_methods
=> ["champion?", "name", "name="]

Aliases

Don’t forget you can alias in irb. The only one I have is:

alias q exit

It makes irb feel more Vim-ish.

Create test fixtures for a third party service

Your app uses the Flickr API. You want your test suite to use legitimate data but not hit the service.

Given the actual call to the service is:

FlickrClient.search("Bruce Springsteen")

Run the query in script/console and convert the data into yaml:

yaml = FlickrClient.search("Bruce Springsteen").to_yaml

Then use your old Ruby friend, File.open with the write option (“w”) to dump it to a file:

File.open("test/fixtures/flickr/springsteen.yml", "w") { |file| file << yaml }

Add this to your test_helper.rb:

def load_yaml_fixture(path)
  absolute_path = File.join(RAILS_ROOT, "test", "fixtures", path)
  YAML::load_file absolute_path
end

Voila, when you need to mock out calls to the third party service, you can now use this in your test code:

load_yaml_fixture("springsteen.yml")

This pattern is repeatable for any third party service. Just replace the actual call with whatever you’re working on, and name the fixture something intention-revealing.

Print SQL to standard out

# Log to STDOUT if in Rails
 if ENV.include?('RAILS_ENV') && !Object.const_defined?('RAILS_DEFAULT_LOGGER')
   require 'logger'
   RAILS_DEFAULT_LOGGER = Logger.new(STDOUT)
 end

This gets you:

>> videos = Video.limited(3)
  Video Load (0.4ms)   SELECT * FROM `videos` LIMIT 3
=> [#<Video id: 1, ...>]

Without the ~/.irbrc love, you’d be missing:

Video Load (0.4ms)   SELECT * FROM `videos` LIMIT 3

pp

After you’ve required rubygems for Wirble, require pretty print, too:

require 'pp'

This is just a nice way to have better formatting at your fingertips. It’s especially helpful when dealing with an Array of ActiveRecord objects:

>> pp videos
[#<Video id: 1, youtube_id: "TcMklv40YMY", name: "Merb, Rubinius and the Engine Yard Stack", upload_time: nil, view_count: 6171, description: "Google Tech Talks\nOctober 20, 2008\n\nABSTRACT\n\nIn th...", created_at: "2008-12-21 00:32:33", updated_at: "2008-12-21 00:32:33">,
 #<Video id: 2, youtube_id: "JySmT-dGOj0", name: "MERB SPORTS Team-Vorstellung 2008", upload_time: nil, view_count: 495, description: "MERB SPORTS stellt sich für die VDRM-Saison 2008 v...", created_at: "2008-12-21 00:32:33", updated_at: "2008-12-21 00:32:33">,
 #<Video id: 3, youtube_id: "6bc-FNNWIsM", name: "Merb && Moi", upload_time: nil, view_count: 58, description: "Amusing yes?", created_at: "2008-12-21 00:32:33", updated_at: "2008-12-21 00:32:33">]