giant robots smashing into other giant robots

Written by thoughtbot

May 19, 2010

dancroak

Recipe: Ajax searching and filtering

A recipe for adding searching and filtering to your Rails app.

Why?

Provide faster feedback for users. Increase the chance that they’ll find what they want.

Ingredients

The meal

Feature

This feature is testing the non-JavaScript path through the app. I’m not going to show any JavaScript testing in this recipe.

Scenario: Search by fieldnotes
  Given a project exists with a name of "Big Dig"
  And the following reports exist:
    | name            | fieldnotes | project       |
    | Traffic Control | Barricade  | name: Big Dig |
    | Gases, Vapors   | Dust       | name: Big Dig |
  When I sign in as a safety manager of "Big Dig"
  And I go to the "Traffic Control" report page
  And I search for "Dust"
  Then I should be on the reports page
  And I should see "Gases, Vapors"
  And I should not see "Traffic Control"
  And the search field should contain "Dust"

I know I’m going to be writing multiple search scenarios so I created some step definitions for search:

When /^I search for "([^\"]*)"$/ do |query|
  When %{I fill in "search-field" with "#{query}"}
  And %{I press "Search"}
end

Then /^the search field should contain "([^\"]*)"$/ do |query|
  field_with_id("search-field").value.should == query
end

I know I want search-field because the designer has already sliced the HTML and CSS.

HTML

The search box:

- form_tag reports_path, method: :get do
  = text_field_tag 'query', params[:query], id: 'search-field'
  = submit_tag 'Search', id: 'search-reports-button'

Only thing that might be of note is using params[:query] to get the ‘And the search field should contain “Dust”’ test to pass.

The filters:

#filter-list{ style: 'display: none;' }
  %form#filters
    / a bunch of checkboxes

The results:

%tbody.striped#reports-tbody
  = render partial: 'report', collection: @reports

Controller

The index action:

def index
  @reports = Report.search(params)

  if request.xhr?
    render partial: '/reports/report', collection: @reports
  end
end

There’s some authorization around scoping results to the current user’s project that I’ve removed for brevity.

Model specs

There’s about a dozen different ways this data can be filtered. There’s specs for each kind that look something like this:

it 'finds reports assigned to any of a set of safety inspectors' do
  first = create(:safety_inspector)
  second = create(:safety_inspector)
  third = create(:safety_inspector)
  find_me = create(:report, safety_inspector: first)
  me_too = create(:report, safety_inspector: second)
  not_me = create(:report, safety_inspector: third)

  reports = Report.search(safety_inspectors: "#{first.id},#{second.id}")

  expect(reports).to include(find_me)
  expect(reports).to include(me_too)
  expect(reports).to_not include(not_me)
end

There’s one for sending a text query, and a giant one that tests combinations.

Searchlogic

The method through which all searching and filtering passes:

def self.search(params)
  full_text_search(params[:query].to_s).
  location_in(params[:locations].to_s).
  safety_inspector_in(params[:safety_inspectors].to_s).
  safety_category_in(params[:safety_categories].to_s).
  created_after(params[:from].to_s).
  created_before(params[:to].to_s).
  severity_in(params[:severities].to_s).
  state_in(params[:states].to_s).
  descend_by_severity_and_created_at.
  distinct
end

Most items in this chain are custom class methods that wrap Searchlogic:

def self.location_in(locations)
  if locations.blank?
    scoped
  else
    location_id_equals_any(locations.to_array_of_ints)
  end
end

def self.distinct
  select("distinct reports.*")
end

The scoped is a way of maintaining chainability in the common cases where most searching and filtering criteria is blank.

The array of integers is a custom String extension:

class String
  def to_array_of_ints
    split(',').map { |integer| integer.to_i }
  end
end

The idea here is to do all filtering on highly indexed integers, which Postgres handles quickly. It’s also easy to pass in comma-separated ids from jQuery as we’ll see later.

“Full text search” is just SQL ILIKEing while avoiding SQL injection:

def self.full_text_search(query)
  if query.blank?
    scoped
  else
    text_search(query)
  end
end

def self.text_search(query)
  joins(
    <<-eosql
      INNER JOIN locations ON locations.id = reports.location_id
      INNER JOIN users ON users.id IN (
        reports.safety_inspector_id, reports.supervisor_id, reports.subcontractor_id
      )
    eosql
  ).where(
    <<-eosql
      (reports.fieldnotes ILIKE :query) OR
      (reports.name ILIKE :query) OR
      (locations.name ILIKE :query) OR
      (users.name ILIKE :query)
    eosql, { query: "#{query}%" }
  )
end

jQuery

Bind all the necessary events:

$(document).ready(function() {
  $('#clear-filters-btn').click(function() {
    $('#filters :checked').attr('checked', false);
    $('#slider').slider('values', 0, $('#slider').slider('option', 'min'));
    $('#slider').slider('values', 1, $('#slider').slider('option', 'max'));
    searchReports();
  });

  $("#filter-list-btn").click(function(){
    $(this).toggleClass("active");
    $("#filter-list").slideToggle("500");
    return false;
  });

  $('#search-field').keyup(searchReports);

  $('#filters :checkbox').click(searchReports);
  $('#filters :text').focus(searchReports);
});

Build the Ajax call with jQuery.param and call it:

function searchReportsURL(){
  var params = {
    'query' : escape($('#search-field').val()),
    'locations' : checkedIdsForFilter('location'),
    'supervisors' : checkedIdsForFilter('supervisor'),
    'safety_categories' : checkedIdsForFilter('category'),
    'severities' : checkedIdsForFilter('severity'),
    'states' : checkedIdsForFilter('state'),
    'risk_profiles' : checkedIdsForFilter('risk-profile'),
    'from' : $('.left-handle').text(),
    'to' : $('.right-handle').text()
  }

  return '/reports?' + $.param(params) + '&' + (new Date()).getTime();
}

var searchReportsTimeout = null;

function searchReports() {
  if (searchReportsTimeout) {
    clearTimeout(searchReportsTimeout);
  }

  searchReportsTimeout = setTimeout(function() {
    $.get(searchReportsURL(),
      function(data) {
        $('#reports-tbody').html(data);
        $(document).trigger('stripeRows');
      }
    );
  }, 500);
}

For brevity, I’ve omitted some helpers like checkedIdsForFilter that build a comma-separated list of ids for each criteria. You can figure that out based on your own markup and JavaScript’s replace() method.

I kept the timeout, however, because I think it’s important for the user experience. Without it, the search will happen too fast on every keyup(), resulting in a herky-jerky experience. There’s a half-second pause now, but that’s preferred over “strobe-lighting” the user.

Appending the date to the end of the URL is a cache-buster for Internet Explorer.

Bon appétit!

Written by .

May 20, 2008

emill

Jester 1.6: Modern REST

I’m sticking this version number flag into the ground to bring attention to some crucial Jester updates. Namely, full compatibility with IE7, Safari, and Prototype 1.6. If you’ve been following Jester over on Github, or its SVN trunk, then you’ve already gotten these, some from as far back ago as January. There’s not a whole lot past those compatibility fixes, but you can see what there is over at the commit list for Jester at Github, which naturally includes the commit history prior to Jester’s migration from SVN.

Download a tarball of 1.6 (via Github), or check out the trunk in Git with:
git clone git://github.com/thoughtbot/jester.git.

Jester, if you don’t remember, is a little JavaScript library we developed here to act as a REST client. We modeled it after ActiveResource and its syntax, released it a little over a year ago this time, and it got a modest amount of attention. People seem to widely appreciate its syntax, and a less wide but consistent core of people have found some use out of it, and continue to contribute patches and bug reports to the project. We made a landing site, and a little walkthrough of its syntax and uses there.

The next major development step would be to detach Jester from Prototype and bridge it to JQuery and Mootools, if there’s community interest. Also, once Firefox 3 comes out and has genuine cross site XMLHttpRequests, that would be interesting to support. Let me know any other features you think Jester would benefit from, and might make it more useful to the JavaScript world at large.

Mailing List
Bug Tracker
Landing Site

April 2, 2007

emill

Jester: JavaScriptian REST

Jester is our implementation of REST, in JavaScript. It provides (nearly) identical syntax to ActiveResource for using REST to find, update, and create data, but from the client side.

Update, 6/16/07: We have released version 1.3 of Jester. You may want to view its release description.

Jester is available from SVN in trunk form, or a 1.1 release form. You can also download a zipped copy of 1.1. Jester is released under the MIT License.

Syntax

All examples below are taken from inside the JavaScript console of Firebug, the best JavaScript development tool you could possibly have.

First, declare a model in Jester by calling model on Base:

>>> Base.model("User")
>>> User
Object _name=User _singular=user _plural=users

This creates a global variable called User. It assumes that the URL prefix it uses to base its HTTP requests from is the current domain and port, and assumes “user” and “users” as single and plural forms to make these URLs. There’s no “people/person” intelligence here, so make sure to override these defaults if you need to, like so:

>>> Base.model("Child", "http://www.thoughtbot.com", "child", "children")
>>> Child
Object _name=Child _singular=child _plural=children

If you want to capture the model created in a local variable, or simply prefer more traditional JavaScript syntax, you can do:

>>> var Child = new Base("Child", "http://www.thoughtbot.com", "child", "children")
>>> Child
Object _name=Child _singular=child _plural=children

Find will retrieve a particular instance of your model. Attributes are auto-converted to integer or boolean types if that’s what they are on the server side. The “GET” line is not a return value, just Firebug’s report of activity, but relevant to understanding what’s happening.

>>> eric = User.find(1)
GET http://localhost:3000/users/1.xml
Object _name=User _singular=user _plural=users

>>> eric.attributes
["active", "email", "id", "name"]

>>> eric.id
1
>>> eric.name
"Eric Mill"
>>> eric.active
true

Create takes a hash of attribute values. After calling create, the model will fetch its new ID from the return headers.

>>> floyd = User.create({name: "Floyd Wright", email: "tfwright@thoughtbot.com"})
POST http://localhost:3000/users.xml
Object _name=User _singular=user _plural=users

>>> floyd.id
9

>>> User.find(9).name
GET http://localhost:3000/users/9.xml
"Floyd Wright"

Updating is as simple as changing one of the properties and calling save

.

>>> eric = User.find(1)
GET http://localhost:3000/users/1.xml
Object _name=User _singular=user _plural=users

>>> eric.email
"emill@thoughtbot.com"
>>> eric.email = "sandybeach@wintermute.com"
"sandybeach@wintermute.com"

>>> eric.save()
POST http://localhost:3000/users/1.xml
true

>>> User.find(eric.id).email
GET http://localhost:3000/users/1.xml
"sandybeach@wintermute.com"

Sadly, there’s one area where Jester’s syntax can’t match ActiveResource’s perfectly. The method “new” has been renamed to build, due to “new” being an illegal method name in JavaScript up to 1.6. Hopefully this can be updated as the browser landscape evolves. Build was chosen because it is similarly used in ActiveRecord to replace “new” on an association array, where “new” cannot be used.

>>> chad = User.build({email: "cpytel@thoughtbot.com", name: "Chad Pytel"})
Object _name=User _singular=user _plural=users

>>> chad.new_record()
true
>>> chad.save()
POST http://localhost:3000/users.xml
true

>>> chad.id
9
>>> chad.new_record()
false

Error validations are supported. If a model fails to save, save returns false, and the model’s errors property is set with an array of the error messages returned.

>>> jared = User.build({name: "", email: ""})
Object _name=User _singular=user _plural=users

>>> jared.save()
POST http://localhost:3000/users.xml
false

>>> jared.errors
["Name can't be blank", "Email can't be blank"]
>>> jared.valid()
false

>>> jared.name = "Jared Carroll"
"Jared Carroll"
>>> jared.email = "emill@thoughtbot.com"
"emill@thoughtbot.com"

>>> jared.save()
POST http://localhost:3000/users.xml
false

>>> jared.errors
["Email has already been taken"]
>>> jared.email = "jcarroll@thoughtbot.com"
"jcarroll@thoughtbot.com"

>>> jared.save()
POST http://localhost:3000/users.xml
true

Lastly, associations are also supported. If the association data is included in the XML, they’ll be loaded into the returned model as Jester models of their own, using the same assumptions on naming and URL prefix described above. They’re full models, so you can edit and save them as you would the parent. Has_many relationships come back as simple arrays, has_one relationships as a property. In this example, User has_many :posts, and Post belongs_to :user.

>>> eric = User.find(1)
GET http://localhost:3000/users/1.xml
Object _name=User _singular=user _plural=users

>>> eric.posts
[Object _name=Post _singular=post _plural=posts, Object _name=Post _singular=post _plural=posts]

>>> eric.posts.first().body
"Today I passed the bar exam. Tomorrow, I make Nancy my wife."
>>> eric.posts.first().body = "Today I *almost* passed the bar exam. The ring waits one more day."
"Today I *almost* passed the bar exam. The ring waits one more day."

>>> eric.posts.first().save()
POST http://localhost:3000/posts/1.xml
true

>>> post = Post.find(1)
GET http://localhost:3000/posts/1.xml
Object _name=Post _singular=post _plural=posts

>>> post.body
"Today I *almost* passed the bar exam. The ring waits one more day."
>>> post.user
Object _name=User _singular=user _plural=users
>>> post.user.name
"Eric Mill"

Using Jester

Jester depends on two libraries: Prototype, which comes with Rails and most people are familiar with, and ObjTree, a nice DOM parsing engine for JavaScript. Both of these are packaged along with Jester in its SVN repository, so you don’t have to hunt for them yourself. Just make sure you’re including all three in your test file.

<script type="text/javascript" src="/javascripts/prototype.js"></script>
<script type="text/javascript" src="/javascripts/ObjTree.js"></script>
<script type="text/javascript" src="/javascripts/jester.js"></script>

JavaScript in the browser is limited to requests with in only the same domain as the script is running in, so without iframe hackery, Jester is probably only useful for writing client code in your own apps, to talk to itself. We’re investigating whether Jester can use this hackery to make cross-domain requests, but it’s not clear if this will be feasible.

There are also some basic unit tests included inside Jester’s repository, which run using JsUnit. To run them yourself, from Jester’s repository open the file test/jsunit/testRunner.html in your browser, and choose test/jester_test.html as the test file.

The Server Side

These examples are talking with a Rails application whose controllers were generated with ”./script generate scaffold_resource”—in other words, the ideal RESTful controllers. It’s very easy to make your controller RESTful. Here’s the source for the User controller I’m using. The lines that deal with returning HTML have been removed, and I have added “(:include => :posts)” as an argument to to_xml in two places, so associations are included (it’s that easy!).


&lt;script type="text/javascript" src="/javascripts/prototype.js"&gt;&lt;/script&gt;
&lt;script type="text/javascript" src="/javascripts/ObjTree.js"&gt;&lt;/script&gt;
&lt;script type="text/javascript" src="/javascripts/jester.js"&gt;&lt;/script&gt;

An example of the XML produced here, of a User with one Post, at /users/2.xml:

<user>
  <active type="boolean">true</active>
  <email>cpytel@thoughtbot.com</email>
  <id type="integer">2</id>
  <name>Chad Pytel</name>
  <posts>
    <post>
      <title>Life as a Jester</title>
      <body>It's not as hard as Master said it would be.  Today I made 200 dollars.</body>
      <created-at type="datetime">2007-04-01T04:01:56-04:00</created-at>
      <id type="integer">2</id>
      <user-id type="integer">2</user-id>
    </post>
  </posts>
</user>

To see some real live examples, the Beast forum is currently implenting some of ActiveResource. Here’s technoweenie’s User account, in XML, and an XML list of selected Users. Pretty much any URL in Beast can have ”.xml” appended to it.

ActiveResource Reference

Taking ARes Out for a Test Drive—Great introduction to ActiveResource, by one of its authors.

ActiveResource and Testing—A post I made here discussing how I tested ActiveResource models.

ActiveResource’s Subversion Repository—Current ActiveResource trunk, at svn.rubyonrails.org.

Thanks

Thanks go to Chad for the original idea, and Jared for writing Jester’s tests.

Jester is new, so we’d love to hear feedback on its strengths and weaknesses. We’re using it ourselves, so it’s under active development and getting plenty of love and attention. Please tell us what you think!

January 2, 2007

jyurek

Keep your mongrels on a leash

We’re fans of Mac OS X over here, and a number of us have MacBooks or MacBook Pros on which we do our daily development work. And since we’re a Rails shop, we tend to keep around a few Mongrel processes on our local machines running hither and thither with our code. Normally, this would simply mean you keep Terminal.app open and have it running in there.

However, I’ve taken to using Visor as of late, with the modification to make it run transparently in the background (which I’ll get to in my next post). So, since I only have one terminal at any given time, and since (for this app, at least) I have to restart mongrel a lot to effect changes from the lib directory, I’d prefer to not give over my terminal simply for the ease of ctrl-c’ing my mongrel process.

Enter Dogtag, a Dashboard widget I made when I saw the announcement of Apple’s beta release of Dashcode. It runs, and keeps track of, one mongrel process. Give it an environment, a port, and your rails root, then click start. You can restart easily via the restart button. And since it’s nicely multi-instance aware, you can have a few running on different ports and keep track of them all.

It’s not terribly feature-packed, but that’s what dashboard widgets are for. I figure, if I wanted to tweak every setting, I’d make a real Cocoa app to do it. This was about a day’s work and is quite useful to me. I hope someone else finds it useful too.

October 13, 2006

cpytel

offsetHeight especially slow in Firefox for Mac

At thoughtbot, we’re working on an exciting piece of code that will someday be shared with the rest of the world. Suffice to say, its a very intensive bit of javascript code that stresses the boundaries of all browsers. After initially solving a lot of the performance problems by offloading a lot of the calculations to CSS instead of javascript (lots of the elements on the page effect the position of all the others), we were cruising along, only to hit a brick wall.

While performance in IE, Firefox on Windows, and Safari was acceptable (the performance of Safari has been running circles around the other browsers), performance on Firefox on Mac was incredibly poor. Amazingly so.

After the initial panic, I set to tracking the precise cause of the performance issues. By commenting out large sections of the code, I was able to determine that we were calling offsetHeight on some DIVs repeatedly each time an event fired (and it fires a lot).

A quick google search indicated that yes, some people have documented performance issues with offsetHeight (here and here).

While I can understand why offsetHeight is slow, I don’t understand why the performance of it in Firefox on Mac (Macbook Pro) was so much worse than any other browser.