Sign up to receive a weekly recap from Giant Robots
You’re here for the link juice, amirite?
Search engines “crawl” and “index” web content through programs called robots (a.k.a. crawlers or spiders). Here are some approaches to blocking them in Ruby on Rails apps.