The Hoptoad Database is Friggen Huge

As of today, Hoptoad has processed over 28 million errors. As those of your who use Hoptoad may know, we store the full details of not only the exception that occurred, including the backtrace, but also the full application session, request data, and server environment – forever.

This has caused the Hoptoad database to rapidly grow to about 300GB, and its still growing. We’re currently receiving about a million errors a week now. The bulk of this data is the backtrace, session, request, and environment information of each individual error.

The Solution

While all this data hasn’t really caused us any performance problems, the shear size of the data has caused some problems, as we outgrew our first database cluster, and needed to be moved to another one, and if we keep things they way they are now, we’d shortly have to move to another cluster.

So, we can’t keep the unchecked data growth in place, especially since, quite frankly, knowing what the backtrace on an error was that happened 6 months ago, that you’ve already resolved, isn’t very useful.

So, we’re in the process now of making some changes to Hoptoad so that backtrace, session, request data, and server environment is removed after 30 days. The individual notices will remain, all of the counts will remain accurate, and you’ll always be able to see backtrace, session, request data, and server environment on the latest occurrence of the error, no matter how long ago it occurred – its only this other data that will be removed after its 30 days old.

The Conclusion

As we’ve made the changes to support this upcoming change over the past week, its caused the database load to spike, at times causes slowness in Hoptoad. If you seen this, we wanted to let you know that we’re aware of the issue, and to explain the cause. After this process is completed, Hoptoad should be back to its speedy self.

Finally, I’d just like to give yet another shout out to Engine Yard, who provides hosting to Hoptoad. Their support and assistance on this continues to be outstanding.

Thanks for your patience and understanding as we make sure Hoptoad is a sustainable service that will be around for a long time to come.

I accidentally the whole SMTP exception

You have a slick, exclusive, invite-only Web app for sharing Tor URLs, with an Android client and specialty hardware. You use validates_email_format_of in the Invitation model, but still something slips through and your Hoptoad errors pile up, showing your user the beautifully-designed 500 page instead of an error explanation.

There are two types of exceptions that ActionMailer will raise when you attempt to deliver an email: user input problems and server problems.

User input problems are those such as incorrect or invalid email addresses; the exceptions raised are Net::SMTPFatalError and Net::SMTPSyntaxError. These are issues that the user can fix and as such the error message should indicate that everything’s fine, nothing is ruined.

Server problems could be anything from a non-existent server to an authentication issue; the exceptions raised are: TimeoutError, IOError, Net::SMTPUnknownError, Net::SMTPServerBusy, and Net::SMTPAuthenticationError. These issues are outside the power of the user and should indicate that we screwed up.

So in config/initializers/errors.rb:

SMTP_SERVER_ERRORS = [TimeoutError,
                          IOError,
                          Net::SMTPUnknownError,
                          Net::SMTPServerBusy,
                          Net::SMTPAuthenticationError]

SMTP_CLIENT_ERRORS = [Net::SMTPFatalError, Net::SMTPSyntaxError]

SMTP_ERRORS = SMTP_SERVER_ERRORS.concat(SMTP_CLIENT_ERRORS)

SMTP_CLIENT_ERROR_FLASH = 'The email address supplied is invalid.  Please check for spelling mistakes.'
SMTP_SERVER_ERROR_FLASH = 'We encountered an internal issue while attempting to deliver this email.  Please try again in a few minutes.'

We can test it with invitations_controller_test.rb:

class InvitationsController; def rescue_action(e) raise e end; end
class InvitationsControllerTest < Test::Unit::TestCase
  SMTP_CLIENT_ERRORS.each do |exn|
    should "handle #{exn}" do
      InvitationsMailer.expects(:deliver_invitation).raises(exn)
      post :create, :invitation => {:email => 'invalid email'}
      assert_match /#{SMTP_CLIENT_ERROR_FLASH}/i, @response.flash[:warning]
      assert_template 'new'
    end
  end

  SMTP_SERVER_ERRORS.each do |exn|
    should "handle #{exn}" do
      InvitationsMailer.expects(:deliver_invitation).raises(exn)
      post :create, :invitation => {:email => 'invalid email'}
      assert_match /#{SMTP_SERVER_ERROR_FLASH}/i, @response.flash[:warning]
      assert_template 'new'
    end
  end
end

And in invitations_controller.rb:

class InvitationsController < ApplicationController
  def create
    @invitation.new(params[:invitation])
    if @invitation.save
      redirect_to root_url
    else
      render :action => 'new'
    end
  rescue *SMTP_CLIENT_ERRORS
    flash[:warning] = SMTP_CLIENT_ERROR_FLASH
    render :action => 'new'
  rescue *SMTP_SERVER_ERRORS => error
    notify_hoptoad error
    flash[:warning] = SMTP_SERVER_ERROR_FLASH
    render :action => 'new'
  end
end

If you use Suspenders you’ll be pleased to find that we’ve included config/initializers/errors.rb for you pre-populated with both SMTP and HTTP exceptions.

Let’s Put Some Lipstick on this Toad

Its been a little over a week since we officially launched Hoptoad, our centralized error catching application. The feedback we’ve received has been overwhelmingly positive, and we’ve caught about 45,000 non-test errors from 115 applications. If you haven’t given Hoptoad a try yet, come on in, the water’s fine.

Yesterday we deployed several improvements:

One of the more noticeable changes we’ve made is that the default error listing now excludes errors that you’ve marked as resolved.

In Hoptoad, you can mark an error that your application has thrown as “resolved”. Hoptoad detects similar errors, and only notifies you about the first error when it occurs. Until now, the primary benefit to marking an error as resolved was that if an error you thought was resolved occurs again, we’d specifically notify you about it again. Previously, we defaulted the main error listing to show you all errors regardless of their resolved state, and you had to click “Hide Resolved Errors” in order to remove the errors from the listing.

We originally made this design decision in order to err on the side of more visibility into all of the errors that were occurring in your application. But, based on user feedback, and a change of heart internally, we’ve reversed this so that resolved errors are now excluded from the listing, and you click “Show Resolved Errors” to view them.

We’ve also added a simple visual cue for resolved errors in the listing.

This release also includes documentation on the Hoptoad API, and the server side changes necessary to get it going. The API is RESTful, so you can use Active Resource, or use it directly if you want. We’ve also provided basic documentation on how to submit errors directly to Hoptoad without using the Rails plugin we provide – so the door is now officially open for you to create your own clients for Hoptoad – maybe Merb, or .Net strikes your fancy. For complete details on the API, check out the documentation.

We’ve also included a few other refinements and bug fixes as well, you can take a look at the changes in our lighthouse to see the full list.

Finally, thanks to Tony Pitale for his writeup on hoptoad.

We’ve got some more improvements in the works, stay tuned.