Updates on Hoptoad

We recently made the decision to change our hosting infrastructure, but that doesn’t mean we were off the hook until we’ve moved! Over the weekend our traffic increased greatly: we saw a 200% to 400% increase in traffic over the past few days, and it doesn’t show signs of stopping. Naturally, we were floored. No, really, the site dropped on the floor. We simply couldn’t keep up. Everything from memory and swapping issues, to I/O contention, to nginx not being able to hold open incoming connections. We were providing you with very sub-par service.

We realize this and completely apologize. In fact, anyone who wants a refund for March can have it, just let us know.

The good news is that we’ve provisioned a ton of capacity as well as removed some of the other bottlenecks and we appear to be running smoothly now. This hasn’t stopped our migration plans, either. Starting tomorrow, we’re stress testing our new setup with nothing less than full, real traffic from the live site, using the ever-flexible em_proxy. We’re planning on cutting over to the new infrastructure next week, and we will definitely keep you posted during the move.

Hoptoad, the cloud, and the pond ahead

For the last couple of months, we’ve seen considerable growth on Hoptoad accounts and traffic. Thank you all! But this introduced new traffic patterns and challenges. During this time we’ve been mostly keeping up with this growth and making sure we can provide as reliable a service as possible. There have been some bumps along the way. This is what has happened, what we’ve done about it, and what is yet to come.

The error process queue

For over a year, Hoptoad has stored exception details as a gzipped XML on Amazon S3. When an error is POSTed to our API endpoint, we validate it, group it with similar errors, and store it on the app server’s file system. Every five minutes there was a cron job that would upload all these XML files to S3. These details were only available for viewing on the UI after they made it to S3. This is why, more often than we had liked, you would see the dreaded message “Details for this error are still being processed”. This served us well for some time, but we knew it was time to rethink this architecture.

There were many problems with this approach. The most obvious was that this “still processing” error was becoming more and more common, and this degraded the experience of viewing error messages for our users (us included). The first thing we did to improve that experience was rather simple and did not require wholesome architectural changes: Instead of trying to display the last notice that we got for that error group, we showed you the last processed error for that group. So therefore, instead of seeing the processing message, you would see actionable data for that exception so that you can get back to work fixing bugs.

Even though this helped the situation and the number of support requests greatly decreased, we always knew this was a temporary solution and we could do better. We needed a way to store error details in the life cycle of the request, in such a way that it was available immediately afterwards for viewing. Uploading to S3 became too slow for our needs.

Furthermore, this was not the only problem with this architecture. The larger problem is that because of our high traffic, we started running into all sorts of issues with either disk space filling up before our workers were able to push notice details to S3, or even worse, an application instance failing completely thus losing any unprocessed details. In those rare cases, another application instance would be automatically provisioned, and the XML on that filesystem would be lost.

Enter MongoDB

In order to display exception details quickly, we decided to make use of MongoDB, removing temporary file system and S3 storage alltogether. When an exception hits our API, we do the same processing we’ve always done but store it in a MongoDB collection instead. The three main advantages to you are:

• Error details are always available, immediately after we receive them. Therefore you can click on the error URL that you receive on the notification emails and start seeing details for the error with no delay.
• A more robust storage approach, where app instance failures will never cause details to be completely lost. With careful planning, disk space is not an issue either.
• Better response times: A nice by-product of this change has been that both storing and reading the data has improved the response time of the application by roughly 30%.

A hybrid future

We can’t stop here. We have encountered numerous problems with our current environment, and we are working to improve our infrastructure. This has been our primary focus for the last couple of months.

We plan on migrating our application to a more traditional hosting environment. While we will continue to use virtualization for application servers and other utilities, our databases will now run on bare metal. We are confident that this will increase our overall performance even more, and provide a predictable path for growth. Among other things, this solves:

• The bad neighbor problem, where other instances in the cloud steal precious CPU cycles. For high traffic applications like Hoptoad the cost of this problem is very real. On our planned setup, our app servers will run under our own hypervisor, so it is impossible for other applications to steal our CPU.
• I/O contingency - while most apps can run just fine on the cloud, it is underprovisioned for an application like Hoptoad. We will gain superior I/O bandwidth by designing an infrastructure with faster disks that can support our needs.

Looking forward to a brighter pond

We have been forced to focus our efforts on performance improvements and architectural changes that can support the growth we’ve seen. We are very sorry for the bumps on the road along the way. We are also tired of feeling apologetic. Enough is enough. We have made changes to improve your experience as a customer, and we will continue to do so. Please bear with us until we’ve migrated our infrastructure. We’ll keep you updated as to the timeline for the hosting move. We look forward to being able to stop worrying about performance, and start worrying about how to improve the service by providing better features that make more use of the data, and help you handle your app’s bugs efficiently.

Hoptoad is now available as a Heroku Addon

We’ve been working with Heroku to bring you an even easier and more convenient way to add Hoptoad to your Heroku apps:

$ heroku addons:add hoptoad:plus

• $7/month, SSL, ratelimit of 30 errors/minute, plus Github, Lighthouse, and deploy integration

$ heroku addons:add hoptoad:basic

• $2/month, SSL, ratelimit of 5 errors/minute

Or, if you prefer, you can add Hoptoad to your app with a single click over on the Hoptoad Heroku Addon page.

Once you’ve configured the addon, you’ll want to install the Hoptoad Notifier in your project, and you can use a shiny, new, Heroku-specific configuration flag in the generator:

$ rails generate hoptoad --heroku

Then, you can click through from your Heroku dashboard to the Hoptoad addon, and get the same Hoptoad interface you’re come to know and love:

The Heroku Addon Experience

I would highly recommend building your own Heroku addon. The experience is getting very polished, and there is a large amount of documentation at the Heroku Addon Provider site. They take care of billing and provide a single-sign on API for addon providers, and generally make the process seamless and pain-free. We’ve been working with them for a while, through the development of their addon platform, and they have a very solid offering now. Go forth and develop!

The Dashboard and the Toad

We’re happy to introduce several UI improvements to Hoptoad today: a new account dashboard and project navigation.

Previously, the main page of the app showed a long list of the latest errors across all your projects.

Among developers at thoughtbot, a common workflow was to bypass this combined list for the project specific listing, since greater detail was available there. We decided to make the main page of the app a higher level overview of your projects: the current rate limit, the number of unresolved errors, the latest deploy, and the last three errors for that project. We think these changes do a better job of giving you an immediate sense of what’s happening with your apps.

The dashboard shows summaries for up to five projects. If you have more than five projects on your account, then we will show the five projects you have most recently viewed through the site. This will make sure that you are always seeing the most relevant projects first. The dropdown menu in the navigation is sorted the same way, but will always show the full list of your projects.

In the process of changing the app home page, we removed the sidebar that listed all the projects on your account. Instead, we’ve made the “Projects” link in the main navigation a dropdown that contains a list of all your projects, as well as a button to quickly add a new project. We took great care in crafting the user experience of this menu and think it should help you move around the site with less effort.

Along with these changes, there are a number of other refinements to the UI: the main navigation makes greater use of CSS3, the blank state has been improved, and more.

There were a handful of ideas that we played with and ultimately decided against including in this release (some were cut just a few days ago), all with the aim of presenting the most focused, easy-to-use UI possible.

We’re really pleased with these changes and are excited to roll them out to all our users. If you do happen to have any problems with anything, please let us know through the Hoptoad support site.

Hoptoad and Javascript, Sitting in a Tree, S-E-N-D-I-N-G

With an open API, Hoptoad, the app error app, can track errors from any sort of app you want: Rails, Sinatra, Rack, PHP, C#… the list goes on. We didn’t want to leave front-end developers in the dark when their Javascript throws errors, so we’ve added a Hoptoad notifier for Javascript!

I’m sure you’re asking, “How do I participate in this awesomeness?”. It’s easy.

If you’re using the hoptoad_notifier gem, upgrade to the latest version (2.3.7 at the time of this update) and update your config/initializers/hoptoad.rb:

HoptoadNotifier.configure do |config|
  config.api_key = 'YOUR-PROJECT-API-KEY'
  config.js_notifier = true # ADD ONE LINE FOR JAVASCRIPT ERROR TRACKING!
end

Yes, that’s it.

The gem injects Javascript right after your <head> tag. Done!

Not using hoptoad_notifier gem?

If you’re not using the hoptoad_notifier gem, you’ll just need to include a couple of lines (at the top of your <head>):

<script type="text/javascript" src="http://hoptoadapp.com/javascripts/notifier.js"></script>
<script type="text/javascript">
  Hoptoad.setKey('YOUR-PROJECT-API-KEY');
</script>

The details

You may be proclaiming, “Oh no, Javascript at the top of <head>! What are you thinking?” I’ll let you in on a little secret: we want our Javascript loaded before any other Javascript. That way, if one of those libraries throws an exception, we’re there to let you know!

The nitty-gritty details: we’re currently only supporting browsers that support the onerror event (Firefox 2+, IE6+). You’ll also be including your project API key on the page. This is different from your user API key, so there’s no need to worry about someone deleting your projects or users. We’ll continue to expand this feature and let you know as we progress. If you have any suggestions or spot a bug, let us know!

So, what’re you waiting for? It takes 30 seconds to sign up; go grab an account and start tracking your Javascript errors now!