We recently made the decision to change our hosting infrastructure, but that doesn’t mean we were off the hook until we’ve moved! Over the weekend our traffic increased greatly: we saw a 200% to 400% increase in traffic over the past few days, and it doesn’t show signs of stopping. Naturally, we were floored. No, really, the site dropped on the floor. We simply couldn’t keep up. Everything from memory and swapping issues, to I/O contention, to nginx not being able to hold open incoming connections. We were providing you with very sub-par service.
We realize this and completely apologize. In fact, anyone who wants a refund for March can have it, just let us know.
The good news is that we’ve provisioned a ton of capacity as well as removed some of the other bottlenecks and we appear to be running smoothly now. This hasn’t stopped our migration plans, either. Starting tomorrow, we’re stress testing our new setup with nothing less than full, real traffic from the live site, using the ever-flexible em_proxy. We’re planning on cutting over to the new infrastructure next week, and we will definitely keep you posted during the move.
For the last couple of months, we’ve seen considerable growth on Hoptoad accounts and traffic. Thank you all! But this introduced new traffic patterns and challenges. During this time we’ve been mostly keeping up with this growth and making sure we can provide as reliable a service as possible. There have been some bumps along the way. This is what has happened, what we’ve done about it, and what is yet to come.
For over a year, Hoptoad has stored exception details as a gzipped XML on Amazon S3. When an error is POSTed to our API endpoint, we validate it, group it with similar errors, and store it on the app server’s file system. Every five minutes there was a cron job that would upload all these XML files to S3. These details were only available for viewing on the UI after they made it to S3. This is why, more often than we had liked, you would see the dreaded message “Details for this error are still being processed”. This served us well for some time, but we knew it was time to rethink this architecture.
There were many problems with this approach. The most obvious was that this “still processing” error was becoming more and more common, and this degraded the experience of viewing error messages for our users (us included). The first thing we did to improve that experience was rather simple and did not require wholesome architectural changes: Instead of trying to display the last notice that we got for that error group, we showed you the last processed error for that group. So therefore, instead of seeing the processing message, you would see actionable data for that exception so that you can get back to work fixing bugs.
Even though this helped the situation and the number of support requests greatly decreased, we always knew this was a temporary solution and we could do better. We needed a way to store error details in the life cycle of the request, in such a way that it was available immediately afterwards for viewing. Uploading to S3 became too slow for our needs.
Furthermore, this was not the only problem with this architecture. The larger problem is that because of our high traffic, we started running into all sorts of issues with either disk space filling up before our workers were able to push notice details to S3, or even worse, an application instance failing completely thus losing any unprocessed details. In those rare cases, another application instance would be automatically provisioned, and the XML on that filesystem would be lost.
In order to display exception details quickly, we decided to make use of MongoDB, removing temporary file system and S3 storage alltogether. When an exception hits our API, we do the same processing we’ve always done but store it in a MongoDB collection instead. The three main advantages to you are:
We can’t stop here. We have encountered numerous problems with our current environment, and we are working to improve our infrastructure. This has been our primary focus for the last couple of months.
We plan on migrating our application to a more traditional hosting environment. While we will continue to use virtualization for application servers and other utilities, our databases will now run on bare metal. We are confident that this will increase our overall performance even more, and provide a predictable path for growth. Among other things, this solves:
We have been forced to focus our efforts on performance improvements and architectural changes that can support the growth we’ve seen. We are very sorry for the bumps on the road along the way. We are also tired of feeling apologetic. Enough is enough. We have made changes to improve your experience as a customer, and we will continue to do so. Please bear with us until we’ve migrated our infrastructure. We’ll keep you updated as to the timeline for the hosting move. We look forward to being able to stop worrying about performance, and start worrying about how to improve the service by providing better features that make more use of the data, and help you handle your app’s bugs efficiently.
We’ve been working with Heroku to bring you an even easier and more convenient way to add Hoptoad to your Heroku apps:
$ heroku addons:add hoptoad:plus
$ heroku addons:add hoptoad:basic
Or, if you prefer, you can add Hoptoad to your app with a single click over on the Hoptoad Heroku Addon page.
Once you’ve configured the addon, you’ll want to install the Hoptoad Notifier in your project, and you can use a shiny, new, Heroku-specific configuration flag in the generator:
$ rails generate hoptoad --heroku
Then, you can click through from your Heroku dashboard to the Hoptoad addon, and get the same Hoptoad interface you’re come to know and love:
I would highly recommend building your own Heroku addon. The experience is getting very polished, and there is a large amount of documentation at the Heroku Addon Provider site. They take care of billing and provide a single-sign on API for addon providers, and generally make the process seamless and pain-free. We’ve been working with them for a while, through the development of their addon platform, and they have a very solid offering now. Go forth and develop!
We’re happy to introduce several UI improvements to Hoptoad today: a new account dashboard and project navigation.
Previously, the main page of the app showed a long list of the latest errors across all your projects.
Among developers at thoughtbot, a common workflow was to bypass this combined list for the project specific listing, since greater detail was available there. We decided to make the main page of the app a higher level overview of your projects: the current rate limit, the number of unresolved errors, the latest deploy, and the last three errors for that project. We think these changes do a better job of giving you an immediate sense of what’s happening with your apps.
The dashboard shows summaries for up to five projects. If you have more than five projects on your account, then we will show the five projects you have most recently viewed through the site. This will make sure that you are always seeing the most relevant projects first. The dropdown menu in the navigation is sorted the same way, but will always show the full list of your projects.
In the process of changing the app home page, we removed the sidebar that listed all the projects on your account. Instead, we’ve made the “Projects” link in the main navigation a dropdown that contains a list of all your projects, as well as a button to quickly add a new project. We took great care in crafting the user experience of this menu and think it should help you move around the site with less effort.
Along with these changes, there are a number of other refinements to the UI: the main navigation makes greater use of CSS3, the blank state has been improved, and more.
There were a handful of ideas that we played with and ultimately decided against including in this release (some were cut just a few days ago), all with the aim of presenting the most focused, easy-to-use UI possible.
We’re really pleased with these changes and are excited to roll them out to all our users. If you do happen to have any problems with anything, please let us know through the Hoptoad support site.
I’m sure you’re asking, “How do I participate in this awesomeness?”. It’s easy.
If you’re using the hoptoad_notifier gem, upgrade to the latest version (2.3.7 at the time of this update) and update your
Yes, that’s it.
<head> tag. Done!
If you’re not using the hoptoad_notifier gem, you’ll just need to include a couple of lines (at the top of your
The nitty-gritty details: we’re currently only supporting browsers that support the
onerror event (Firefox 2+, IE6+). You’ll also be including your project API key on the page. This is different from your user API key, so there’s no need to worry about someone deleting your projects or users. We’ll continue to expand this feature and let you know as we progress. If you have any suggestions or spot a bug, let us know!