We recently made the decision to change our hosting infrastructure, but that doesn’t mean we were off the hook until we’ve moved! Over the weekend our traffic increased greatly: we saw a 200% to 400% increase in traffic over the past few days, and it doesn’t show signs of stopping. Naturally, we were floored. No, really, the site dropped on the floor. We simply couldn’t keep up. Everything from memory and swapping issues, to I/O contention, to nginx not being able to hold open incoming connections. We were providing you with very sub-par service.
We realize this and completely apologize. In fact, anyone who wants a refund for March can have it, just let us know.
The good news is that we’ve provisioned a ton of capacity as well as removed some of the other bottlenecks and we appear to be running smoothly now. This hasn’t stopped our migration plans, either. Starting tomorrow, we’re stress testing our new setup with nothing less than full, real traffic from the live site, using the ever-flexible em_proxy. We’re planning on cutting over to the new infrastructure next week, and we will definitely keep you posted during the move.