We recently made the decision to change our hosting infrastructure, but that doesn’t mean we were off the hook until we’ve moved! Over the weekend our traffic increased greatly: we saw a 200% to 400% increase in traffic over the past few days, and it doesn’t show signs of stopping. Naturally, we were floored. No, really, the site dropped on the floor. We simply couldn’t keep up. Everything from memory and swapping issues, to I/O contention, to nginx not being able to hold open incoming connections. We were providing you with very sub-par service.
We realize this and completely apologize. In fact, anyone who wants a refund for March can have it, just let us know.
The good news is that we’ve provisioned a ton of capacity as well as removed
some of the other bottlenecks and we appear to be running smoothly now. This
hasn’t stopped our migration plans, either. Starting tomorrow, we’re stress
testing our new setup with nothing less than full, real traffic from the live
site, using the ever-flexible em_proxy. We’re planning on cutting over to the
new infrastructure next week, and we will definitely keep you posted during the
move.
FYI: Hoptoad/Airbrake was sold to RackSpace and is now called Airbrake Bug Tracker.
