Hi Everyone,
Some of you may have noticed that over the past two weeks Hoptoad has had a number of, sometimes brief, occurrences of downtime. We sincerely apologize for any trouble this may have caused you, and we’re working on putting a series of fixes in place to make sure it doesn’t continue to happen.
Each of these occurrences except one (more on that below) was caused by a Hoptoad client sending many thousands of exceptions very rapidly, effectively creating a denial of service attack. Each time, we’ve taken measures to identify the client and disable it, contact them to let them know that we’ve disabled their account and to let us know when the exception is fixed so that we can re-enable it.
Before these last two weeks, we haven’t had this problem, and everything with Hoptoad has been pretty much fine, so now that we know where the pain points are, we’re actively working to put changes in place so that we can been handle this when it happens and so that it doesn’t effect other users of Hoptoad.
This is our highest priority for Hoptoad right now.
As is often the case when a problem like this arises, it happens the first time and you think Well, that was awful, but its the first time it happened, and its never happened before, so it probably won’t again. Well it has happened again, a few times now, and its obvious that we need to take action.
Additionally, we had one outage which was caused by a problem with the shared file system between our web slices. We (Engine Yard and us) are not exactly sure what caused it, but we’re working with Engine Yard to make sure it doesn’t happen again.
Again, we sincerely apologize for the trouble, and we’re actively working with Engine Yard on ensuring that the Hoptoad service stays up and is reliable. We’ll be sure to let you know about the fixes here as we put them in place.
Thanks very much for your patience.
FYI: Hoptoad/Airbrake was sold to RackSpace and is now called Airbrake Bug Tracker.
