---
title: Hoptoad Status Update
teaser: Update on recent Hoptoad downtime.
tags: news,airbrake
author: Chad Pytel
published_on: 2009-03-11
---

Hi Everyone,

Some of you may have noticed that over the past two weeks Hoptoad has had a
number of, sometimes brief, occurrences of downtime. We sincerely apologize for
any trouble this may have caused you, and we're working on putting a series of
fixes in place to make sure it doesn't continue to happen.

Each of these occurrences except one (more on that below) was caused by a
Hoptoad client sending many thousands of exceptions very rapidly, effectively
creating a denial of service attack.  Each time, we've taken measures to
identify the client and disable it, contact them to let them know that we've
disabled their account and to let us know when the exception is fixed so that we
can re-enable it.

Before these last two weeks, we haven't had this problem, and everything with
Hoptoad has been pretty much fine, so now that we know where the pain points
are, we're actively working to put changes in place so that we can been handle
this when it happens and so that it doesn't effect other users of Hoptoad.

This is our highest priority for Hoptoad right now.

As is often the case when a problem like this arises, it happens the first time
and you think Well, that was awful, but its the first time it happened, and its
never happened before, so it probably won't again.  Well it has happened again,
a few times now, and its obvious that we need to take action.

Additionally, we had one outage which was caused by a problem with the shared
file system between our web slices.  We (Engine Yard and us) are not exactly
sure what caused it, but we're working with Engine Yard to make sure it doesn't
happen again.

Again, we sincerely apologize for the trouble, and we're actively working with
Engine Yard on ensuring that the Hoptoad service stays up and is reliable.
We'll be sure to let you know about the fixes here as we put them in place.

Thanks very much for your patience.

FYI: _Hoptoad/Airbrake was sold to RackSpace and is now called [Airbrake Bug Tracker](http://airbrake.io)_.