---
title: The Hoptoad Database is Friggen Huge
teaser: Would you look at all those errors? Wow it's a lot of errors.
tags: airbrake,web
author: Chad Pytel
published_on: 2009-02-05
---

As of today, Hoptoad has processed over 28 million errors.  As those of your who
use Hoptoad may know, we store the full details of not only the exception that
occurred, including the backtrace, but also the full application session,
request data, and server environment - forever.

This has caused the Hoptoad database to rapidly grow to about 300GB, and its
still growing.  We're currently receiving about a million errors a week now.
The bulk of this data is the backtrace, session, request, and environment
information of each individual error.

### The Solution

While all this data hasn't really caused us any performance problems, the shear
size of the data has caused some problems, as we outgrew our first database
cluster, and needed to be moved to another one, and if we keep things they way
they are now, we'd shortly have to move to another cluster.

So, we can't keep the unchecked data growth in place, especially since, quite
frankly, knowing what the backtrace on an error was that happened 6 months ago,
that you've already resolved, isn't very useful.

So, we're in the process now of making some changes to Hoptoad so that
backtrace, session, request data, and server environment is removed after 30
days.  The individual notices will remain, all of the counts will remain
accurate, and you'll always be able to see backtrace, session, request data, and
server environment on the _latest_ occurrence of the error, no matter how long
ago it occurred - _its only this other data that will be removed after its 30
days old_.

### The Conclusion

As we've made the changes to support this upcoming change over the past week,
its caused the database load to spike, at times causes slowness in Hoptoad.  If
you seen this, we wanted to let you know that we're aware of the issue, and to
explain the cause.  After this process is completed, Hoptoad should be back to
its speedy self.

Finally, I'd just like to give yet another shout out to [Engine
Yard](http://www.engineyard.com),  who provides hosting to Hoptoad.  Their
support and assistance on this continues to be outstanding.

Thanks for your patience and understanding as we make sure Hoptoad is a
sustainable service that will be around for a long time to come.

FYI: _Hoptoad/Airbrake was sold to RackSpace and is now called [Airbrake Bug Tracker](http://airbrake.io)_.