As of today, Hoptoad has processed over 28 million errors. As those of your who use Hoptoad may know, we store the full details of not only the exception that occurred, including the backtrace, but also the full application session, request data, and server environment - forever.
This has caused the Hoptoad database to rapidly grow to about 300GB, and its still growing. We’re currently receiving about a million errors a week now. The bulk of this data is the backtrace, session, request, and environment information of each individual error.
The Solution
While all this data hasn’t really caused us any performance problems, the shear size of the data has caused some problems, as we outgrew our first database cluster, and needed to be moved to another one, and if we keep things they way they are now, we’d shortly have to move to another cluster.
So, we can’t keep the unchecked data growth in place, especially since, quite frankly, knowing what the backtrace on an error was that happened 6 months ago, that you’ve already resolved, isn’t very useful.
So, we’re in the process now of making some changes to Hoptoad so that backtrace, session, request data, and server environment is removed after 30 days. The individual notices will remain, all of the counts will remain accurate, and you’ll always be able to see backtrace, session, request data, and server environment on the latest occurrence of the error, no matter how long ago it occurred - its only this other data that will be removed after its 30 days old.
The Conclusion
As we’ve made the changes to support this upcoming change over the past week, its caused the database load to spike, at times causes slowness in Hoptoad. If you seen this, we wanted to let you know that we’re aware of the issue, and to explain the cause. After this process is completed, Hoptoad should be back to its speedy self.
Finally, I’d just like to give yet another shout out to Engine Yard, who provides hosting to Hoptoad. Their support and assistance on this continues to be outstanding.
Thanks for your patience and understanding as we make sure Hoptoad is a sustainable service that will be around for a long time to come.
FYI: Hoptoad/Airbrake was sold to RackSpace and is now called Airbrake Bug Tracker.
