Here’s a brief introduction to a new application we’ve been working on. The application is called hoptoad, and it’s a hosted web service which can be used as an error reporting and analysis tool by Rails applications. The service is running in the Engine Yard cloud and will be completely free when it launches, so go ahead and sign up to that mailing list and we’ll let you know when it’s live.
For the academically curious, the word hoptoad is railroad industry slang for a derailed train. At least, it used to be when the railroad industry had slang.
Background
We’ve been using Jamis Buck’s Exception Notifier Rails plugin for years now, and for the most part it’s accomplished what we wanted. For the uninitiated, the Exception Notifier plugin catches all errors that haven’t already been caught, and sends an email with a formatted error report about what happened, where it happened, the session data, the params, the environment info, etc.
We’ve used this with great success, probably on ~30 projects or so, and it’s been a reliable way to know when there is an issue with one of the production applications you are maintaining.
Problems
The problems we ultimately ran into here…
- We would occasionally get completely hammered with a stream of app error emails. In most cases, this was something completely innocent (like a bunch of RoutingError’s caused by a new release version changing the url structure, or by the rails2 upgrade sending InvalidAuthenticityToken, or by a search engine doing something weird like sending us an HTTP ‘OPTIONS’ verb in a request, etc). In other cases, we had legitimately screwed up and had an app which was throwing off errors left and right - but in that case, once the first 10 emails trickled in, we already knew there was an urgent problem to solve, and the next 1000 emails didn’t make it any more urgent, they just clogged up our inboxes.
- There’s no way to measure success over time. Am I correcting the errors I had last deploy? Am I getting more or less errors each month?
- The errors are a big stream. If I get error A 10 times, error B 40 times and error C 50 times - sure, I have 100 errors - but I really want to know that I have three errors, and how many times eacbh of them have occurred.
Solution
As you may have guessed, we’ve solved these problems, and replaced the Exception Notifier in most of the apps we manage, with a homegrown solution called hoptoad. Hoptoad is two things:
- A smallish Rails plugin, which installs into
vendor/plugins, and defines a#rescue_action_in_publicmethod which does nothing but report the error it received (via HTTP POST to a web service), and then return control to the framework so that Rails can render the 500 (or 422, or 404, or whatever) page that’s appropriate for the error. - The web service, which accepts incoming error reports from various applications running the hoptoad plugin, and stores them for later review.
Plugin
By default, the hoptoad notifier plugin requires only one configuration option: the API key for the project the plugin is installed into, so that it can be properly stored. In addition to that, it is possible to specify a set of errors that you’d like the plugin to ignore (it has a default set to ignore and not report on, which you may override). The plugin, and sample configuration, are available from the project configuration area within the application.
Web application
The application is responsible for grouping errors, marking inactive errors, providing rss feeds of groups of errors, managing user access to error reports, handling incoming errors from the plugin via an API, etc.
We expect to the public launch to be sometime next week, so stay tuned.
If my train goes off the track, pick it up, pick it up, pick it up back back back to the grill again, the grill again friends how many MC’s must get dissed?
-KRS One
FYI: Hoptoad/Airbrake was sold to RackSpace and is now called Airbrake Bug Tracker.
