Update: Our quest for full-stack testing continued with writing the capybara-webkit driver. We also wrote about capybara-webkit more recently.
Stop! Who would cross the Bridge of Death must answer me these questions three, ‘ere the other side he see.
Ask me the questions, bridge-keeper. I’m not afraid.
What is your name?
My name is cpytel of thoughtbot.
What is your favorite color?
Red.
What is your quest?
To seek a reliable, robust way of integration testing our entire application to prevent against regressions and to build better systems.
With apologies to Monty Pyton, we’ve been on this quest for several years now. After many false starts and tribulations I feel we’ve reached an important plateau in this quest, and I feel its prudent to take a step back, look at where we’ve been, where we are now, and pass on some of this information for the benefit of all.
Selenium
Throughout this quest, Selenium has been the siren song that continually calls out to us. Unfortunately, in practice we’ve been unable to get Selenium to run reliably for real applications, on both developers machines and on the continuous integration server.
This failure with Selenium has caused us to search for alternative solutions.
Holy Grail
The first promising solution was the aptly named, Holy Grail. This library uses Harmony, which in turn wraps Johnson, env.js and Envjs to execute browser-less, console-based, javascript and DOM code right from within your Rails test suite. Our own Jason Morrison then wrote cucumber-holygrail to allow Cucumber to drive the Holy Grail integration tests.
Unfortunately, this solution wasn’t really designed to accomplish what we wanted; Holy Grail lets you run javascript on a single page and routes xhr requests to a controller in a functional test, and therefore it didn’t work like we expected: we wanted to drive a virtual browser session from action to action that supported javascript and ui interactions.
Envjs
From there, we discovered capybara-envjs. Capybara is a replacement for Webrat (the driver underneath Cucumber) which has a more flexible driver subsystem. capybara env-js provides more of what we needed, it had the goal of being a virtual browser, like we wanted. In order to make it work like we wanted, we needed to provide a number of fixes which we encapsulated into the capybara-envjs-fixes gem (as a holding place while we waited for pull requests to be merged in).
Capybara and capybara-envjs, and capybara-envjs-fixes were (and mostly still are) a deadly combination. We’re actively using it on projects still. However, the more we used it, the more we started to discover cracks in the foundation.
- It doesn’t work with all javascript (for example, we had issues on sites that still use Prototype)
- It doesn’t support jQuery live
- It is sometimes very inconsistent with how real-world browsers behave, or how things should properly work, forcing us to write our code like we wouldn’t normally write it
The three above points, particularly the lack of support for jQuery live (which is used by more and more of the internal of jQuery, jQuery mobile, and even the new Rails unobtrusive javascript) caused us to continue to search for a better solution.
Akephalos
Thankfully, we then found Akephalos. Akephalos provides a Capybara driver that allows you to run your cucumber integration tests in the headless browser HtmlUnit. HtmlUnit is a “GUI-Less browser for Java programs”. It models HTML documents and provides an API that allows you to invoke pages, fill out forms, click links, etc… just like you do in your “normal” browser. With our fork of Akephalos to resolve a couple of issues that we ran into along the way, we were up and running with very reliable, headless browser tests.
HtmlUnit is written in Java, and Akephalos uses jruby-jars to start up and interact with the HtmlUnit browser. It has fairly good JavaScript support (it was able to deal with everything we were able to throw at it, including jQuery 1.4.2 and 1.4.3, jQuery Mobile, and jQuery live).
Note: We also tried culerity which is a cucumber driver that works with Celerity, which also is underpinned by HtmlUnit, but we couldn’t get it to really work well.
One of the great things about HtmlUnit and Akephalos is that it uses technology which has been existence in a while, is actively developed, and well documented. We were able to look at the code and understand what was going on and contributate changes if necessary much more easily than we were able to do with any of the previous attempts.
If you want to get started with Akephalos, I recommend that you use our fork on github until the changes are pulled in. We’ve used Akephalos on both Rails 2 and 3.
The relevant part of our Gemfile looks like this:
group :test, :cucumber do
gem 'akephalos', :git => 'git://github.com/thoughtbot/akephalos.git'
gem "cucumber-rails"
gem "capybara"
gem "database_cleaner"
gem "treetop"
gem "launchy"
end
Then, to a features/support/akephalos.rb add:
require 'akephalos'
You can then tag any scenarios you want to run with Akephalos with the @akephalos tag. If you want to be able to tag your scenarios with @javascript and have it execute in Akephalos (the default javascript driver in Capybara is Selenium) you’ll want to add the following additional line to your features/support/akephalos.rb file.
Capybara.javascript_driver = :Akephalos
There are a couple of important gotchas right now that will effect you, particularly if you have an existing application.
There is a known bug in HtmlUnit where any jQuery live events bound to the click
event stop any jQuery live events bound to submit defined after them. In
particular, this means that if you’re using Rails 3 unobtrusive jQuery form
helpers, you’ll want to edit your rails.js file and move the definition of the
follow event:
$('form[data-remote]').live('submit')
Up above any jQuery live click bindings.
Additionally, ajax requests execute asynchronously, just like in a real browser. In order for the tests to not get out of sequence and fail, ajax requests need to be executed synchronously instead. You can do this by adding the following code after your inclusion of jQuery in only your test or cucumber environments:
$.ajaxSetup({ async: false });
Akephalos doesn’t have the ability to set cookies, or access the session. These are capabilities in other drivers, and so you may find that your cucumber steps are doing this, and they’ll need to be rewritten. For example, the built in clearance step definitions read the cookies in order to expose a current_user helper for step definitions to use, and deleted cookies in order to force a sign out. There is newly committed support for reading cookies in Akephalos, but I couldn’t get it to work, and I figured as a matter of practice it’d be better not to rely on it anyway.
Have we reached the end of our quest
We’re not the only ones on this quest, I’m sure, and I encourage those of you who are to take a look at Akephalos and give it a try, I don’t think you’ll be disappointed.
Unfortunately, I don’t think the quest is over. But Akephalos represents a significant step forward in a reliable headless browser that can allow us to fully integration test our applications. There are probably otther options available to us to try, and I expect there will be more as time goes on. For example, we’ve recently started to use evergreen to do javascript unit testing right alongside the normal tests.
We’ll continue to report back on what we find and consider the current state-of-the-art. Let us know if you’ve found and love something we’ve missed here.
