Headless Capybara Feature Specs with Chrome

I’ve been a happy user of Capybara-WebKit for many years now, but its dependence on Qt can make it frustrating to install on macOS, particularly following macOS or Xcode updates. One such recent issue lead me to experiment with running my tests in Chrome via ChromeDriver and Selenium.

I found the installation of ChromeDriver and Capybara-Selenium to be significantly faster than the installation of Qt and Capybara-WebKit, and my teammates reported no difficulties with those steps either.

While tests executing with Capybara-WebKit are headless, I initially did not have this option when configuring ChromeDriver. In this initial configuration, I was reminded that it is occasionally useful to watch a test execute without having to resort to save_and_open_screenshot. I was also pleasantly surprised to find that ChromeDriver never seems to steal focus from my active Chrome session or any other applications, which is a refreshing change from my memories of executing tests in Firefox via Selenium.

Last week saw the stable channel release of Chrome 59, which supports headless operation on macOS. Headless operation on Linux was already possible as of Chrome 57 and will be coming to Windows soon as well. I updated my Capybara configuration to run ChromeDriver with headless support and am now a mostly satisfied ChromeDriver convert.

ChromeDriver Installation

Executing your feature specs in Chrome requires that you have Chrome and ChromeDriver installed. Web developers and designers are likely to have Chrome installed already, so that leaves us needing to install ChromeDriver.

ChromeDriver is installed via Homebrew with brew install chromedriver and is similarly available in your package manager of choice on Linux. If you already have ChromeDriver installed, be sure to install an up-to-date version for headless support. I’m currently using ChromeDriver version 2.30.

If you’re not comfortable making this a prerequisite to running your application’s tests, you can also install ChromeDriver by adding chromedriver-helper to your Gemfile. On install this will download a platform-appropriate binary for ChromeDriver and add it to your gem path. If you opt for this approach, be sure to read the documentation on updating ChromeDriver and the open issue on Windows support.

Capybara Configuration

Add capybara-selenium to the test group of your Gemfile and optionally remove capybara-webkit while you’re at it. All that’s left for us to do now is to configure our drivers. In your rails_helper.rb or some file required by that file, add the following:

require "selenium/webdriver"

Capybara.register_driver :chrome do |app|
  Capybara::Selenium::Driver.new(app, browser: :chrome)
end

Capybara.register_driver :headless_chrome do |app|
  capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
    chromeOptions: { args: %w(headless disable-gpu) }
  )

  Capybara::Selenium::Driver.new app,
    browser: :chrome,
    desired_capabilities: capabilities
end

Capybara.javascript_driver = :headless_chrome

This configures chrome and headless_chrome drivers and sets Capybara to use headless_chrome for JavaScript tests by default. If you’d like to watch the tests execute while debugging, you can change the driver to chrome.

You may notice that the headless_chrome driver also passes the disable-gpu option. The documentation for the headless Chrome indicates this is only temporarily necessary but does not specify why. It’s not clear if this is necessary now that the feature is stable, but it doesn’t seem to hurt performance.

If you use the headless_chrome driver on an older version of Chrome or on Windows, you will get a NetReadTimeout error. I had hoped ChromeDriver would ignore the option on older browsers, but this is unfortunately not the case. You may need to make Capybara.javascript_driver configurable via an environment variable until such time as Chrome 59 is ubiquitous across your team.

On CI

The project I tried this on uses CircleCI which required no changes to its configuration. Chrome and ChromeDriver are already available in that environment.

If your project uses Travis, you will need to enable the Chrome addon. If you’re trying out Heroku CI, you can use the Chrome buildpack.

Beyond Installation, How Does ChromeDriver Stack Up?

Capybara-WebKit runs our tests on a fork of the WebKit browser engine via Qt. This engine is generally “close enough” but is not functionally equivalent to Safari (built on WebKit), Chrome (built on Blink, another fork of WebKit), or any other browser your users are likely to be using. This has occasionally caused issues in tests. By executing our tests directly in Chrome we are testing with the exact browser many of our users will be using. We’re also just steps away from executing those same tests in Firefox or even in Safari via SafariDriver.

I don’t have any hard science to offer on the matter of performance. I can anecdotally report that Capybara-Webkit seems significantly faster. I used an application with a dozen JavaScript-dependent specs to compare performance and found that Capybara-Webkit runs rspec --tag js in about 16 seconds, while the same command takes about 22 seconds using ChromeDriver. It’s not clear to me if the performance difference is dominated by startup cost or if it will scale linearly with the number of specs.

Capybara-Webkit also offers the block_unknown_urls configuration setting which prevents loading potentially slow external assets in your tests, such as external web fonts or analytics packages. I’ve yet to find a similar configuration in ChromeDriver. ChromeDriver allows specifying a proxy which could be used to accomplish this, but it would mean needing to run that proxy process as well.

Finally, I’ve noticed that save_and_open_screenshot produces an empty, gray screenshot when running in headless mode, while it works as expected on Capybara-Webkit. When the need for visual inspection of a test arises, I switch to the chrome Capybara driver, but this may be a significant detriment to your workflow if you rely on automated screenshots after test failures, for example.

What Does This Mean for Capybara-WebKit?

We’re not sure. We’re still in the experimentation phase of our use of ChromeDriver. At this time, our new projects still default to using Capybara-WebKit though this may change as more projects try out headless ChromeDriver.

If you’re a Capybara-Webkit user and give headless Chrome a try, we’d love to hear your experiences. How did the performance of your test suite differ? Did you find Chrome via Selenium and ChromeDriver to be lacking any features you count in Capybara-WebKit? Have you found a way to take screenshots? Tweet us and let us know!