How to Stub External Services in Tests

Requests to external services during test runs can cause several issues:

Tests failing intermittently due to connectivity issues.
Dramatically slower test suites.
Hitting API rate limits on 3rd party sites (e.g. Twitter).
Service may not exist yet (only documentation for it).
Service doesn’t have a sandbox or staging server.

When integrating with external services we want to make sure our test suite isn’t hitting any 3rd party services. Our tests should run in isolation.

Disable all remote connections

We’ll use Webmock, a gem which helps to stub out external HTTP requests. In this example we’ll search the GitHub API for contributors to the FactoryBot repository.

First, let’s make sure our test suite can’t make external requests by disabling them in our spec_helper.rb:

# spec/spec_helper.rb
require 'webmock/rspec'
WebMock.disable_net_connect!(allow_localhost: true)

Now let’s verify that any external requests will raise an exception and break the build:

# spec/features/external_request_spec.rb
require 'spec_helper'

feature 'External request' do
  it 'queries FactoryBot contributors on GitHub' do
    uri = URI('https://api.github.com/repos/thoughtbot/factory_bot/contributors')

    response = Net::HTTP.get(uri)

    expect(response).to be_an_instance_of(String)
  end
end

As expected we now see errors when external requests are made:

$ rspec spec/features/external_request_spec.rb
F

Failures:

  1) External request queries FactoryBot contributors on GitHub
     Failure/Error: response = Net::HTTP.get(uri)
     WebMock::NetConnectNotAllowedError:
       Real HTTP connections are disabled.
       Unregistered request: GET https://api.github.com/repos/thoughtbot/factory_bot/contributors
       with headers {
         'Accept'=>'*/*',
         'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3',
         'Host'=>'api.github.com',
         'User-Agent'=>'Ruby'
       }

       You can stub this request with the following snippet:

       stub_request(:get,     "https://api.github.com/repos/thoughtbot/factory_bot/contributors").
     with(:headers => {'Accept'=>'*/*', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'Host'=>'api.github.com', 'User-Agent'=>'Ruby'}).
     to_return(:status => 200, :body => "", :headers => {})

       ============================================================
     # ./spec/features/external_request_spec.rb:8:in `block (2 levels) in <top (required)>'

Finished in 0.00499 seconds
1 example, 1 failure

We can fix this by stubbing any requests to api.github.com with Webmock, and returning pre-defined content.

# spec/spec_helper.rb
RSpec.configure do |config|
  config.before(:each) do
    stub_request(:get, /api.github.com/).
      with(headers: {'Accept'=>'*/*', 'User-Agent'=>'Ruby'}).
      to_return(status: 200, body: "stubbed response", headers: {})
  end
end

Run the test again and now it will pass.

$ rspec spec/features/external_request_spec.rb
.

Finished in 0.01116 seconds
1 example, 0 failures

VCR

Another approach for preventing external requests is to record a live interaction and ‘replay’ it back during tests. The VCR gem has a concept of cassettes which will record your test suites outgoing HTTP requests and then replay them for future test runs.

Considerations when using VCR:

Communication on how cassettes are shared with other developers.
Needs the external service to be available for first test run.
Difficult to simulate errors.

We’ll go a different route and create a fake version of the GitHub service.

Create a Fake (Hello Sinatra!)

When your application depends heavily on a third party service, consider building a fake service inside your application with Sinatra. This will let us run full integration tests in total isolation, and control the responses to our test suite.

First we use Webmock to route all requests to our Sinatra application, FakeGitHub.

# spec/spec_helper.rb
RSpec.configure do |config|
  config.before(:each) do
    stub_request(:any, /api.github.com/).to_rack(FakeGitHub)
  end
end

Next we’ll create the FakeGitHub application.

# spec/support/fake_github.rb
require 'sinatra/base'

class FakeGitHub < Sinatra::Base
  get '/repos/:organization/:project/contributors' do
    json_response 200, 'contributors.json'
  end

  private

  def json_response(response_code, file_name)
    content_type :json
    status response_code
    File.open(File.dirname(__FILE__) + '/fixtures/' + file_name, 'rb').read
  end
end

Download a sample JSON response and store it in a local file.

# spec/support/fixtures/contributors.json
[
  {
    "login": "joshuaclayton",
    "id": 1574,
    "avatar_url": "https://2.gravatar.com/avatar/786f05409ca8d18bae8d59200156272c?d=https%3A%2F%2Fidenticons.github.com%2F0d4f4805c36dc6853edfa4c7e1638b48.png",
    "gravatar_id": "786f05409ca8d18bae8d59200156272c",
    "url": "https://api.github.com/users/joshuaclayton",
    "html_url": "https://github.com/joshuaclayton",
    "followers_url": "https://api.github.com/users/joshuaclayton/followers",
    "following_url": "https://api.github.com/users/joshuaclayton/following{/other_user}",
    "gists_url": "https://api.github.com/users/joshuaclayton/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/joshuaclayton/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/joshuaclayton/subscriptions",
    "organizations_url": "https://api.github.com/users/joshuaclayton/orgs",
    "repos_url": "https://api.github.com/users/joshuaclayton/repos",
    "events_url": "https://api.github.com/users/joshuaclayton/events{/privacy}",
    "received_events_url": "https://api.github.com/users/joshuaclayton/received_events",
    "type": "User",
    "site_admin": false,
    "contributions": 377
  }
]

Update the test, and verify the expected stub response is being returned.

require 'spec_helper'

feature 'External request' do
  it 'queries FactoryBot contributors on GitHub' do
    uri = URI('https://api.github.com/repos/thoughtbot/factory_bot/contributors')

    response = JSON.load(Net::HTTP.get(uri))

    expect(response.first['login']).to eq 'joshuaclayton'
  end
end

Run the specs.

$ rspec spec/features/external_request_spec.rb
.

Finished in 0.04713 seconds
1 example, 0 failures

Voilà, all green! This now allows us to run a full integration test without ever having to make an external connection.

A few things to consider when creating a fake:

A fake version of a service can lead to additional maintenance overhead.
Your fake could get out of sync with the external endpoint.

What’s next

If you found this useful, you might also enjoy:

Disclaimer:

Looking for FactoryGirl? The library was renamed in 2017. Project name history can be found here.

Disable all remote connections

VCR

Create a Fake (Hello Sinatra!)

What’s next

Sign up to receive a weekly recap from thoughtbot

Upgrade your codebase with a Code Audit