Faking APIs in Development and Staging

You are a developer for startup called Movie Social Network++, building a social network for movie aficionados. Several features are dependent on data about various movies, actors, or directors. You do a quick Google search to figure out where to get this sort of data and come across movie-facts.com. They have already gathered all this data and have a team constantly keeping it up to date. They offer to make all this data available to you via their API for a moderate fee.

This seems like a great solution. The team agrees and one of the founders reaches out to movie-facts.com to negotiate a contract.

Test-driving the first feature

In the meantime, you start working on a feature that depends on this API:

When I reference a movie title in a post, I want to see some quick facts (director, actors).

In test-driven fashion, you start by writing a feature test:

visit root_path

click_on "New Post"
fill_in :post, with "Jurassic World was soooo good!!!"
click_on "Post"

info_box = find(".info-box")
expect(info_box).to have_content("Chris Pratt")

As you try to make this test pass, you eventually need to actually fetch the data from the API. You decide to isolate all of the API-related code in an adapter class:

# app/models/movie_database.rb

class MovieDatabase
  BASE_URL = "http://api.movie-facts.com".freeze

  def actors_for(movie:)
    HTTParty.get(BASE_URL + "/movies/#{movie.id}/actors", format: :json)
  end
end

Now you get an error from Webmock saying that external requests are blocked in tests. You decide to build a test fake for the movie-facts.com API using Sinatra.

# spec/support/fake_movie_facts.rb

module FakeMovieFacts
  class Application < Sinatra::Base
    get "/movies/:movie_name/actors" do
      {
        actors: [
          {
            name: "Actor 1",
            character_played: "Character 1"
          },
          {
            name: "Actor 2",
            character_played: "Character 2"
          }
        ]
      }.to_json
    end
  end
end

There are a few ways to load up a fake in a test.

Capybara Discoball is a gem that allows you to boot other Rack apps in the background of a feature test run. Once you have an app running, the adapter can be pointed to that app.

You decide to use Capybara Discoball because we don’t need to stub anything. The application is literally making HTTP requests to APIs that are running on your localhost.

You add configuration to allow Capybara Discoball to run your fake movie-facts.com API:

# spec/support/fake_movie_facts_runner.rb

FakeMovieFactsRunner =
  Capbybara::Discoball::Runner.new(FakeMovieFacts::Application) do |server|
    MovieDatabase.base_url = "#{server.host}:#{server.port}"
  end

Then in the spec helper:

# spec/spec_helper.rb

FakeMovieFactsRunner.boot

The MovieDatabase adapter doesn’t currently allow its base url to be changed so you change it to use a class accessor rather than a hard coded constant, and default it to the movie-facts.com API.

# app/models/movie_database.rb

class MovieDatabase
  cattr_accessor :base_url
  base_url = "http://api.movie-facts.com"

  def actors_for(movie:)
    HTTParty.get(self.class.base_url + "/movies/#{movie.id}/actors", format: json)
  end
end

Now your tests pass. Mission accomplished!

Fakes in development

Excited by this new feature, you grab a colleague to show off the new functionality your local machine. To your chagrin, it doesn’t work at all. Instead, you get unauthorized errors from the movie-facts.com API. You explain to your colleague that the feature does work in the tests because you are using a fake.

She nods and makes a surprising suggestion: Why not also use the fake in development? You think about it briefly and agree that this is probably the easiest way to get this feature running in development.

You open a new terminal and run the following:

ruby spec/support/fakemoviefacts.rb

This boots up a Sinatra server on localhost:4567.

Now you just need to point the adapter to this url. In addition to allowing the base url to be changed at runtime, you decide to default the value to an environment variable instead of the hardcoded url you are currently using.

# app/models/movie_database.rb

class MovieDatabase
  cattr_accessor :base_url
  self.base_url = ENV.fetch("MOVIE_FACTS_API_BASE_URL")

  def actors_for(movie:)
    HTTParty.get(self.base_url + "/movies/#{movie.id}/actors", format: :json)
  end
end

You try booting up a Rails server again:

MOVIEFACTSAPIBASEURL=localhost:4567 rails server

You open the app in your browser and create a post that references a movie. “HEY, IT WORKS!”

Fakes on staging

Hearing your shout of jubilation, your boss comes over to admire your achievement. He mentions that it would be great if you could deploy this feature to staging so that he can demo it at an investor meeting tomorrow.

You scratch your head. The real movie-facts.com API won’t be available in time you explain to your boss. The demo he’s just seen used a test fake to simulate the real API. “Can you do the same thing on the staging server?”, he asks. You agree that this should be possible and timebox an hour to explore the options.

You need to run both the main application and the fake and have them be able to communicate with each other via HTTP. The simplest way to accomplish this is to have each be its own Heroku application. However, extracting the fake into its own app leads to a dilemma. The tests still need the fake in order to pass. How do you share the code with the main application while still making it easy to deploy as its own application?

After discussing with your colleague, you decide on the following architecture:

Extract to a separate git repository
Wrap it as a gem so it can be used by the tests
Provide a config.ru file in the root of the gem so it can be deployed on Heroku.

Initializing a new gem with bundle gem fakemoviefacts will generate the following structure (via tree fakemoviefacts):

fake_movie_facts
├── Gemfile
├── README.md
├── Rakefile
├── bin
│   ├── console
│   └── setup
├── fake_movie_facts.gemspec
├── lib
│   ├── fake_movie_facts
│   │   └── version.rb
│   └── fake_movie_facts.rb
└── spec
    ├── fake_movie_facts_spec.rb
    └── spec_helper.rb

You extract FakeMovieFacts::Application from the main application into fake_movie_facts/lib/fake_movie_facts/application.rb and add a config.ru file to the root of the repo:

# config.ru

$LOAD_PATH << File.expand_path("../lib", __FILE__)
require "fake_movie_facts/application"

run FakeMovieFacts::Application

Heroku will automatically pickup on the config.ru file when you deploy the gem. You can also run it locally via the rackup command.

You deploy both apps Heroku (staging) and edit the main application’s environment variables to point to the gem:

heroku config:set MOVIEFACTSAPIBASEURL=http://fake-movie-facts.herokuapp.com --remote staging

A test drive confirms that the various components are correctly working with each other. Investor demo, here we come!

Advanced Fakes

With your feature complete, you pull the next one from the top of the queue:

As a user, I want to be able to subscribe to news for an upcoming movie and have it piped into my news feed.

The task mentions that movie-news.com has a free API that allows you to subscribe to events for a given movie via a webhook. As previously, you test-drive via a feature spec and run into Webmock’s “external URL” error, so you write a fake:

module FakeUpcomingMovieEvents
  class Application < Sinatra::Base
    post "subscriptions/:movie_name" do
      successful_subscription.to_json
    end

    private

    def successful_subscription
      {
        subscription_id: "123",
        movie_subscribed_to: params[:movie_name]
      }
    end
  end
end

Great! This fixes your test failure. You are now faced with another problem though. You need some way to trigger an event in your tests. Thinking ahead, you realize that you will probably also need a way to do this in development and staging for demo purposes. Since you have no control over movie-events.com‘s event API, you decide to have the fake automatically trigger the webhook right after each subscription.

module FakeUpcomingMovieEvents
  class Application < Sinatra::Base
    post "subscriptions/:movie_name" do
      trigger_webhook

      successful_subscription.to_json
    end

    private

    def trigger_webhook(callback_url)
      HTTParty.post(params[:callback_url], event_payload_json)
    end

    def event_payload_json
      {
        event_type: "New Trailer",
        url: "http://video-sharing-platform.com/123"
      }.to_json
    end

    # ...
  end
end

This results in an unexpected bug. The subscription deadlocks because the subscription endpoint can’t return until the webhook request succeeds but the webhook can’t be processed by the main app until the subscription request succeeds. This catch-22 situation is caused by having the two tasks be synchronous (blocking).

After giving it some thought you decide to try and background the webhook task. Adding a queueing system such as DelayedJob to a fake seems a bit heavy handed so you try to build something using threads:

module FakeUpcomingMovieEvents
  class Application < Sinatra::Base
    post "subscriptions/:movie_name" do
      async do
        trigger_webhook(params[:callback_url])
      end

      successful_subscription.to_json
    end

    private

    def async
      Thread.new do
        sleep ENV.fetch("WEBHOOK_DELAY").to_i
        yield
      end
    end

    # other methods
  end
end

This fixes your tests. You extract a gem + config.ru as with the previous fake and deploy to heroku/staging. You configure the fake to trigger a “New Trailer” event 15 seconds after subscribing to a movie’s events. Tomorrow’s demo should be a good one.

Conclusion

Fakes are great for testing an application that interacts with 3rd party APIs. Their usefulness extends beyond just testing however. In situations where the API is not currently available, doesn’t have a sandbox mode, or you need more control over events that it emits, fakes can be a great solution in development and staging as well.

Packaging them as Rack-compatible apps plus a config.ru file wrapped in a gem makes it easy to share across environments and servers. It also makes it easy to open source and develop fakes for popular APIs with the community, even though your main app remains closed source.

Test-driving the first feature

Fakes in development

Fakes on staging

Advanced Fakes

Conclusion

Sign up to receive a weekly recap from thoughtbot