Advanced Ruby: The chainable request pattern

In the previous article of my series on error handling and fault tolerance, we discussed how to run external API requests in a transaction while handling errors and making it fault-tolerant. That article is not a requirement to understand this one, so read on in either case!

Here, we will explore some means of abstraction for the transactional pattern. You will learn how to encapsulate API requests for use in an external transaction, including:

Automatic rollback on error;
Support for running custom code in between requests;
Chain multiple API requests with automatic rollback support;
Return a chained result of all API requests.

Let’s call it the “Chainable Request Pattern”.

The request object

The first step is to wrap the API request in a class. The code must return a result object with the outcome:

Result = Struct.new(:status, :error_code, :id, keyword_init: true) do
  def success?
    status == :ok
  end

  def error?
    !success?
  end
end

class LineItemRequest
  BASE_URL = "https://app.moderntreasury.com/api/"
  HEADERS = {"Content-Type" => "application/json"}.freeze

  def self.execute(params)
    response = client.post("ledger_transactions", JSON.dump(params))

    if response.status == 200
      Result.new(status: :ok, id: response.body["id"])
    else
      Result.new(status: :error, error_code: response.status)
    end
  end

  private_class_method def self.client
    Faraday.new(url: BASE_URL, headers: HEADERS) do |client|
      org_id = Configuration.organization_id
      api_key = Configuration.api_key

      client.response :json
      client.request :authorization, :basic, org_id, api_key
    end
  end
end

The result object must return whether the request was successful or not.

The request runner

The role of the request runner is to execute a request object and optionally run arbitrary code on success. Let’s see how it could be implemented:

class RequestRunner
  def self.execute(request, params)
    result = request.execute(params)

    if block_given? && result.success?
      yield result
    end

    result
  end
end

Using the request object is straightforward:

params = {
  description: "Scientific Calculator",
  status: "pending",
  ledger_entries: [
    {
      amount: 10_00,
      direction: "debit",
      ledger_account_id: "368feff6-fe48-44f0-95de-50ee1f2d1e50",
    },
    {
      amount: 10_00,
      direction: "credit",
      ledger_account_id: "eec76db0-b537-47bf-a28d-3aa397f35f69",
    },
  ]
}

RequestRunner.execute LineItemRequest, params do |result|
  puts "Success! ID: #{result.id}"
end

Rollbacks

If you are asking, “How can the request runner be helpful? It seems to be a thin wrapper that provides no value”, that is a great question! And the preliminar answer is rollbacks. The request runner runs a request and executes a block that optionally runs dependent code. If the dependent code fails, the request runner should automatically roll back the request.

The first step to implementing that is to write a rollback method in our request object:

class LineItemRequest
  # ...

  def self.execute(params)
    # Code for 'execute' goes here
  end

  def self.rollback(result, _params)
    return if result.nil?

    # Archiving is how we roll back our particular operation
    # in the Modern Treasury API.
    patch_params = {status: "archived"}
    client.patch "ledger_transactions/#{result.id}", JSON.dump(patch_params)
  end

  # ...
end

If you think this looks familiar, you’re on point, as LineItemRequest implements the Command pattern.

So far, every request object must implement the following interface:

execute(params) - Returns a Struct result that responds to success?.
rollback(result, params) - Returns void. Don’t care for the return value.

Second, we need to hook up rollback to our request runner:

class RequestRunner
  def self.execute(request, params)
    result = request.execute(params)

    if block_given? && result.success?
      begin
        yield result
      rescue => e
        request.rollback(result, params)
        raise e
      end
    end

    result
  end
end

And we do that through an exception handler because we want to revert our successful request if an exception is raised. After rollback, we still need to reraise the exception to signal an error to the application.

Now, let’s execute the request and try to save an Active Record reference:

RequestRunner.execute LineItemRequest, params do result
  OrderLineItem.create! external_id: result.id
end

In Rails, create! raises an exception on error. If that ever happens, our request will automatically roll back. Neat!

Chaining dependent requests

Do you recall that the role of the success block is to run dependent code with rollback support? What if that code is another request? Would it work? Yes!

However, chaining requests poses an exciting challenge: the caller needs to get the chained results of various requests. Managing local variables and overriding them in our success blocks would achieve that, but it’s not practical:

result_1 = nil
result_2 = nil

RequestRunner.call ... do |r1|
  result_1 = r1

  RequestRunner.call ... do |r2|
    result_2 = r2
  end
end

final_result = [result_1, result_2]

Instead, let’s change the request runner to do that work behind the scenes and chain the results in a linked list. By wishful thinking, the API we want to achieve is:

# Different types of request can be chained, not just a single type
result = RequestRunner.execute LineItemRequest, params_1 do |_result_1|
  RequestRunner.execute LineItemRequest, params_2 do |_result_2|
    RequestRunner.execute LineItemRequest, params_3
  end
end

result # Result of the first request
result.next_result # Result of the second request
result.next_result.next_result # Result of the third request

Also, if the second or third request fails, the first result should be an error; after all, it’s an all or nothing transaction.

# Should always return false on error throughout the chain
first_result.success?

Finally, if the second request fails, the chain must halt and not run the third request:

result.success? # false
result.next_result.success? # false
result.next_result.next_result # nil. Third request does not run.

Let’s see how we can achieve that. The first step is to change the request runner to chain the previous result with the next result in case of success:

class RequestRunner
  def self.execute(request, params)
    result = request.execute(params)

    if block_given? && result.success?
      begin
        next_result = yield(result)
      rescue => e
        request.rollback(result, params)
        raise e
      end

      if next_result.respond_to?(:chain_result)
        result = result.chain_result(next_result)

        if next_result.error?
          request.rollback(result, params)
        end
      end
    end

    result
  end
end

We only call chain_result if the current result is successful; otherwise, we return the current result unchanged.

We are also rolling back the current request if the next result is an error, which is a great companion to the exception handler and makes our code work per our intended design. Oh, and if you are asking if that works recursively for every chained request, the answer is yes! This will be clear down below, so read on.

And, of course, we need to implement chain_result in our result object. Because we want the “chain” ability in requests with different requirements, we need a reusable module:

module ResultChainable
  attr_reader :next_result

  def initialize(next_result: nil, **attrs)
    super(**attrs)
    @next_result = next_result
  end

  def chain_result(next_result)
    attrs = {
      status: next_result.status,
      next_result: next_result
    }
    self.class.new(**to_h, **attrs)
  end
end

Note that ResultChainable requires the result object to respond to status and to_h (which comes free with Struct objects). The status of the chained result must become the status of the next result because one error should make everything fail.

Now, let’s include the module in our result class:

Result = Struct.new(:status, :error_code, :id, keyword_init: true) do
  include ResultChainable

  def success?
    status == :ok
  end

  def error?
    !success?
  end
end

And try it out:

result_1 = Result.new(status: :ok, id: "result-1-id")
result_2 = Result.new(status: :ok, id: "result-2-id")
result_3 = Result.new(status: :error, error_code: "101")

chained_result = result_1.chain_result(result_2.chain_result(result_3))

chained_result.status # :error, not :ok
chained_result.id # "result-1-id"
chained_result.success? # false, not true

chained_result.next_result.status # :error, not :ok
chained_result.next_result.id # "result-2-id"
chained_result.next_result.success? # true

chained_result.next_result.next_result.status # :error
chained_result.next_result.next_result.success? # false
chained_result.next_result.next_result.error_code # "101"
chained_result.next_result.next_result.next_result # nil

This works as expected! Let’s review our request chaining code:

chained_result = RequestRunner.execute LineItemRequest, params_1 do |_result_1|
  RequestRunner.execute LineItemRequest, params_2 do |_result_2|
    RequestRunner.execute LineItemRequest, params_3 do |_result_3|
      # ...
    end
  end
end

With our code so far, the results are chained in a fold right fashion. For example, assuming that the three above requests are indeed run, their three results would get implicitly folded:

# This happens behind the scenes
chained_result = result_1.chain_result(result_2.chain_result(result_3))

Note that you can still run custom code before running the next request:

result = RequestRunner.execute LineItemRequest, params_1 do
  # Can still run arbitrary code here

  RequestRunner.execute LineItemRequest, params_2 do
    # Can still run arbitrary code here

    RequestRunner.execute LineItemRequest, params_3
  end
end

Let’s imagine a few scenarios to form a mental picture of how this all works.

Scenario 1: An exception is thrown in the success block of the third request. What happens: Requests 1 and 2 roll back. Chained result is error.

Scenario 2: Request 1 succeeds, request 2 returns a non-200 response. What happens: Requests 1 rolls back. Request 3 does not run. Chained result is error.

Scenario 3: Requests 1 and 2 succeed, request 3 returns a non-200 response. What happens: Requests 1 and 2 roll back. Chained result is error.

Scenario 4: All requests succeed, no exceptions thrown. What happens: No rollbacks. Chained result is success.

Flattening the chain

If you can, you should avoid running too many dependent requests. If you are running just a few requests, though, the nesting imposed by the block structure should not be a big deal. But what if you need to run more than a few requests? Then, unnesting the structure should improve clarity and readability.

It also formalizes the overall pattern, which comes with a benefit regardless if you are running just a few requests or more than a few. Oh, and you can also run requests dynamically with that!

Let’s imagine an API where we could flatten our requests:

on_line_item_success = -> { OrderLineItem.create!(external_id: _1.id) }

array_of_arguments = [
  [LineItemRequest, params_1, on_line_item_success],
  [OrderRequest, params_2],
]

result = TransactionalRequestRunner.execute(array_of_arguments)

# Do something with the result, or not

array_of_arguments is a matrix that has the arguments of each request. For each request, we have the request class, the params, and an optional block to run on the success of that request where we don’t worry about request chaining because TransactionalRequestRunner will already do it for us.

Now let’s write the code to run a batch of requests in a transaction:

class TransactionalRequestRunner
  def self.execute(array_of_arguments)
    request, params, on_success = array_of_arguments.shift

    request.execute(params) do |result|
      on_success.call result if on_success

      unless array_of_arguments.empty?
        execute array_of_arguments
      end
    end
  end
end

As you can see, TransactionalRequestRunner#execute is a recursive method that runs a request, and within the request’s success block, it recursively calls itself to fire the subsequent request. It formalizes the overall pattern and leverages all the features we’ve implemented.

What we haven’t discussed

We can make both essential and nice-to-have improvements to this code, but I’ll leave it as an exercise for the reader. For example:

Make rollback asynchronous for fault tolerance and reducing the overall runtime;
Improve HTTP response handling;
Improve exception handling and isolate exceptions from third-party libraries;
Support rolling back custom code that runs in between requests;
Support traversing the linked list in the result object to collect data;
Allow the on_success blocks in TransactionalRequestRunner to access the result of previous requests.

What other improvements do you have in mind?

Wrapup

You can find the complete code examples here in this gist.

I hope this article inspires you to up your external API code game! A more advanced version of this pattern has been successfully used in production in a system that moves millions of dollars. Regardless of your app’s size, these are all good practices you can adopt to make your code more resilient and fault-tolerant. It’s not specific to the Ruby language, either.

If you want to dive deeper into the theory behind transactional API requests with an eye on error handling and fault tolerance, I highly recommend you read the previous article.

I also recommend the first article in the series about the resumable pattern, which discusses a different approach for dealing with external API requests.

Learn about working with thoughtbot

Partner with us to learn how thoughtbot develops best-in-class Ruby on Rails applications. Let’s talk!

Sign up to receive a weekly recap from thoughtbot

1-on-1 and Group Mentoring