This is your threading bug

A race condition bug is a common class of bugs that occur when a sequence of actions assumes a state, but the state can be modified from an external source. We’ll go in depth in many examples, but a quick example is a mobile app that assumes a data connection exists because you had previously performed an API call; the state of the network is controlled by an external source. These two representations of network state are in a race for the next network call.

A special case of race condition is the time-of-check to time-of-use (TOCTOU) bug. These bugs deal with the distance between checking the state to using the knowledge of the state. Continuing the above example: checking whether you have a network connection, and then sending a HTTP request with the assumption that it will work. That network check might have been the network’s last breath before the phone enters a tunnel, thus presenting a TOCTOU bug.

These bugs are subtle, frequent, tricky to debug, and in some cases a security concern. In this article let’s examine some of the more common ones in the hope that we can internalize the fundamental structure, gain an intuition for recognizing this class of bugs, and avoid these classic examples in our own code.

(“TOCTOU” is not pronouced “toucan”, which is a disappointment to us all.)

Networks

The first example we’ll dive into is the one mentioned in the intro: checking for network connectivity before making the request. Here’s an example in Ruby; it contains a TOCTOU bug:

def request(req)
  if has_connection?(req)
    Net::HTTP.start(req.host, req.port) { |http| http.request(req) }
  end
end

def has_connection?(req)
  Addrinfo.getaddrinfo(req.host, req.port)
rescue SocketError
  false
end

This example exhibits the TOCTOU bug: it has a gap between the time of check and the time of use. We can use a sequence diagram to understand this gap:

The Ruby code seems sequential, but the state the program is tracking is a snapshot of state that is controlled externally. If the Ruby program uses its last network touchpoint to check for access instead of sending the HTTP request, we will see unexpected errors.

The correct way to solve this is hinted at in has_connection?. We already have to handle an exception; that is how the Addrinfo.getaddrinfo method communicates its return value. Therefore, we can make the HTTP call and handle the exception all in one swoop:

HTTP_ERRORS = [
  EOFError,
  Errno::ECONNRESET,
  Errno::EINVAL,
  Net::HTTPBadResponse,
  Net::HTTPHeaderSyntaxError,
  Net::ProtocolError,
  SocketError,
  Timeout::Error,
]

def request(req)
  Net::HTTP.start(req.host, req.port) { |http| http.request(req) }
rescue *HTTP_ERRORS
  nil
end

This has the added advantage of making only one network call, instead of two. The sequence diagram is much simpler now:

The network example is nice because it’s familiar and also a good template. It’s familiar as a user: you’ve definitely experienced bugs in mobile apps that assumed you had network connectivity because you previously had network connectivitity. It’s also familiar as a programmer: the network APIs we use all work via a exception-based pattern, and all examples are like the above. In this way it also provides a good template: move from an explicit conditional (asking permission) to a model where we do the action we want to do and handle the failure (asking forgiveness).

Note that a variant of this pattern holds for languages without exceptions. Many of these are too verbose for a blog post, but for example the Rust reqwest::get function returns a Result<Response> value instead of just Response, indicating that it can have an Error. Similarly in the C libcurl library, the curl_easy_perform(3) function returns CURLE_OK on success and non-zero on error, allowing you to make a request and handle the errors manually.

Files

Let’s play Spot The Bug with this code sample:

def eval_gemfile
  if File.exist?("Gemfile.local")
    eval(File.read("Gemfile.local"))
  end
end

This shape looks identical to the buggy network code above. Here’s an example sequence diagram:

This sequence diagram shows the bug: Ruby’s state of the filesystem does not match the actual state of the filesystem. We can recover this with exceptions, as we did with the network example:

def eval_gemfile
  eval(File.read("Gemfile.local"))
rescue Errno::ENOENT
end

The chance of this but appearing is low, but the ability to debug it even lower. This kind of bug can be exploited due to the complexities of how operating systems schedule processes and threads. After each operation there’s a good chance your program will stop for an unknown amount of time while other processes run. The arbitrary and seemingly random nature of this means that these kind of bugs can occur, and can be impossible to reproduce.

File security

A more subtle, security-sensitive bug can occur when checking permission and file access. For example, this has a bug:

def print_secrets
  if has_access?("secrets")
    puts File.read("secrets")
  end
end

def has_access?(filename)
  status = File.new(filename).stat
  status.file? && status.owned?
end

The TOCTOU bug becomes apparent under a sequence diagram showing the attack:

This example shows a much more difficult consideration than the above. If an attacker can know when File.stat is called vs File.read, they can manipulate the program into opening an unexpected file. The reason this example is much more difficult is because there’s no good solution here. Not all TOCTOU bugs are as trivial to solve as the file existence example. However, we should try to rely on underlying filesystem permissions as much as possible. This is harder to mitigate, which means that this kind of bug does happen in the wild.

def print_secrets
  file = File.new("secrets")
  contents = file.read

  if has_permission?(file)
    puts contents
  end
rescue Errno::EACCES
end

def has_permission?(file)
  status = file.stat
  status.file? && status.owned?
end

This is harder to exploit because the attacker needs to time calls to #print_secrets instead of File.stat, which is presumably more user-dependent. In addition, note that we now handle the exception for a file that we cannot access.

Authorization

The above examples are ones where the external state is under the control of a different entity – the network stack or the filesystem. As a last example let’s talk about what infrastructure you need to build to when plan around TOCTOU bugs.

Let’s use the example of authorization: a user needs to be able to do the things they are able to do, and nothing more (or less!). Some examples of what users can do are read, publish, delete, delete other content, and create admins.

If you check what a user can do when they sign in, you are storing that state for as long as they are authenticated. Here’s an example of buggy authorization code:

class SignInsController < ApplicationController
  def create
    if user = User.authorize_by(params[:email], params[:password])
      session[:user] = {
        id: user.to_param,
        can_read: user.can_read?,
        can_write: user.can_write?,
        can_delete: user.can_delete?,
        can_moderate: user.can_moderate?,
        can_promote: user.can_promote?,
      }
      redirect_to dashboard_path
    else
      flash.now[:error] = I18n.t(".incorrect_username_or_password")
      render :new
    end
  end
end

Setting authorization details once at sign-in means that any future check is static. Here’s an example of a buggy method of checking whether a user is an admin:

def can_promote?
  session[:user][:can_promote]
end

If the user is promoted, they will need to sign out and back in to see the effects. This is something the user wants, so they’ll be encouraged to do so. However, if the user is demoted or otherwise moderated by the admins, they will not feel the effects until they re-authenticate – which they have no interest in doing.

The TOCTOU bug may go deeper. Let’s look at some example code that shows the link to the admin page, the controller backing it, and the model code:

<nav>
  <ul>
    <li><%= link_to "Write new post", new_article_path %></li>
    <% if can_promote? %>
      <li><%= link_to "Promote user", new_promotions_path %></li>
    <% end %>
  </ul>
</nav>

class PromotionsController < ApplicationController
  def new
    @user = User.new
  end

  def create
    @user = User.find(params[:user_id])

    if @user.promote
      redirect_to new_promotions_path
    else
      render :new
    end
  end
end

class User < ActiveRecord::Base
  def promote
    update(can_promote: true)
  end
end

When designing your system, you can avoid introducing TOCTOU bugs by delaying any checks until the last moment, and relying on a “ask forgiveness, not permission” mindset.

A bug-free version of #can_promote? would go directly to the database every time:

def can_promote?
  current_user.can_promote?
end

But it goes deeper: the actual check must happen in the model layer. It is the last safeguard of the data and the keeper of authorization state.

class User < ActiveRecord::Base
  def promote_by!(promoter_id)
    transaction do
      promoter = self.class.lock.find_by(
        id: promoter_id,
        can_promote: true,
      )

      if promoter
        update!(can_promote: true)
      else
        raise ArgumentError, "can only be promoted by an admin"
      end
    end
  end
end

The prior method for promoting a user, User#promote, is replaced with one that tracks details on both users, User#promote_by!. This allows us to perform checks at the lowest level.

We want an atomic world with the latest values, so we run two database queries inside a transaction. The first query is for a user that matches our promoter’s ID and also has the permission required to perform the promotion. The second query is the update itself.

There remains a chance that the promoting user will be updated outside of this transaction, so we lock that row for consistency.

We choose to switch to exceptions here for consistency with other examples in Ruby where we try to avoid TOCTOU bugs, such as in networking or file code, and also because ActiveRecord::Base#transaction requires that.

Conclusion

When dealing with files, keep an eye on stat(2) calls, such as File.exist? in Ruby, test -e in shell, or stat() in C. These indicate potential bugs, and you may be able to re-write them using exceptions instead.

When working with or designing a system, take note of when you’re tracking state and who can control that state. If it can be modified by an external actor – a train tunnel, the user in another shell, the user opening another tab, or a security-minded attacker – then you have the potential for a time-of-check to time-of-use bug.

TOCTOU bugs range in severity from simply being tricky to track down to actual security issues. It is in the best interest for you and your users to build your software to avoid TOCTOU bugs. Take note of the textbook examples to avoid those in general.

The underlying principle to avoid a TOCTOU bug is “ask for forgiveness, not for permission”. This can be seen as another form of “tell, don’t ask”, and can also be seen as a form of Law of Demeter. The object that maintains the resource is the final authority on the resource’s state, and tracking the state elsewhere is setting yourself up for a TOCTOU bug.