Testing Your Edge Cases

We have just added some new permissions logic to a project.

def can_read?
  article.access_policy&.public? || user.admin?
end

We might include some tests that read like:

describe "#can_read?" do
  it "allows admin to access public article"
  it "allows admin to access private article"
  it "allows non-admin to access public article"
  it "disallows non-admin from accessing private article"
end

During code review, one of our colleagues asks

Do we have any tests that cover the nil case?

Hmmm, it looks like we forgot! How can we do a better job of making sure we cover all our edge cases?

Using math

We can use math to calculate how many combinations of inputs we have. In order to do so, we need to identify each of the inputs that could affect our method, calculate how many different states they can have, and multiply them together.

Here, two variables influence the outcome of our method: the access policy and the user’s role. They can be in the following number of states:

1. An article’s public status can be in 3 states (nil, true, or false)
2. A user being an admin can have 2 states (true or false)

Multiplying these gives us 6 possible combinations of inputs. We probably want 6 unit tests to cover all the combinations but our original implementation only had 4. Time to add the missing 2 tests to cover nil!

Visually

Sometimes we may prefer to calculate the different cases visually using a truth table or decision table. Taking this approach is more explicit but doesn’t scale up well to many inputs or many states due to combinatorial explosion.

More than just booleans

Calculating combinations is useful for more than just booleans. Consider the following constructor for a query object:

class ArticleQuery
  def initialize(limit: nil, scope: Article.all)
    # ...
  end
end

There are at least 4 different ways of invoking this method (2 arguments, each of which can be explicitly passed or defaulted). We probably should have a test for each of these cases.

1. new
2. new(limit: 10)
3. new(scope: Article.published)
4. new(limit: 10, scope: Article.published)

There may be even more cases. Sometimes explicitly passing nil may be different than not passing anything. The scope argument could have 3 states (default, explicit scope, explicit nil).

Some types of values like strings or arrays have theoretically infinite states. In practice we probably care about a finite number of slices of that infinite domain. For example, our code might only distinguish between empty, single, and multi-item arrays giving us functionally 3 different states.

Reducing the states

In our original example, we identified 2 edge cases that weren’t covered in our test suite. One solution is to add some tests so that our suite now covers the 6 scenarios handled by our code.

Alternatively, we could reduce our code to only handle the 4 scenarios described in our test suite. Testing pain is often a sign that we should change our implementation.

Pushing uncertainty to the edges

Nils in particular cause a lot of edge cases and pushing them to the edges of our system as much as possible will make both our coding and testing lives easier.

In our original example, we may decide to make access policies required on articles, eliminating the potential nil and bringing us down to 4 possible combinations.

Similarly, we might remove the defaults from the ArticleQuery constructor. Now we have eliminated a lot of uncertainty as there is only one way to invoke it. A win for both our code and our test suite!