---
title: Your flaky tests might be time dependent
teaser: 'Time dependent tests are difficult to diagnose, and the symptoms don''t present
  themselves until it''s too late. This article will help you determine if your test
  suite is time dependent.

  '
tags: testing,rails,web
author: Steve Polito
published_on: 2022-12-05
---

Have you ever dealt with unexpected test failures, and no matter how many times
you re-run the tests, they still fail? You assume that you introduced a bug, so
you stash your changes and checkout `main` and run the same tests, but are
surprised to see that they still fail, even though the latest build is passing.
Dejected, you close your laptop and call it a day, only to return to a passing
test suite the next morning. The problem wasn't the codebase, but the test
suite. More specifically, the problem was with _when_ the test suite was run.

## A Simple Example

Image you're working on a new feature that prevents notifications from being
created outside business hours. Here's what the current state of the codebase
looks like.

```ruby
# app/controllers/notifications_controller.rb
def create
  @notification = Notification.new(notification_params)

    if @notification.save
      ...
    else
      ...
    end
  end
end

# test/controllers/notifications_controller_test.rb
test "should create notification" do
  assert_difference("Notification.count") do
    post notifications_url, params: {...}
  end
end
```

You already have existing test coverage for creating a notification, so you go
ahead and write some tests and build the new feature.

```ruby
# app/controllers/notifications_controller.rb
def create
  @notification = Notification.new(notification_params)

    if within_business_hours? && @notification.save
      ...
    else
      ...
    end
  end

  private

  def within_business_hours?
    Time.use_zone("America/New_York") do
      Time.current.between?(
        Time.zone.parse("06:00"),
        Time.zone.parse("17:59")
      )
    end
  end
end

# test/controllers/notifications_controller_test.rb
test "should not create notification before business hours" do
  travel_to(Time.use_zone("America/New_York") { Time.parse("2022-11-19T05:59") })

  assert_no_difference("Notification.count") do
    post notifications_url, params: {...}
  end
end

test "should not create notification after business hours" do
  travel_to(Time.use_zone("America/New_York") { Time.parse("2022-11-19T18:00") })

  assert_no_difference("Notification.count") do
    post notifications_url, params: {...}
  end
end

test "should create notification" do
  assert_difference("Notification.count") do
    post notifications_url, params: {...}
  end
end
```

All three tests pass, so you create a pull request and request a review. Later
that day, your co-worker in California pulls down your branch, but is not able
to get the test suite to pass locally.

The problem is that it's 3:00 PM local time for your co-worker, which is 6:00 PM
on the East Coast, meaning it's after business hours. You never ran into this
problem during feature development because you are on the East Coast, and work
within the app's business hours.

Fortunately, your West Coast co-worker realizes the problem and updates the
existing test.

```ruby
# test/controllers/notifications_controller_test.rb
test "should create notification within business hours" do
  travel_to(Time.use_zone("America/New_York") { Time.parse("2022-11-19T06:00") })

  assert_difference("Notification.count") do
    post notifications_url, params: {...}
  end
end
```

## How do I know if my test suite is time dependent?

The previous example isn't dissimilar to when a test suite begins to fail after
a change in Daylight Savings. The root cause is that the tests are time
dependent, but the symptoms don't present themselves immediately. It's not until
you are working during an unusual time, or a new team member joins from another
time zone, that the symptoms appear. This is even more relevant when you're on a
globally distributed team.

One method to check if your test suite is time dependent is to temporarily
change your system clock to a time outside normal business hours, and then run
the test suite looking for any new failures. A better approach would be to
leverage Rails' existing [travel_to][] method and set an environment variable
before running the test suite.

```ruby
# test/test_helper.rb
class ActiveSupport::TestCase
  ...
  if ENV["TRAVEL_TO"]
    def setup
      travel_to Time.use_zone(ENV.fetch("TIME_ZONE", "America/New_York")) {
        Time.parse(ENV["TRAVEL_TO"])
      }
    end
  end
end
```

```
TRAVEL_TO=2022-11-18T05:00 rails test
```

The above example would run all tests at 5AM local time in the East Coast. This
approach can be improved by creating a binstub to run the test suite against
_each_ hour of the day.

```ruby
#!/usr/bin/env ruby
require "time"

failures = []

0.upto(24) do |i|
  hour = sprintf("%02d", i)
  date = "2022-11-19T#{hour}:00"

  puts "\n== Testing with #{date} =="

  unless system("TRAVEL_TO=#{date} rails test")
    failures << Time.parse(date).strftime("%I:%M%p %Z")
  end
end

if failures.any?
  puts "\n== The following times of day resulted in failures =="
  puts failures
end

```

Had we run this binstub in our example from earlier, we would have seen that our
test suite was time dependent between 6:00AM and 5:59PM local time EST.

```text
== The following times of day resulted in failures ==
12:00AM EST
01:00AM EST
02:00AM EST
03:00AM EST
04:00AM EST
05:00AM EST
06:00PM EST
07:00PM EST
08:00PM EST
09:00PM EST
10:00PM EST
11:00PM EST
12:00AM EST
```

I would encourage you to occasionally run this script against your test suite in
an effort to catch unexpected time dependencies before they prevent an early
morning deploy due to CI failures.

[travel_to]: https://api.rubyonrails.org/classes/ActiveSupport/Testing/TimeHelpers.html#method-i-travel_to