I’ve had enough of Rails testing, coming from other languages/frameworks that know the importance of testing, the more I test in Rails the more I miss ‘em. I don’t care how quick and easy I can make a web app in Rails, when it comes down to it, language, framework, whatever, it’s all about testing.
So I decided to try real testing in Rails. By real testing, I mean tests where each test, only tests 1 thing (class). The UserTest is the test for just the User model. The UsersControllerTest is the test for just the UsersController. You say yeah I know, that’s what I’m already doing. Not if you’re using Rails you aren’t. In Rails functional and unit tests are not real unit tests; a real unit test tests a single unit i.e. 1 class, all collaborators (other objects, database, etc) are mocked up. Forget the word functional, controllers and models are classes and you unit test classes. Outside Rails the word functional means more like what Rails calls integration tests, courser grained/high-level tests that test more end-to-end functionality that just a single unit (class).
In essence both functional and unit testing in rails are like mini-integration tests since you’re not just testing one controller, your testing that controller, the models/libraries in interacts with, the database adapters your model interacts with, etc, etc. This means that a bug in a model that a controller interacts with could cause your controller test to break. Or a bug in a Rails database adapter could cause your model tests to fail. Or even better, a change to one fixture for one model can cause a test to fail in another model that collaborates with that model. An example:
# A model
class User < ActiveRecord::Base
def foo
f = Foo.new
f.blah
end
end
# Foo is a library defined in 'lib'
class Foo
def blah
'blah'
end
end
User collaborates with Foo. In rails testing you mock nothing, so your tests for User are dependent on the actual implementation of Foo. Uh-oh there’s a bug in Foo#blah, that causes the Foo tests to break…but wait that’s not all that’s going to break, so does your User tests because User collaborates with Foo. If we had mocked out Foo in our User test then the only failing test would be in Foo’s tests. Too bad we didn’t write our User test to just test the User model and not indirectly test its collaborators (Foo).
instead of
def test_should_return_blah_when_sent_foo
user = User.new
assert_equal 'blah', user.foo
end
we should of did (using Mocha a mocking library for ruby)
def test_should_return_blah_when_sent_foo
foo = mock
foo.expects(:blah)
Foo.expects(:new).returns foo
user = User.new
user.foo
end
Foo is mocked out, we care nothing about its implementation, just its interface i.e. it responds to #blah. Therefore a change in Foo’s implementation does not effect our User test even though User collaborates with Foo.
This gets even more fun when your UsersController functional tests are failing because you changed a single attribute value in one of your User model fixtures and now your assertions in your functional tests are failing because your UsersController collaborates with your User model and loads in its fixtures.
You want to talk about brittle! Now that’s brittle baby!
A lot of Rails users are running into various bugs with Rails not cleaning up
your database tables on #teardown of your Test::Unit::TestCases leading to
cases were the test passes when I run it individually however when I run it with
'rake test:units’ it fails. Some of my fellow t-boters’ (myself NOT included)
solution:
Add the following class method (pseudo code) to all tests
def self.load_all_fixtures
fixtures :users, :articles, etc., etc.
end
Defined in test/test_helper.rb, it does what you think it does. Loads every
fixture for every model at the start of every test (not each test method because
we use transactional fixtures). That is pure trash! No more strange bugs but
any real tester would be disgusted. I just love sitting around watching tests
run all day, that’s a lot more fun that actually coding features.
Ok, then lets see what I really should be doing then…
What we want is each of our Test::Unit::TestCases to test 1 class and 1 class only. I don’t want it indirectly testing its collaborators. So I want to mock all objects that it collaborates with.
An example for testing the #search action in the UsersController
def test_should_find_all_users_whos_name_matches_the_given_search_phrase_on_GET_to_search
users = []
User.expects(:find).with(:all,
:conditions => ['name like ?', '%a%']).returns users
get :search, :q => 'a'
assert_equal users, assigns(:users)
assert_response :success
assert_template 'search'
end
class UsersController
def search
@users = User.find :all,
:conditions => ['name like ?', "%#{params[:q]}%"]
end
end
The above functional test uses mocha again.
This test is written in an interaction based style as opposed to a state based
style, all I care about is that the #search action sends #find to the User class
with the given parameters and assigns the return value it to an instance
variable named ‘users’. I’m not going to iterate through all the results and
run regexp’s against each of their names. According to the Rails doc, calling
#find on an ActiveRecord::Base model with those parameters will generate the
SQL that will find records
according to how I want to search (wildcard matching on the name column in the
users table). Rails has already tested #find, I don’t need to indirectly test
it in my UsersControllerTest. At the end of the test all my expectations set on
my mocks will be verified automatically by Mocha, if any expectation is not met
the test will fail.
This test can run without any tables in your test database (it cant run without a test database at all because somewhere when running a functional test Rails tries to connect to the test database even though this functional tests specifies it uses no fixtures, guess rails needs to initialize its test environment)
Here at t-bot my colleague and I have used this approach in all our functional tests in a recent application and we can run them all without any tables in our test database. This would be a huge benefit in terms of speed in any other framework that knew what real testing was about, however in Rails it’s necessary that Rails is initialized and that we have a database running every time you run a test; so its more of a I knew we could do it, but in any other framework our functional tests would absolutely smoke. No more watching tests run, now we can actually get some work done.
But wait…the test is exactly the same as the implementation. Thats no good.
This is true, the test is coupled to the implementation and the test is brittle, making refactoring basically not possible at all without breaking your tests. Also you could just write the implementation wrong, but I’d argue that when testing using mocks you still need more course-grained/high level end-to-end tests to catch instances like that.
What is implementation? Lets call implementation just the objects state i.e. instance variables and not how it interacts with other objects (behavior), then our tests are no longer coupled to an objects implementation because we only care about the messages its sends to other objects not their state. When testing like this the code results in a tell don’t ask (real OO) style of coding in which objects care very little about each other’s implementation i.e. state.
By mocking all collaborators, a bug in one class will only cause the test for that class to fail, it will not ripple out into any of its collaborators because all the tests for the collaborators have mocked it (the class with the failing test) out (like above, if we mocked Foo out in our User test, a Foo bug would of only broken the Foo test and not our User test as well). I argue that coupling a single test to its implementation better isolates your code from bugs in other parts of the code that it collaborates with. Not only that, someone else might be developing one of your object’s collaborators and you don’t even have access to its code. You don’t want to sit around and wait for them, so you agree on an interface w/ them for the object, and then you mock it out. Developers should never be waiting like that, it wastes time.
When first coming to Rails I never had seen fixtures used in a test environment, and thought (and still do) that they were ridiculous, not to mention the idea of a test environment and a test database. Come on, you mean everyone that runs these tests has to have MySQL installed with the right database and user accounts setup. Nice portability there Rails.
Agile development is about being agile; not about making sure you have MySQL installed correctly w/ the right user accounts, that you have the in-memory database setup correctly, or that your tests can contact some internal server running a fake credit card processing server, all that takes and wastes valuable time and slows your tests down big time, which results in more time lost.
In closing, I just want to say to, relax and stop drinking so much of the Rails Kool-aid. Everyone has just accepted Rails’ fixtures and test environment and the fact they need MySQL running to run tests. Real testers way before Rails did testing right, no fixtures, no database, no ‘mini’ web server, no test environment; they knew and know what it means to be agile. Their tests ran on any machine because they had no external dependencies and their tests ran fast, making their development process and software that much more agile.
Rails didn’t do testing right and as a result this so called ‘agile’ framework isn’t as agile as it really could be.
References:
The difference between state-based and interaction-based testing by Martin Fowler, an object master and huge influence on Rails.
Nat Pryce - expert interaction-based tester and creator of jMock (one of the most popular mocking library in Java) and Ruby/Mock an extension to Test::Unit ‘implementation is really just state - not how an objects interacts with other objects’
Mocha - mocking and stubbing library for ruby.