I have heard—and been in the middle of—arguments which make the claim that the tests is the stupid, simple part and the “real code” is the hard part. One such person made the claim that interns should write the test code and “real programmers” should write code to make the tests pass.
This is backwards.
The tests are the program. Without the tests, the program does not work. Tests are not something that should be left for the inexperienced; tests are the hard part.
To quote Code Complete:
Test cases are often as likely or more likely to contain errors than the code being testing (Weiland 1983, Jones 1986a, Johnson 1994 [Johnson, Mark. 1994. “Dr. Boris Beizer on Software Testing: An Interview”]). The reasons are easy to find—especially when the developer writes the test case. Test cases tend to be created on the fly rather than through a careful design and construction process. They are often viewed as one-time tests and are developed with the care commensurate with something to be thrown away.
For example, the following test does not have nearly enough setup to be useful
(assigns(:posts) could be empty, causing this test to be something of a
no-op). This is a common problem; sometimes it happens because it relies on
fixtures that get lost in a refactoring, and sometimes it happens out of just
not thinking hard enough. The non-test code behind this, of course, is part of
the Rails scaffold and dead simple.
context "a GET to index" do
setup { get :index }
should "have a link for each post" do
assigns(:posts).each do |post|
assert_select 'a[href=?]', post_path(post)
end
end
end
The next set of tests could use a loop over %w(index show edit) and
should_redirect_to to shorten it without any loss of knowledge (and could even
make it more clear). This is a case of an accurate series of tests (for a
one-line piece of code) that has become difficult to maintain because the
developer failed to notice the pattern.
context "logged out" do
setup { session[:user_id] = nil }
context "GET to index" do
setup { get :index }
should "redirect to the root_url" do
assert_redirected_to root_url
end
end
context "GET to show" do
setup { get :show }
should "redirect to the root_url" do
assert_redirected_to root_url
end
end
context "GET to edit" do
setup { get :edit }
should "redirect to the root_url" do
assert_redirected_to root_url
end
end
end
It’s interesting to note that the problem in the first example could be caused by refactoring, and the problem in the second example is caused by not refactoring. This is a fair point, and another indication of the difficulty in writing tests: when refactoring non-test code, anything you break is pointed out quickly by the tests; when refactoring test code, you need to make sure that it will continue to fail when needed, and continue to pass when needed.
Some tests are just so arcane or trivial that they would normally be overlooked, but if it really needs to be a specific way then it needs to be tested.
For example, ActiveRecord::Base#to_xml is a lovely part of Rails that is often
used for part of the application’s public API. If #to_xml changes, many
customers will be upset. Recently we had to override #to_xml with lots of
special-casing; while the special-casing did get tested, some of the more subtle
parts of #to_xml were ignored. Thus, this test was recently added in response
to a bug report:
should "dasherize any element with underscores when sent #to_xml" do
xml = Factory(:message).to_xml
assert_no_match /<\w*_\w*>/, xml
end
Some code needs a complex maze of tests. Advanced search is a common feature
that requires careful thought to test all the options. The
Album.advanced_search method may be large and complicated, but the tests for
it must be necessarily more so. No time for laziness or ignorance here; this
test suite requires the knowledge of an advanced search expert, and the patience
of a person who has a lot of patience.
context "when sent #advanced_search" do
setup { create_all_sorts_of_data }
%w(producer artist mixing_engineer).each do |str_opt|
context "with the #{str_opt} option" do
# ...
end
end
[true,false].each do |include_compilation|
context "with :include_compilations option set to #{include_compilation}" do
# ...
end
end
# ... and then permutations of these, too
end
(While researching this post I found these two interesting links: a WardsWiki page about bugs in tests, in which people argue over buggy tests, refactoring tests, and how this affects non-test code; and a paper titled Refactoring Test Code (PDF]), which discusses “code smells” relevant to testing and how best to refactor these.)
