Overmocking makes tests brittle
Let’s look at the controller action from the last post:
And the proposed test:
So what happens if we decide that the call to
find should really be a
searchbyname method on the
User model? We refactor, the controller action becomes:
And our tests break. Note that the functionality is the same, and normal tests would have passed to prove it. This means that every time we refactor, we have to change this and every other functional test on
search, easily five times the work. This would also happen if we added a call to
User.count (for whatever reason). That is seriously unagile.
By now you should have noticed that the test name above is misleading…
Overmocking gives a false sense of security
The test above is named
testshouldfindalluserswhosnamematchesthegivensearchphraseonGETtosearch, but that’s not actually what it’s testing. A more accurate name would be
testshouldcallUserfindwithallandconditionssettonamelikeparamsqonGETto_search. This isn’t just about accurate naming, but about your expectations on what a test means. If I refactor the search method again:
The test would fail, and the fixed test would be:
Now the tests pass, the test does what it espouses to do, and your application is broken (did you catch the bug? ‘
==’ isn’t valid SQL). This is they key: Effective tests only care about the user-visible behavior of an action, regardless of how it goes about doing it. This is the essence of modern computer science, and what all that talk in college about black boxes, objects, APIs and contracts was trying to show you. As a coworker put it: our tests shouldn’t care whether our method calls
find, or produces a fairy princess to do the work for us. Our tests should only care that the work gets done.
Interaction testing is TDD, but backwards
The above tests are examples of what the kids in the know call Interaction Testing.
The problem with overdoing this method of testing is that it encourages you to make assumptions about how your method will work before you write it. The whole point of TDD is to force you to take a step back, pretend you’re a client of your own API, and write only the code necessary for that client. Writing a test that asserts that
find is called before you write the method encourages you to use
find instead of
findbyid, which is not something the client cares about. This is not the purpose of TDD.
So you’re hatin' on Interaction Testing?
Absolutely not. Interaction testing is necessary if you don’t have full control of or access to a module your code works with. If you work in a large shop, it’s completely legit to test the interactions between your team’s module and the modules of the team down the hall. It’s the only practical way to keep coding without waiting on them.
Likewise, if you’re application works against Amazon’s API, or does some intense filesystem access, you had better mock that stuff out for your tests. Nobody’s arguing against these points.
So are there problems with rails testing?
Rails testing is worlds easier than in any other framework I’ve dealt with, but of course there are problems.
rake can be dog slow on a large application. Using in memory sqlite3 databases roughly halves your test times. Also, the rspec folk have a really good idea with their spec server, which loads the rails environment and serves access to tests through DRB. These are much better solutions than sacrificing developer time through brittle and ineffective testing.
Rails tests tend to be very wet. We’ve been working on some testing helpers that are addressing that. I’m not quite ready to release it (soon, soon), but here’s a quick peek:
We’re hoping to do the same thing to controller tests fairly soon, and we promise to let you (our loyal fans) know when we do.