In my last post I have written about the main and secondary uses of unit tests and how they compare to other kinds of automated tests. This time I will explore what implications that has on how unit tests and the code under test should be written, how unit tests should be used and what should and can be unit tested.

Granularity

Ever had hours to debug some code because the cause of an error had been buried somewhere deep under several layers of abstraction? You don’t want that with unit tests.

As I had written in the post about unit test purposes, unit tests fail frequently, especially when using test driven development. Ideally, each programming error will result in only very few (preferably one) test to fail, and it will be obvious in which part of your code the error originates.

The implication is that each unit should be tested separately from the rest of the code. Dependencies should be replaced by mock or dummy classes.

To be able to do this, the code under test has to be very modular, so a clean design and architecture of the code under test is a prerequisite for “unit-testability”.

You don’t want to have an error in some utility class cause hundreds of tests to fail, because hundreds of units have some indirect dependency of that class that you can’t mock away in the tests. You don’t want to debug through tons of code because the error is buried somewhere deep in the guts of your dependency graph.

Modularize your source code, so you can test your units in isolation.

Speed

Maybe even more important than granularity is execution speed. Unit tests should be run often. Very often. Extremely often, if you use TDD. I have done TDD sessions where I ran the whole test suite every 2-4 minutes.

Unit tests should execute in a matter of seconds.

Obviously, this impossible if your unit tests themselves take that long, so a must-have for unit tests is speed. I’d say a hundred or more tests per second is the bare minimum, because you will have lots of tests.

Speed through granularity

I have covered granularity already, but it also contributes to speed. If you have to build up an environment of hundreds of objects for a class to be tested, the tests are not going to be fast enough.

Stay away from I/O

Don’t use I/O in your unit tests, because it tends to be slow. This includes console output, writing to files or databases, and network connections.

Of course you should build tests that do such things, and you can even use your unit test framework of choice to implement them, but keep them out of your overall unit test suite.

Put I/O tests away in a special suite that does not have to be run so often. Usually the I/O code does not need to be changed once it is set up properly, so there is no need to always run it during development of business logic.

Of course I/O should be factored into a separate module to achieve that. During the unit tests you then can use dummy database interfaces, stringstreams and other things to mock away dependencies on I/O facilities.

Don’t use big data volumes in unit tests

Unit tests should confirm that the logic is right, not that the performance is good. A handful of data points should be enough to prove correctness. Use stress tests to prove that the program is fast enough if you have thousands or millions of data points.

Code coverage

Some people say that you should strive for 100% code coverage, i.e. every line of your code should be executed during the unit tests. This is blind perfectionism and I don’t fully agree.

That does not mean that code coverage is not important. You should get as high a coverage as is possible and sensible. However, with some code you have to go to extreme lengths to bring them under test.

In some cases you have exception handling in places where it’s hard or close to impossible to simulate the exception, for example at the boundary to third party libraries.

Sometimes you have pure data structures. It would not be worth the time to write and run tests for a simple setter that does nothing but setting the corresponding member variable.

However, not unit testing a piece of code should be a conscious decision which should not be taken lightly. Such code should be extremely simple. Set the bar very low at which you tolerate untested code.

Strive for high code coverage, but don’t be dogmatic about reaching 100%.

Maintainability

Test code is code. And like all code, test code should be maintainable, i.e. readable, understandable and easily changeable.

If a piece of code under test changes, tests have to be adjusted. You’d better make sure those adjustments are easy to do. A rotten test code base leads to a sloppy attitude towards the tests, which leads to missing tests and unnecessary risks.

I sometimes hear that test code is not as important as production code, because it won’t be shipped to the customer. I disagree. Tests complement the product as they make sure it does what it’s supposed to do. So Test and production code are of equal importance.

In fact, the argument that the test code won’t be shipped is phony. If you write a library, you better ship your tests with it, especially if it’s open source. They might give good hints if the library does not work in a different environment.

Even for enterprise software you might one day decide to ship your source code, e.g. if you sell it. If the test code is not maintainable, the whole software is less maintainable and you might have to support the new maintainers longer that you would like to.

A last argument for maintainability of the test code is the documentation of the code under test. If your test cases are easily read and understood, a reader can see how to use the class under test and how it will behave.

Treat test code like production code, it is of equal importance.

Conclusion

Unit tests should be modular, fast, readable and covering most, but not all of your code. To get an impression of what unit tests should not be like, read my earlier post “One Hour of Unit Tests?”