If you have followed my blog, you know that I am a proponent of automated testing. I like TDD and have a very strong opinion about code that has not been tested. However, there are situations, e.g. prototypes, where not writing any tests is the right thing to do.

Not to test is not an option

Wait. Didn’t I just write “there are situations where not writing any tests is the right thing to do.”? Yes, I did. My point is, that if you don’t write any tests, which can be perfectly sensible, that means you should do some manual tests.

Even if your code is not even meant to be executed, e.g. like example code on a blog, try to compile it and see if it does what you intended it to do. You might be surprised about bugs you find, but at least you’ll catch embarrassing typos.

When to test manually

But still, when I like automated testing so much, why would I settle for manual testing? It’s a matter of pragmatism and efficiency. Writing automated tests takes time. If you are not going to execute those tests too often, doing them manually might save time.

So, if you are writing a script to do something once, e.g. convert some data, then it might not pay off to also write a test for it. But don’t underestimate the number of times you are going to run that script. You might have forgotten something, so you have to run it again.

The more often we are going to run a program, the more likely it is that at least some automated tests pay off. We also have to take into account our ability to concentrate. If we have to do the same manual test over and over again, we are likely to make errors or do something different than the last time, leading to different test results.

In addition to the cost tradeoff, we also have to make some risk assessment in order to determine how thorough our tests should be. A local script sending the wrong file to the printer usually is a low risk. A shell script erasing your work of the last few hours not so much. A bug in a piece of software we ship that could cost a customer lots of money or even lives is an extremely high and intolerable risk.

Production software versus prototypes and toy programs

The rule of thumb for risk assessment usually is that software that will go into production should have a good test coverage. On the other hand, if we are just experimenting with some new language features or writing a small prototype to show in a demo, the risk is low. We may well decide to not write any tests for that kind of software.

Prototypes…

However, if we decide that a piece of software we write will be only a prototype without test coverage, we have to be clear to stick with that decision. Slapping a test suite onto the prototype after the fact is in almost all cases a bad decision:

Often we do not have a plan where we will end up with the prototype. We then probably don’t write tests, because they would have to change too often and slow down the process of producing some throwaway code quickly.

Our code base will then contain the remnants of many wrong things we tried. After all, if we don’t have tests, we also won’t refactor to clear those things out of our code base.

In this case, it’s cleaner to start from scratch, without all the deviations and with proper tests, than to try to polish throwaway code and bolt on a test suite that might or might not catch all the oddities we built in.

It is important to be clear about this. I have seen several teams using their prototypes as a building block for the later product. Using something that has no tests and no clearly factored structure as a starting point is a way to disaster.

If it looks too big to write from scratch, it will also look too big to clean it up properly and support it with an extensive test suite. Starting a project with that kind of code base means building on sand.

… or otherwise

Sometimes we do, in fact, have a plan and build a prove of concept in a clean, maintainable way. If the plan includes to eventually build on this code and ship the result into production, then we should write automated tests from the start. Delaying the test automation would mean that we waste the time needed to do manual tests.

Of course, there is a middle ground. What if management decides to not go ahead and evolve the prove of concept into something shippable? Investing in an extensive test suite that might not get used enough to pay off is a risk we might not want to take.

Still, if we plan to build on something later, we have to make it solid. Write sufficient tests without over-investing into something that has a chance to be discontinued. If you later decide to continue with that code, stuff out the test suite first. This will make it robust enough to serve as the foundation of the later product.

Don’t tempt yourself

If you are writing throwaway code, be it to just experiment with a new framework or as a quick prototype, then keep that code away from your production code. Put it into its own repository, and give that repository a name that makes its intent clear to not be used in production.

It is way too tempting for management and our future self to “just take what we already have” and continue. Don’t fool yourself – you don’t have anything to build on. A wooden model is not fit to be a cornerstone of your spaceship.