honest.knave

From the this-should-be-obvious-by-now dept.: I recently added an integration test suite to our codebase, which gives us the ability to write some tests that were impossible when we only had unit and system test suites. Once it was stable, I added it to our nightly build process, so it would be tested every night.

I updated our Linux build machine with the new nightly build script. At the same time, I was configuring our new, faster Windows build machine to replace our old, pokey one. Since the old Windows machine is expected to be rendered obsolete, I didn’t bother installing the new build script on it. So Windows just continued running the normal build, unit tests, and system tests. We knew the integration tests worked on Windows anyway, because we ran them by hand just recently (right?).

This Sunday, about a week later, the new Windows machine came online. I installed all the prerequisites (which only took a few hours, because we have an awesome wiki full of HOWTOs), connected it to our network, and installed the updated build script. By now, you know what happened next: I ran the build script, and everything worked except the integration tests. The code checked out, compiled, unit tested, and system tested just fine. But the one thing that wasn’t tested nightly — the integration test suite on Windows — was broken.

Our strategy of automating everything continues to pay dividends. And it shows what happens when you don’t. If you aren’t able to regularly test your software in some kind of automated fashion, it is broken. You just don’t know it yet.

Title is a quote attributed to Bruce Eckel.