When the Code Is Too Obvious to Check

A bat and a ball cost $1.10. The bat costs a dollar more than the ball. How much do the bat and the ball cost individually?

Did you immediately think that the bat costs $1 and the ball costs $0.10? Nope. Then the difference would be only ninety cents. The correct answer is the bat costs $1.05 and the ball costs $0.05.

Every time I hear this riddle, the wrong answer reflexively comes to mind. When I first heard it, I was irritated when the wrongness of my answer was proved to me.

Why was I so committed to the wrong answer? I didn't check. It seemed too obvious to check.

But we're software engineers, and our work goes way beyond story problems with bats and balls. We’re trained to think with rigor, so we never make obvious mistakes. Right?

Consider a prehistoric example scrawled on a cave wall:

#include <stdio.h>
#include <malloc.h>
int main (int argc, char[] argv) {
            char * hello = malloc (80);
            strcpy (hello, "Hello, World");
            char * HELLO = Capitalize (hello);
            printf ("%s\n", HELLO);
            free(hello);
}

This raises the question, what is Capitalize()?

Imagine for a moment that you get to write this function—and test its correctness. Maybe it capitalizes each letter in place and returns the input pointer. Or maybe it preserves the input string, allocates new memory, capitalizes, then returns a pointer to the new string.

What if Capitalize() has an error that raises an exception? Will free() still get called? These are obvious things to check—after they've burned you a few times. The trouble comes in the obviously "right" things that need not be checked. (Did your Capitalize() take into account Turkish character encoding?)

How many times do we know something but it seems too obvious to check? Most of the time this normal human response is a handy shortcut. We can't go about verifying every instance of "red light means stop" and "green light means go." Your brain tries to save you time—but you can’t always trust it.

When you code up a software solution, you've invested part of yourself in the code. Your mind has drawn dozens or hundreds of conclusions about your code that are all "too obvious to check." If your code malfunctions, each of those "too obvious to check" thoughts will bias your thinking about what caused the malfunction. If the underlying flaw in the code is something "too obvious to check," you're going to take longer to correct that flaw.

This is because you have two flaws: the flaw in your code and the flaw in your mind.

As a result, we have to commit up front, before our thinking crystalizes, that the code will have to prove to us that it is correct.

This isn't particularly straightforward. Any test of the code never demonstrates it is correct, merely that it isn't wrong, given the circumstances of the test. Nevertheless, passing tests manage to constrain the software to failures where we're not looking. And the last places we'll look are the “obviously” correct functions.

It would be helpful to have something more powerful than our capacity to dream up unit tests.

The first codes I ever learned to debug were mathematics proofs. I noticed after making the exact same mistake several times in completely different contexts that I frequently failed to consider the case where I might divide by zero. And I frequently lost points for that reason. This taught me to always check for this problem.

Likewise, as a young programmer, I learned a few common failure modes, like off-by-one errors in looping structures. I had to make sure every malloc() had a free(). I distinctly recall several memory leaks because of something "too obvious to check."

I was thus grateful when I started using languages with built-in garbage collection, such as Java and C#. Years later I came to appreciate the way LINQ expressions could replace explicit loops.

Sure, my code became more expressive in each of these languages, but the greater benefit was how the structure of the language precludes obvious errors. The boffins who devised LINQ handled all the pesky starting and ending index conditions that always tripped me up.

When something is "too obvious to check," maybe the compiler should do the checking.

This is why I rely heavily upon third-party libraries. I figure that if Bill Gates distributes a library with a Capitalize() function in it, enough people ahead of me will have flushed out Turkish “gotchas.”

Nevertheless, there's still a chance that I'm the special snowflake who programs something the library has never seen before. And I might win the lottery. That said, Sherlock Holmes reminds us that the truth may be the quite improbable remainder after we've looked everywhere else.

We've long had compilers that save us the embarrassment of putting a floating-point value into an integer variable. But nowadays, languages like TypeScript go light years beyond Modula or Pascal's strong typing. It notices if you initialize a variable with a certain type, then yells at you if subsequent usage is inconsistent with this. The benefit is increased by the fact that such usage errors are "too obvious to check."

This tells us something about compilers and human reasoning: We are prone to false confidence in our code's goodness. Each thing the compiler does to catch "obvious" errors has an amplified benefit.

Then there are combinator libraries, like Haskell's QuickCheck or F#'s FsCheck. Combinator libraries promise a lot because they automatically generate thousands of "too obvious to check" test cases. This is nice, but the thing that has me excited is the second step, where they thoughtfully reduce the failure to its simplest form. It's one thing to see that the code fails somewhere in a twisted thicket of data. It's much more helpful to have all the brush pruned away to see the simplest way to evoke the failure.

Keep this in mind as you curate your test suites.

When we don't have a combinator library to generate test cases for us, we should emulate the process. We should manually construct test case generators in the same way, then reduce them manually. The process of thinking “combinatorically” about test inputs when you're writing a test case generator uses a different part of the brain than the part that wrote the code.

This switching between thinking with different of parts of the brain may be a helpful way to escape "too obvious to check" pitfalls. Your code looks different at different levels of abstraction, and you think about it differently, too.

So the next time you test your code, ask yourself the really obvious questions about its correctness. Ask it the impossible. Think differently. Think of your code like one of those annoying questions: Where was the Declaration of Independence signed?

At the bottom.