Being a detective with data
“When you have eliminated the impossible, whatever remains, however improbable, must be the truth.”
— Sherlock Holmes in The Sign of Four, by Arthur Conan Doyle
Open up any major newspaper, flip to the health section, and you will find the latest medical study showing something is linked to something else. Wine and cancer. Coffee and anxiety. Smartphones and poor sleep. Pick a habit, pick a condition, and somewhere out there a scientist has found a link between them. Sometimes the headlines are even phrased as a question. Are eggs bad for your heart?
I want to know the answer. I love eggs. So I click on the article, nervously scanning for the conclusion. Instead, I find another study showing that people who eat more eggs have a higher rate of heart disease. Or a lower rate. It seems to depend on the week.
Someone once joked that whenever a headline is phrased as a question, the answer is almost always “no.” Are eggs bad for your heart? No. Does your morning coffee cause anxiety? No. It is a surprisingly useful rule of thumb.
Scientists develop a similar reflex. We see the words “linked to,” roll our eyes, mumble something about confounding, and get on with our day. Somewhere around the time we learn how to calculate a mean, another lesson gets drilled into us: association is not causation. It is one of the first things every statistics student learns, and rightly so.
Every morning, I sit in my office looking out at the large green elm outside my window. There is often a squirrel in it. Over the years, I have watched that squirrel happily munch away on whatever seeds or fruits a green elm produces. (I should probably know what they are called.) I have never once seen it eat a walnut. Based on my observations alone, I might conclude that squirrels prefer green elms to walnut trees. Until I realize that there isn’t a walnut tree anywhere outside my office. The squirrel was never given the choice.
In a nutshell, that is what we call an observational study. The squirrel is simply living its life. I am not telling it where to climb or what to eat. I am just sitting back, watching, and taking notes. Nature decides what the squirrel does; I simply observe.
Much of science works exactly this way. A few years back, a study reported that people who regularly skipped meals were more likely to develop heart problems later in life (Chen and Zhong 2024). Some friends were concerned for me because I was experimenting with intermittent fasting at the time. But stop and think for a moment about the people who skip meals. Busy people. Stressed people. People sleeping too little, exercising too little, grabbing fast food when they do find time to eat. Is it the fasting that leads to heart disease, or everything else that tends to come along for the ride? Just like the squirrel, we never saw what would have happened had those same people lived different lives.
That is what makes observational studies both fascinating and frustrating. When nature writes the story, things can appear related for all sorts of reasons. Sometimes the relationship is genuine. Sometimes it is little more than coincidence. From the data alone, it is surprisingly difficult to tell these possibilities apart.
Statisticians have a word for associations that are misleading: spurious. I always have to look that one up. The Cambridge Dictionary defines it as “false and not what it appears to be.” It makes the data sound almost mischievous, as though they are trying to trick us. And sometimes they do. There is even an entire website devoted to these deceptive relationships: Tyler Vigen’s Spurious Correlations.
One of my favorite examples comes from my home state of New Hampshire. There is a strong association between the annual number of burglaries and the number of babies named Johnny. I even had a friend named Johnny growing up. I can assure you he was not secretly masterminding statewide burglary rings. The association is real, but the explanation is nonsense. The phrase association is not causation exists for a reason.
But I wonder if we learned that lesson a little too well?
After enough contradictory nutrition studies and enough absurd examples like Johnny and burglaries, it becomes tempting to dismiss observational studies altogether. Another association? Next. That reaction is understandable. But it can make us timid. We become so careful not to overclaim that we sometimes stop short of the harder, more useful question: What would it take for this association to be causal?
Imagine Holmes arriving at the scene of a crime. He notices fingerprints on the murder weapon.
“Fingerprints don’t prove murder.”
That’s true. They don’t. The butler may be the murderer. Or perhaps he handled the weapon earlier that day. Or perhaps someone planted the evidence. But imagine if Holmes shrugged, closed his notebook, and went home. That would make for a very short detective novel. Sure, fingerprints are not proof. But they are evidence. Holmes’ job is not to dismiss the evidence because another explanation is possible. His job is to investigate them until only the most convincing explanation remains.
Science is much the same. The hard part is not dreaming up theories. That is easy. Whenever we observe an association, there is almost always multiple stories we could tell. Perhaps one variable causes the other. Or perhaps some third factor causes them both. The hard part is deciding whether that alternative explanation can withstand careful scrutiny. Like any good detective, we are not trying to find an explanation. We are trying to find the explanation that best accounts for the evidence after reasonable alternatives have been investigated.
Few stories illustrate this better than the debate over smoking and lung cancer. Today, it is difficult to imagine the question ever being controversial. Cigarette packages carry warning labels. Physicians routinely advise patients to quit. Yet in the early 1950s, none of that was settled. Scientists debated whether smoking truly caused lung cancer. One pathologist observed the nicotine stains on the fingers of many cigarette smokers, remarking that cancers of the fingers were essentially unheard of (Hueper 1956). If tobacco tar really caused cancer, why did it seem so devastating to the lungs but not to the skin on the hands?
To help settle the debate, the American Cancer Society turned to two statistically minded epidemiologists: E. Cuyler Hammond, its director of statistical research and assistant director Daniel Horn. With the help of roughly 20,000 volunteers, they enrolled 187,783 men between the ages of 50 and 69. At enrollment, each man completed a questionnaire about his smoking habits. Then the researchers did something wonderfully simple: they waited. For nearly four years, volunteers tracked who died and, whenever possible, what they died from. By the end of the study, 11,870 men had died. They compared those deaths across smoking groups (Hammond and Horn 1958).
Differences in mortality were striking. Overall, cigarette smokers experienced 397 lung cancer deaths, compared with only 37 that would have been expected had they experienced the same lung cancer rate as men who never smoked. This corresponds to a relative risk of 10.7. Even more striking was what happened as cigarette consumption increased. The more people smoked, the greater their risk became. Men who had never smoked experienced about 3.4 lung cancer deaths per 100,000 person-years. Among men smoking two or more packs of cigarettes each day, the rate climbed to 217.3—roughly 64 times higher. It was a remarkably strong association, and it became stronger with increasing exposure. If I told you today that a new medication increased your risk of cancer ninefold—let alone sixtyfold—I suspect you would stop taking it before you finished reading this chapter (Hammond and Horn 1958).
But this was still an observational study. One prominent skeptic was the statistician Ronald Fisher, so influential that he was eventually knighted. He did not dispute the data. Instead, he questioned whether the observations alone were enough to conclude that smoking caused lung cancer (Fisher 1957; Fisher 1958). He reasoned:
” any statistical association, observed without the precautions of a definite experiment, always allows-namely, (1) that the supposed effect is really the cause, or in this case that incipient cancer, or a pre-cancerous condition with chronic inflammation, is a factor in inducing the smoking of cigarettes, or (2) that cigarette-smoking and lung cancer, though not mutually causative, are both influenced by a common cause, in this case the individual genotype.” (Fisher 1957)
Let’s unpack this. We’ll use a graph. Later, we’ll make these graphs precise; for now, think of them as sketches of two competing explanations. Fisher argued that more than one explanation was consistent with the data. One is the explanation we now accept: smoking causes lung cancer. This is shown on the left. The arrow runs from Smoking to Cancer. If smoking causes cancer, then people who smoke more should develop lung cancer more often, exactly as Hammond and Horn observed.
The second explanation is is that both smoking and lung cancer might be consequences of some third factor. Because of his interest in genetics, he suggested genotype as a candidate. Some people might inherit a genotype that makes smoking more enjoyable, more addictive, or easier to tolerate. If those same genes also increase the risk of lung cancer, then smokers would still develop lung cancer more often than non-smokers, even if smoking itself played no causal role. This alternative is shown on the right, where Genotype points to both Smoking and Cancer.
In the second explanation, genotype is a confounder: a common cause of both the exposure (smoking) and the outcome (lung cancer). (Confounders are the villains of this textbook, forever threatening our ability to draw causal conclusions from the associations we observe.) The mere possibility of confounding was enough to cast doubt on the claim that smoking causes lung cancer. Perhaps smokers were older. Perhaps they worked in occupations with hazardous exposures. Perhaps they lived in more polluted cities. Fortunately, possibilities like these could be investigated. Researchers repeatedly compared smokers and non-smokers within the same age group, occupation, or region, yet the association persisted (Doll and Hill 1956; Doll and Hill 1964). One by one, the measured confounders were ruled out.
Genotype, however, was an entirely different beast. It was unmeasured. An unmeasured confounder is an invisible explanation, lurking in your data, forever capable of casting doubt on an observed association. Fisher understood this perfectly. For him, the mere possibility of an unmeasured confounder was enough to remain skeptical. Because genotype could not be measured, he argued that the observational evidence alone could never distinguish between the two competing explanations.
If both explanations predict exactly the same association, how can we ever decide between them? This question is not historical. Open almost any modern observational study. Somewhere near the end, you will find an acknowledgment that unmeasured confounding cannot be ruled out, followed by a call for future studies. It is practically a ritual.
In 1959, Jerome Cornfield and several of his colleagues had a clever idea (Cornfield et al. 2009). Rather than arguing that Fisher’s explanation was impossible, he asked what the world would have to look like if Fisher were right. Suppose the entire association between smoking and lung cancer were explained by an unmeasured genotype. What would necessarily have to be true?
The argument that follows is one of my favorites because, once you draw the right picture, the answer is almost impossible to miss. Let \(R_S\) and \(R_N\) be the risk of lung cancer among smokers and non-smokers. We begin by plotting the point \((R_N,R_S)\). The horizontal coordinate is the risk among non-smokers; the vertical coordinate is the risk among smokers.
The rectangle helps us locate the point. Because smokers have a higher risk of lung cancer than non-smokers, the rectangle is taller than it is wide. Now connect the origin to the point we just plotted.
The slope of this line is \(R_S/R_N,\) which is exactly the relative risk. Steeper lines correspond to larger relative risks. Thus, instead of thinking of the relative risk as a number, we can think of it as the steepness of a line.
Now suppose Fisher’s explanation is correct. Smoking itself has no effect on lung cancer. Instead, everyone faces the same background risk of lung cancer, which we denote by \(B\). Carrying the genotype adds an additional risk, which we denote by \(\Delta\). Suppose a fraction \(w_S\) of smokers carry the genotype. Then the lung cancer risk among smokers is simply the common background risk plus the additional risk contributed by the genotype: \[R_S = B + w_S\Delta.\] Likewise, if a fraction \(w_N\) of non-smokers carry the genotype, \[R_N = B + w_N\Delta.\]
Imagine removing the common background risk \(B\) from everyone. Graphically, this shifts our point left by \(B\) units and downward by \(B\) units, leaving the point \[(w_N\Delta,\;w_S\Delta).\]
We add this new point to the graph.
The two points differ only by the background risk. Moving from one point to the other means subtracting exactly the same quantity, \(B\), from both coordinates. Thus, the two points lie at opposite corners of a square of side length \(B\).
Connect the origin to this second point.
The slope of the original line is \[\frac{R_S}{R_N} = \frac{B+w_S\Delta}{B+w_N\Delta},\] while the slope of the new line is \[\frac{w_S\Delta}{w_N\Delta} = \frac{w_S}{w_N}.\] The picture now tells the rest of the story. Removing the common background risk moves the point left and downward by the same amount. This makes the line from the origin steeper. Therefore, \[ \frac{w_S}{w_N} > \frac{R_S}{R_N}.\] This is Cornfield’s inequality.
Its interpretation is striking. If smokers have 11 times the lung cancer risk of non-smokers, then Fisher’s hypothetical genotype must be more than 11 times as common among smokers. If heavy smokers have 64 times the risk of never smokers, then the genotype must be more than 64 times as common among heavy smokers. Cornfield summarized the result this way:
“the excess lung-cancer risk among cigarette smokers is so great that the results can not be interpreted as arising from an indirect association … since this hypothetical agent would have to be at least as strongly associated with lung cancer as cigarette use; no such agent has been found or suggested.”
Cornfield’s argument did not prove that smoking causes lung cancer. Cornfield accepted Fisher’s alternative explanation (at least temporarily) and followed it to its logical conclusion. The alternative was certainly possible, but only if one accepted an extraordinarily implausible hidden genotype. The burden had shifted. It was no longer enough to say that another explanation could exist. That explanation now had to bear the weight of its own consequences.
That is the broader lesson. Faced with competing explanations, we need not shrug our shoulders and conclude that “association is not causation.” Instead, we write down each explanation as clearly as we can and follow it wherever logic leads. Once we do, our assumptions become tools for reasoning - objects we can manipulate, question, and follow to their logical conclusions. A good detective does not wait for a confession. He keeps asking questions until one story holds together and the others fall apart.