Why probability is not enough
Suppose we want to understand the relationship between coffee and anxiety. Focus on one person, and let \(X\) represent the number of cups of coffee a person drinks each day. Let \(Y\) be that person’s score on an anxiety scale, with higher values meaning more severe symptoms. We collect data on coffee consumption and anxiety from several people, and then fit a regression model:
\[Y = \alpha + \beta X + \varepsilon.\]
Here, \(\alpha\) is the intercept, \(\beta\) is the slope, and \(\varepsilon\) is an error with mean zero within each level of \(X\).
Optional technical note: Where are the person subscripts?
Suppose our sample contains \(n\) people. For person \(j\), let \((X_j,Y_j)\) denote the pair of random variables. Once that person is observed, the realized values are written in lowercase as \((x_j,y_j)\). Thus, the observed dataset is
\[ (x_1,y_1),\ldots,(x_n,y_n). \]
Under the usual independent and identically distributed—or i.i.d.—sampling model, the random pairs
\[ (X_1,Y_1),\ldots,(X_n,Y_n) \]
are independent copies of the same pair of random variables \((X,Y)\).
Because the same model applies to each person, we usually write \(X\) and \(Y\) when discussing the generic probability model and reserve \(X_j\) and \(Y_j\) for occasions when a person-specific random variable matters. Lowercase \(x_j\) and \(y_j\) always refer to the values that were actually observed for person \(j\). This lets us drop unnecessary subscripts from the probability model without confusing random variables with their realizations.
The simplification is not appropriate when observations do not follow a common distribution, such as when a single patient contributes multiple observations, patients clustered within hospitals, or people connected through households or social networks. In those settings, the indices and the dependence structure must be represented explicitly.
How should we interpret \(\beta\)?
The coefficient \(\beta\) is the expected difference in anxiety score associated with drinking one additional cup of coffee per day. If \(\beta=2\), then people who drink one additional cup of coffee per day have an expected anxiety score that is two points higher.
A regression model is useful. It summarizes the pattern in our data. It can help us predict anxiety scores for people who drink different amounts of coffee. But notice the care in the wording: an additional cup of coffee is associated with a \(\beta\) difference in average anxiety. We did not say that drinking another cup causes anxiety to increase by two points.
Why are we so cautious? For one thing, we could fit another regression with the variables reversed:
\[X = \alpha' + \beta'Y + \varepsilon'.\]
Now anxiety is being used to predict coffee consumption. Here, \(\alpha'\) is the intercept, \(\beta'\) is the slope, and \(\varepsilon'\) has mean zero within each level of \(Y\).
Suppose \(\alpha'=1\) and \(\beta'=0.25\). How should we interpret \(\beta'\)?
The slope \(\beta'=0.25\) says that a one-point higher anxiety score is associated with an expected difference of 0.25 additional cups of coffee per day. It does not say that increasing anxiety by one point would cause someone to drink 0.25 additional cups.
Both regressions can describe the association. They may also invite opposite causal stories. When coffee is regressed onto anxiety, you can imagine coffee increasing a person’s anxiety. When anxiety is regressed onto coffee, you imagine people drinking more coffee because anxiety interferes with sleep and leaves them tired the next day. The data do not label one variable as the cause and the other as the effect.
This is not merely a quirk of linear regression. The following general fact states the idea more precisely.
A general fact. For any joint distribution of \(X\) and \(Y\), we can draw a sample from \((X,Y)\) in two ways:
- Draw \(X\) first. Independently draw a uniform random number \(U\), and then use \(X\) and \(U\) to specify \(Y\).
- Draw \(Y\) first. Independently draw a uniform random number \(U\), and then use \(Y\) and \(U\) to specify \(X\).
From the data on \(X\) and \(Y\) alone, we cannot tell whether the data came from the first sampling strategy or the second. We therefore need information beyond the data to determine which causal direction, if either, is correct.
Optional: Why does the uniform random number work?
For real-valued \(X\) and \(Y\), we can choose a rule that assigns a conditional distribution of \(Y\) to each value of \(X\). After drawing \(X\), use the conditional distribution assigned to the value that was drawn. A uniform random number \(U\) chooses a percentile from that distribution. For example, \(U=0.25\) selects its 25th percentile and \(U=0.90\) selects its 90th percentile.
Repeating this procedure, first drawing \(X\), then drawing an independent \(U\), and finally selecting the corresponding conditional percentile of \(Y\), reproduces the joint distribution of \((X,Y)\).
We can reverse the procedure: draw \(Y\), independently draw \(U\), and use it to select a percentile from the distribution of \(X\) conditional on that value of \(Y\). This reproduces the same joint distribution in the opposite order.
The picture below summarizes the problem as a snake eating its tail. A probability model can describe \(Y\) using \(X\), and it can describe \(X\) using \(Y\). If we interpreted both descriptions causally, we would conclude that \(X\) causes \(Y\) and that \(Y\) causes \(X\). The circular picture is not meant to propose a plausible causal mechanism. It shows why the probability model alone cannot tell us which causal arrow, if either, is appropriate.
Main point
The joint distribution of the observed variables does not, by itself, determine how those variables are causally related.
At first, this ambiguity may not seem especially troubling. Perhaps the data cannot tell us whether \(X\) causes \(Y\) or \(Y\) causes \(X\), but we might still conclude that one must cause the other.
We have already seen why that conclusion does not follow. In the smoking and lung cancer example, a confounder could, in principle, explain the entire association. The association alone could not distinguish among three possibilities: smoking causes lung cancer, lung cancer causes smoking, or a third variable causes both.
The following case studies show how consequential this problem can be. In the first, researchers identified a confounder that provided a compelling explanation for an apparent association. In the second, accounting for a confounder did more than explain the association: it reversed its direction.
Night lights and childhood myopia
Example 1
In May 1999, Nature published a short report by Graham Quinn and colleagues on night-time lighting and childhood myopia (Quinn et al. 1999). The researchers asked the parents of children seen in a pediatric ophthalmology clinic about the lighting conditions in which their children had slept during their first two years of life. They reported a strong association: children exposed to more ambient light at night were more likely to develop myopia.
The study included 479 children. The reported prevalence of myopia was lowest among children who slept in darkness, higher among children who slept with a night light, and highest among children who slept with room lighting. The pattern suggested that greater exposure to light at night was associated with a greater probability of myopia.
The result attracted attention because it suggested a simple environmental cause of myopia. Let \(X\) indicate the nighttime lighting condition and let \(Y\) indicate whether the child develops myopia. It is easy to tell a causal story in which exposure to light at night affects the development of the eye.
What do you think might confound the relationship between night-light use and childhood myopia? Why?
Parental myopia could confound the relationship. Parents with myopia may be more likely to use a night-light in their child’s room, and their children may inherit a greater risk of developing myopia. Parental myopia could therefore help explain both the exposure and the outcome.
In March 2000, however, Nature published a response from Jane Gwiazda and colleagues (Gwiazda et al. 2000). They were unable to confirm the original association. They also found that myopic parents were more likely to use night-time lighting for their children, providing evidence for the confounding explanation above.
Parental myopia could be a common cause of the exposure and outcome. As we saw in Data Detective, this is what we call a confounder. Confounding can create an association even when the exposure does not cause the outcome.
Sometimes a common cause does something even more surprising: it can reverse the apparent direction of an association.
Kidney-stone treatments
Example 2
Our second case study begins in a London hospital during a period when the treatment of kidney stones was changing. From 1972 through 1980, patients in the study were treated with open surgery. From 1980 through 1985, physicians increasingly used a less invasive procedure called percutaneous nephrolithotomy. We will call it the percutaneous procedure. In 1986, C. R. Charig and colleagues published a historical comparison of these and other kidney-stone treatments (Charig et al. 1986).
In the overall comparison, the percutaneous procedure had a higher observed success rate than open surgery.
Eight years later, Steven Julious and Mark Mullee revisited these data (Julious and Mullee 1994). They pointed out that kidney stones vary in size and that stone size was related both to treatment and to the chance of success. When they compared patients with similarly sized stones, the pattern reversed.
Within both the small-stone and large-stone groups, open surgery had a slightly higher observed success rate. The direction of the association was therefore the opposite of the overall comparison.
Open surgery was used much more often for difficult, large stones, while the percutaneous procedure was used much more often for easier, small stones. Combining the two stone sizes therefore compares treatment groups with very different mixtures of patients. Once we compare patients with similarly sized stones, open surgery has the higher observed success rate in both groups.
Based on these results, which treatment would you recommend?
If successful stone removal were our only consideration, the comparisons within kidney-stone size would favor open surgery. These comparisons are more informative than the overall comparison because stone size influenced treatment choice and was also related to treatment success.
A clinical recommendation would require more than these success rates. The percutaneous procedure is less invasive, and clinicians would also consider complications, recovery time, cost, other characteristics of the stone, and the patient’s preferences. Moreover, these data came from a historical comparison rather than a randomized trial.
Simpson’s paradox
We have already seen that an association or regression can invite a causal interpretation that the data alone cannot support. It can mislead us in another way: even the direction of an association can depend on how we group the data. Simpson’s paradox occurs when the direction of an association in the combined data is opposite to its direction within groups defined by a third variable. In the kidney-stone example, the overall comparison favored the percutaneous procedure, while the comparisons within both stone sizes favored open surgery.
Example 3: Graduate admissions at Berkeley
A classic example comes from graduate admissions at the University of California, Berkeley. In 1973, women were less likely than men to be admitted when all departments were combined. Viewed one way, the association appeared to show an admissions process that favored men. Once department was taken into account, however, the association went slightly in the other direction: it appeared to favor women. Women were more likely to apply to departments with lower admission rates for everyone (Bickel et al. 1975).
The third variable—department—therefore matters greatly. Yet this example also warns us not to treat stratification as an automatic causal solution. Department choice may itself reflect earlier social and educational forces, so the departmental comparisons answer a narrower question than the overall comparison.
Simpson’s paradox is not limited to binary exposures and outcomes. The same reversal can occur when \(X\) and \(Y\) are continuous. In the picture below, color indicates membership in a third group. Within every color, the points move downward as \(X\) increases, showing a negative association between \(X\) and \(Y\). If we ignore the colors and combine all the points, the dark arrow moves upward, showing a positive association.
Something more than statistics
Statistical inference uses observed data to learn about a probability model. From data, we can infer means and probabilities. We can also infer measures of association, such as risk differences, relative risks, odds ratios, and correlations. These quantities tell us how variables are distributed and how they are related.
Causal inference aims at a different target. We want to answer causal questions, such as: How much might an exposure cause an outcome to occur or change? A probability model alone cannot answer this question. It can tell us how the exposure and outcome occur together, but it cannot tell us how the outcome would change if the exposure changed.
To answer causal questions, we need to say more than what is contained in the joint distribution of the variables. One important part of this “more” is directionality. We specify which variables can cause which other variables. A causal model makes this directionality explicit by describing each variable in terms of its direct causes. Because it adds causal structure to a probability model, a causal model subsumes a probability model.
The bottom of the figure represents statistical inference. We use observed data to learn about a probability model. The top represents causal inference. We use data to learn about a causal model. This lets us infer quantities such as how much an exposure causes an outcome to change.
How do we add this directionality? What, more precisely, is a causal model?