The Econometric Argument

Association and intervention

A firm offers optional training. Employees who take more of it tend to produce more. A manager asks whether another training hour would pay for itself.

The scatterplot answers a question about employees who chose different amounts of training. The manager asks what would happen if the firm changed an employee’s training. Those questions can have different answers, including different signs.

Start by writing the target in a sentence: “The average change in next month’s output if each eligible employee received one additional hour of this training, with staffing and equipment held fixed.” The intervention, population, outcome, and horizon belong in that sentence. “The effect of training” leaves all four open.

An estimand is the mathematical quantity corresponding to that target. An estimator is a rule for calculating a number from observations. An estimate is the number the rule produces this time. A spreadsheet can return an estimate before anybody has agreed on an estimand. That is an efficient way to answer the wrong question.

Observationally equivalent models

A quick notation bridge

$E[Y]$ means the population average of $Y$. The bar in $E[Y\mid D=d]$ restricts that average to observations with $D=d$. Variance, $\operatorname{Var}(Y)$, averages the squared distance from the mean, so it measures spread in squared units. Covariance, $\operatorname{Cov}(D,Y)$, averages the product of the two variables’ deviations from their means; its sign describes linear co-movement.

Two variables are independent when learning the value of one leaves the other’s distribution unchanged. A standard normal variable has a bell-shaped distribution with mean zero and variance one. These ingredients keep the following thought experiment easy to calculate. Chapter 2 develops the averaging rules.

Here is a deliberately stripped-down, synthetic economy. Let $U$ denote an unobserved employee characteristic and $V$ an independent output shock. Both have mean zero and variance one; specifically, they are independent standard normal variables. Training intensity $D$ is measured relative to its mean, so a negative value means below-average intensity.

For any real number $b$, suppose the economy obeys

$$D=U,\qquad Y=bD+(1-b)U+V.$$

The coefficient $b$ describes the direct response to changing training while leaving the employee’s underlying characteristic unchanged. Substitute the first equation into the second:

$$\begin{aligned} Y&=bU+(1-b)U+V\\ &=U+V=D+V. \end{aligned}$$

Every value of $b$ produces exactly the same joint distribution of observed training and output. In all these worlds,

$$E[Y\mid D=d]=d,\qquad \frac{\operatorname{Cov}(D,Y)}{\operatorname{Var}(D)}=1.$$

Now intervene. The operation $\operatorname{do}(D=d)$ replaces the training equation with $D=d$. It leaves the distribution of $U,V$ and the output equation alone. Consequently,

$$E[Y\mid \operatorname{do}(D=d)]=bd.$$

At $b=1$, moving training from zero to one increases expected output by one unit. At $b=0$, it changes nothing. At $b=-1$, it reduces expected output by one unit. The observational regression is upward-sloping in all three cases.

Identification

A target is identified within a stated model class if every model in that class consistent with the observable distribution gives the same target value. Identification is therefore a joint statement about data and restrictions. The identified set collects the target values of every model in the class that matches the observable distribution, so it is always taken relative to the maintained model class.

In this example, the observable distribution cannot identify $b$. An estimator is consistent if it converges in probability to its target as the sample grows; chapter 4 makes that convergence precise. No estimator that sees only $(D,Y)$ can consistently recover the correct $b$ in every one of these worlds. The reason is stronger than “the sample is small”: such an estimator receives data with the same distribution whichever world is true. Chapters 2 and 5 use the word in a second sense, for a causal assumption about how observed outcomes relate to the outcomes a unit would have under each treatment. That sense has nothing to do with sample size.

The same-data experiment lets you change $b$ while preserving the observed sample. Its slider runs from $-1$ to $2$ as a teaching choice. In the unrestricted construction above, every real $b$ is observationally compatible, so the identified set for $b$ is the whole real line. If the model class were restricted to $b\in[-1,2]$, the identified set would be that whole interval: the restriction would have produced it, and the observations would not narrow it.

There is plenty of variation in observed training: the normal distribution has support on the entire real line. That does not remove confounding. Conditional on $U$, training is deterministic. Even measuring $U$ would leave no within-$U$ training variation in this toy economy. A credible randomized intervention would add the variation needed to distinguish these worlds.

Numerical check

Run this with Python and NumPy. Reusing the same shocks is essential: it shows that changing $b$ preserves every observed point, not merely a rounded correlation.

import numpy as np

rng = np.random.default_rng(20260904)
u, v = rng.normal(size=(2, 1000))
d = u.copy()
reference = d + v

for b in (-1.0, 0.0, 1.0, 2.0):
    observed_y = b*d + (1-b)*u + v
    intervention_y = b*1.0 + (1-b)*u + v
    slope = np.polyfit(d, observed_y, 1)[0]
    print(b, np.max(np.abs(observed_y-reference)),
          round(slope, 4), round(np.mean(intervention_y), 4))

Each line prints $b$, the maximum observed-data difference, the fitted slope, and the simulated intervention mean. The maximum difference is zero up to floating-point rounding. The fitted slope is 1.0241 in every world, because every world produces the same observations. The intervention mean is close to $b$; its departures, such as 0.0762 when $b=0$, are sampling error from 1,000 draws. The slope’s finite-sample departure from one is an estimation issue; its inability to reveal $b$ is an identification issue.

Components of an empirical analysis

For each investigation, keep six linked statements:

1. Target: the quantity needed for the decision.
2. Observations: what was measured, for whom, and when.
3. Identification: restrictions connecting observations to the target.
4. Estimation: the calculation implementing that connection.
5. Uncertainty: sampling variation and credible departures from assumptions.
6. Decision: the action, costs, and threshold at which the recommendation changes.

These statements can disagree. A prediction may be excellent while a causal claim remains unidentified. A causal effect may be precisely estimated while the business case fails because implementation costs are high.

Reporting a regression coefficient as a causal effect requires an identification argument.

Exercises

1. In the synthetic model, the observational mean rises by two units when $D$ rises from zero to two. What is the corresponding intervention contrast when $b=0.4$?

Work through the solution

The intervention mean is $bd$, so the contrast is $0.4(2-0)=0.8$. The observational contrast is two because changing the observed value of $D$ also selects a different value of $U$. The intervention changes $D$ while retaining the population distribution of $U$.

2. Somebody proposes collecting a million additional observations from the same system. Which parts of the argument could improve? Which cannot?

Work through the solution

The joint distribution of observed training and output can be estimated more precisely. Sampling error in its regression slope can shrink. But all values of $b$ still produce that distribution. More observations from this same process do not identify $b$. A new design, additional defensible restrictions, or a narrower target is required.

Further reading

For an alternative introduction to causal questions, see Scott Cunningham’s Foundational Ideas in Causal Inference. For the distinction between association and intervention developed through potential outcomes, see Hernán and Robins’s Causal Inference: What If. These established frameworks inform the course; the recurring training example and its teaching sequence are original constructions.

Group composition and aggregate averages

Suppose a report says that 68% of training participants meet their output target, compared with 32% of nonparticipants. Before explaining the difference, find out who is inside each denominator.

Here is a completely constructed population of 100 employees. Let $D=1$ mean participation, $K=1$ mean prior experience, and $Y=1$ mean meeting the output target. Each entry describes people, not estimated probabilities.

Prior experience	Participation	Employees	Meet target
Experienced	Participate	40	32
Experienced	Do not participate	10	8
New	Participate	10	2
New	Do not participate	40	8

Among experienced employees, the success rate is 80% in both participation groups. Among new employees, it is 20% in both groups. Yet participants are mostly experienced, and nonparticipants are mostly new.

The aggregate difference is 36 percentage points. Every within-experience difference is zero. This is a composition calculation; a causal interpretation will require additional assumptions.

Conditioning changes the reference group

Imagine drawing one employee uniformly from the table. Probability describes the resulting uncertainty. For events $A$ and $B$ with $P(B)>0$,

$$P(A\mid B)=\frac{P(A\cap B)}{P(B)}.$$

Conditioning on participation restricts attention to the 50 participants. Forty are experienced, so $P(K=1\mid D=1)=0.8$. The probability of meeting the target is

$$\begin{aligned} &P(Y=1\mid D=1)\\ &\quad=P(Y=1\mid K=1,D=1)\\ &\qquad\times P(K=1\mid D=1)\\ &\qquad+P(Y=1\mid K=0,D=1)\\ &\qquad\times P(K=0\mid D=1)\\ &\quad=0.8(0.8)+0.2(0.2)\\ &\quad=0.68. \end{aligned}$$

The same calculation for nonparticipants gives $0.8(0.2)+0.2(0.8)=0.32$. The within-group success rates are identical. The weights differ.

Probabilities with different conditioning information answer different questions. $P(K=1\mid D=1)=0.8$ says how many participants are experienced. $P(D=1\mid K=1)=0.8$ says how many experienced employees participate. They happen to agree here because both marginal group sizes are 50. That coincidence is not a general symmetry.

Expectation is a weighted sum

For a discrete variable $Y$, expectation is

$$E[Y]=\sum_y yP(Y=y).$$

The expectation of a zero-one variable equals its probability of being one. Expectations of income, output, and losses use the same rule, with the outcome values included in the weighted sum. For continuous variables, integration replaces summation.

The law of iterated expectations says, for an integrable outcome,

$$E[Y]=E\{E[Y\mid K]\}.$$

Here integrable means $E[|Y|]<\infty$. In the table, half the workforce is experienced:

$$E[Y]=0.5(0.8)+0.5(0.2)=0.5.$$

You can also count directly: $32+8+2+8=50$ successes among 100 employees. Conditioning reorganizes the calculation without changing the overall population average, provided the outer weights are the correct population weights.

Standardizing both participation groups to a half-experienced, half-new workforce gives a success rate of 0.5 for each, so the standardized contrast is zero. That zero measures the causal effect of participation only under three conditions. First, each employee’s recorded success must be the success that employee would have under the participation status actually recorded. This causal condition is called consistency; chapter 5 states it with potential outcomes, and it is unrelated to the estimator property of the same name in chapter 1. Second, each experience group must contain both participants and nonparticipants, a condition called overlap; here the experienced employees split 40 and 10, and the new employees 10 and 40. Third, nothing besides experience may influence both participation and success. The table can check the second condition. It cannot check the first or the third, and no identity about probabilities can certify them.

Where variation comes from

Let $m(K)=E[Y\mid K]$. If $E[Y^2]<\infty$, then

$$\begin{aligned} \operatorname{Var}(Y) &=E\{\operatorname{Var}(Y\mid K)\}\\ &\quad+\operatorname{Var}\{m(K)\}. \end{aligned}$$

The first term is variation within experience groups. The second is variation in their conditional means. In our population, $Y$ is binary with mean 0.5, so its variance is $0.5(1-0.5)=0.25$. Within either experience group, the variance is $0.8(0.2)=0.16$. The group means lie 0.3 above or below 0.5, so their variance is $0.09$. Thus $0.25=0.16+0.09$.

The Distribution Lab opens on a binomial distribution with one trial and success probability 0.5, which is this binary outcome. Its bench reports the mean 0.5 and the variance 0.25. Moving the success probability to 0.8 reproduces the within-group variance of 0.16. The same bench covers eighteen other distributions.

Why the decomposition works

Write

$$\begin{aligned} Y-E[Y]&=\{Y-m(K)\}\\ &\quad+\{m(K)-E[Y]\}. \end{aligned}$$

Square and take expectations. The cross term is zero because

$$\begin{aligned} &E[(Y-m(K))(m(K)-E[Y])]\\ &\quad=E[(m(K)-E[Y])\\ &\qquad\times E[Y-m(K)\mid K]]\\ &\quad=0. \end{aligned}$$

The remaining two squared terms are precisely the within-group and between-group components.

This also explains why the conditional mean minimizes mean squared prediction error among functions of $K$. For any square-integrable $g(K)$,

$$\begin{aligned} &E[(Y-g(K))^2]\\ &\quad=E[(Y-m(K))^2]\\ &\qquad+E[(m(K)-g(K))^2]. \end{aligned}$$

The second term cannot be negative. This is a prediction result and makes no causal claim.

A census, a sample, and a missing employee

The table is a finite population we constructed. Its proportions are exact. If a survey samples 20 of these employees, its proportions become random because a different sample could contain different people. If the intended population is next year’s workforce, even a census of these 100 employees leaves a transport question.

Now suppose experienced employees answer a follow-up survey with probability 0.8 and new employees with probability 0.2. Assuming response is independent of success within experience groups, the respondent population is 80% experienced and has a success rate of 68%. The whole eligible population still has a 50% success rate.

A million responses produced by the same selection mechanism would estimate the respondent rate with great precision. It would not turn respondents into a representative workforce.

Calculating the weighted average

# Standard Python; all quantities describe the synthetic table.
rates = [0.8, 0.2]  # experienced, new
weights = {
    "participants": [0.8, 0.2],
    "nonparticipants": [0.2, 0.8],
    "eligible population": [0.5, 0.5],
}
for name, w in weights.items():
    print(name, sum(p*q for p, q in zip(rates, w)))

Exercises

1. A future workforce is 75% experienced. Keep the two group-specific success rates fixed. What success rate should you predict?

Work through the solution

Use the future population weights: $0.75(0.8)+0.25(0.2)=0.65$. Using 0.5 would silently assume the old composition. Using 0.68 would silently use participant composition. Keeping the group-specific rates fixed is itself a transport assumption.

2. Does zero covariance between two variables establish independence?

Work through the solution

No. Let $X$ be equally likely to equal $-1,0,1$ and let $Y=X^2$. Then $E[X]=0$ and $E[XY]=E[X^3]=0$, so $\operatorname{Cov}(X,Y)=0$. But $Y$ is determined by $X$: for example, $P(Y=0\mid X=0)=1$, whereas $P(Y=0)=1/3$. A zero linear association can conceal complete nonlinear dependence.

Further reading

Blitzstein and Hwang’s Introduction to Probability develops conditioning, expectation, and their calculations. For the econometric use of conditional expectations and projections, see Bruce Hansen’s Econometrics.

Least squares

Least squares minimizes the sum of squared residuals. Interpreting the fitted coefficients requires assumptions about how the data were generated.

The six-person example below can be followed with averages and arithmetic. The derivative and matrix steps explain why its calculation generalizes; readers new to those tools can return to them after working through the numbers.

For observations $(d_i,y_i)$, a regression with an intercept chooses $a,b$ to minimize

$$\sum_{i=1}^n(y_i-a-bd_i)^2.$$

Differentiate with respect to $a$ and $b$. At the optimum,

$$\begin{gathered} \sum_i \widehat u_i=0,\qquad \sum_i d_i\widehat u_i=0,\\ \text{where }\widehat u_i=y_i-\widehat a-\widehat b d_i. \end{gathered}$$

The first equation gives $\widehat a=\bar y-\widehat b\bar d$. Substituting it into the second gives

$$\widehat b= \frac{\sum_i(d_i-\bar d)(y_i-\bar y)} {\sum_i(d_i-\bar d)^2},$$

provided training varies in the sample. These residual properties follow from the optimization. They cannot serve as evidence that the error in a causal model is unrelated to training. The exogeneity panel of the OLS assumptions lab simulates the slope when the error is correlated with the regressor.

A six-person example

All numbers below are synthetic. Here $d$ is training hours, $k$ indicates prior experience, and $y$ is output in arbitrary units.

Employee	Training $d$	Experience $k$	Output $y$
A	0	0	1
B	1	0	0
C	2	0	5
D	2	1	8
E	3	1	7
F	4	1	12

The overall means are $\bar d=2$ and $\bar y=5.5$. The centered cross-product is 29 and the centered training sum of squares is 10. Regressing output on training alone produces a slope of $2.9$.

Now remove each experience group’s mean from both variables. The group training means are 1 and 3; output means are 2 and 9. The residualized observations are

$$\begin{gathered} \widetilde d=(-1,0,1,-1,0,1),\\ \widetilde y=(-1,-2,3,-1,-2,3). \end{gathered}$$

Their cross-product is 8 and the training sum of squares is 4. The adjusted slope is $8/4=2$.

This is the training coefficient from regressing output jointly on an intercept, training, and experience. Residualizing makes the comparison visible: within each experience group, how do deviations in training line up with deviations in output?

The general projection

Put the outcome in an $n$-vector $\mathbf y$ and the regressors, including any intercept, in an $n\times p$ matrix $\mathbf X$. If $\mathbf X$ has full column rank,

$$\widehat{\boldsymbol\beta} =(\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf y,\qquad \mathbf X'\widehat{\mathbf u}=0.$$

The fitted vector lies in the space spanned by the columns of $\mathbf X$. The residual is perpendicular to that space. Adding a regressor enlarges the available space, so in-sample squared error cannot increase. This says nothing by itself about prediction on new data.

In a population, let $\mathbf x$ be a random $p$-vector. With finite second moments and nonsingular $Q=E[\mathbf x\mathbf x']$, the best linear predictor minimizes $E[(Y-\mathbf x'\boldsymbol\beta)^2]$ and satisfies

$$\boldsymbol\beta=Q^{-1}E[\mathbf xY],\qquad E[\mathbf xu]=0.$$

The conditional mean $E[Y\mid\mathbf x]$ may still be nonlinear. Orthogonality to the chosen regressors is weaker than $E[u\mid\mathbf x]=0$.

The OLS assumptions lab simulates each classical assumption separately. Its correct-specification panel fits a straight line to a curved conditional mean, and its collinearity panel shows what happens to the coefficients as the full-rank condition above nearly fails.

The residualization theorem

Partition the design into a variable of interest $\mathbf d$ and controls $\mathbf W$, including an intercept if needed. Define

$$M_W=I-\mathbf W(\mathbf W'\mathbf W)^{-1}\mathbf W'.$$

Assume $\mathbf W$ has full rank and $\mathbf d'M_W\mathbf d>0$. For a proposed coefficient $b$ on $\mathbf d$, fitting the controls leaves residual $M_W(\mathbf y-\mathbf d b)$. Minimizing its squared length gives

$$\widehat b=\frac{\mathbf d'M_W\mathbf y}{\mathbf d'M_W\mathbf d}.$$

Since $M_W$ is symmetric and idempotent, this is the regression of residualized $\mathbf y$ on residualized $\mathbf d$. This algebraic result is the Frisch–Waugh–Lovell theorem. It requires no causal assumptions.

Where an omitted variable enters

Suppose a substantive model is

$$\begin{gathered} Y=\alpha+\beta D+\gamma K+\varepsilon,\\ \operatorname{Cov}(D,\varepsilon)=0. \end{gathered}$$

Then the population slope from omitting $K$ is

$$b_{\mathrm{short}} =\frac{\operatorname{Cov}(D,Y)}{\operatorname{Var}(D)} =\beta+\gamma \frac{\operatorname{Cov}(D,K)}{\operatorname{Var}(D)}.$$

The extra term requires both an output relationship and a training relationship for $K$. In the six-person construction, $y=2d+3k+e$ with $e=(1,-2,1,1,-2,1)$. The residual vector has zero sample covariance with $d$ and $k$. The short slope is therefore exactly $2+3(3/10)=2.9$.

Knowing the construction lets us interpret the coefficient. With actual company records, that construction is unknown. Experience may be insufficient to address self-selection. A variable measured after training may be a mediator, which carries part of training’s effect to output, or a collider, which is caused both by training and by something else that affects output. Chapter 6 shows how adjusting for either changes the comparison. More controls do not automatically produce a better causal argument.

Reproduce the adjustment

# Standard Python.
d = [0, 1, 2, 2, 3, 4]
k = [0, 0, 0, 1, 1, 1]
y = [1, 0, 5, 8, 7, 12]

def demean_by_group(values):
    means = {
        g: sum(v for v, h in zip(values, k) if h == g)
           / k.count(g)
        for g in set(k)
    }
    return [v-means[g] for v, g in zip(values, k)]

rd, ry = demean_by_group(d), demean_by_group(y)
print(sum(a*b for a, b in zip(rd, ry))
      / sum(a*a for a in rd))  # 2.0

Exercises

1. Does a regression fitted without an intercept necessarily have residuals that sum to zero?

Work through the solution

No. Residuals are orthogonal to the included columns. Without a constant column, orthogonality to the vector of ones is not generally imposed. For example, fitting $y=bx$ to $x=(1,2)$ and $y=(1,1)$ gives $\widehat b=3/5$ and residuals $(0.4,-0.2)$, which sum to 0.2.

2. Let $X$ be uniform on $[-1,1]$ and $Y=X^2$. What does the population linear projection with an intercept predict? Is its residual conditionally mean zero?

Work through the solution

Symmetry gives $E[X]=E[X^3]=0$, so the slope is zero. The intercept is $E[X^2]=1/3$. The residual is $u=X^2-1/3$, with $E[u]=E[Xu]=0$. But $E[u\mid X]=X^2-1/3$, generally nonzero. The projection is the best available straight line, although the conditional mean is a curve.

Further reading

Bruce Hansen’s Econometrics develops population projection, least squares, and their sampling theory. Angrist and Pischke’s Mostly Harmless Econometrics resources provide a route from regression arguments to inspectable empirical applications.

The sampling process

An estimate is a number from one dataset. A standard error describes how that estimate would vary across repetitions of a specified sampling or assignment process.

That process matters. Repeatedly sampling employees differs from repeatedly randomizing training across the same employees. Sampling 1,000 employees from one office differs from sampling one employee from each of 1,000 independent offices.

A standard-error calculation depends on which quantities are fixed and which are random across repetitions.

A case where the interval is exact

Let $Y_1,\ldots,Y_n$ be independent draws from a normal population with unknown mean $\mu$ and known standard deviation $\sigma$. Then

$$\bar Y\sim N\left(\mu,\frac{\sigma^2}{n}\right), \qquad \frac{\bar Y-\mu}{\sigma/\sqrt n}\sim N(0,1).$$

Let $z_{0.975}\approx1.959964$ be the standard normal 97.5th percentile. The interval

$$C(Y)= \left[\bar Y-z_{0.975}\frac{\sigma}{\sqrt n}, \bar Y+z_{0.975}\frac{\sigma}{\sqrt n}\right]$$

contains $\mu$ in 95% of repeated samples. The sampling experiment deliberately uses this known-$\sigma$, Gaussian setting. Its simplicity lets you inspect coverage without mixing in estimated variance or asymptotic approximations.

With $\bar Y=2.4$, $\sigma=4$, and $n=100$, the standard error is $0.4$. The interval is approximately $[1.616,3.184]$.

The parameter is fixed in this frequentist calculation; the interval is random before sampling. After observing a particular interval, the procedure’s 95% coverage does not become a 95% posterior probability that $\mu$ lies in that interval. A Bayesian probability statement would require a specified probability model for the unknown parameter.

If $\sigma$ is unknown and the observations remain independent normal draws, replacing it with the sample standard deviation yields a Student $t$ statistic with $n-1$ degrees of freedom. Using a normal critical value with an estimated standard error is usually an approximation, not that exact result. The Constructions tab of the Distribution Lab builds the $t$ distribution draw by draw from a standard normal and an independent chi-square, which shows where its heavier tails come from.

The limits behind large-sample reasoning

For independent, identically distributed observations with $E[|Y_i|]<\infty$, the law of large numbers gives $\bar Y\to_p\mu$. With finite, positive variance as well, the central limit theorem gives

$$\sqrt n(\bar Y-\mu)\ \xrightarrow{d}\ N(0,\sigma^2).$$

Convergence in probability concerns closeness to a target. Convergence in distribution concerns the shape of scaled fluctuations. Neither statement says every dataset of size 30, or 300, behaves approximately normally. Heavy tails, rare influential observations, and dependence can make an approximation poor. The lab’s central-limit construction averages draws from skewed and heavy-tailed sources, including a Cauchy source whose average never settles; the normal-errors panel of the OLS assumptions lab checks a regression test under skewed errors.

The same distinction carries into regression. Under suitable sampling and moment conditions, a coefficient becomes increasingly precise around its population projection. That projection can differ from the causal parameter a manager cares about.

Confidence-interval coverage under bias

Suppose the desired target is $\theta$, but the observations satisfy

$$Y_i=\theta+b+\varepsilon_i,\qquad \varepsilon_i\overset{\mathrm{iid}}{\sim}N(0,\sigma^2).$$

The persistent bias is $b$. An analyst mistakenly reports the ordinary mean interval as an interval for $\theta$. Its actual coverage is

$$\begin{aligned} P(\theta\in C) &=\Phi\left(z_{0.975}-\frac{b\sqrt n}{\sigma}\right)\\ &\quad-\Phi\left(-z_{0.975}-\frac{b\sqrt n}{\sigma}\right), \end{aligned}$$

where $\Phi$ is the standard normal cumulative distribution function.

Take $b=1$ and $\sigma=4$. Coverage is about 29.5% at $n=100$ and 0.12% at $n=400$. With nonzero fixed bias, coverage approaches zero as the sample grows. The interval becomes excellent at locating $\theta+b$ and terrible at covering $\theta$.

# Standard Python; 1.96 approximates the exact normal quantile.
from math import erf, sqrt
Phi = lambda z: (1 + erf(z/sqrt(2))) / 2

for n in (25, 100, 400):
    bias_in_se = sqrt(n) / 4
    coverage = Phi(1.96-bias_in_se) - Phi(-1.96-bias_in_se)
    print(n, round(coverage, 4))

Match the standard error to the dependence

Heteroskedasticity allows the error variance to differ across observations. For independent, identically distributed observations, let $\mathbf x_i$ be the regressor vector and $u_i$ the population projection error. The conditional error variance can still vary with $\mathbf x_i$. If the relevant moments exist, $Q=E[\mathbf x_i\mathbf x_i']$ is nonsingular, and a suitable central limit theorem applies,

$$\begin{gathered} \sqrt n(\widehat{\boldsymbol\beta}-\boldsymbol\beta) \xrightarrow{d} N(0,Q^{-1}\Omega Q^{-1}),\\ \Omega=E[\mathbf x_i\mathbf x_i'u_i^2]. \end{gathered}$$

The sample counterpart is the heteroskedasticity-robust sandwich variance estimator. Its job is to estimate sampling variance under unequal variances. It does not correct selection, measurement error, or omitted-variable bias. The constant-variance panel of the OLS assumptions lab shows how far the conventional standard error can understate the spread of a slope when error variance grows with a skewed regressor.

Employees within an office may share shocks. A cluster-robust calculation aggregates score contributions within office:

$$\begin{aligned} \widehat{\operatorname{Var}}(\widehat{\boldsymbol\beta}) &=(\mathbf X'\mathbf X)^{-1} \left(\sum_{g=1}^G\mathbf S_g\mathbf S_g'\right)\\ &\qquad\times(\mathbf X'\mathbf X)^{-1},\\ \mathbf S_g&=\sum_{i\in g}\mathbf x_i\widehat u_i. \end{aligned}$$

This displays the uncorrected form; software often adds finite-sample adjustments. The standard justification requires sufficiently many independent clusters and conditions preventing a few clusters from dominating. A large employee count cannot substitute for those conditions. The lab’s independent-errors panel shows the analogous failure for serially correlated errors: the slope stays centered while the conventional standard error understates its spread.

As an illustration for a simple mean with equally sized independent clusters, cluster size $m=25$ and within-cluster correlation $\rho=0.1$ inflate variance relative to independent observations by $1+(m-1)\rho=3.4$. This formula depends on that exchangeable covariance structure; it is not a universal regression correction.

Exercises

1. How much must sample size increase to halve the standard error of an independent sample mean, keeping population variance fixed?

Work through the solution

The standard error is $\sigma/\sqrt n$. Replacing $n$ with $4n$ divides it by two. Doubling $n$ multiplies it by $1/\sqrt2\approx0.707$. The claim assumes the additional observations follow the same independent sampling process.

2. A training estimate is 3 with a standard error of 0.1. A credible selection mechanism could contribute between 1 and 4 output units to that association. Can the narrow interval establish a positive causal effect?

Work through the solution

No. Ignoring sampling variation for the moment, subtracting the selection contribution gives causal effects between $3-4=-1$ and $3-1=2$. Sampling uncertainty would add another layer. Reporting $3\pm1.96(0.1)$ addresses precision around the association and leaves the identification uncertainty unresolved.

Further reading

Hansen’s Econometrics develops asymptotic regression theory. Cameron and Miller’s A Practitioner’s Guide to Cluster-Robust Inference explains why the clustering level and the number of clusters deserve substantive attention.

Treatment and comparison groups

The assignment process determines which treatment comparisons the data can support.

Suppose 100 employees are eligible for the same four-hour training session. Before enrollment, the firm randomly selects 50 to receive it this month; the others receive it later. Output is measured for everyone over the same following month. The intervention, eligibility rule, outcome window, and assignment mechanism have now become concrete.

Define $Y_i(1)$ as employee $i$‘s output under training and $Y_i(0)$ as output without it during that window. The individual effect is $Y_i(1)-Y_i(0)$. Only one outcome is observed:

$$Y_i=D_iY_i(1)+(1-D_i)Y_i(0),$$

where $D_i$ indicates assigned and delivered training in this initial, perfect-compliance example. This equation is the consistency assumption: the observed outcome equals the potential outcome under the treatment actually received. It is a causal restriction, distinct from the consistency of an estimator defined in chapter 1.

These definitions assume that each treatment condition has a specified version and that another employee’s assignment does not affect $i$‘s outcome. These are the usual no-hidden-versions and no-interference components associated with SUTVA. If trained employees teach their colleagues, the second assumption is doubtful. The intervention and analysis may need to move to teams.

Why random assignment identifies a comparison

For a population formulation, suppose

$$\begin{gathered} D\ \perp\ (Y(1),Y(0)),\\ 0<P(D=1)<1, \end{gathered}$$

and observed outcomes satisfy the consistency relation above. Then

$$\begin{aligned} &E[Y\mid D=1]-E[Y\mid D=0]\\ &\quad=E[Y(1)\mid D=1]\\ &\qquad-E[Y(0)\mid D=0]\\ &\quad=E[Y(1)]-E[Y(0)]. \end{aligned}$$

The first equality uses consistency. The second uses random assignment. Writing both steps makes it possible to see which part fails when training is optional.

For a fixed set of $N$ employees, complete randomization selects exactly $n_1$ for treatment and $n_0=N-n_1$ for control. The difference in observed group means,

$$\widehat\tau=\bar Y_1-\bar Y_0,$$

is unbiased over the randomization for the finite-population average effect

$$\tau_N=\frac1N\sum_{i=1}^N\{Y_i(1)-Y_i(0)\}.$$

Randomization balances the potential outcomes in expectation. It does not guarantee perfect balance in every realized assignment.

The exact randomization variance

Let $S_d^2$ be the finite-population variance of $Y_i(d)$, using denominator $N-1$, and let $S_\tau^2$ be the corresponding variance of individual effects. Under complete randomization,

$$\operatorname{Var}_D(\widehat\tau) =\frac{S_1^2}{n_1}+\frac{S_0^2}{n_0} -\frac{S_\tau^2}{N}.$$

The random subscript emphasizes that assignments vary while potential outcomes remain fixed. A single realized assignment reveals only one potential outcome per employee, so $S_\tau^2$ generally cannot be calculated from it without additional restrictions. The usual estimator $s_1^2/n_1+s_0^2/n_0$ has expectation at least as large as the true variance. This is a statement about conservative variance estimation, not a guarantee of exact coverage for a normal-approximation interval in a tiny experiment.

An experiment small enough to enumerate

Consider four employees with observed outputs $(1,3,5,7)$. Exactly two were randomized to training; the employees with outputs 5 and 7 received it. The observed difference in means is $6-2=4$.

Test the sharp null that training affects nobody: $Y_i(1)=Y_i(0)$ for every $i$. Under that null, each employee’s observed output would remain the same under any assignment. All six possible treated pairs are equally likely. Their differences in means are

$$-4,\ -2,\ 0,\ 0,\ 2,\ 4.$$

The exact two-sided randomization $p$-value, using absolute difference as the statistic, is $2/6=1/3$. A large observed difference can coexist with weak evidence when the design permits very few distinct comparisons.

from itertools import combinations

y = [1, 3, 5, 7]
statistics = []
for treated in combinations(range(4), 2):
    controls = [i for i in range(4) if i not in treated]
    stat = (sum(y[i] for i in treated)/2
            - sum(y[i] for i in controls)/2)
    statistics.append(stat)

print(sorted(statistics))
print(sum(abs(t) >= 4 for t in statistics) / len(statistics))

This test targets the sharp null. A null of zero average effect permits positive and negative individual effects and does not reveal every missing potential outcome. The same enumeration cannot be carried over without justification.

Noncompliance and attrition

Covariates. Measure experience before assignment. Blocking randomization by experience can improve balance. Prespecified regression adjustment can improve precision too. A useful large-sample approach includes centered baseline covariates and their treatment interactions, with appropriate robust uncertainty. It does not make finite-sample bias impossible, and a large search over adjustments compromises the advertised analysis.

Uptake. If some assigned employees skip training, preserve assignment in the main comparison. Randomization then identifies the effect of offering or assigning training, the intention-to-treat effect. Comparing attendees with nonattendees reintroduces selection. Moving from assignment to receipt requires additional assumptions, developed in the instruments chapter.

Attrition. Collect outcomes for both groups whether employees attend or not. Randomizing assignment does not randomize which outcomes go missing afterward. If disappointed participants stop reporting output, complete-case analysis can break the original comparison.

Power. Under an independent, equal-variance approximation, 200 people in each arm with outcome standard deviation 4 give a difference-in-means standard error of $4\sqrt{1/200+1/200}=0.4$. Approximate 80% power for a two-sided 5% test requires an effect of $(1.96+0.84)(0.4)=1.12$. Clustering, attrition, unequal allocation, and baseline adjustment alter this planning calculation.

Exercises

1. In the four-person example, what is the exact one-sided $p$-value for a prespecified positive-effect alternative?

Work through the solution

Only one of the six assignments yields a statistic at least as large as 4, so the value is $1/6$. Choosing the direction after seeing the sign would not justify reporting this as a prespecified one-sided test.

2. Twenty employees were assigned treatment. Eighteen report an average outcome of 8; two outcomes are missing. Outcomes are bounded between 0 and 10. All 20 controls report an average of 7. Bound the full-assignment difference in observed group means.

Work through the solution

The full treatment mean lies between $(18\cdot8+2\cdot0)/20=7.2$ and $(18\cdot8+2\cdot10)/20=8.2$. Subtracting 7 gives $[0.2,1.2]$. These bounds address the missing outcomes for this realized assignment. Randomization uncertainty about the causal average is an additional issue.

Further reading

Cunningham’s Potential Outcomes and Randomization develops the design-based argument. Winston Lin’s Agnostic Notes on Regression Adjustments to Experimental Data supplies the formal basis and qualifications for fully interacted adjustment.

Variable definitions

The firm’s records contain a column called training hours. One employee reports hours scheduled, another reports hours attended, and a third includes time practicing afterward. Output is recorded only for employees still present at the next performance review.

Those details change the problem. They define the measured exposure, determine whose outcomes enter the dataset, and may create associations that would be absent in the full workforce.

An estimator operates on the variables supplied to it. A careful measurement model explains how those variables relate to the quantities in the question.

Classical measurement error

Suppose true, centered training intensity is $D$, but the recorded value is

$$D^*=D+\eta.$$

Assume the outcome obeys $Y=\beta D+\varepsilon$. Require finite second moments, $\operatorname{Var}(D)>0$, and

$$\begin{gathered} \operatorname{Cov}(D,\varepsilon)=0,\qquad \operatorname{Cov}(D,\eta)=0,\\ \operatorname{Cov}(\eta,\varepsilon)=0. \end{gathered}$$

These are classical-error restrictions relevant to this single-regressor linear calculation. The population regression slope using the recorded variable is

$$\begin{aligned} b^* &=\frac{\operatorname{Cov}(D^*,Y)}{\operatorname{Var}(D^*)}\\ &=\frac{\beta\operatorname{Var}(D)} {\operatorname{Var}(D)+\operatorname{Var}(\eta)} =\lambda\beta, \end{aligned}$$

where $\lambda$ is the reliability ratio. Under these assumptions, $0<\lambda\le1$, so the coefficient is attenuated toward zero.

If the true effect is 2 output units per training hour, true-hour variance is 9 hours squared, and error variance is 3 hours squared, then $\lambda=9/12=0.75$ and the recorded-hours slope is 1.5.

The units help catch mistakes: the reliability ratio is dimensionless; the resulting coefficient retains output units per hour.

This result has boundaries. Correlated reporting errors can move a coefficient in either direction. With several regressors, error in one variable can distort several coefficients without a simple attenuation pattern. Independent additive noise in the outcome leaves this simple population slope unchanged, although it generally increases sampling variance. A validation sample, repeat measurements with justified error assumptions, or a defensible instrument may supply information about measurement quality.

Missing outcomes change the population

Let $S=1$ indicate that output is observed, and let $X$ contain fully observed baseline information. With one potentially missing outcome, useful distinctions are:

• Missing completely at random: $S$ is independent of the pair $(Y,X)$. Independence from $Y$ and from $X$ taken one at a time is weaker, because it still allows response to depend on how the two combine.
• Missing at random given $X$: $S$ is independent of $Y$ conditional on $X$.
• Missing not at random: response still depends on the missing outcome after conditioning on the observed information in the model.

“At random” in the second phrase does not mean every employee has the same response probability. Experience may predict response strongly. The restriction is that, within the observed information, response does not further select on the outcome.

Suppose experienced and new employees each constitute half the workforce, with target-attainment rates 0.8 and 0.2. If response rates are 0.8 and 0.2 respectively, and response is independent of attainment within experience groups, respondents are 80% experienced. Their attainment rate is 0.68; the workforce rate is 0.5.

Why inverse response weighting can recover the mean

Define $\pi(X)=P(S=1\mid X)$. Assume $S\perp Y\mid X$, positive response probability wherever the target population has support, and integrability. Then

$$\begin{aligned} E\left[\frac{SY}{\pi(X)}\right] &=E\left[\frac{E[SY\mid X]}{\pi(X)}\right]\\ &=E\left[\frac{\pi(X)E[Y\mid X]}{\pi(X)}\right]\\ &=E[Y]. \end{aligned}$$

The recorded product $SY$ contributes zero for nonrespondents, so their unobserved outcomes do not need to be filled in to implement the sample counterpart. In practice $\pi$ is often estimated, adding estimation error and model dependence. Very small response probabilities produce unstable weights.

If response depends on unobserved output within $X$, the factorization in the second line fails. Missing-at-random assumptions generally cannot be verified from observed outcomes alone. Sensitivity analysis should vary the unresolved outcome distribution, not simply run another imputation package.

Three reasons a variable might matter

A causal graph is an explicit proposal about how variables arise. An arrow denotes a possible direct causal relationship in the proposed model; absence of an arrow is a substantive restriction. Graphs help expose the assumptions behind a control list.

Common cause: $X\to D$ and $X\to Y$. Prior experience might influence both participation and output. Adjustment can close this confounding path, provided the selected variables are sufficient and treatment has overlap within their values.

Mediator: $D\to M\to Y$. Training may improve skill, which raises output. Controlling for post-training skill removes part of the pathway included in a total-effect question. The remaining coefficient does not automatically identify a direct effect; mediator–outcome confounding introduces further requirements.

Collider: $D\to S\leftarrow U\to Y$, with $S$ the inclusion indicator defined above. Inclusion in a performance review, and therefore in the outcome data, may depend on training and latent readiness. Conditioning on review inclusion can associate randomized training with readiness.

Here is an exact collider example. Let $D$ and $U$ be independent fair zero-one variables, let $Y=U$, and include employees if $S=1$ whenever $D=1$ or $U=1$. Training has no effect on output. Among included employees with $D=0$, everyone has $U=1$, so their mean output is 1. Among included employees with $D=1$, half have $U=1$, so their mean is 0.5. The selected-sample difference is $-0.5$.

from itertools import product

# Four equally likely types; D has no causal role in Y.
rows = [{"d": d, "y": u}
        for d, u in product((0, 1), repeat=2)
        if d == 1 or u == 1]

means = {d: sum(r["y"] for r in rows if r["d"] == d)
            / sum(r["d"] == d for r in rows)
         for d in (0, 1)}
print(means[1] - means[0])  # -0.5

Documenting variable construction

For each analysis variable, record its substantive definition, units, measurement window, source system, transformations, missing-value codes, and whether it precedes assignment. For each exclusion, record the rule and the number removed by treatment group. Retain raw values separately from derived fields.

Then ask a harder question: could the outcome or treatment have affected the variable used to select these rows? A clean-looking dataset can represent a badly selected comparison.

Exercises

1. True-hour variance remains 9, but measurement-error variance rises from 3 to 9. Under the classical assumptions above, what happens to the slope when $\beta=2$?

Work through the solution

Reliability becomes $9/(9+9)=0.5$, so the slope becomes 1. The calculation applies to the stipulated classical-error model. It is not evidence that an observed coefficient should be doubled without external information about reliability.

2. Should a total-effect analysis of training control for an employee’s post-training certification score?

Work through the solution

Usually that requires changing the question or giving a specific causal argument. If training affects certification and certification affects output, the score is a mediator. Adjusting for it blocks part of the total effect. If unobserved motivation affects both certification and output, conditioning on certification can also introduce selection along a collider path. A baseline certification score has a different temporal and causal role.

Further reading

The measurement investigation asks an earlier question: what makes an observation a measure of the thing we care about? Enter through a narrative essay, work the undergraduate examples, or examine the doctoral companion’s proofs and research proposals. Its two browser experiments separate ordinal recoding, empirical uncertainty, and disagreement about values.

Rubin’s Inference and Missing Data established the missingness framework. Cunningham’s Directed Acyclic Graphs and Hernán and Robins’s Causal Inference: What If develop graphical reasoning about selection and adjustment.

An invitation is not participation

A firm randomly sends some employees an invitation to a training program. Invited employees may decline; employees without an invitation can still find a place. Comparing participants with nonparticipants reintroduces self-selection. Comparing invited with uninvited employees retains the lottery.

In a synthetic population, participation is 60% among invited employees and 20% among uninvited employees. Average output is 108 versus 100 units. The invitation raises participation by 0.40 and output by 8. Its effect on output, the intention-to-treat effect, answers a useful policy question on its own: what does offering this invitation accomplish?

The ratio $8/0.40=20$ has a different interpretation. Obtaining that interpretation requires an argument about how the invitation works.

Compliance types

Let $Z\in\{0,1\}$ indicate invitation, $D(z)$ indicate participation under invitation status $z$, and $Y(d)$ denote output under participation status $d$. Observed participation is $D=D(Z)$ and observed output is $Y=Y(D)$ under exclusion and consistency. The latter equality says the invitation affects output through participation alone.

The standard binary-instrument argument requires:

• Random assignment: $Z$ is independent of $(Y(0),Y(1),D(0),D(1))$, with both invitation states having positive probability.
• Exclusion: invitation has no additional effect on output, including through morale, scheduling, or alternative training.
• Consistency and no interference: recorded treatment corresponds to the stated intervention, and one employee’s invitation or participation does not alter another employee’s outcome.
• Monotonicity: $D(1)\geq D(0)$ for every employee. Nobody participates only when uninvited.
• Relevance: the invitation changes participation for a positive share of employees.

Monotonicity leaves always-participants, never-participants, and compliers. A complier participates when invited and does not participate when uninvited. Compliance is a pair of potential choices; an observed participant cannot generally be labeled a complier.

Write $\tau=Y(1)-Y(0)$. Randomization and exclusion give

$$\begin{aligned} &E[Y\mid Z=1]-E[Y\mid Z=0]\\ &\quad=E[(D(1)-D(0))\tau]. \end{aligned}$$

For a binary choice, changing participation changes output by $\tau$; unchanged participation contributes zero. Similarly, the first-stage difference is $E[D(1)-D(0)]$. Under monotonicity, this difference equals the complier share. Therefore the Wald estimand is

$$\begin{aligned} &\frac{E[Y\mid Z=1]-E[Y\mid Z=0]} {E[D\mid Z=1]-E[D\mid Z=0]}\\ &\quad=E[\tau\mid D(1)>D(0)]. \end{aligned}$$

This is the local average treatment effect, or LATE. In the example, 40% are compliers, 20% always participate, and 40% never participate. Compliers gain 20 units on average. Effects for the other groups remain unknown. A compulsory-training policy may consequently have an effect very different from 20.

From a ratio to moments

For a constant-coefficient model $Y=\alpha+\beta D+u$, an instrument supplies the restriction $E[Zu]=0$ along with an intercept restriction $E[u]=0$. Centering variables gives

$$\beta=\frac{\operatorname{Cov}(Z,Y)}{\operatorname{Cov}(Z,D)}$$

when the denominator is nonzero. This constant-effect formulation and the heterogeneous-effect LATE argument are distinct routes to an IV interpretation.

With a regressor vector $X_i$, an instrument vector $Z_i$, and parameter vector $\theta$, generalized method of moments starts from $E[Z_i(Y_i-X_i'\theta)]=0$. Define

$$\begin{gathered} \bar g_n(\theta)=\frac1n\sum_i Z_i(Y_i-X_i'\theta),\\ \hat\theta=\arg\min_\theta \bar g_n(\theta)'W_n\bar g_n(\theta). \end{gathered}$$

$W_n$ is a positive-definite weighting matrix. Identification requires sufficient rank in $E[Z_iX_i']$. Consistency of $\hat\theta$ as an estimator also requires valid moments, suitable sampling conditions, and convergence of the criterion. Two-stage least squares is one familiar linear implementation. Additional moments permit specification tests under further regularity conditions. Passing such a test does not establish that every instrument satisfies exclusion.

A small denominator changes inference

Sampling noise affects both parts of a Wald ratio. When the population first stage is close to zero, the estimated denominator can approach or cross zero. A symmetric normal interval around the ratio can then behave very poorly. A large sample is helpful only relative to the weakness of the first stage.

The Anderson–Rubin idea avoids dividing by the first stage. To test $H_0:\beta=\beta_0$ in the constant-coefficient model, form $Y-\beta_0D$ and test whether the excluded instrument predicts it, including the specified exogenous controls. Under the null, the instrument should have no explanatory role. In the classical linear model with conditionally independent Gaussian homoskedastic errors, the appropriate F test has a finite-sample reference distribution. Heteroskedasticity or clustering requires a corresponding valid implementation and justification.

Collecting all unrejected values produces a confidence set that may be very wide, disconnected, or unbounded. That can be an accurate account of the information available. Neither a first-stage F rule of thumb nor an identification-robust test repairs an invalid instrument. Andrews, Stock, and Sun’s review develops these distinctions.

Let exclusion fail by a stated amount

Suppose the invitation also raises output directly by $\gamma$ units. In a simple additive sensitivity model,

$$8=0.40\beta+\gamma, \qquad \beta=\frac{8-\gamma}{0.40}.$$

Allowing $|\gamma|\leq2$ yields $\beta\in[15,25]$. These are population sensitivity bounds under the stated model. Sampling uncertainty has not been included.

# Arithmetic for the synthetic population, not estimated standard errors.
reduced_form, first_stage, direct_bound = 8, 0.40, 2
print(reduced_form / first_stage)              # 20.0
print(((reduced_form - direct_bound) / first_stage,
       (reduced_form + direct_bound) / first_stage))  # (15.0, 25.0)

The denominator also magnifies exclusion violations. With a 0.02 first stage, a one-unit direct effect shifts the inferred coefficient by 50 units.

Exercises

1. The same invitation effects are observed, but always-participants gain 5 units and never-participants gain 0. What is the population average treatment effect?

Work through the solution

The shares are 0.20 always-participants, 0.40 compliers, and 0.40 never-participants. The supplied additional information gives $0.20(5)+0.40(20)+0.40(0)=9$ units. The instrument alone identified the complier effect of 20; the effects of 5 and 0 were new information.

2. An encouragement raises participation by 0.02 and output by 1 unit. Calculate the Wald ratio and its sensitivity to a direct effect of 0.5 units.

Work through the solution

The ratio is $1/0.02=50$. With the specified direct effect, the adjusted value is $(1-0.5)/0.02=25$. This calculation concerns validity. Without sample information we cannot calculate sampling precision or diagnose weak-instrument severity from a test statistic.

Further reading

Cunningham’s IV treatment in The Remix develops the compliance argument. Hansen’s Econometrics supplies the formal IV/GMM framework.

Difference-in-differences

In a synthetic firm, offices adopting training increase output from 50 to 62 units per employee. Comparison offices increase from 40 to 46. The trained offices improve by 12; the comparison offices improve by 6. Difference-in-differences reports $12-6=6$.

Why subtract six? Because we are proposing that the comparison group’s change describes what would have happened to the training group without training. We can observe three relevant pieces: the training group’s earlier outcome, its later treated outcome, and the comparison group’s change. The training group’s later untreated outcome is missing.

A second date supplies a comparison over time. It does not automatically supply the missing counterfactual.

Derive the two-by-two result

Let $G=1$ identify the group that starts treatment between dates 0 and 1; $G=0$ identifies the untreated comparison group. Let $Y_t(0)$ be an outcome without treatment and $Y_1(1)$ the treated outcome at date 1. Assume consistency, no spillovers, and no anticipation, so the training group is untreated at date 0. Its target effect is

$$ATT=E[Y_1(1)-Y_1(0)\mid G=1].$$

Define the observable difference-in-differences as

$$\begin{aligned} DD&=E[Y_1-Y_0\mid G=1]\\ &\quad-E[Y_1-Y_0\mid G=0]. \end{aligned}$$

Add and subtract $E[Y_1(0)\mid G=1]$. The result is

$$DD=ATT+\delta,$$

where

$$\begin{aligned} \delta&=E[Y_1(0)-Y_0(0)\mid G=1]\\ &\quad-E[Y_1(0)-Y_0(0)\mid G=0]. \end{aligned}$$

Parallel trends sets $\delta=0$. The groups may have different initial levels. What must match is their mean untreated change over the relevant interval. The observed ten-unit baseline difference causes no algebraic difficulty.

For a panel, we also need the sample and outcome definitions to represent the intended population at both dates. A repeated cross-section can identify a similar target under appropriate stable-composition and sampling assumptions. Replacing departing workers with more productive recruits can otherwise produce an apparent training effect through composition.

Fixed effects remove a particular kind of difference

A familiar panel regression is

$$Y_{it}=\alpha_i+\lambda_t+\tau D_{it}+\varepsilon_{it}.$$

$\alpha_i$ captures time-invariant unit differences, $\lambda_t$ common date effects, and $D_{it}$ treatment status. Within-unit transformations remove $\alpha_i$. They leave time-varying omitted determinants intact.

In a balanced two-group, two-date setting with the appropriate group and time indicators, the treatment-interaction coefficient equals the difference-in-differences contrast. Its causal meaning still comes from the counterfactual assumptions. In more general panels, a constant causal interpretation of $\tau$ needs additional restrictions on effects and on the relation between treatment and disturbances. Writing “unit and year fixed effects” is not a substitute for stating those restrictions.

Consider offices selected for training because management already expects their orders to recover. That expectation may cause a differential untreated trend. Neither office effects nor a common year effect removes it.

Staggered adoption changes the comparison

When offices adopt at different dates, define $G_i=g$ as first treatment date and consider

$$\begin{aligned} &ATT(g,t)\\ &\quad=E[Y_{it}(g)-Y_{it}(\infty)\mid G_i=g],\\ &\quad t\geq g. \end{aligned}$$

$Y_{it}(g)$ is the outcome under adoption at $g$; $Y_{it}(\infty)$ is the outcome under no adoption. Identification requires specifying valid untreated comparisons for each cohort and date, such as never-treated or still-untreated units, with suitable no-anticipation, parallel-trends, and overlap conditions.

A conventional two-way fixed-effects coefficient can compare newly treated units with already treated units. If effects change with exposure time or differ across cohorts, those comparisons need not recover a useful average causal effect; implied weights can be problematic. Callaway and Sant’Anna organize estimation around cohort-time effects and explicit aggregation.

Aggregation itself answers a question. Weighting each office equally, each worker equally, or each cohort equally produces different targets. State the weights and which cohort-time cells are supported by untreated comparisons.

Pre-trend diagnostics

Pre-treatment outcomes can expose implausible comparisons. A visible differential pre-trend is a warning. An insignificant pre-trend test, however, may simply be imprecise. It cannot demonstrate that an unobserved post-treatment counterfactual satisfies parallel trends.

Every plotted coefficient also has a construction: reference period, comparison group, and aggregation. Modern estimators’ pre-treatment coefficients need not be built symmetrically with their post-treatment effects. Check the construction before applying a familiar visual heuristic.

Inference must reflect the dependence structure. Repeated observations in an office are not independent replications of the treatment assignment. Clustered inference often matters; few assignment clusters require methods and caution suited to that design. A large employee count does not manufacture a large number of independently treated offices.

Sensitivity to parallel trends

Suppose substantive knowledge supports $|\delta|\leq M$ output units. Since $ATT=DD-\delta$, the population sensitivity interval is

$$ATT\in[DD-M,DD+M].$$

At $DD=6$ and $M=4$, the interval is $[2,10]$. If each extra output unit is worth $100 and training costs $300 per employee, the corresponding net-benefit interval is $[-100,700]$ dollars. A positive baseline effect need not imply a decision robust to the specified trend uncertainty.

# Synthetic group means and an assumed population trend bound.
treated_change = 62 - 50
comparison_change = 46 - 40
dd, M = treated_change - comparison_change, 4
effect_bounds = (dd - M, dd + M)
net_bounds = tuple(100 * effect - 300 for effect in effect_bounds)
print(dd, effect_bounds, net_bounds)  # 6 (2, 10) (-100, 700)

The design experiment exposes this calculation. Its interval excludes sampling uncertainty. It does not implement HonestDiD. Rambachan and Roth develop formal restrictions and inference that connect pre-treatment evidence to allowed post-treatment violations. Choosing a restriction remains substantive work.

Exercises

1. A firm’s $DD$ is 6. Suppose the training offices would have improved two units more than comparison offices without training. What is $ATT$?

Work through the solution

The differential untreated change is $\delta=2$. Thus $ATT=6-2=4$. The observed training-group change remains 12. Its implied untreated change is $6+2=8$, leaving a treatment effect of $12-8=4$.

2. Retain the $100 value per output unit and $300 cost. What largest symmetric trend bound guarantees nonnegative net benefit when $DD=6$, ignoring sampling uncertainty?

Work through the solution

The worst permitted effect is $6-M$. Require $100(6-M)-300\geq0$, so $M\leq3$. At $M=3$, worst-case net benefit is zero. Strictly positive worst-case benefit requires $M<3$. An estimated $DD$ would introduce sampling uncertainty into a real decision.

Further reading

Use The Remix for extended design examples, Callaway and Sant’Anna for multiple-period targets, and Rambachan and Roth for structured sensitivity.

Treatment at a cutoff

A synthetic firm guarantees training to employees whose assessment score reaches 70. People scoring 69.9 and 70.1 may differ little in underlying skill, yet the rule changes their access sharply. A regression discontinuity design asks whether the outcome has a jump at that same threshold.

The argument rests on what would happen at 70 without the treatment rule. If expected untreated output changes smoothly through 70, a jump in observed output has a causal interpretation. The relevant comparison is local. Employees scoring 40 or 95 do not enter that interpretation simply because their records are available.

The sharp discontinuity result

Let $R$ be the running variable, $c=70$ the cutoff, and $D=1\{R\geq c\}$. Potential outcomes are $Y(0)$ and $Y(1)$, with observed $Y=DY(1)+(1-D)Y(0)$. Suppose the conditional means

$$\mu_d(r)=E[Y(d)\mid R=r],\quad d\in\{0,1\},$$

are continuous at $c$, with observations supported on both sides. Under consistency and no interference,

$$\begin{aligned} &\lim_{r\downarrow c}E[Y\mid R=r] -\lim_{r\uparrow c}E[Y\mid R=r]\\ &\quad=\mu_1(c)-\mu_0(c). \end{aligned}$$

The first limit approaches the threshold from above; the second approaches from below. Above the threshold we observe the treated conditional mean. Below it we observe the untreated conditional mean. Continuity carries each to the same score. Their difference is the treatment effect at the cutoff, using these continuous extensions.

Suppose those limits are 63 and 55 output units. The local effect is 8. If another benefit starts at 70, the jump may instead represent a bundle of changes. A clean-looking discontinuity cannot separate components whose assignment rules coincide.

With imperfect take-up, divide the outcome jump by the participation jump only under additional local IV assumptions, including exclusion, monotonicity, and a nonzero participation discontinuity. An outcome jump of 8 and participation jump of 0.40 give a local complier effect of 20. The target has changed from the sharp-design effect at the cutoff.

Estimating a limit from finite data

A local linear fit on each side estimates separate intercepts at the cutoff. For a given side, minimize

$$\begin{aligned} &\sum_{i\text{ on that side}}K\!\left(\frac{R_i-c}{h}\right)\\ &\qquad\times\{Y_i-a-b(R_i-c)\}^2. \end{aligned}$$

$K$ weights nearby observations and $h$ is the bandwidth. The estimated jump is the right intercept minus the left intercept. Separate slopes allow different local trends.

The bandwidth controls a real tradeoff. A wide neighborhood supplies more observations but may poorly approximate a curved conditional mean. A narrow neighborhood reduces approximation error but raises variance. Under suitable smoothness, density, moment, and independent-sampling conditions, local linear boundary bias is of order $h^2$, while variance is of order $1/(nh)$.

Balancing squared bias and variance gives $h$ of order $n^{-1/5}$. At that rate, bias and standard error both have order $n^{-2/5}$. Consequently, ignoring bias in a conventional interval is generally unjustified at a mean-squared-error-optimal bandwidth. Bias-aware inference, including properly constructed robust bias correction, addresses this problem. A plot with two fitted lines does not implement that inference. See Cattaneo, Idrobo, and Titiunik’s Foundations.

Continuity and local randomization

Continuity-based RD and local-randomization RD make different commitments. Continuity-based analysis takes limits as the score approaches the cutoff. Local randomization posits a window in which assignment behaves according to a specified randomization mechanism and potential outcomes satisfy the corresponding restrictions. Selecting a narrow window does not establish random assignment within it.

Inspect score manipulation, heaping, covariate jumps, other rules at the cutoff, and sensitivity to reasonable analysis choices. A density discontinuity may undermine the institutional story; a nonsignificant density test does not prove it. Discrete scores also demand care: if only integer values are possible, one cannot observe arbitrarily close scores on either side. Extrapolation or finite-window assumptions must then be made explicit.

Synthetic control

A different institution may introduce training throughout one region at once, leaving no score threshold. Synthetic control constructs a comparison from untreated regions:

$$\begin{gathered} \hat Y_{1t}(0)=\sum_{j=2}^{J+1}w_jY_{jt},\\ w_j\geq0,\quad\sum_jw_j=1. \end{gathered}$$

Weights are selected using pre-treatment information, often to approximate outcomes and predictors. Nonnegative weights summing to one make the comparison a convex combination of donors.

Consider this synthetic example:

Region	Pre 1	Pre 2	Pre 3	Post
Treated	45	47.5	50	60
Donor A	40	44	48	52
Donor B	60	58	56	54

Weights 0.75 on A and 0.25 on B reproduce all three pre-treatment observations. The synthetic post-treatment outcome is 52.5, leaving a gap of 7.5.

a, b = [40, 44, 48, 52], [60, 58, 56, 54]
synthetic = [0.75 * x + 0.25 * y for x, y in zip(a, b)]
print(synthetic)          # [45.0, 47.5, 50.0, 52.5]
print(60 - synthetic[-1]) # 7.5

Why might the comparison persist? In a factor model $Y_{it}(0)=\delta_t+\lambda_t'\mu_i+\varepsilon_{it}$, matching the treated region’s factor loadings $\mu_1$ with $\sum_jw_j\mu_j$ cancels the common factor component. Let $\mathcal F_{\mathrm{pre}}$ contain the pre-treatment information used to choose the weights. For a post-treatment date, also require

$$E\!\left[\varepsilon_{1t}-\sum_jw_j\varepsilon_{jt} \mid\mathcal F_{\mathrm{pre}}\right]=0.$$

Exact loading balance and this conditional restriction yield a zero expected untreated gap given the weight-selection information. Unconditional mean-zero disturbances alone are insufficient because fitted weights depend on pre-treatment outcomes. Good pre-fit may support the loading approximation. It does not prove it or rule out a new region-specific shock.

Donor contamination, spillovers, anticipation, and a post-treatment shock unique to the treated unit can all break the interpretation. Placebo comparisons are informative diagnostics; their inferential meaning depends on an exchangeability or assignment argument. Abadie’s methodological review emphasizes feasibility and contextual requirements.

Exercises

1. Training and a cash bonus both start at score 70. Output jumps by 8 units. What does the sharp RD identify?

Work through the solution

Under continuity and the other design assumptions, it identifies the local effect of eligibility for the combined intervention. It does not separately identify training’s effect. Separating the components requires further variation or substantive restrictions.

2. Every donor’s pre-treatment outcome lies below 40, while the treated region’s outcome is 50. Can nonnegative weights summing to one fit that date exactly?

Work through the solution

No. A convex combination cannot exceed the largest donor outcome. Exact fit would require a changed donor pool or a different model allowing extrapolation or adjustment. Such a change brings additional assumptions; it does not recover missing overlap by algebra alone.

Prediction and causal estimation

A model predicts employee output extremely well from past performance, job role, manager ratings, and training participation. That may help staffing. It does not establish what would happen if the firm changed training assignments.

Prediction evaluates a forecast against an observed outcome under a stated data distribution and loss. Causal estimation needs a target and identifying restrictions. Flexible learning is valuable within that argument: it can estimate complicated conditional relationships that would be difficult to specify by hand. It cannot supply missing randomization or remove unmeasured confounding by predictive accuracy alone.

Training and evaluation samples

For squared-error prediction, define risk as $E[(Y-\hat f(X))^2]$ for a new observation and the specified training procedure. More flexibility can reduce approximation bias while increasing estimation variability. The direction of the net change is an empirical question for honest evaluation.

Use training data to fit, validation data to select, and untouched evaluation data to estimate performance. Preprocessing, feature selection, and tuning belong inside the appropriate training split. Looking repeatedly at the final holdout and changing the model spends its independence. Split by the unit of dependence and by time when the deployment task requires it.

For causal work, excluding post-treatment variables and information unavailable at the decision date comes before a prediction contest.

A partial-linear target

Consider independent observations $W=(Y,D,X)$ satisfying

$$\begin{gathered} Y=\theta_0D+g_0(X)+\varepsilon,\\ E[\varepsilon\mid D,X]=0, \end{gathered}$$

where $D$ is training dose, $X$ contains pre-treatment covariates, and $\theta_0$ is a constant partial-linear coefficient. For a causal interpretation, the conditional mean restriction and structural interpretation must be credible, including the absence of relevant unmeasured confounding. This model is restrictive; arbitrary heterogeneous effects need different target definitions.

Define $m_0(X)=E[D\mid X]$, $\ell_0(X)=E[Y\mid X]$, and $V=D-m_0(X)$. Iterated expectations imply $\ell_0=\theta_0m_0+g_0$, hence

$$Y-\ell_0(X)=\theta_0V+\varepsilon.$$

If $J=E[V^2]>0$, multiplying by $V$ and taking expectations identifies $\theta_0$. Covariates may predict much of training, but some residual treatment variation must remain.

Why the score is orthogonal

Use the residual-on-residual score

$$\begin{aligned} &\psi(W;\theta,\ell,m)\\ &\quad=(D-m(X))\\ &\qquad\times\{Y-\ell(X)-\theta(D-m(X))\}. \end{aligned}$$

At the true functions and coefficient, $\psi=V\varepsilon$, with mean zero. Now perturb the nuisance functions along fixed square-integrable directions: $\ell_t=\ell_0+ta$ and $m_t=m_0+tb$. At $\theta_0$,

$$\psi_t=(V-tb)\{\varepsilon-ta+\theta_0tb\}.$$

Differentiate its expectation at zero:

$$\left.\frac{dE[\psi_t]}{dt}\right|_{t=0} =E[-b\varepsilon-Va+\theta_0Vb]=0.$$

Each term vanishes because $E[\varepsilon\mid X]=0$ and $E[V\mid X]=0$. This is Neyman orthogonality: small nuisance errors have no first-order effect on the population moment at the truth. It is a local insensitivity result, not immunity to arbitrary model errors.

What error remains?

Let $a=\hat\ell-\ell_0$ and $b=\hat m-m_0$ be nuisance errors trained on independent data. The same expansion gives the exact conditional population drift

$$E[\psi(W;\theta_0,\hat\ell,\hat m)] =E[ab]-\theta_0E[b^2].$$

By Cauchy–Schwarz, its magnitude is bounded by

$$\|a\|_2\|b\|_2+|\theta_0|\|b\|_2^2,$$

where $\|f\|_2=(E[f(X)^2])^{1/2}$. A random sequence $A_n$ is $o_p(r_n)$ when $A_n/r_n$ converges in probability to zero. Sufficient rate conditions make both terms $o_p(n^{-1/2})$. Taking both nuisance errors to be $o_p(n^{-1/4})$ suffices; other combinations are possible provided the product and the squared treatment-model error satisfy the condition. This parameterization should not be silently exchanged for a score using the structural function $g_0$.

Together with nuisance consistency, suitable moments and sampling conditions, and $J$ bounded away from zero, these restrictions support asymptotically linear inference. Orthogonality alone is insufficient.

Cross-fit, then estimate

Partition observations into a fixed number of folds. For every fold, estimate $\ell$ and $m$ using the other folds and predict the held-out observations. Let $\tilde Y_i=Y_i-\hat\ell_{-k(i)}(X_i)$ and $\tilde D_i=D_i-\hat m_{-k(i)}(X_i)$. Pool the held-out residuals:

$$\hat\theta=\frac{\sum_i\tilde D_i\tilde Y_i}{\sum_i\tilde D_i^2}.$$

Cross-fitting limits own-observation overfitting bias while reusing data. Under the required conditions,

$$\sqrt n(\hat\theta-\theta_0) =\frac{1}{\sqrt n}\sum_i\frac{V_i\varepsilon_i}{J}+o_p(1).$$

The asymptotic variance is $E[V^2\varepsilon^2]/J^2$. A corresponding standard-error estimate is

$$\begin{gathered} \widehat{SE}^2=\frac{n^{-1}\sum_i\hat\psi_i^2}{n\hat J^2},\\ \hat J=n^{-1}\sum_i\tilde D_i^2,\\ \hat\psi_i=\tilde D_i(\tilde Y_i-\hat\theta\tilde D_i). \end{gathered}$$

Dependence or clustered assignment needs an appropriate extension. None of these calculations measures uncertainty from omitted confounding.

Residual regression example

Suppose held-out residuals in a synthetic example are $\tilde D=(-2,-1,1,2)$ and $\tilde Y=(-5,-1,2,4)$. Their cross-product sums to 21 and squared treatment residuals sum to 10, so $\hat\theta=2.1$.

d, y = [-2, -1, 1, 2], [-5, -1, 2, 4]
theta = sum(a * b for a, b in zip(d, y)) / sum(a * a for a in d)
print(theta)  # 2.1

This example calculates the final residual regression. A full DML analysis also requires nuisance estimation, cross-fitting, and inference.

Exercises

1. Suppose $\|a\|_2$ has order $n^{-0.3}$ and $\|b\|_2$ has order $n^{-0.2}$. Do the displayed sufficient remainder conditions follow?

Work through the solution

No. Their product has order $n^{-0.5}$, which alone does not give the required little-o rate. More importantly, $\|b\|_2^2$ has order $n^{-0.4}$, generally too slow when $\theta_0\ne0$. Faster treatment-model learning or a different valid construction is needed.

2. Managers use unrecorded motivation to select training. Cross-fitted output prediction is excellent. Which identifying condition remains doubtful?

Work through the solution

If motivation affects output beyond recorded $X$, the disturbance may satisfy $E[\varepsilon\mid D,X]\ne0$. Cross-fitting cannot make that conditional mean restriction true. Seek additional design information, a valid instrument, sensitivity analysis, or a more limited predictive claim.

Further reading

Chernozhukov and coauthors’ original DML paper provides the framework and regularity conditions. Ahrens and coauthors’ introduction, revised in February 2026, is a current explanatory companion to this established methodology.