Sampling uncertainty and standard errors

The sampling process

An estimate is a number from one dataset. A standard error describes how that estimate would vary across repetitions of a specified sampling or assignment process.

That process matters. Repeatedly sampling employees differs from repeatedly randomizing training across the same employees. Sampling 1,000 employees from one office differs from sampling one employee from each of 1,000 independent offices.

A standard-error calculation depends on which quantities are fixed and which are random across repetitions.

A case where the interval is exact

Let $Y_1,\ldots,Y_n$ be independent draws from a normal population with unknown mean $\mu$ and known standard deviation $\sigma$. Then

$$\bar Y\sim N\left(\mu,\frac{\sigma^2}{n}\right), \qquad \frac{\bar Y-\mu}{\sigma/\sqrt n}\sim N(0,1).$$

Let $z_{0.975}\approx1.959964$ be the standard normal 97.5th percentile. The interval

$$C(Y)= \left[\bar Y-z_{0.975}\frac{\sigma}{\sqrt n}, \bar Y+z_{0.975}\frac{\sigma}{\sqrt n}\right]$$

contains $\mu$ in 95% of repeated samples. The sampling experiment deliberately uses this known-$\sigma$, Gaussian setting. Its simplicity lets you inspect coverage without mixing in estimated variance or asymptotic approximations.

With $\bar Y=2.4$, $\sigma=4$, and $n=100$, the standard error is $0.4$. The interval is approximately $[1.616,3.184]$.

The parameter is fixed in this frequentist calculation; the interval is random before sampling. After observing a particular interval, the procedure’s 95% coverage does not become a 95% posterior probability that $\mu$ lies in that interval. A Bayesian probability statement would require a specified probability model for the unknown parameter.

If $\sigma$ is unknown and the observations remain independent normal draws, replacing it with the sample standard deviation yields a Student $t$ statistic with $n-1$ degrees of freedom. Using a normal critical value with an estimated standard error is usually an approximation, not that exact result. The Constructions tab of the Distribution Lab builds the $t$ distribution draw by draw from a standard normal and an independent chi-square, which shows where its heavier tails come from.

The limits behind large-sample reasoning

For independent, identically distributed observations with $E[|Y_i|]<\infty$, the law of large numbers gives $\bar Y\to_p\mu$. With finite, positive variance as well, the central limit theorem gives

$$\sqrt n(\bar Y-\mu)\ \xrightarrow{d}\ N(0,\sigma^2).$$

Convergence in probability concerns closeness to a target. Convergence in distribution concerns the shape of scaled fluctuations. Neither statement says every dataset of size 30, or 300, behaves approximately normally. Heavy tails, rare influential observations, and dependence can make an approximation poor. The lab’s central-limit construction averages draws from skewed and heavy-tailed sources, including a Cauchy source whose average never settles; the normal-errors panel of the OLS assumptions lab checks a regression test under skewed errors.

The same distinction carries into regression. Under suitable sampling and moment conditions, a coefficient becomes increasingly precise around its population projection. That projection can differ from the causal parameter a manager cares about.

Confidence-interval coverage under bias

Suppose the desired target is $\theta$, but the observations satisfy

$$Y_i=\theta+b+\varepsilon_i,\qquad \varepsilon_i\overset{\mathrm{iid}}{\sim}N(0,\sigma^2).$$

The persistent bias is $b$. An analyst mistakenly reports the ordinary mean interval as an interval for $\theta$. Its actual coverage is

$$\begin{aligned} P(\theta\in C) &=\Phi\left(z_{0.975}-\frac{b\sqrt n}{\sigma}\right)\\ &\quad-\Phi\left(-z_{0.975}-\frac{b\sqrt n}{\sigma}\right), \end{aligned}$$

where $\Phi$ is the standard normal cumulative distribution function.

Take $b=1$ and $\sigma=4$. Coverage is about 29.5% at $n=100$ and 0.12% at $n=400$. With nonzero fixed bias, coverage approaches zero as the sample grows. The interval becomes excellent at locating $\theta+b$ and terrible at covering $\theta$.

# Standard Python; 1.96 approximates the exact normal quantile.
from math import erf, sqrt
Phi = lambda z: (1 + erf(z/sqrt(2))) / 2

for n in (25, 100, 400):
    bias_in_se = sqrt(n) / 4
    coverage = Phi(1.96-bias_in_se) - Phi(-1.96-bias_in_se)
    print(n, round(coverage, 4))

Match the standard error to the dependence

Heteroskedasticity allows the error variance to differ across observations. For independent, identically distributed observations, let $\mathbf x_i$ be the regressor vector and $u_i$ the population projection error. The conditional error variance can still vary with $\mathbf x_i$. If the relevant moments exist, $Q=E[\mathbf x_i\mathbf x_i']$ is nonsingular, and a suitable central limit theorem applies,

$$\begin{gathered} \sqrt n(\widehat{\boldsymbol\beta}-\boldsymbol\beta) \xrightarrow{d} N(0,Q^{-1}\Omega Q^{-1}),\\ \Omega=E[\mathbf x_i\mathbf x_i'u_i^2]. \end{gathered}$$

The sample counterpart is the heteroskedasticity-robust sandwich variance estimator. Its job is to estimate sampling variance under unequal variances. It does not correct selection, measurement error, or omitted-variable bias. The constant-variance panel of the OLS assumptions lab shows how far the conventional standard error can understate the spread of a slope when error variance grows with a skewed regressor.

Employees within an office may share shocks. A cluster-robust calculation aggregates score contributions within office:

$$\begin{aligned} \widehat{\operatorname{Var}}(\widehat{\boldsymbol\beta}) &=(\mathbf X'\mathbf X)^{-1} \left(\sum_{g=1}^G\mathbf S_g\mathbf S_g'\right)\\ &\qquad\times(\mathbf X'\mathbf X)^{-1},\\ \mathbf S_g&=\sum_{i\in g}\mathbf x_i\widehat u_i. \end{aligned}$$

This displays the uncorrected form; software often adds finite-sample adjustments. The standard justification requires sufficiently many independent clusters and conditions preventing a few clusters from dominating. A large employee count cannot substitute for those conditions. The lab’s independent-errors panel shows the analogous failure for serially correlated errors: the slope stays centered while the conventional standard error understates its spread.

As an illustration for a simple mean with equally sized independent clusters, cluster size $m=25$ and within-cluster correlation $\rho=0.1$ inflate variance relative to independent observations by $1+(m-1)\rho=3.4$. This formula depends on that exchangeable covariance structure; it is not a universal regression correction.

Exercises

1. How much must sample size increase to halve the standard error of an independent sample mean, keeping population variance fixed?

Work through the solution

The standard error is $\sigma/\sqrt n$. Replacing $n$ with $4n$ divides it by two. Doubling $n$ multiplies it by $1/\sqrt2\approx0.707$. The claim assumes the additional observations follow the same independent sampling process.

2. A training estimate is 3 with a standard error of 0.1. A credible selection mechanism could contribute between 1 and 4 output units to that association. Can the narrow interval establish a positive causal effect?

Work through the solution

No. Ignoring sampling variation for the moment, subtracting the selection contribution gives causal effects between $3-4=-1$ and $3-1=2$. Sampling uncertainty would add another layer. Reporting $3\pm1.96(0.1)$ addresses precision around the association and leaves the identification uncertainty unresolved.

Further reading

Hansen’s Econometrics develops asymptotic regression theory. Cameron and Miller’s A Practitioner’s Guide to Cluster-Robust Inference explains why the clustering level and the number of clusters deserve substantive attention.