Conditional expectation and composition

Group composition and aggregate averages

Suppose a report says that 68% of training participants meet their output target, compared with 32% of nonparticipants. Before explaining the difference, find out who is inside each denominator.

Here is a completely constructed population of 100 employees. Let $D=1$ mean participation, $K=1$ mean prior experience, and $Y=1$ mean meeting the output target. Each entry describes people, not estimated probabilities.

Prior experience	Participation	Employees	Meet target
Experienced	Participate	40	32
Experienced	Do not participate	10	8
New	Participate	10	2
New	Do not participate	40	8

Among experienced employees, the success rate is 80% in both participation groups. Among new employees, it is 20% in both groups. Yet participants are mostly experienced, and nonparticipants are mostly new.

The aggregate difference is 36 percentage points. Every within-experience difference is zero. This is a composition calculation; a causal interpretation will require additional assumptions.

Conditioning changes the reference group

Imagine drawing one employee uniformly from the table. Probability describes the resulting uncertainty. For events $A$ and $B$ with $P(B)>0$,

$$P(A\mid B)=\frac{P(A\cap B)}{P(B)}.$$

Conditioning on participation restricts attention to the 50 participants. Forty are experienced, so $P(K=1\mid D=1)=0.8$. The probability of meeting the target is

$$\begin{aligned} &P(Y=1\mid D=1)\\ &\quad=P(Y=1\mid K=1,D=1)\\ &\qquad\times P(K=1\mid D=1)\\ &\qquad+P(Y=1\mid K=0,D=1)\\ &\qquad\times P(K=0\mid D=1)\\ &\quad=0.8(0.8)+0.2(0.2)\\ &\quad=0.68. \end{aligned}$$

The same calculation for nonparticipants gives $0.8(0.2)+0.2(0.8)=0.32$. The within-group success rates are identical. The weights differ.

Probabilities with different conditioning information answer different questions. $P(K=1\mid D=1)=0.8$ says how many participants are experienced. $P(D=1\mid K=1)=0.8$ says how many experienced employees participate. They happen to agree here because both marginal group sizes are 50. That coincidence is not a general symmetry.

Expectation is a weighted sum

For a discrete variable $Y$, expectation is

$$E[Y]=\sum_y yP(Y=y).$$

The expectation of a zero-one variable equals its probability of being one. Expectations of income, output, and losses use the same rule, with the outcome values included in the weighted sum. For continuous variables, integration replaces summation.

The law of iterated expectations says, for an integrable outcome,

$$E[Y]=E\{E[Y\mid K]\}.$$

Here integrable means $E[|Y|]<\infty$. In the table, half the workforce is experienced:

$$E[Y]=0.5(0.8)+0.5(0.2)=0.5.$$

You can also count directly: $32+8+2+8=50$ successes among 100 employees. Conditioning reorganizes the calculation without changing the overall population average, provided the outer weights are the correct population weights.

Standardizing both participation groups to a half-experienced, half-new workforce gives a success rate of 0.5 for each, so the standardized contrast is zero. That zero measures the causal effect of participation only under three conditions. First, each employee’s recorded success must be the success that employee would have under the participation status actually recorded. This causal condition is called consistency; chapter 5 states it with potential outcomes, and it is unrelated to the estimator property of the same name in chapter 1. Second, each experience group must contain both participants and nonparticipants, a condition called overlap; here the experienced employees split 40 and 10, and the new employees 10 and 40. Third, nothing besides experience may influence both participation and success. The table can check the second condition. It cannot check the first or the third, and no identity about probabilities can certify them.

Where variation comes from

Let $m(K)=E[Y\mid K]$. If $E[Y^2]<\infty$, then

$$\begin{aligned} \operatorname{Var}(Y) &=E\{\operatorname{Var}(Y\mid K)\}\\ &\quad+\operatorname{Var}\{m(K)\}. \end{aligned}$$

The first term is variation within experience groups. The second is variation in their conditional means. In our population, $Y$ is binary with mean 0.5, so its variance is $0.5(1-0.5)=0.25$. Within either experience group, the variance is $0.8(0.2)=0.16$. The group means lie 0.3 above or below 0.5, so their variance is $0.09$. Thus $0.25=0.16+0.09$.

The Distribution Lab opens on a binomial distribution with one trial and success probability 0.5, which is this binary outcome. Its bench reports the mean 0.5 and the variance 0.25. Moving the success probability to 0.8 reproduces the within-group variance of 0.16. The same bench covers eighteen other distributions.

Why the decomposition works

Write

$$\begin{aligned} Y-E[Y]&=\{Y-m(K)\}\\ &\quad+\{m(K)-E[Y]\}. \end{aligned}$$

Square and take expectations. The cross term is zero because

$$\begin{aligned} &E[(Y-m(K))(m(K)-E[Y])]\\ &\quad=E[(m(K)-E[Y])\\ &\qquad\times E[Y-m(K)\mid K]]\\ &\quad=0. \end{aligned}$$

The remaining two squared terms are precisely the within-group and between-group components.

This also explains why the conditional mean minimizes mean squared prediction error among functions of $K$. For any square-integrable $g(K)$,

$$\begin{aligned} &E[(Y-g(K))^2]\\ &\quad=E[(Y-m(K))^2]\\ &\qquad+E[(m(K)-g(K))^2]. \end{aligned}$$

The second term cannot be negative. This is a prediction result and makes no causal claim.

A census, a sample, and a missing employee

The table is a finite population we constructed. Its proportions are exact. If a survey samples 20 of these employees, its proportions become random because a different sample could contain different people. If the intended population is next year’s workforce, even a census of these 100 employees leaves a transport question.

Now suppose experienced employees answer a follow-up survey with probability 0.8 and new employees with probability 0.2. Assuming response is independent of success within experience groups, the respondent population is 80% experienced and has a success rate of 68%. The whole eligible population still has a 50% success rate.

A million responses produced by the same selection mechanism would estimate the respondent rate with great precision. It would not turn respondents into a representative workforce.

Calculating the weighted average

# Standard Python; all quantities describe the synthetic table.
rates = [0.8, 0.2]  # experienced, new
weights = {
    "participants": [0.8, 0.2],
    "nonparticipants": [0.2, 0.8],
    "eligible population": [0.5, 0.5],
}
for name, w in weights.items():
    print(name, sum(p*q for p, q in zip(rates, w)))

Exercises

1. A future workforce is 75% experienced. Keep the two group-specific success rates fixed. What success rate should you predict?

Work through the solution

Use the future population weights: $0.75(0.8)+0.25(0.2)=0.65$. Using 0.5 would silently assume the old composition. Using 0.68 would silently use participant composition. Keeping the group-specific rates fixed is itself a transport assumption.

2. Does zero covariance between two variables establish independence?

Work through the solution

No. Let $X$ be equally likely to equal $-1,0,1$ and let $Y=X^2$. Then $E[X]=0$ and $E[XY]=E[X^3]=0$, so $\operatorname{Cov}(X,Y)=0$. But $Y$ is determined by $X$: for example, $P(Y=0\mid X=0)=1$, whereas $P(Y=0)=1/3$. A zero linear association can conceal complete nonlinear dependence.

Further reading

Blitzstein and Hwang’s Introduction to Probability develops conditioning, expectation, and their calculations. For the econometric use of conditional expectations and projections, see Bruce Hansen’s Econometrics.