Probability

A screening test is $99\%$ accurate in both directions, and the disease it looks for affects $1$ person in $10,000$. You test positive. What is the probability that you have the disease?

Almost everyone answers $99\%$. The correct answer is about $1\%$.

Imagine a million people. About $100$ of them have the disease, and the test catches $99$ of those. The other $999,900$ are healthy, and the test wrongly flags $1\%$ of them — nearly $10,000$ people. So of roughly $10,099$ positive results, only $99$ are genuine:

$P\left(D\mid +\right)=\frac{99}{99+9999}\approx \mathrm{0.0098.}$

The accuracy of the test was never in question. What the naive answer ignores is the base rate: healthy people are so much more numerous that even a small error rate among them swamps the true positives.

This is the shape of nearly every hard probability question at Advanced level. The information given is a conditional probability in one direction, and the question asks for it in the other. Those two are not the same number, and Bayes' theorem is the only bridge between them.

1. Sample spaces and the assumption inside them

A probability model is a sample space together with a rule assigning probabilities that are non-negative and sum to $1$. Counting favourable outcomes and dividing by the total is legitimate only when the outcomes listed are equally likely, and choosing a sample space in which they are is half the work.

Rolling two dice illustrates the point. If the sample space is the $21$ unordered pairs, they are not equally likely: $\{1,2\}$ arises two ways and $\{1,1\}$ only one. Using the $36$ ordered pairs restores equal likelihood, and every standard result about two dice assumes that space.

Illustration 1

Two dice are thrown. Find the probability that the sum is $7$, and that the sum is $2$.

In the $36$-outcome space, six ordered pairs sum to $7$, giving $\frac{1}{6}$, and one sums to $2$, giving $\frac{1}{36}$. In the $21$-outcome space the naive counts would give $\frac{3}{21}$ and $\frac{1}{21}$, both wrong. The arithmetic was never the difficulty; the sample space was.

2. Adding, multiplying, and two words that are not synonyms

For any two events, $P\left(A\cup B\right)=P(A)+P(B)-P\left(A\cap B\right)$, and for three the inclusion-exclusion pattern of the counting chapter reappears unchanged.

Two events are mutually exclusive when $P\left(A\cap B\right)=0$ and independent when $P\left(A\cap B\right)=P(A)P(B)$. These are not the same idea, and they are close to opposites: if two events both have positive probability, then being mutually exclusive makes them dependent, because knowing that one occurred tells you the other did not.

Illustration 2

$A$ and $B$ satisfy $P(A)=0.4$, $P(B)=0.5$ and $P\left(A\cup B\right)=0.7$. Are they independent? Are they mutually exclusive?

From the addition rule, $P\left(A\cap B\right)=0.4+0.5-0.7=0.2$. Since $P(A)P(B)=0.2$ as well, they are independent. They are certainly not mutually exclusive, since the intersection has positive probability — and indeed no pair of events with positive probabilities can be both.

3. Conditional probability

The conditional probability $P\left(A\mid B\right)=\frac{P\left(A\cap B\right)}{P(B)}$ is simply probability computed inside a shrunken sample space: $B$ becomes the new universe, and everything is rescaled so that it has probability $1$.

Rearranged, this gives the multiplication rule $P\left(A\cap B\right)=P(B)P\left(A\mid B\right)$, which is how sequential experiments — drawing cards without replacement, choosing a bag then a ball — are built up one stage at a time.

Illustration 3

Two cards are drawn without replacement from a standard pack. Find the probability that both are kings, and the probability that the second is a king.

For both, $\frac{4}{52}\cdot \frac{3}{51}=\frac{1}{221}$.

For the second alone, condition on the first: $\frac{4}{52}\cdot \frac{3}{51}+\frac{48}{52}\cdot \frac{4}{51}=\frac{12+192}{2652}=\frac{204}{2652}=\frac{1}{13}$.

The second answer equals the probability that the first is a king. That is not a coincidence: before any card is seen, every position in the shuffled pack is equally likely to hold a king, so the order of drawing is irrelevant. Recognising this symmetry answers many questions without any conditioning at all.

Illustration 4

A family has two children, at least one of whom is a boy. What is the probability that both are boys?

The sample space of two children in birth order is $\{BB,BG,GB,GG\}$, all equally likely. Conditioning on at least one boy removes $GG$, leaving three outcomes of which one is $BB$, so the answer is $\frac{1}{3}$.

The instinctive answer $\frac{1}{2}$ treats the information as being about a specific child. Had the question said "the elder is a boy", the space would reduce to $\{BB,BG\}$ and the answer would indeed be $\frac{1}{2}$. Which child the information identifies changes the answer, and questions are worded to test exactly that.

4. Probability by counting

When every outcome of an experiment is equally likely, a probability is a ratio of two counts, and the whole of the previous chapter becomes available. The discipline is the same as it was there: the numerator and the denominator must be counted in the same way, either both as ordered arrangements or both as unordered selections.

Two phrasings recur. "Exactly $k$ of a kind" is a product of two binomial coefficients over one, and "at least one" is almost always shortest through the complement, since the opposite of "at least one" is the single event "none".

It is also worth knowing the language of odds, which questions occasionally use without explanation. Odds in favour of an event are the ratio of favourable to unfavourable cases, so odds of $a:b$ correspond to a probability of $\frac{a}{a+b}$, not $\frac{a}{b}$.

Illustration 5

A bag contains $5$ white and $4$ black balls, and three are drawn together. Find the probability that exactly two are white.

Choosing $2$ of the $5$ white and $1$ of the $4$ black, out of all ways to choose $3$ from $9$:

$\frac{\left(\genfrac{}{}{0ex}{}{5}{2}\right)\left(\genfrac{}{}{0ex}{}{4}{1}\right)}{\left(\genfrac{}{}{0ex}{}{9}{3}\right)}=\frac{10\times 4}{84}=\frac{10}{21}.$

Both counts are unordered, which is what makes the ratio legitimate. Counting the numerator as ordered and the denominator as unordered is the standard way to produce an answer greater than $1$.

Illustration 6

Four people enter a lift in a building with five floors above the ground. Assuming each chooses a floor at random and independently, find the probability that they all get out at different floors.

The total number of choice patterns is $5^4=625$, all equally likely. Those with all floors distinct number $5\times 4\times 3\times 2=120$, so the probability is

$\frac{120}{625}=\frac{24}{125}\approx \mathrm{0.192.}$

Here both counts are ordered, since the people are distinguishable. The complement, that at least two share a floor, has probability about $0.81$ — the same surprisingly high figure that makes the classic birthday problem counterintuitive.

5. Total probability and Bayes

If $E_1,\dots ,E_n$ partition the sample space, then any event $A$ decomposes along that partition:

$P(A)={\sum }_{i}P\left(E_i\right)P\left(A\mid E_i\right).$

Bayes' theorem reverses the conditioning by dividing one term by that whole sum:

$P\left(E_k\mid A\right)=\frac{P\left(E_k\right)P\left(A\mid E_k\right)}{{\sum }_{i}P\left(E_i\right)P\left(A\mid E_i\right)}.$

The reliable way to apply it is a tree: branch on the partition, multiply along each path to get the probability of that path, and then compare the paths that end in $A$.

Illustration 7

Bag I holds $3$ red and $4$ black balls; Bag II holds $5$ red and $6$ black. A bag is chosen at random and a ball drawn from it turns out to be red. Find the probability that it came from Bag I.

The two paths ending in "red" have probabilities $\frac{1}{2}\cdot \frac{3}{7}=\frac{3}{14}$ and $\frac{1}{2}\cdot \frac{5}{11}=\frac{5}{22}$. So

$P\left(\text{I}\mid \text{red}\right)=\frac{\frac{3}{14}}{\frac{3}{14}+\frac{5}{22}}=\frac{\frac{33}{154}}{\frac{33}{154}+\frac{35}{154}}=\frac{33}{68}.$

The answer is a little below $\frac{1}{2}$, which is right: Bag II has the higher proportion of red balls, so a red draw is weak evidence in its favour.

Illustration 8

A factory has three machines producing $50\%$, $30\%$ and $20\%$ of output, with defect rates $3\%$, $4\%$ and $5\%$. A defective item is found. Which machine most probably made it?

The path probabilities are $0.5(0.03)=0.015$, $0.3(0.04)=0.012$ and $0.2(0.05)=0.010$, totalling $0.037$. So the posterior probabilities are $\frac{15}{37}$, $\frac{12}{37}$ and $\frac{10}{37}$, and the first machine is most likely despite having the lowest defect rate — because it makes the most items. This is the base-rate effect of the opening in a milder form.

6. Independence of three or more events

Three events are mutually independent when every pair multiplies correctly and, in addition, $P\left(A\cap B\cap C\right)=P(A)P(B)P(C)$. The last condition does not follow from the others.

Illustration 9

Toss two fair coins. Let $A$ be "first is heads", $B$ be "second is heads" and $C$ be "the two match". Show that these are pairwise independent but not mutually independent.

Each has probability $\frac{1}{2}$. Each pairwise intersection is a single outcome of the four, so has probability $\frac{1}{4}=\frac{1}{2}\cdot \frac{1}{2}$: all three pairs are independent.

But $A\cap B\cap C$ is the single outcome $HH$, with probability $\frac{1}{4}$, while $P(A)P(B)P(C)=\frac{1}{8}$. The triple condition fails. Knowing any one of the three tells you nothing, but knowing two tells you the third exactly.

A useful consequence is worth stating on its own. If $A$ and $B$ are independent, then so are $A$ and $B^{\prime }$, and so are $A^{\prime }$ and $B^{\prime }$ — independence survives complementation.

That is what licenses the standard treatment of "at least one" in repeated independent trials: the probability that none of $n$ independent events with probabilities $p_i$ occurs is $\prod \left(1-p_i\right)$, and one minus that is the answer. Applying the same product rule to events that are merely pairwise independent, or not independent at all, is a genuine error rather than an approximation.

7. Random variables, expectation and variance

A random variable assigns a number to each outcome. Its expectation is the probability-weighted average $E(X)=\sum x_i{p}_i$, and its variance is

$\mathrm{Var}(X)=E\left(X^2\right)-{\left[E(X)\right]}^2,$

which is almost always faster to compute than the definition as a mean squared deviation.

Expectation is linear whatever the dependence between variables: $E(X+Y)=E(X)+E(Y)$ always. Variance adds only when the variables are independent.

Illustration 10

A die is thrown once and $X$ is the score. Find $E(X)$ and $\mathrm{Var}(X)$.

$E(X)=\frac{1+2+\cdots +6}{6}=3.5$ and $E\left(X^2\right)=\frac{1+4+9+16+25+36}{6}=\frac{91}{6}$, so

$\mathrm{Var}(X)=\frac{91}{6}-{\left(\frac{7}{2}\right)}^2=\frac{182-147}{12}=\frac{35}{12}.$

Note that $E(X)=3.5$ is not a possible score. An expectation is a long-run average, not a prediction of any single trial.

Illustration 11

Two fair dice are thrown. Find the expected value of the sum.

By linearity, $E\left(X_1+X_2\right)=3.5+3.5=7$, with no need to enumerate the $36$ outcomes or their eleven possible sums. Linearity holds even for dependent variables, which is what makes it the most economical tool in the chapter.

8. The binomial distribution

If a trial with success probability $p$ is repeated $n$ times independently, the number of successes $X$ satisfies

$P(X=r)=\left(\genfrac{}{}{0ex}{}{n}{r}\right)p^r{q}^{n-r},q=1-p,$

with $E(X)=np$ and $\mathrm{Var}(X)=npq$. Since $q<1$, the variance of a binomial variable is always less than its mean — a quick check that rules out impossible data.

Illustration 12

A biased coin shows heads with probability $\frac{1}{3}$. It is tossed $5$ times. Find the probability of at least $4$ heads.

$P(X=4)+P(X=5)=\left(\genfrac{}{}{0ex}{}{5}{4}\right){\left(\frac{1}{3}\right)}^4\frac{2}{3}+{\left(\frac{1}{3}\right)}^5=\frac{10}{243}+\frac{1}{243}=\frac{11}{243}.$

Computing "at least $4$" directly is shorter than the complement here, because only two terms are involved. For "at least $1$" the complement is always shorter, since it is a single term.

Illustration 13

A binomial variable has mean $8$ and variance $4$. Find $n$ and $p$.

From $np=8$ and $npq=4$, dividing gives $q=\frac{1}{2}$, so $p=\frac{1}{2}$ and $n=16$.

If instead the variance had been given as $10$, the data would be impossible: variance can never exceed the mean for a binomial variable, and noticing that saves a fruitless calculation.

Illustration 14

Six dice are thrown. Find the probability of getting at least two sixes.

With $p=\frac{1}{6}$ and $n=6$, use the complement:

$1-P(X=0)-P(X=1)=1-{\left(\frac{5}{6}\right)}^6-6\left(\frac{1}{6}\right){\left(\frac{5}{6}\right)}^5=1-\frac{5^6}{6^6}-\frac{5^5}{6^5}.$

Numerically this is $1-0.3349-0.4019\approx 0.263$. The expected number of sixes is $np=1$, so a result a little above a quarter for "two or more" is the right order of magnitude.

Summary

A conditional probability in one direction is not the same number as the conditional probability in the other, and Bayes' theorem is the only bridge. When one branch of a partition is far more populous than another, its errors can outnumber the other branch's successes, which is why a $99\%$ accurate test for a rare condition usually raises a false alarm.

Counting favourable cases over total cases is valid only in a sample space of equally likely outcomes, and constructing such a space — ordered pairs rather than unordered, for two dice — is often the whole difficulty.

Mutually exclusive and independent are close to opposite conditions: events with positive probabilities cannot be both, since disjointness makes each one informative about the other. For three events, pairwise independence does not give mutual independence, as two coins and the event that they match demonstrate.

Where outcomes are equally likely, a probability is a ratio of counts, and the numerator and denominator must be counted the same way — both ordered or both unordered. Odds of $a:b$ mean a probability of $\frac{a}{a+b}$.

Total probability decomposes an event along a partition and Bayes reverses the conditioning; a tree makes both mechanical. Expectation is linear whatever the dependence, which frequently removes the need to enumerate anything, while variance adds only for independent variables and is computed as $E\left(X^2\right)-{\left[E(X)\right]}^2$. For a binomial variable the mean is $np$ and the variance $npq$, so the variance is always the smaller of the two.