1.9 Bayes’ Rule

Key Terms

Bayes’ rule（貝氏定理）

A rule for finding a reversed conditional probability.

Prior probability（先驗機率）

The probability of a possible case before the new event is observed.

Posterior probability（後驗機率）

The updated probability of a possible case after the new event is observed.

Possible case（可能情況）

One event in a partition, such as \(B_1,B_2,\ldots,B_k\).

Observed event（觀察到的事件）

The event that is known to have occurred.

1.9.2 Learning Outcomes

After completing this section, students should be able to:

• explain why \(P(A\mid B)\) and \(P(B\mid A)\) answer different questions;

• recognize when the condition in a probability question has been reversed;

• use Bayes’ rule for two or more possible cases;

• calculate the overall probability needed in the denominator using total probability;

• interpret the numerator of Bayes’ rule as one desired path;

• interpret the denominator as all possible paths leading to the observed event;

• distinguish prior probability from posterior probability;

• explain how an observed event can update a starting probability;

• connect total probability and Bayes’ rule.

1.9.3 Reverse the Condition

Conditional probability depends on which event is already known.

For example,

\[P(H\mid C_2)\]

means:

probability of heads, given that Coin \(C_2\) was chosen.

Here, the coin is known first.

The direction is

known coin -> probability of heads.

Now consider

\[P(C_2\mid H).\]

This means:

probability that Coin \(C_2\) was chosen, given that heads was observed.

Here, heads is known first.

The direction is

observed heads -> probability of the coin.

These two probabilities answer different questions.

Note

The condition has been reversed

\[P(H\mid C_2)\]

means probability of heads given \(C_2\).

\[P(C_2\mid H)\]

means probability of \(C_2\) given heads.

In general,

\[P(H\mid C_2)\neq P(C_2\mid H).\]

The event after the vertical bar is always the information that is already known.

1.9.4 Example 1: Choose One of Two Coins

Suppose one of two coins is chosen at random.

• \(C_1\) is a fair coin.

• \(C_2\) is a double-headed coin.

The selected coin is tossed once.

Because \(C_2\) is double-headed,

\[P(H\mid C_2)=1.\]

This is a forward conditional probability.

It asks:

If Coin \(C_2\) was selected, how likely is heads?

Now suppose instead that heads is observed, but we do not know which coin was selected.

The question becomes

\[P(C_2\mid H).\]

This asks:

Given that heads was observed, how likely is it that Coin \(C_2\) was selected?

This is the type of reversed conditional probability that Bayes’ rule helps us calculate.

1.9.5 Bayes’ Rule

Start with the conditional probability formula:

\[P(C_2\mid H) = \frac{P(C_2\cap H)}{P(H)}.\]

From the multiplication rule,

\[P(C_2\cap H) = P(C_2)P(H\mid C_2).\]

Substituting this into the conditional probability formula gives

\[P(C_2\mid H) = \frac{P(C_2)P(H\mid C_2)} {P(H)}.\]

Note

Bayes’ Rule

\[P(C_2\mid H) = \frac{P(C_2)P(H\mid C_2)} {P(H)}.\]

The numerator is the probability of one particular path:

\[C_2\rightarrow H.\]

The denominator is the overall probability of heads from all possible paths.

A useful way to remember the structure is:

desired path to the observed event

divided by

all paths to the observed event

1.9.6 Example 2: Which Coin Was Chosen?

One of two coins is chosen at random.

• Coin \(C_1\) is fair.

• Coin \(C_2\) is double-headed.

Each coin is selected with probability

\[P(C_1)=\frac{1}{2}\]

and

\[P(C_2)=\frac{1}{2}.\]

The selected coin is tossed and heads is observed.

Find

\[P(C_2\mid H).\]

From Section 1.8, the overall probability of heads is

\[P(H)=\frac{3}{4}.\]

The probability of the path

\[C_2\rightarrow H\]

is

\[P(C_2\cap H) = P(C_2)P(H\mid C_2).\]

Since

\[P(C_2)=\frac{1}{2}\]

and

\[P(H\mid C_2)=1,\]

we get

\[P(C_2\cap H) = \frac{1}{2}\times1 = \frac{1}{2}.\]

Now use conditional probability:

\[P(C_2\mid H) = \frac{P(C_2\cap H)}{P(H)}.\]

Therefore,

\[P(C_2\mid H) = \frac{1/2}{3/4} = \frac{2}{3}.\]

So, after observing heads, the probability that the double-headed coin was selected is

\[\frac{2}{3}.\]

Notice that

\[P(H\mid C_2)=1\]

but

\[P(C_2\mid H)=\frac{2}{3}.\]

The two conditional probabilities are not the same.

1.9.7 General Form

Suppose the possible cases are

\[B_1,B_2,\ldots,B_k,\]

and these cases form a partition of the sample space.

Now suppose event \(A\) is observed.

We want the probability of one particular case \(B_j\) after observing \(A\).

Starting from conditional probability,

\[P(B_j\mid A) = \frac{P(B_j\cap A)}{P(A)}.\]

Using the multiplication rule,

\[P(B_j\cap A) = P(B_j)P(A\mid B_j).\]

Therefore,

\[P(B_j\mid A) = \frac{P(B_j)P(A\mid B_j)} {P(A)}.\]

Note

Bayes’ Rule

\[P(B_j\mid A) = \frac{P(B_j)P(A\mid B_j)} {P(A)}.\]

The numerator is the desired path

\[B_j\rightarrow A.\]

The denominator is the probability of observing \(A\) through all possible cases.

1.9.8 Bayes’ Rule for Several Cases

From Section 1.8, if

\[B_1,B_2,\ldots,B_k\]

form a partition, then total probability gives

\[P(A) = \sum_{i=1}^{k}P(B_i)P(A\mid B_i).\]

Substituting this expression for \(P(A)\) into Bayes’ rule gives

\[P(B_j\mid A) = \frac{ P(B_j)P(A\mid B_j) }{ \displaystyle \sum_{i=1}^{k}P(B_i)P(A\mid B_i) }.\]

Note

Bayes’ Rule for Several Cases

\[P(B_j\mid A) = \frac{ P(B_j)P(A\mid B_j) }{ \displaystyle \sum_{i=1}^{k}P(B_i)P(A\mid B_i) }.\]

The interpretation is:

desired case leading to \(A\)

divided by

all cases leading to \(A\).

The numerator is one of the path probabilities included in the denominator.

1.9.9 Applying Bayes’ Rule

A basic Bayes problem usually contains two parts.

First, calculate the overall probability of the observed event.

Second, divide the desired path probability by that overall probability.

The process is:

possible cases

-> calculate all paths to the observed event

-> add the paths to find the overall probability

-> divide the desired path by the overall probability

1.9.10 Example 3: Two Card Piles

One of two card piles is selected.

Pile \(B_1\) is selected with probability

\[P(B_1)=\frac{1}{3}\]

and contains:

• 3 red cards;

• 1 black card.

Pile \(B_2\) is selected with probability

\[P(B_2)=\frac{2}{3}\]

and contains:

• 1 red card;

• 3 black cards.

One card is drawn from the selected pile and is found to be red.

Find

\[P(B_1\mid R).\]

From the pile compositions,

\[P(R\mid B_1)=\frac{3}{4}\]

and

\[P(R\mid B_2)=\frac{1}{4}.\]

Step 1: Find the overall probability of red

Using total probability,

\[P(R) = P(B_1)P(R\mid B_1) + P(B_2)P(R\mid B_2).\]

Therefore,

\[P(R) = \frac{1}{3}\times\frac{3}{4} + \frac{2}{3}\times\frac{1}{4}.\]

So,

\[P(R) = \frac{1}{4} + \frac{1}{6} = \frac{5}{12}.\]

Step 2: Find the desired path

The desired path is

\[B_1\rightarrow R.\]

Its probability is

\[P(B_1)P(R\mid B_1) = \frac{1}{3}\times\frac{3}{4} = \frac{1}{4}.\]

Step 3: Apply Bayes’ rule

\[P(B_1\mid R) = \frac{ P(B_1)P(R\mid B_1) }{ P(R) }.\]

Therefore,

\[P(B_1\mid R) = \frac{1/4}{5/12} = \frac{3}{5}.\]

So, after observing a red card, the probability that Pile \(B_1\) was selected is

\[\frac{3}{5}.\]

1.9.11 Example 4: Three Card Piles

One of three card piles is selected.

Pile \(B_1\) is selected with probability

\[P(B_1)=\frac{1}{2}\]

and contains:

• 1 red card;

• 3 black cards.

Pile \(B_2\) is selected with probability

\[P(B_2)=\frac{1}{3}\]

and contains:

• 2 red cards;

• 2 black cards.

Pile \(B_3\) is selected with probability

\[P(B_3)=\frac{1}{6}\]

and contains:

• 3 red cards;

• 1 black card.

One card is drawn from the selected pile and is found to be red.

Find

\[P(B_3\mid R).\]

From the pile compositions,

\[P(R\mid B_1)=\frac{1}{4},\]

\[P(R\mid B_2)=\frac{1}{2},\]

and

\[P(R\mid B_3)=\frac{3}{4}.\]

Step 1: Find the overall probability of red

Using total probability,

\[\begin{split}\begin{aligned} P(R) ={}& \frac{1}{2}\times\frac{1}{4} + \frac{1}{3}\times\frac{1}{2} + \frac{1}{6}\times\frac{3}{4}\\ ={}& \frac{1}{8} + \frac{1}{6} + \frac{1}{8}\\ ={}& \frac{5}{12}. \end{aligned}\end{split}\]

Step 2: Find the desired path

The desired path is

\[B_3\rightarrow R.\]

Its probability is

\[P(B_3)P(R\mid B_3) = \frac{1}{6}\times\frac{3}{4} = \frac{1}{8}.\]

Step 3: Apply Bayes’ rule

\[P(B_3\mid R) = \frac{1/8}{5/12} = \frac{3}{10}.\]

Therefore,

\[P(B_3\mid R)=\frac{3}{10}.\]

With three cases, the idea is unchanged:

add all paths leading to the observed event

then divide the desired path by that total.

1.9.12 Prior and Posterior Probabilities

Bayes’ rule updates the probability of a possible case after an event is observed.

The probability before observing the event is called the prior probability.

For example, in the two-coin problem,

\[P(C_2)=\frac{1}{2}.\]

This is the prior probability that Coin \(C_2\) was selected.

After heads is observed, we calculate

\[P(C_2\mid H)=\frac{2}{3}.\]

This is the posterior probability.

Note

Prior probability

Probability before observing event \(A\):

\[P(B_i).\]

Posterior probability

Updated probability after observing event \(A\):

\[P(B_i\mid A).\]

The update can be written conceptually as

prior -> observe \(A\) -> posterior

In the coin example,

\[\frac{1}{2} \quad\longrightarrow\quad \text{observe heads} \quad\longrightarrow\quad \frac{2}{3}.\]

Observing heads changes the probability because heads is more likely under the double-headed coin than under the fair coin.

1.9.13 Effect of the Starting Probability

The observed event is important, but the starting probability also matters.

Bayes’ rule uses both

\[P(B_j)\]

and

\[P(A\mid B_j).\]

So an event with a high conditional probability does not automatically become the most likely case after \(A\) is observed.

The starting probability of the case is also part of the calculation.

1.9.14 Example 5: Unequal Starting Probabilities

One of two card piles is selected.

Pile \(B_1\) is selected with probability

\[P(B_1)=\frac{3}{4},\]

and

\[P(R\mid B_1)=\frac{1}{4}.\]

Pile \(B_2\) is selected with probability

\[P(B_2)=\frac{1}{4},\]

and

\[P(R\mid B_2)=\frac{3}{4}.\]

A red card is observed.

Find

\[P(B_2\mid R).\]

First, find the overall probability of red:

\[P(R) = P(B_1)P(R\mid B_1) + P(B_2)P(R\mid B_2).\]

Therefore,

\[P(R) = \frac{3}{4}\times\frac{1}{4} + \frac{1}{4}\times\frac{3}{4} = \frac{3}{16}+\frac{3}{16} = \frac{3}{8}.\]

The desired path is

\[B_2\rightarrow R.\]

Its probability is

\[P(B_2)P(R\mid B_2) = \frac{1}{4}\times\frac{3}{4} = \frac{3}{16}.\]

Now apply Bayes’ rule:

\[P(B_2\mid R) = \frac{3/16}{3/8} = \frac{1}{2}.\]

Before observing red,

\[P(B_2)=\frac{1}{4}.\]

After observing red,

\[P(B_2\mid R)=\frac{1}{2}.\]

The observed red card increases the probability of \(B_2\).

However, the starting probability still matters.

Even though

\[P(R\mid B_2)=\frac{3}{4},\]

the posterior probability is not \(3/4\).

1.9.15 Connecting Total Probability and Bayes’ Rule

Total probability and Bayes’ rule use the same probability tree, but they answer different questions.

Total probability

Start with the possible cases and move toward the final event.

The direction is

cases -> event \(A\).

The goal is to find the overall probability

\[P(A).\]

Bayes’ rule

Start with the observed event and ask which case was responsible.

The direction is

observed event \(A\) -> one possible case.

The goal is to find

\[P(B_j\mid A).\]

Note

Total Probability

Add all paths leading to \(A\).

Bayes’ Rule

Divide the desired path by all paths leading to \(A\).

For the partition

\[B_1,B_2,\ldots,B_k,\]

first calculate each path:

\[P(B_i)P(A\mid B_i).\]

Then add all paths:

\[P(A) = \sum_{i=1}^{k}P(B_i)P(A\mid B_i).\]

Finally, for the desired case \(B_j\),

\[P(B_j\mid A) = \frac{ P(B_j)P(A\mid B_j) }{ P(A) }.\]

This gives a useful workflow:

possible cases

-> path probabilities

-> total probability of \(A\)

-> posterior probability of the desired case

1.9.16 A Simple Method

For a basic Bayes’ rule problem, use the following steps.

Step 1: Identify the possible cases.

Write

\[B_1,B_2,\ldots,B_k.\]

The cases should form a partition.

Step 2: Identify the observed event.

Call the observed event \(A\).

The question usually asks for something like

\[P(B_j\mid A).\]

Step 3: Write the starting probability of each case.

Find

\[P(B_1),P(B_2),\ldots,P(B_k).\]

These are the prior probabilities.

Step 4: Write the probability of the observed event within each case.

Find

\[P(A\mid B_1), P(A\mid B_2), \ldots, P(A\mid B_k).\]

Step 5: Multiply within each path.

Calculate

\[P(B_i)P(A\mid B_i).\]

Step 6: Add all paths to find the overall probability.

\[P(A) = \sum_{i=1}^{k}P(B_i)P(A\mid B_i).\]

Step 7: Divide the desired path by the total.

\[P(B_j\mid A) = \frac{ P(B_j)P(A\mid B_j) }{ P(A) }.\]

Step 8: Interpret the answer.

The result is the updated probability of the case after the event is observed.

1.9.17 Common Mistakes

A common mistake is to reverse the conditional probability incorrectly.

In general,

\[P(A\mid B)\neq P(B\mid A).\]

Always identify which event is after the vertical bar.

Another mistake is to use

\[P(A\mid B_j)\]

as the answer when the question asks for

\[P(B_j\mid A).\]

These are different directions.

A third mistake is to calculate only the desired path and forget the denominator.

Bayes’ rule requires

\[\frac{\text{desired path}} {\text{all paths leading to the observed event}}.\]

A fourth mistake is to add the conditional probabilities directly when finding \(P(A)\).

The correct total probability calculation uses

\[P(B_i)P(A\mid B_i)\]

for each path.

Another mistake is to forget one possible case in the denominator.

Every case that can produce the observed event must be included in \(P(A)\).

A final mistake is to ignore the prior probabilities.

A case with a large value of

\[P(A\mid B_i)\]

does not automatically have the largest posterior probability.

Bayes’ rule also uses

\[P(B_i).\]

1.9.18 Summary

Bayes’ rule is used when the direction of a conditional probability is reversed.

In general,

\[P(A\mid B)\neq P(B\mid A).\]

For one possible case \(B_j\) and an observed event \(A\),

\[P(B_j\mid A) = \frac{P(B_j)P(A\mid B_j)} {P(A)}.\]

If

\[B_1,B_2,\ldots,B_k\]

form a partition, total probability gives

\[P(A) = \sum_{i=1}^{k} P(B_i)P(A\mid B_i).\]

Therefore,

\[P(B_j\mid A) = \frac{ P(B_j)P(A\mid B_j) }{ \displaystyle \sum_{i=1}^{k}P(B_i)P(A\mid B_i) }.\]

The numerator is the desired path to the observed event.

The denominator is the probability of all possible paths to the observed event.

A prior probability is the probability before the event is observed:

\[P(B_i).\]

A posterior probability is the updated probability after the event is observed:

\[P(B_i\mid A).\]

The main connection is:

Total probability: cases -> observed event

Bayes’ rule: observed event -> desired case

A useful memory rule is:

multiply along each path

add all paths to the observed event

divide the desired path by the total