1.9 Bayes' Rule
===============

:bolditalic:`Key Terms`

:bolditalic:`Bayes' rule（貝氏定理）`
   A rule for finding a reversed conditional probability.

:bolditalic:`Prior probability（先驗機率）`
   The probability of a possible case before the new event is observed.

:bolditalic:`Posterior probability（後驗機率）`
   The updated probability of a possible case after the new event is observed.

:bolditalic:`Possible case（可能情況）`
   One event in a partition, such as :math:`B_1,B_2,\ldots,B_k`.

:bolditalic:`Observed event（觀察到的事件）`
   The event that is known to have occurred.

1.9.2 Learning Outcomes
-----------------------

After completing this section, students should be able to:

- explain why :math:`P(A\mid B)` and :math:`P(B\mid A)` answer different questions;
- recognize when the condition in a probability question has been reversed;
- use Bayes' rule for two or more possible cases;
- calculate the overall probability needed in the denominator using total probability;
- interpret the numerator of Bayes' rule as one desired path;
- interpret the denominator as all possible paths leading to the observed event;
- distinguish prior probability from posterior probability;
- explain how an observed event can update a starting probability;
- connect total probability and Bayes' rule.

1.9.3 Reverse the Condition
---------------------------

Conditional probability depends on which event is already known.

For example,

.. math::

   P(H\mid C_2)

means:

    probability of heads, given that Coin :math:`C_2` was chosen.

Here, the coin is known first.

The direction is

    known coin -> probability of heads.

Now consider

.. math::

   P(C_2\mid H).

This means:

    probability that Coin :math:`C_2` was chosen, given that heads was
    observed.

Here, heads is known first.

The direction is

    observed heads -> probability of the coin.

These two probabilities answer different questions.

.. note::

   **The condition has been reversed**

   .. math::

      P(H\mid C_2)

   means probability of heads given :math:`C_2`.

   .. math::

      P(C_2\mid H)

   means probability of :math:`C_2` given heads.

   In general,

   .. math::

      P(H\mid C_2)\neq P(C_2\mid H).

The event after the vertical bar is always the information that is already
known.

1.9.4 Example 1: Choose One of Two Coins
----------------------------------------

Suppose one of two coins is chosen at random.

- :math:`C_1` is a fair coin.
- :math:`C_2` is a double-headed coin.

The selected coin is tossed once.

Because :math:`C_2` is double-headed,

.. math::

   P(H\mid C_2)=1.

This is a forward conditional probability.

It asks:

    If Coin :math:`C_2` was selected, how likely is heads?

Now suppose instead that heads is observed, but we do not know which coin
was selected.

The question becomes

.. math::

   P(C_2\mid H).

This asks:

    Given that heads was observed, how likely is it that Coin :math:`C_2`
    was selected?

This is the type of reversed conditional probability that Bayes' rule helps
us calculate.

1.9.5 Bayes' Rule
-----------------

Start with the conditional probability formula:

.. math::

   P(C_2\mid H)
   =
   \frac{P(C_2\cap H)}{P(H)}.

From the multiplication rule,

.. math::

   P(C_2\cap H)
   =
   P(C_2)P(H\mid C_2).

Substituting this into the conditional probability formula gives

.. math::

   P(C_2\mid H)
   =
   \frac{P(C_2)P(H\mid C_2)}
        {P(H)}.

.. note::

   **Bayes' Rule**

   .. math::

      P(C_2\mid H)
      =
      \frac{P(C_2)P(H\mid C_2)}
           {P(H)}.

The numerator is the probability of one particular path:

.. math::

   C_2\rightarrow H.

The denominator is the overall probability of heads from all possible
paths.

A useful way to remember the structure is:

    desired path to the observed event

    divided by

    all paths to the observed event

1.9.6 Example 2: Which Coin Was Chosen?
---------------------------------------

One of two coins is chosen at random.

- Coin :math:`C_1` is fair.
- Coin :math:`C_2` is double-headed.

Each coin is selected with probability

.. math::

   P(C_1)=\frac{1}{2}

and

.. math::

   P(C_2)=\frac{1}{2}.

The selected coin is tossed and heads is observed.

Find

.. math::

   P(C_2\mid H).

From Section 1.8, the overall probability of heads is

.. math::

   P(H)=\frac{3}{4}.

The probability of the path

.. math::

   C_2\rightarrow H

is

.. math::

   P(C_2\cap H)
   =
   P(C_2)P(H\mid C_2).

Since

.. math::

   P(C_2)=\frac{1}{2}

and

.. math::

   P(H\mid C_2)=1,

we get

.. math::

   P(C_2\cap H)
   =
   \frac{1}{2}\times1
   =
   \frac{1}{2}.

Now use conditional probability:

.. math::

   P(C_2\mid H)
   =
   \frac{P(C_2\cap H)}{P(H)}.

Therefore,

.. math::

   P(C_2\mid H)
   =
   \frac{1/2}{3/4}
   =
   \frac{2}{3}.

So, after observing heads, the probability that the double-headed coin was
selected is

.. math::

   \frac{2}{3}.

Notice that

.. math::

   P(H\mid C_2)=1

but

.. math::

   P(C_2\mid H)=\frac{2}{3}.

The two conditional probabilities are not the same.

1.9.7 General Form
------------------

Suppose the possible cases are

.. math::

   B_1,B_2,\ldots,B_k,

and these cases form a partition of the sample space.

Now suppose event :math:`A` is observed.

We want the probability of one particular case :math:`B_j` after observing
:math:`A`.

Starting from conditional probability,

.. math::

   P(B_j\mid A)
   =
   \frac{P(B_j\cap A)}{P(A)}.

Using the multiplication rule,

.. math::

   P(B_j\cap A)
   =
   P(B_j)P(A\mid B_j).

Therefore,

.. math::

   P(B_j\mid A)
   =
   \frac{P(B_j)P(A\mid B_j)}
        {P(A)}.

.. note::

   **Bayes' Rule**

   .. math::

      P(B_j\mid A)
      =
      \frac{P(B_j)P(A\mid B_j)}
           {P(A)}.

The numerator is the desired path

.. math::

   B_j\rightarrow A.

The denominator is the probability of observing :math:`A` through all
possible cases.

1.9.8 Bayes' Rule for Several Cases
-----------------------------------

From Section 1.8, if

.. math::

   B_1,B_2,\ldots,B_k

form a partition, then total probability gives

.. math::

   P(A)
   =
   \sum_{i=1}^{k}P(B_i)P(A\mid B_i).

Substituting this expression for :math:`P(A)` into Bayes' rule gives

.. math::

   P(B_j\mid A)
   =
   \frac{
      P(B_j)P(A\mid B_j)
   }{
      \displaystyle
      \sum_{i=1}^{k}P(B_i)P(A\mid B_i)
   }.

.. note::

   **Bayes' Rule for Several Cases**

   .. math::

      P(B_j\mid A)
      =
      \frac{
         P(B_j)P(A\mid B_j)
      }{
         \displaystyle
         \sum_{i=1}^{k}P(B_i)P(A\mid B_i)
      }.

The interpretation is:

    desired case leading to :math:`A`

    divided by

    all cases leading to :math:`A`.

The numerator is one of the path probabilities included in the denominator.

1.9.9 Applying Bayes' Rule
--------------------------

A basic Bayes problem usually contains two parts.

First, calculate the overall probability of the observed event.

Second, divide the desired path probability by that overall probability.

The process is:

    possible cases

    -> calculate all paths to the observed event

    -> add the paths to find the overall probability

    -> divide the desired path by the overall probability

1.9.10 Example 3: Two Card Piles
--------------------------------

One of two card piles is selected.

Pile :math:`B_1` is selected with probability

.. math::

   P(B_1)=\frac{1}{3}

and contains:

- 3 red cards;
- 1 black card.

Pile :math:`B_2` is selected with probability

.. math::

   P(B_2)=\frac{2}{3}

and contains:

- 1 red card;
- 3 black cards.

One card is drawn from the selected pile and is found to be red.

Find

.. math::

   P(B_1\mid R).

From the pile compositions,

.. math::

   P(R\mid B_1)=\frac{3}{4}

and

.. math::

   P(R\mid B_2)=\frac{1}{4}.

**Step 1: Find the overall probability of red**

Using total probability,

.. math::

   P(R)
   =
   P(B_1)P(R\mid B_1)
   +
   P(B_2)P(R\mid B_2).

Therefore,

.. math::

   P(R)
   =
   \frac{1}{3}\times\frac{3}{4}
   +
   \frac{2}{3}\times\frac{1}{4}.

So,

.. math::

   P(R)
   =
   \frac{1}{4}
   +
   \frac{1}{6}
   =
   \frac{5}{12}.

**Step 2: Find the desired path**

The desired path is

.. math::

   B_1\rightarrow R.

Its probability is

.. math::

   P(B_1)P(R\mid B_1)
   =
   \frac{1}{3}\times\frac{3}{4}
   =
   \frac{1}{4}.

**Step 3: Apply Bayes' rule**

.. math::

   P(B_1\mid R)
   =
   \frac{
      P(B_1)P(R\mid B_1)
   }{
      P(R)
   }.

Therefore,

.. math::

   P(B_1\mid R)
   =
   \frac{1/4}{5/12}
   =
   \frac{3}{5}.

So, after observing a red card, the probability that Pile :math:`B_1` was
selected is

.. math::

   \frac{3}{5}.

1.9.11 Example 4: Three Card Piles
----------------------------------

One of three card piles is selected.

Pile :math:`B_1` is selected with probability

.. math::

   P(B_1)=\frac{1}{2}

and contains:

- 1 red card;
- 3 black cards.

Pile :math:`B_2` is selected with probability

.. math::

   P(B_2)=\frac{1}{3}

and contains:

- 2 red cards;
- 2 black cards.

Pile :math:`B_3` is selected with probability

.. math::

   P(B_3)=\frac{1}{6}

and contains:

- 3 red cards;
- 1 black card.

One card is drawn from the selected pile and is found to be red.

Find

.. math::

   P(B_3\mid R).

From the pile compositions,

.. math::

   P(R\mid B_1)=\frac{1}{4},

.. math::

   P(R\mid B_2)=\frac{1}{2},

and

.. math::

   P(R\mid B_3)=\frac{3}{4}.

**Step 1: Find the overall probability of red**

Using total probability,

.. math::

   \begin{aligned}
   P(R)
   ={}&
   \frac{1}{2}\times\frac{1}{4}
   +
   \frac{1}{3}\times\frac{1}{2}
   +
   \frac{1}{6}\times\frac{3}{4}\\
   ={}&
   \frac{1}{8}
   +
   \frac{1}{6}
   +
   \frac{1}{8}\\
   ={}&
   \frac{5}{12}.
   \end{aligned}

**Step 2: Find the desired path**

The desired path is

.. math::

   B_3\rightarrow R.

Its probability is

.. math::

   P(B_3)P(R\mid B_3)
   =
   \frac{1}{6}\times\frac{3}{4}
   =
   \frac{1}{8}.

**Step 3: Apply Bayes' rule**

.. math::

   P(B_3\mid R)
   =
   \frac{1/8}{5/12}
   =
   \frac{3}{10}.

Therefore,

.. math::

   P(B_3\mid R)=\frac{3}{10}.

With three cases, the idea is unchanged:

    add all paths leading to the observed event

    then divide the desired path by that total.

1.9.12 Prior and Posterior Probabilities
----------------------------------------

Bayes' rule updates the probability of a possible case after an event is
observed.

The probability **before** observing the event is called the **prior
probability**.

For example, in the two-coin problem,

.. math::

   P(C_2)=\frac{1}{2}.

This is the prior probability that Coin :math:`C_2` was selected.

After heads is observed, we calculate

.. math::

   P(C_2\mid H)=\frac{2}{3}.

This is the **posterior probability**.

.. note::

   **Prior probability**

   Probability before observing event :math:`A`:

   .. math::

      P(B_i).

   **Posterior probability**

   Updated probability after observing event :math:`A`:

   .. math::

      P(B_i\mid A).

The update can be written conceptually as

    prior -> observe :math:`A` -> posterior

In the coin example,

.. math::

   \frac{1}{2}
   \quad\longrightarrow\quad
   \text{observe heads}
   \quad\longrightarrow\quad
   \frac{2}{3}.

Observing heads changes the probability because heads is more likely under
the double-headed coin than under the fair coin.

1.9.13 Effect of the Starting Probability
-----------------------------------------

The observed event is important, but the starting probability also matters.

Bayes' rule uses both

.. math::

   P(B_j)

and

.. math::

   P(A\mid B_j).

So an event with a high conditional probability does not automatically
become the most likely case after :math:`A` is observed.

The starting probability of the case is also part of the calculation.

1.9.14 Example 5: Unequal Starting Probabilities
------------------------------------------------

One of two card piles is selected.

Pile :math:`B_1` is selected with probability

.. math::

   P(B_1)=\frac{3}{4},

and

.. math::

   P(R\mid B_1)=\frac{1}{4}.

Pile :math:`B_2` is selected with probability

.. math::

   P(B_2)=\frac{1}{4},

and

.. math::

   P(R\mid B_2)=\frac{3}{4}.

A red card is observed.

Find

.. math::

   P(B_2\mid R).

First, find the overall probability of red:

.. math::

   P(R)
   =
   P(B_1)P(R\mid B_1)
   +
   P(B_2)P(R\mid B_2).

Therefore,

.. math::

   P(R)
   =
   \frac{3}{4}\times\frac{1}{4}
   +
   \frac{1}{4}\times\frac{3}{4}
   =
   \frac{3}{16}+\frac{3}{16}
   =
   \frac{3}{8}.

The desired path is

.. math::

   B_2\rightarrow R.

Its probability is

.. math::

   P(B_2)P(R\mid B_2)
   =
   \frac{1}{4}\times\frac{3}{4}
   =
   \frac{3}{16}.

Now apply Bayes' rule:

.. math::

   P(B_2\mid R)
   =
   \frac{3/16}{3/8}
   =
   \frac{1}{2}.

Before observing red,

.. math::

   P(B_2)=\frac{1}{4}.

After observing red,

.. math::

   P(B_2\mid R)=\frac{1}{2}.

The observed red card increases the probability of :math:`B_2`.

However, the starting probability still matters.

Even though

.. math::

   P(R\mid B_2)=\frac{3}{4},

the posterior probability is not :math:`3/4`.

1.9.15 Connecting Total Probability and Bayes' Rule
---------------------------------------------------

Total probability and Bayes' rule use the same probability tree, but they
answer different questions.

**Total probability**

Start with the possible cases and move toward the final event.

The direction is

    cases -> event :math:`A`.

The goal is to find the overall probability

.. math::

   P(A).

**Bayes' rule**

Start with the observed event and ask which case was responsible.

The direction is

    observed event :math:`A` -> one possible case.

The goal is to find

.. math::

   P(B_j\mid A).

.. note::

   **Total Probability**

   Add all paths leading to :math:`A`.

   **Bayes' Rule**

   Divide the desired path by all paths leading to :math:`A`.

For the partition

.. math::

   B_1,B_2,\ldots,B_k,

first calculate each path:

.. math::

   P(B_i)P(A\mid B_i).

Then add all paths:

.. math::

   P(A)
   =
   \sum_{i=1}^{k}P(B_i)P(A\mid B_i).

Finally, for the desired case :math:`B_j`,

.. math::

   P(B_j\mid A)
   =
   \frac{
      P(B_j)P(A\mid B_j)
   }{
      P(A)
   }.

This gives a useful workflow:

    possible cases

    -> path probabilities

    -> total probability of :math:`A`

    -> posterior probability of the desired case

1.9.16 A Simple Method
----------------------

For a basic Bayes' rule problem, use the following steps.

**Step 1: Identify the possible cases.**

Write

.. math::

   B_1,B_2,\ldots,B_k.

The cases should form a partition.

**Step 2: Identify the observed event.**

Call the observed event :math:`A`.

The question usually asks for something like

.. math::

   P(B_j\mid A).

**Step 3: Write the starting probability of each case.**

Find

.. math::

   P(B_1),P(B_2),\ldots,P(B_k).

These are the prior probabilities.

**Step 4: Write the probability of the observed event within each case.**

Find

.. math::

   P(A\mid B_1),
   P(A\mid B_2),
   \ldots,
   P(A\mid B_k).

**Step 5: Multiply within each path.**

Calculate

.. math::

   P(B_i)P(A\mid B_i).

**Step 6: Add all paths to find the overall probability.**

.. math::

   P(A)
   =
   \sum_{i=1}^{k}P(B_i)P(A\mid B_i).

**Step 7: Divide the desired path by the total.**

.. math::

   P(B_j\mid A)
   =
   \frac{
      P(B_j)P(A\mid B_j)
   }{
      P(A)
   }.

**Step 8: Interpret the answer.**

The result is the updated probability of the case after the event is
observed.

1.9.17 Common Mistakes
----------------------

A common mistake is to reverse the conditional probability incorrectly.

In general,

.. math::

   P(A\mid B)\neq P(B\mid A).

Always identify which event is after the vertical bar.

Another mistake is to use

.. math::

   P(A\mid B_j)

as the answer when the question asks for

.. math::

   P(B_j\mid A).

These are different directions.

A third mistake is to calculate only the desired path and forget the
denominator.

Bayes' rule requires

.. math::

   \frac{\text{desired path}}
        {\text{all paths leading to the observed event}}.

A fourth mistake is to add the conditional probabilities directly when
finding :math:`P(A)`.

The correct total probability calculation uses

.. math::

   P(B_i)P(A\mid B_i)

for each path.

Another mistake is to forget one possible case in the denominator.

Every case that can produce the observed event must be included in
:math:`P(A)`.

A final mistake is to ignore the prior probabilities.

A case with a large value of

.. math::

   P(A\mid B_i)

does not automatically have the largest posterior probability.

Bayes' rule also uses

.. math::

   P(B_i).

1.9.18 Summary
--------------

Bayes' rule is used when the direction of a conditional probability is
reversed.

In general,

.. math::

   P(A\mid B)\neq P(B\mid A).

For one possible case :math:`B_j` and an observed event :math:`A`,

.. math::

   P(B_j\mid A)
   =
   \frac{P(B_j)P(A\mid B_j)}
        {P(A)}.

If

.. math::

   B_1,B_2,\ldots,B_k

form a partition, total probability gives

.. math::

   P(A)
   =
   \sum_{i=1}^{k}
   P(B_i)P(A\mid B_i).

Therefore,

.. math::

   P(B_j\mid A)
   =
   \frac{
      P(B_j)P(A\mid B_j)
   }{
      \displaystyle
      \sum_{i=1}^{k}P(B_i)P(A\mid B_i)
   }.

The numerator is the desired path to the observed event.

The denominator is the probability of all possible paths to the observed
event.

A **prior probability** is the probability before the event is observed:

.. math::

   P(B_i).

A **posterior probability** is the updated probability after the event is
observed:

.. math::

   P(B_i\mid A).

The main connection is:

    Total probability:
    cases -> observed event

    Bayes' rule:
    observed event -> desired case

A useful memory rule is:

    multiply along each path

    add all paths to the observed event

    divide the desired path by the total

.. 1.9.19 Short Review Questions
.. -----------------------------

.. 1. What is the difference between :math:`P(A\mid B)` and :math:`P(B\mid A)`?
.. 2. What does it mean to reverse the condition?
.. 3. State Bayes' rule using :math:`B_j` and :math:`A`.
.. 4. What does the numerator of Bayes' rule represent?
.. 5. What does the denominator of Bayes' rule represent?
.. 6. How is :math:`P(A)` found when there are several possible cases?
.. 7. Why is total probability usually calculated before Bayes' rule?
.. 8. What is a prior probability?
.. 9. What is a posterior probability?
.. 10. In the two-coin example, why is :math:`P(H\mid C_2)` different from :math:`P(C_2\mid H)`?
.. 11. What are the main steps for solving a Bayes' rule problem?
.. 12. Why must all possible paths to the observed event be included in the denominator?
.. 13. Why does a large value of :math:`P(A\mid B_i)` not automatically make :math:`B_i` the most likely case after observing :math:`A`?
.. 14. How are total probability and Bayes' rule connected?