The Dawn Treader

Imagine there is a bag of black and white marbles.  There is an equal mix of black marbles and white marbles.  If you were to reach in and pull out a marble, what is probability that you will pull out a black marble?

The answer is 50 percent.

That was a confidence booster for warm up.  Probabilities are not that hard, right?

One more confidence builder before moving on.

What is the probability that you will pull a Jack from a deck of cards?

Answer: 4/52 = 1/13.  Because there are 4 Jacks and 52 cards.

Now, if you pulled a card from the deck, and before looking, was told that it was a face card, what is the probability that you pulled a Jack?

Answer: 4/12 = 1/3.  Because there are 4 Jacks and 12 Face cards in a deck.  The probability of pulling a Jack has been altered.  This altered probability is known as a conditional probability.

Ok, now for Bayes Theorem.

Suppose Wal Mart sells two different kinds of marble bags.  The first marble bag contains all black marbles.  The second bag contains 20 percent black marbles.

For the sake of argument, assume the bags are opaque and have no visible marking on them to indicate the contents of the bag.

Your math teacher buys a bag of marbles from Wal Mart and brings it in to class.  She tells you that you are not allowed to look inside.  She asks, "what are the odds that the bag I have is the one with only black marbles?"

With no additional information, it would kinda hard to say.  There is not enough information to make a good calculation.

She calls on you, and you respond sheepishly, "50 percent".

She responds, "ok ... good try.  Let me give you a little help with your estimate."

She reaches in and pulls out one marble.  It is a black marble.

She looks at you, and asks "Ok.  Now what is the probability that I have the marble bag with all black marbles?"

Do we know conclusively that she bought the bag with all black marbles?  Nope.  However, we saw some evidence that she did.  Not a lot of evidence, but some.  Our belief that she did in fact buy the all black marble bag is strengthened, is it not?  After all, the odds that she would have pulled a black marble out of the bag with only twenty percent black marbles is unlikely ... not impossible, but unlikely.

Having recently studied Bayes Theorem, you proudly answer "there is an 83 percent probability that you bought the bag with all black marbles."

"Excellent answer", she replies.  "Here is another clue."

She pulls out a second marble.  It is black too.

Once again, she asks, "ok, now what is the probability that I have the all black marbles bag?"

Things are looking stronger and stronger, are they not?  Two black marbles in a row.

You close your eyes, squint, and then reply ... "96 percent!"

"Impressive", she exclaims.

I will explain where the estimates come from, but first notice what is going on.  We are revising our probability estimate based on new information.  As she continues to pull out black marbles, our belief that she indeed purchased the all black marble bag is strengthened.  It gets revised upward each time she pulls out another black marble.  More evidence, stronger belief.  That is the way Bayes works.

Guess what would happen the moment she pulled out a green marble?  We would know that she did not buy the all black marble bag, and the probability of her buying the all black marble would plummet to zero.  In other words, our belief that she bought the black marble bag is zero percent.

Ok, now for the math.

Bayes formula is an equation that produces a result called a posterior probability -- iow, a revised probability based on new evidence.

Bayes Formula is :

P(T|E) = P(E|T) x P(T) / [ P(E|T) x P(T) + P(E|~T) x P(~T) ]

where

P stands for probability
T stands for theory
E stands for event
| stands for "given that"
~ stands for "not"

P(T|E) = the probability of theory T given the fact that we observed event E.  Theory T is the "theory" that the teacher purchased the all black marble bag.  Event E is pulling a black marble from the bag.  Bayesians call P(T|E) a "posterior" probability.  It is the revised probability after observing new evidence (hence the term posterior).  The posterior probability is the whole point of Bayes Theorem.

P(T) = our best "a priori" guess for the probability that the theory is true.  In our case, the probability that the teacher bought the all black marble bag.  Bayesians call this estimate a "prior belief".

P(E|T) = the probability that event E will occur given that theory T is true.  In our case, the probability that a black marble will be drawn from the bag if the bag is the "all black marble" bag.

P(E|~T) = the probability that event E will occur given the opposite of theory T.  In our case, the probability that a black marble will be drawn from the bag with only 20 percent black marbles.

Still hanging in there?  Let's plug in some numbers.

P(T) = our initial best guess at probability that she bought the all black marble bag ... which was 50 % or 0.5
P(E|T) = the probability of pulling a black marble from the all black marble bag ... which is 100% or 1.
P(E|~T) = the probability of pulling a black marble from the other bag ... which is 20% or 0.2.

Here is how the equation looks:

1 - P(T|E) = (0.5) x ( 1 ) / [ ( 0.5 ) x ( 1 ) + ( 0.5 ) x ( 0.2 ) ]
2 - 0.5 / [ 0.5 + 0.1 ]
3 - 0.5 / 0.6
4 - 0.83 ... or 83 % probability

So our "posterior" probability estimate is 83%.  We are 83% certain that we are dealing with the all black marble bag after seeing the teacher pull a black marble out of the bag.

Here is where it gets cool.

Bayes will revise our probability based on the second black marble too.  All we have to do is plug 83% into the formula.

Watch what happens.

P(T) = 0.83 ... our new "prior" belief  ... note: this means we only believe there is a 0.17 chance that the bag is the one with 20 percent black marbles.
P(E|T) = 1 ... no change here
P(E|~T) = 0.2 ... no change here

1 - P(T|E) = (0.83) x ( 1 ) / [ ( 0.83 ) x ( 1 ) + ( 0.17 ) x ( 0.2 ) ]
2 - 0.83 / [ 0.83 + 0.034 ]
3 - 0.83 / 0.864
4 - 0.96 ... or 96 % probability

Conclusion:

Bayes formula keeps revising our probabilities based on evidence.  That is the whole idea.  Our probabilities get better based on what we observe.  That is why Bayes turns out to be so useful in so many applications.

For further reading, .