Overview

This post is chapter 6 in continuation of my coverage of Data Science from Scratch by Joel Grus. We will work our way towards understanding conditional probability by understanding preceding concepts like marginal and joint probabilities.

At the end, we'll tie all concepts together through code. For those inclined, you can jump to the code towards the bottom of this post.

Challenge

The first challenge in this section is distinguishing between two conditional probability statements.

Here's the setup. We have a family with two (unknown) children with two assumptions. First, each child is equally likely to be a boy or a girl. Second, the gender of the second child is independent of the gender of the first child.

Outcome 1: What is the probability of the event "both children are girls" (B) conditional on the event "the older child is a girl" (G)?

The probability for statement one is roughly 50% or (1/2).

Outcome 2: What is the probability of the event "both children are girls" (B) conditional on the event "at least one of the children is a girl" (L)?

The probability for statement two is roughly 33% or (1/3).

But at first glance, they look similar.

Marginal and Joint Probabilities

The book jumps straight to conditional probabilities, but first, we'll have to look at marginal and joint probabilities. Then we'll create a joint probabilities table and sum probabilities to help us figure out the differences. We'll then resume with conditional probabilities.

Before anything, we need to realize the situation we have is one of independence. The gender of one child is independent of a second child.

The intuition for this scenario will be different from a dependent situation. For example, if we draw two cards from a deck (without replacement), the probabilities are different. The probability of drawing one King ♠️ is (4/52) and the probability of drawing a second King ♣️ is now (3/51); the probability of the second event (a second King) is dependent on the result of the first draw.

Back to the two unknown children.

We can say the probability of the first child being either a boy or a girl is 50/50. Moreover, the probability of the second child, which is independent of the first, is also 50/50. Remember, our first assumption is that each child is equally likely to be a boy or a girl.

Let's put these numbers in a table. The (1/2) probabilities shown here are called marginal probabilities (note how they're at the margins of the table).

Since we have two gender (much like two sides of a flipped coin), we can intuitively figure out all possible outcomes:

1. first child (Boy), second child (Boy)
2. first child (Boy), second child (Girl)
3. first child (Girl), second child (Boy)
4. first child (Girl), second child (Girl)

There are 4 possible outcomes so the probability of getting any one of the four outcomes is (1/4). We can actually write these probabilities in the middle of the table, the joint probabilities:

To recap, the probability of the first child being either boy or girl is 50/50, simple enough. The probability of the second child being either boy or girl is also 50/50. When put in a table, this yielded the marginal probability.

Now we want to know the probability of say, 'first child being a boy and second child being a girl'. This is a joint probability because is is the probability that the first child take a specific gender (boy) AND the second child take a specific gender (girl).

If two event are independent, and in this case they are, their joint probabilities are the product of the probabilities of each one happening.

The probability of the first child being a Boy (1/2) and second child being a Girl (1/2); The product of each marginal probability is the joint probability (1/2 * 1/2 = 1/4).

This can be repeated for the other three joint probabilities.

Conditional Probability

Now we get into conditional probability which is the probability of one event happening (i.e., second child being a Boy or Girl) given that or on conditional that another event happened (i.e., first child being a Boy).

At this point, it might be a good idea to begin writing probability statements similar to how it is expressed in mathematics.

A joint probability is the product of each individual event happening (assuming they are independent events). For example we might have two individual events:

• P(1st Child = Boy): 1/2
• P(2nd Child = Boy): 1/2

Here is their joint probability:

• P(1st Child = Boy, 2nd Child = Boy)
• P(1st Child = Boy) * P(2nd Child = Boy)
• (1/2 * 1/2 = 1/4)

There is a relationship between conditional probabilities and joint probabilities.

Here is their conditional probability:

• P(2nd Child = Boy | 1st Child = Boy)
• P(1st Child = Boy, 2nd Child = Boy) / P(1st Child = Boy)

This works out to:

• (1/4) / (1/2) = 1/2 or
• (1/4) * (2/1) = 1/2

In other words, the probability that the second child is a boy, given that the first child is a boy is still 50% (this implies that with respect to conditional probability, if the events are independent it is not different from a single event).

Now we're ready to tackle the two outcomes posed at the beginning of this post.

Outcome 1: What is the probability of the event "both children are girls" (B) conditional on the event "the older child is a girl" (G)?

Let's break it down. First we want the probability of the event that "both children are girls". We'll take the product of two events; the probability that the first child is a girl (1/2) and the probability that the second child is a girl (1/2). So for both child to be girls, 1/2 * 1/2 = 1/4

• P(1st Child = Girl, 2nd Child = Girl) = 1/4

Second, we want that to be given that the "older child is a girl".

• P(1st Child = Girl) = 1/2

Conditional probability:

• P(1st Child = Girl, 2nd Child = Girl) / P(1st Child = Girl)
• (1/4) / (1/2) = (1/4) * (2/1) = (2/4) = 1/2 or roughly 50%

Now let's break down the second outcome:

Outcome 2: What is the probability of the event "both children are girls" (B) conditional on the event "at least one of the children is a girl" (L)?

Again, we start with "both children are girls":

• P(1st Child = Girl, 2nd Child = Girl) = 1/4

Then, we have "on condition that at least one of the children is a girl". We'll reference a joint probability table. We see that when trying to figure out the probability that "at least one of the children is a girl", we rule out the scenario where both children are boys. This is actually the compliment to at least one child is a girl. The remaining 3 out of 4 probabilities, fit the condition.

The probability of at least one children being a girl is:

• (1/4) + (1/4) + (1/4) = 3/4

So:

• P(1st Child = Girl, 2nd Child = Girl) / P("at least one child is a girl")
• (1/4) / (3/4) = (1/4) * (4/3) = (4/12) = 1/3 or roughly 33%

Key Take-away

When two events are independent, their joint probability is the product of each event:

• P(E,F) = P(E) * P(F)

Their conditional probability is the joint probability divided by the conditional (i.e., P(F)).

• P(E|F) = P(E,F) / P(F)

And so for our two challenge scenarios, we have:

Challenge 1:

• B = probability that both children are girls
• G = probability that the older children is a girl

This can be stated as: P(B|G) = P(B,G) / P(G)

Challenge 2:

• B = probability that both children are girls
• L = probability that at least one children is a girl

This can be stated as: P(B|L) = P(B,L) / P(L)

Python Code

Now that we have an intuition and have worked out the problem on paper, we can use code to express conditional probability:

import enum, random
class Kid(enum.Enum):
    BOY = 0
    GIRL = 1

def random_kid() -> Kid:
    return random.choice([Kid.BOY, Kid.GIRL])

both_girls = 0
older_girl = 0
either_girl = 0
random.seed(0)
for _ in range(10000):
    younger = random_kid()
    older = random_kid()
    if older == Kid.GIRL:
        older_girl += 1
    if older == Kid.GIRL and younger == Kid.GIRL:
        both_girls += 1
    if older == Kid.GIRL or younger == Kid.GIRL:
        either_girl += 1

print("P(both | older):", both_girls / older_girl)   # 0.5007089325501317
print("P(both | either):", both_girls / either_girl) # 0.3311897106109325

We can see that code confirms our intuition by looking at each of the joint probabilities

either_girl #7,464 / 10,000 ~ roughly 75% or 3/4 probability that there is at least one girl
both_girls  #2,472 / 10,000 ~ roughly 25% or 1/4 probability that both children are girls
older_girl  #4,937 / 10,000 ~ roughly 50% or 1/2 probability that the first child is a girl

Code Breakdown

Challenge 1:

• P(B|G) = P(B,G) / P(G) or more explicitly:
• P(both_girls | older_girl) = P(both_girls) / P(older_girl)

Challenge 2:

• P(B|L) = P(B,L) / P(L) or more explicitly:
• P(both_girls | either_girl) = P(both_girls) / P(either_girl)

Conditional probabilities are conditional statements in code.

First, we establish a random function that assigns a random.choice method to assign gender such that each child (i.e.,Kid class instance) is equally likely to be a boy or a girl. This is the first assumption of our scenario.

import enum, random

class Kid(enum.Enum):
    BOY = 0
    GIRL = 1

def random_kid() -> Kid:
    return random.choice([Kid.BOY, Kid.GIRL])

Next we create variables representing joint distributions; one variable for both children being girls (both_girls), one variable for only the older child being a girl (older_girl), and one for at least one child being a girl (either_girl).

First the probability of any one child being a girl is (1/2), consistent with our assumption, we'd expect:

older_girl  #4,937 / 10,000 ~ roughly 50% or 1/2 probability that the first child is a girl

Recall that when we take the product of each child being a girl (1/2), we can figure out the joint probability of both child being a girl (1/4). Thus, we'd expect:

both_girls  #2,472 / 10,000 ~ roughly 25% or 1/4 probability that both children are girls

Finally, recall that if we're trying to calculate that at least one (of two) children is a girl, we can rule-out the (1/4) probability that both children are boys leaving (1/4 + 1/4 + 1/4 = 3/4) (see table above). Thus, we'd expect:

either_girl #7,464 / 10,000 ~ roughly 75% or 3/4 probability that there is at least one girl

To arrive at the numbers we see above, we create 10,000 simulations of scenarios where the 1st Child and 2nd Child (see table above) are randomly assigned gender in each of the scenario and conditionally filter through the code to see if certain outcomes are True.

random.seed(0)

for _ in range(10000):
    younger = random_kid()
    older = random_kid()
    if older == Kid.GIRL:
        older_girl += 1
    if older == Kid.GIRL and younger == Kid.GIRL:
        both_girls += 1
    if older == Kid.GIRL or younger == Kid.GIRL:
        either_girl += 1

This simulation yields the joint probabilities which are then used to find the conditional probabilities of the two outcomes above:

print("P(both | older):", both_girls / older_girl)   # 0.5007089325501317
print("P(both | either):", both_girls / either_girl) # 0.3311897106109325

For more content on data science, machine learning, R, Python, SQL and more, find me on Twitter.