Probability Theory — Quantifying Uncertainty with Numbers

Definition of Probability

Probability: A number between 0 and 1 representing the likelihood that an event occurs.

Three Interpretations:

Classical probability: Ratio of favorable outcomes to equally-likely outcomes in the sample space
Frequentist probability: Relative frequency over many repeated trials
Subjective probability: An individual’s degree of belief (Bayesian perspective)

Probability Axioms (Kolmogorov):

P(A) ≥ 0 (non-negativity)
P(entire sample space) = 1
Mutually exclusive events A, B: P(A∪B) = P(A) + P(B)

Sets and Events

Notation	Meaning
A∪B (Union)	A or B occurs
A∩B (Intersection)	Both A and B occur
Aᶜ (Complement)	A does not occur
∅ (Empty set)	An impossible event

Mutually Exclusive: A∩B = ∅
Collectively Exhaustive: A∪B = entire sample space

Addition Rule

\begin{aligned} P(A∪B) &= P(A) + P(B) − P(A∩B) \\ \text{If mutually exclusive}: P(A∪B) &= P(A) + P(B) \end{aligned}

Conditional Probability

Conditional Probability P(A|B): The probability that A occurs given that B has occurred

P(A \mid B) = P(A∩B) / P(B) (\text{provided} P(B) > 0)

Example: A die is rolled and shows an even number (B). What is the probability that it is 4 or higher (A)?

B = 6: P(B) = 3/6
A∩B = 6: P(A∩B) = 2/6
P(A|B) = (2/6) / (3/6) = 2/3

Independent Events

Independence: A’s occurrence does not affect the probability of B

\begin{aligned} \text{Independence condition}: P(A∩B) &= P(A) \times P(B) \\ P(A \mid B) &= P(A) \end{aligned}

Important: Mutually exclusive ≠ Independent
If events are mutually exclusive, knowing one occurred makes the other impossible → they are strongly dependent

Multiplication Rule (Chain Rule)

\begin{aligned} P(A \cap B) &= P(A) \times P(B \mid A) = P(B) \times P(A \mid B) \\ \text{Three events}: P(A \cap B \cap C) &= P(A) \times P(B \mid A) \times P(C \mid A \cap B) \end{aligned}

Law of Total Probability

When events B₁, B₂, …, Bₙ are collectively exhaustive and mutually exclusive:

P(A) = \sum P(A|B_{i}) \times P(B_{i})

Bayes’ Theorem

A formula for updating prior beliefs in light of new evidence.

P(B \mid A) = P(A \mid B) \times P(B) / P(A)

Real-World Example — Medical Diagnosis:

Disease prevalence (prior): $P(\text{Disease}) = 0.01$
Probability of actually having the disease given a positive test (posterior): ?
Sensitivity (true positive rate): $P(\text{Positive} \mid \text{Disease}) = 0.99$
Specificity (true negative rate): $P(\text{Negative} \mid \text{Healthy}) = 0.95$

P(\text{Positive } \mid \text{Healthy}) = 0.05

P(\text{Disease } \mid \text{Positive}) = \frac{0.99 \times 0.01}{0.99 \times 0.01 + 0.05 \times 0.99}

≈ 16.7% → When prevalence is low, even a positive test result is likely a false positive!

Key Concept Cards

Conditional Probability ★★★★★ : P(A|B) = P(A∩B)/P(B). Probability of A given that B has occurred. Core concept in many statistical problems. Memory tip: conditional = denominator is the probability of the conditioning event

Independent Events ★★★★★ : P(A∩B) = P(A)×P(B) if independent. One event’s occurrence does not affect the other’s probability. Memory tip: independent = intersection = product of probabilities

Bayes’ Theorem ★★★★☆ : Updates prior probability with new evidence. Core principle of spam filters, medical diagnosis, and AI classification. Memory tip: posterior = likelihood × prior / evidence

Practice Questions

Q. From a standard deck, drawing a red card (R) and drawing a heart (H) — are these events independent?

Not independent. P(H) = 13/52 = 1/4. P(R) = 26/52 = 1/2. P(H∩R) = 13/52 = 1/4. P(H)×P(R) = 1/4 × 1/2 = 1/8 ≠ 1/4. → Not independent! (If a card is a heart it must be red, so these events are dependent.)

Q. Factory A produces 60% of output with a 2% defect rate; Factory B produces 40% with a 5% defect rate. What is the probability that a randomly selected product is defective?

P(Defective) = P(Defective|A)×P(A) + P(Defective|B)×P(B) = 0.02×0.6 + 0.05×0.4 = 0.012 + 0.020 = 0.032 = 3.2%.