Probability: The Language of Uncertainty (and Bayes' Theorem)

Hook #

Everything you've learned in this ML-math year so far has been about certainty. Linear algebra transforms a vector into another vector — deterministically, exactly. Calculus computes a gradient — one definite answer. But machine learning is fundamentally about uncertainty: the data is noisy, the model is never sure, the prediction is a probability ("87% cat"), the loss is an expected value over a distribution, and the entire enterprise is "reason well under uncertainty from limited, noisy data." Probability is the mathematics of uncertainty — and it is, honestly, the genuinely hard part of ML math. Not because the rules are complicated (there are only a handful of them), but because human beings are bad at probabilistic thinking: we evolved to reason in certainties and stories, not in distributions, so probability constantly says things that feel wrong even when they're provably right. This lesson lays the foundation: the three axioms every probability obeys, conditional probability (the probability of A given that B happened — the concept that breaks the most brains), independence (when knowing B tells you nothing about A), and the crown jewel, Bayes' theorem — the rule for updating a belief when new evidence arrives, which is the deepest idea in the whole subject and the engine of a huge swath of machine learning. Expect this to feel harder than the linear algebra or calculus did. That's not you failing; that's the subject. Sit with the discomfort — it's the price of admission to reasoning well about a noisy world.

What you'll be able to do by the end of this lesson #

• State what a probability is — a number between 0 and 1 measuring how likely an event is — and the three axioms it obeys (non-negativity, the whole sample space has probability 1, and probabilities of mutually-exclusive events add).
• Explain conditional probability — P(A | B), "the probability of A given that B happened" — as updating the sample space to the world where B is true, and why it is not symmetric (P(A|B) ≠ P(B|A) in general — the confusion at the heart of most probability mistakes).
• Explain independence — A and B are independent when knowing one tells you nothing about the other (P(A|B) = P(A)) — and why the assumption of independence is everywhere in ML (and often wrong).
• State and interpret Bayes' theorem — how to flip a conditional (P(A|B) from P(B|A)) — and read it as belief updating: start with a prior, see evidence, get a posterior. Recognize the base-rate fallacy (why a "99% accurate" medical test can still mean you probably don't have the disease).

A quick try before we start #

Here's the puzzle that has humbled doctors, lawyers, and statisticians for decades. A disease affects 1 in 1,000 people. There's a test that is 99% accurate: if you have the disease, it says "positive" 99% of the time; if you don't, it says "negative" 99% of the time (so it has a 1% false-positive rate). You take the test. It comes back positive. What's the probability you actually have the disease?

Most people — including most physicians, in real studies — say "99%." The real answer is about 9%. Here's the intuition (this is Bayes' theorem, which we'll formalize): imagine 100,000 people. About 100 of them have the disease (1 in 1,000), and the test correctly flags ~99 of those. But of the 99,900 healthy people, the 1% false-positive rate wrongly flags ~999 of them. So among everyone who tests positive (~99 + ~999 ≈ 1,098 people), only ~99 actually have the disease — about 9%. The false positives from the huge healthy population swamp the true positives from the tiny sick population. The test being "99% accurate" felt like it should mean "99% chance you're sick," but that ignored the base rate — the disease is rare, so most positives are false alarms. This is the single most important probabilistic reasoning error humans make, and Bayes' theorem is the tool that gets it right every time. Hold this puzzle; by the end of the lesson the "9%" will feel obvious rather than shocking.

Why this matters here #

This lesson opens Probability and Statistics — the third and final pillar of ML math, completing the toolkit that linear algebra (the geometry of data) and calculus (the mechanics of improvement) began. And it earns a warning the previous two courses didn't need: this is the hard part. The honest framing from the start of this year applies most sharply here — you need less probability than gatekeepers claim but far more than "just call .fit()" admits, and the reason it's hard is not the machinery but that probabilistic thinking is counterintuitive to the human brain. Conditional probability breaks people. Bayesian reasoning breaks people. Not because they're not smart enough — because evolution built us to think in certainties and narratives, not distributions and conditionals. So the emotional task of this course is patience with confusion: you will read "the probability of A given B is not the probability of B given A" and it will feel like a technicality, and then you'll get the medical-test puzzle wrong, and that's when it becomes real.

Why does an engineer need this, concretely? Because machine learning is applied probability. A classifier doesn't output "cat" — it outputs a probability distribution over classes (P(cat) = 0.87, P(dog) = 0.13), and to train it you maximize the likelihood of the observed data (next lessons — that's probability). The most-used loss function in all of deep learning, cross-entropy, is a probability/information-theory quantity (last lesson of this course). A shockingly effective family of classifiers is literally called Naive Bayes — it's Bayes' theorem with an independence assumption baked in. Every "confidence score," every "the model is 87% sure," every calibration question ("does 87% actually mean 87%?"), every A/B test's p-value, every "is this result statistically significant" — all of it is the probability you're learning here. And the reasoning skill matters as much as the formulas: the base-rate fallacy from the puzzle above is exactly why a fraud detector that's "99% accurate" can still be useless (fraud is rare, so most flags are false), why a rare-disease screen produces mostly false positives, why an anomaly detector drowns you in false alarms. Understanding conditional probability and Bayes is what lets you reason correctly about any rare-event detection — which is a huge fraction of real ML. The engineer who thinks probabilistically can reason about model confidence, evaluate a classifier honestly, and avoid the base-rate trap; the one who can't will misread every "accuracy" number they see.

The engineer's lens #

The first lens is probability as a number between 0 and 1, governed by three simple axioms — the rules are easy; the thinking is what's hard. A probability is just a measure of how likely an event is: 0 means impossible, 1 means certain, 0.5 means "as likely as not." Everything rests on three axioms, and they are genuinely almost trivially simple: (1) probabilities are never negative; (2) the probability of the entire sample space (the set of all possible outcomes — something happens) is exactly 1; and (3) for mutually exclusive events (ones that can't both happen), the probability of "A or B" is P(A) + P(B) (probabilities of disjoint outcomes just add). From these three, every rule of probability follows — the complement rule (P(not A) = 1 − P(A)), the general addition rule (P(A or B) = P(A) + P(B) − P(A and B), subtracting the double-counted overlap), all of it. This is worth internalizing because it's reassuring: probability is not a bottomless pit of arbitrary formulas; it's a tiny set of axioms plus their consequences. The difficulty is never the axioms. It's applying them to situations where your intuition screams the wrong answer — which is why the emphasis in a good probability course (Stat 110) is on stories and puzzles, not memorizing rules. The rules you could learn in an afternoon; the judgment to apply them correctly under uncertainty takes the whole course. Hold the reassurance (the machinery is small and clean) alongside the warning (the intuition is the hard part).

The second lens is conditional probability — the concept that breaks the most brains, and the one that matters most — read as updating your world to the one where the condition is true. P(A | B), read "the probability of A given B," asks: if I already know B happened, how likely is A now? The definition is P(A | B) = P(A and B) / P(B) — and the geometric reading makes it intuitive: knowing B happened shrinks your world to just the outcomes where B is true, and within that smaller world, you ask what fraction also has A. You've conditioned — restricted attention to the B-universe and recomputed. Here is the trap, and it is the central probability error: P(A | B) is not the same as P(B | A). The probability you have the disease given a positive test is wildly different from the probability of a positive test given you have the disease (9% vs. 99% in the puzzle!). "Probability of a positive test given you're sick" (99%, a property of the test) and "probability you're sick given a positive test" (9%, what you actually want to know) are different questions with different answers, and confusing them is the base-rate fallacy. This asymmetry is the single most important thing to internalize in this lesson, because almost every real-world probability mistake is some version of swapping these two. The prosecutor who says "the chance of this DNA match if the defendant is innocent is one in a million, so there's a one-in-a-million chance he's innocent" has committed exactly this swap (the "prosecutor's fallacy") — and it has sent innocent people to prison. For the engineer: whenever you see a conditional probability, stop and ask which direction it runs — "probability of evidence given hypothesis" or "probability of hypothesis given evidence"? They are different, and the whole game of Bayes (next) is converting the one you can measure into the one you actually want.

The third lens is Bayes' theorem — the rule for updating a belief when evidence arrives — because it is the deepest idea in probability and the engine of a large fraction of machine learning. Bayes' theorem flips a conditional: it computes P(A | B) (the direction you want) from P(B | A) (the direction you can measure), and its form is P(A | B) = P(B | A) · P(A) / P(B). But the formula matters far less than the interpretation, which is one of the most profound ideas in all of quantitative reasoning: Bayes' theorem is how you rationally update a belief in light of new evidence. Read the pieces as a story: P(A) is your prior — your belief in hypothesis A before seeing the evidence (in the puzzle: "1 in 1,000 people have the disease" — the base rate, your belief before the test). P(B | A) is the likelihood — how probable the evidence is if the hypothesis is true (a positive test is 99% likely if you're sick). P(A | B) is the posterior — your updated belief after seeing the evidence (given the positive test, how likely are you to be sick — the 9%). Bayes tells you exactly how to combine your prior belief with new evidence to get a rationally updated belief: posterior ∝ likelihood × prior. And this is why the base-rate fallacy happens — people ignore the prior (the base rate) and reason only from the likelihood (the test accuracy), so they update from a blank slate instead of from the (very informative) fact that the disease is rare. Bayes forces the prior back into the calculation, and the rare prior is what pulls "99% accurate test" down to "9% chance you're sick." For the engineer, Bayes is everywhere and worth deep familiarity: it's the literal foundation of Naive Bayes classifiers (spam filtering, text classification — compute P(spam | words) from P(words | spam) and the prior spam rate); it's Bayesian inference (this course's later lesson — treating model parameters themselves as having distributions you update with data); it's the correct frame for any evidence-updating problem (medical diagnosis, fraud detection, A/B testing, sensor fusion). The synthesis of the lesson: probability is the mathematics of uncertainty, built on three simple axioms; conditional probability (P(A|B), "given B") restricts your world to where B is true — and crucially is not the same as P(B|A); and Bayes' theorem flips between the two, giving you the one rule for updating a belief (prior) with evidence (likelihood) into a rational new belief (posterior) — the engine of Bayesian ML and the antidote to the base-rate fallacy that fools human intuition. This is the hard, counterintuitive, load-bearing core of probability, and everything else in the course rests on it.

What to focus on in the resources #

• Harvard Stat 110 (Blitzstein) — primary, and the antidote to the difficulty. Watch the early lectures (sample spaces, conditional probability, Bayes' rule). Blitzstein's story-and-puzzle approach is what makes the counterintuitive parts click — this is why it's the primary resource for the hardest math in the year. Expect to rewatch; that's normal here.
• 3Blue1Brown on Bayes' theorem (free). Watch for the visual "thinking in areas" picture of Bayes and the medical-test puzzle. The geometric intuition and Blitzstein's story intuition reinforce each other — do both.
• Deisenroth (MML) Ch 6, free. Read after Stat 110 builds your intuition — it's the terse, ML-facing version that connects probability to the estimation and information-theory ideas coming later in this course. Don't start here; it'll feel like dense notation without the intuition first.
• Skip on first pass: measure-theoretic foundations (sigma-algebras), combinatorics beyond basic counting, and the zoo of named distributions (that's the next lesson). Get: the three axioms (the machinery is small), conditional probability as conditioning / restricting the world (and P(A|B) ≠ P(B|A)), independence, and Bayes as prior → likelihood → posterior (belief updating) — plus a gut-level grasp of the base-rate fallacy.

Explain it back #

Explain to a colleague why a "99% accurate" test for a rare disease can leave you only ~9% likely to be sick after a positive result — using Bayes' theorem. A strong answer: probability measures uncertainty (0 to 1), built on three simple axioms; the machinery is small, but the thinking is counterintuitive because humans reason in certainties, not distributions. Conditional probability P(A|B) ("A given B") means restricting your world to where B is true and asking how likely A is there — and crucially, P(A|B) ≠ P(B|A): "positive test given sick" (99%, a property of the test) is a different question from "sick given positive test" (what you want to know). Bayes' theorem flips between them: P(sick | positive) = P(positive | sick) · P(sick) / P(positive) — read as posterior ∝ likelihood × prior. The key is the prior: the disease is rare (1 in 1,000), so before any test your belief you're sick is tiny. Even a 99%-accurate test, applied to a huge healthy population, produces far more false positives than true positives (of 100,000 people, ~99 true positives among the sick but ~999 false positives among the healthy), so a positive test only raises your probability to ~9%. The base-rate fallacy is ignoring that rare prior and reasoning only from the test's accuracy. Bayes forces the prior back in, which is why it's the correct tool for any evidence-updating problem — and why it underlies Naive Bayes classifiers, Bayesian inference, and honest reasoning about any rare-event detector (fraud, anomalies, medical screens).

Where this connects #

Backward: This closes the loop the year's math opened. Linear algebra gave the geometry of data, calculus gave the mechanics of improvement (gradients, descent), and probability now gives the reasoning under uncertainty that ties them to real, noisy data. Counting and combinatorics (from discrete math) is the substrate under basic probability (probabilities are often ratios of counts). And the "less than gatekeepers claim, more than .fit() admits — and geometric/intuitive where possible" posture carries over, now with the honest addition that this pillar is the genuinely hard one.

Forward: Conditional probability and Bayes are the seed of the whole course. The next lesson (random variables and distributions) gives probability its quantitative vocabulary — the named distributions ML relies on. Bayes returns as the engine of MAP estimation and Bayesian inference (lesson 4 — treating model parameters as having distributions you update with data). And the whole apparatus underlies cross-entropy (lesson 5), the loss function that trains nearly every classifier and language model in the deep-learning courses ahead. Probability is where ML stops being deterministic math and starts being reasoning about an uncertain world — which is what machine learning actually is.