Thinking in Probabilities

Every unit so far gave crisp answers. Solve $A\mathbf x=\mathbf b$: here is $\mathbf x$. Project onto a line: here is the shadow. Unit 13's support vector machine says "$+1$" or "$-1$".

But look at how the SVM says it. A point deep inside the blue group gets "$+1$". A point that only just grazes the street also gets "$+1$". Same answer, same confident voice. The machine cannot say "probably blue, but I am not sure".

The world is full of such "probably". Will it rain? Is this patient sick? Is this photo a cat? The honest answer is a number between 0 and 1. So we need a language for uncertainty. That language is probability, and this whole unit teaches it.

Start with one word. A random variable is a rule that turns an uncertain outcome into a number. Toss a coin and write 1 for heads, 0 for tails. Roll two dice and write their sum. Measure tomorrow's rainfall in millimetres. Each time, you cannot know the number in advance. But you can say how likely each number is.

Here is a machine that makes a random variable. It is called a Galton board. A bead drops in at the top and meets a peg. It bounces left or right, like a coin toss. Then it meets another peg, and another — twelve rows of them. At the bottom it falls into a slot. The slot number is simply how many times it bounced right.

Where will one bead land? Nobody can say. But drop two thousand, and something surprising happens.

Why a bell? Count the routes. To reach slot 0 a bead must bounce left twelve times in a row: there is only one such route. To reach the middle slot, 6, it needs six lefts and six rights in any order, and there are 924 different orders. Out of all $2^{12}=4096$ routes, 924 end in the middle. So about 23% of the beads land there, and almost none at the edges.

No bead knows about bells. Each one just flips a coin at each peg. The shape comes from counting. That is the first big idea of this unit: a single outcome is unpredictable, but the pattern of many outcomes is not. The pattern is called a distribution, and it is the thing we can know.

The road ahead. The unit has four acts.

1. Uncertainty has a shape (§1–§3): distributions, their balance point and spread, and Bayes' rule for turning evidence around.
2. The Gaussian (§4–§6): the bell curve, why it keeps appearing, what it looks like in many dimensions (Unit 12's ellipse!), and how a computer draws samples from it.
3. Learning = the least surprising world (§7–§9): likelihood, fitting by maximum likelihood, and the discovery that least squares was maximum likelihood all along.
4. Scores into probabilities (§10–§13): softmax, surprise and entropy, cross-entropy and KL, and a classifier that trains itself with the cleanest gradient in machine learning.