Unit 20 · From Noise to Pictures: VAEs and Diffusion

Drawing from a hat of noise

Imagine this

Before Ganesh Chaturthi, a sculptor in a small workshop makes one idol after another. Every morning his helper hands him a lump of clay. No two lumps are the same: one is a little bigger, one a little lopsided, one a little wetter.

The sculptor's hands make the same moves every time. Yet every idol comes out slightly different, and every one of them still looks like Ganesha. The skill is fixed. The lump is random. Put the two together and new idols keep coming.

The question. How can a machine draw a face that has never existed?

Here is the whole idea of this unit in one line: to create is to sample. A machine that creates is a machine that takes easy randomness — the lump of clay — and turns it into something that looks like the real data. We call such a machine a generator. The randomness goes in, the sculptor's fixed skill is a function $g$ , and a new example comes out: $x=g(z)$ .

By hand, with heights. Say the heights of grown men in a town follow a bell curve with mean 170 cm and spread 7 cm, written $N(170,\ 7^2)$ . How do we make up a new, realistic height? Draw a standard random number $z$ from $N(0,1)$ — the easiest bell curve there is, centred at 0 with spread 1 — and then stretch and shift it:

$x=\mu+\sigma z=170+7z.$

Four draws of the hat give $z=-1,\ 0,\ 0.5,\ 2$ . The machine turns them into

$\begin{aligned}170+7(-1)&=163\\ 170+7(0)&=170\\ 170+7(0.5)&=173.5\\ 170+7(2)&=184.\end{aligned}$

Four new men who never existed, each with a believable height. The skill ( $\times7$ , then $+170$ ) never changed. Only the lump $z$ did.

Not every shape is a bell. Waiting times at a bus stop, say, follow an exponential curve: short waits are common and long waits are rare. With $\lambda=1$ the chance that the wait is at most $x$ minutes is $F(x)=1-e^{-x}$ . Now take the plainest noise of all, a number $u$ spread evenly between 0 and 1, and solve $u=F(x)$ for $x$ :

$x=-\ln(1-u).$

With $u=0.5$ : $x=-\ln 0.5\approx0.6931$ minutes. This trick is called the inverse CDF (the CDF is the "chance of at most $x$ " curve, $F$ ). It works for any shape of data in one dimension: run the curve $F$ backwards on even noise.

The hard part. For heights and waiting times someone can write the curve down. For pictures, nobody can. A 20 × 20 grey picture is already 400 numbers, and there is no formula for "all the pictures that look like a face". So the generator has to be learned from examples. That is what the whole unit is about: two ways to learn the sculptor's hands — the VAE (Act II) and diffusion (Act III).

Try: On heights, press the four draws of the text: the balls come out at 163, 170, 173.5 and 184. Press draw 500 and watch the green pile take the bell shape of $N(170,\ 7^2)$ . Switch to waiting times and press draw u = 0.5: it comes out at 0.6931. Then try two bumps: plain bell-curve noise in, two humps out — the steep part of the curve splits the pile in two.

Why does this work?

Look at the curve as a funnel. Where it is steep, a narrow band of inputs is stretched over a wide band of outputs, so the balls spread thin. Where it is flat, a wide band of inputs is squeezed into a narrow band of outputs, so the balls pile up. So the curve decides where the pile is high and where it is low: output density = input density ÷ slope. The inverse CDF picks exactly the curve whose slope makes the pile come out right. A learned generator does the same thing in hundreds of dimensions, with a neural network as the curve.

Four ways to learn the sculptor's hands. Every modern generator turns noise into data. They differ in how the noise goes in and how the machine is trained.

family	how it draws	where the noise goes in	good at	weak at
autoregressive (Unit 19)	one piece at a time, each piece chosen from a probability list	a dice roll at every token	text; exact likelihood	slow: one step per piece
VAE (§3–§7)	pick a point on a small map, decode it in one go	the point $z\sim N(0,I)$	fast; a smooth map you can walk	blurry pictures
GAN	one pass of a generator network	a random vector $z$	sharp pictures, fast	unstable training; can forget whole kinds of data
diffusion (§8–§13)	start from static, remove a little noise many times	the starting static, and a little at every step	sharp, varied pictures; easy to steer	many steps (§12 makes it faster)

A GAN in one paragraph. A GAN (a generative adversarial network) trains two networks against each other, like a forger and a detective. The forger turns noise into fake pictures. The detective looks at real pictures and fakes and says "real" or "fake". The forger improves by fooling the detective, the detective improves by catching the forger. It gives sharp pictures, but the contest is hard to balance, and the forger may learn to draw only a few kinds of picture that fool the detective. We will not build one here: the VAE and diffusion are the ones whose maths this course has prepared you for.

Rule of thumb. Text: autoregressive. A fast generator with a smooth map of the data: VAE. The best pictures today: diffusion — often run inside a VAE's map (§13).

Trap

A machine that hands back the training pictures is not creating — it is copying. We want new examples that come from the same distribution as the training data: new heights that are believable, not the heights of the men we measured. A generator is judged by whether its samples look like they came from the same hat, not by whether they match any one example.

The realization

$x=g(z),\qquad z\sim\text{easy noise}$

A generator is a fixed function applied to fresh noise. In one dimension we can write the function down: stretch-and-shift for a bell curve, the inverse CDF for anything else. For pictures the function must be learned — and learning it is the whole art.

Pause & predict

In the heights machine you change the curve from $170+7z$ to $170+14z$ . What happens to the green pile of outputs?

Pause & predict

In the waiting-times machine $x=-\ln(1-u)$ , where do most of the outputs pile up, and why?

If you want the algebra · 2 proofs, step by step

Prove it · running the CDF backwards gives the right shape

Claim. Let $F$ be a CDF that rises steadily (so it has an inverse), and let $u$ be spread evenly on $(0,1)$ . Then $x=F^{-1}(u)$ has exactly the CDF $F$ .

Ask for the chance that the output is at most some value

a

\begin{aligned}&P(x\le a)\\ &=P\big(F^{-1}(u)\le a\big)\\ &=P\big(u\le F(a)\big).\end{aligned}

F

rises steadily, so applying it to both sides keeps the "

\le

" the right way round.

For even noise on

(0,1)

, the chance of landing below any number

c

between 0 and 1 is just

c

. So

P(u\le F(a))=F(a)

— exactly the CDF we wanted. ∎ Exponential with

\lambda=1

F(x)=1-e^{-x}

, so

u=1-e^{-x}

gives

x=-\ln(1-u)

;

u=0.5

gives

\ln2\approx0.6931

Prove it · output density = input density ÷ slope

Claim. If $x=g(z)$ with $g$ increasing, and $z$ has density $p_z$ , then $x$ has density $p_x(x)=p_z(z)\,/\,g'(z)$ at $x=g(z)$ .

A thin band of inputs

[z,\ z+dz]

holds probability

p_z(z)\,dz

. The curve sends it to the band

[x,\ x+g'(z)\,dz]

, which must hold the same probability:

p_x(x)\,g'(z)\,dz=p_z(z)\,dz.

Balls are neither made nor lost on the way through the curve.

Divide by

g'(z)\,dz

p_x(x)=p_z(z)/g'(z)

. Steep curve, thin pile; flat curve, tall pile. ∎ For

x=170+7z

g'=7

, so the output bell is 7 times wider and 7 times lower than

N(0,1)

— that is

N(170,\ 7^2)

. In many dimensions the slope becomes a Jacobian determinant (Unit 4's volume factor).

The road ahead. The unit has four acts.

To create is to sample (§1–§2): the noise machine, and the two facts about Gaussian noise that everything later uses.
Squeeze, then rebuild (§3–§7): the autoencoder, the VAE's cloud, the reparameterisation trick, the ELBO, and walking the map of faces.
Noise in, noise out (§8–§12): diffusion — destroy a picture with noise, learn to guess the noise, the score as a compass, walk back from static, and the fast jump.
Steering, and the whole course (§13–§14): steering with words, latent diffusion, and every tool of the course inside the two machines.

In one sentence: A generator is a sculptor's fixed skill applied to a random lump of clay — $x=g(z)$ with fresh noise $z$ — and since nobody can write $g$ down for pictures, the machine has to learn it.

Free preview · Unit 20 of 20

That was section 1. The rest of the unit opens when you unlock it.

13 more sections and the practice arena — 14 widgets, 32 checks and 16 solved problems in the whole unit (this preview had 1 widget and 2 checks).

Unlock Unit 20

2
The Gaussian toolkit for noise
3
The autoencoder: squeeze and rebuild
4
The VAE: a cloud, not a point
5
The reparameterisation trick
6
The ELBO: the score a VAE maximises
7
Walking the latent map
8
Forward: destroy a picture one small step at a time
9
The denoiser: guess the noise
10
The score: a compass pointing at the data
11
Walking back from static
12
Faster: DDIM and the smooth path
13
Steering with words: guidance and latent diffusion
14
What to carry forward — and the whole course inside one picture
15
Practice arena — sixteen problems, solved in full

Unlock this unit for ₹299, or all seven paid units for ₹999 — one-time payment, full refund within 7 days. See pricing.

Drawing from a hat of noise

That was section 1. The rest of the unit opens when you unlock it.

The Gaussian toolkit for noise

The autoencoder: squeeze and rebuild

The VAE: a cloud, not a point

The reparameterisation trick

The ELBO: the score a VAE maximises

Walking the latent map

Forward: destroy a picture one small step at a time

The denoiser: guess the noise

The score: a compass pointing at the data

Walking back from static

Faster: DDIM and the smooth path

Steering with words: guidance and latent diffusion

What to carry forward — and the whole course inside one picture

Practice arena — sixteen problems, solved in full