The Math Behind the Machine/ Unit 19 · The Maths Inside an LLM Checks 0/2 Sign in
Unit 19 of 20 · by Prof. Saurabh

The Maths Inside an LLM

When you type a question and a chatbot answers, what is the machine actually computing? Less than you might think. It reads everything written so far, gives a probability to every possible next token, picks one, adds it to the text — and does it again. This unit opens that loop. How a whole sentence is scored (cross-entropy from Unit 14). How the next word is picked (temperature, top-k, top-p). The tricks that make writing fast and long (the KV cache, rotary clocks). How training grows predictably with size (scaling laws). And how a raw next-word guesser becomes a helpful assistant: a thin low-rank patch (Unit 5), a reward learned from people's choices, and a leash that keeps it sensible.

≈ 150 min read + play 13 interactive widgets · 4 in 3D · a sampler that writes with your settings 32 inline checks 🧾 17 proofs, folded away — open "if you want the algebra" when you are ready ✍ 16 solved practice problems

← Unit 18 · Attention and Transformers

one token at a time · drag to orbit
1

The only thing an LLM does

Imagine this

You type on your phone, "I will reach home in", and the keyboard offers "10 minutes". It has not read your mind. It has seen millions of messages, and after "reach home in" a number of minutes usually comes.

Now imagine that keyboard grown enormous. It has read a large part of the internet, and it does not stop at one suggestion. It takes its own suggestion, adds it to the message, and suggests again. And again. That is a chatbot.

The question. When a chatbot writes a whole paragraph for you, what is it actually doing, step by step?

First, a word about words. The model does not read letters or whole words. It reads tokens: common words and pieces of rarer words, cut by the byte-pair method of Unit 16. GPT-2's list of tokens, its vocabulary, has 50 257 of them. In this unit a token is nearly always a whole word, to keep the pictures simple.

Here is everything a large language model (LLM) does. It is one loop of four moves.

  1. Read. Take every token written so far. Your question, the earlier chat and the part of the reply already written are all one long list of tokens.
  2. Score every token in the vocabulary. The transformer of Unit 18 reads the list and gives each of the 50 257 tokens a score, called a logit: "how well would you fit next?"
  3. Turn the scores into probabilities with softmax (Unit 14). They are positive and add up to 1.
  4. Pick one token, add it to the list, and go back to move 1. Stop when the model picks a special "end" token.

By hand. The text so far is "I drink". To keep it tiny, say the vocabulary has only five tokens: chai, coffee, water, cricket, the. The model gives them the scores (3, 2, 1, −1, 0)(3,\ 2,\ 1,\ -1,\ 0). (We made these scores up by hand; a real model computes them.)

e3≈20.086,e2≈7.389e1≈2.718,e−1≈0.368e0=1total≈31.561\begin{aligned}e^{3}&\approx20.086,\quad e^{2}\approx7.389\\ e^{1}&\approx2.718,\quad e^{-1}\approx0.368\\ e^{0}&=1\\ \text{total}&\approx31.561\end{aligned}

Divide each by the total:

(0.636, 0.234, 0.086,  0.012, 0.032).\begin{aligned}&(0.636,\ 0.234,\ 0.086,\\ &\ \ 0.012,\ 0.032).\end{aligned}

Chai gets almost two thirds. Say we pick chai. The text is now "I drink chai", and the whole model runs again, from the start, on these three tokens, to choose the fourth. One run of the model gives one new token. A reply of 300 tokens is 300 runs.

The chain rule. How likely is the whole sentence? Multiply the guesses, each one given everything before it. This is the chain of guesses from Unit 16:

P(chai, every∣I drink)=P(chai∣I drink)×P(every∣I drink chai).\begin{aligned}&P(\text{chai, every}\mid\text{I drink})\\ &=P(\text{chai}\mid\text{I drink})\\ &\quad\times P(\text{every}\mid\text{I drink chai}).\end{aligned}

The n-gram model of Unit 16 kept only the last one or two words of the "given" part. An LLM keeps all of them: attention (Unit 18) looks back at every earlier token, and the causal mask (Unit 18) makes sure each guess sees only the past.

The write loopTop: every token written so far — all of it goes into the model. Middle: the model, run once. Bottom: its probability for each of the eight tokens in this tiny vocabulary (pink bars). The gold bar is the one picked; the dashed gold arrow adds it to the text, and on the next run it goes into the model with the rest. The scores are hand-made.

Try: The picture opens after the first run: chai got 0.636 and coffee 0.234 — the numbers above. Press ▶ write and count the runs: four runs of the model write four tokens, "chai every morning .". Watch the arrows: at every run the whole text goes back into the model, including the words it just wrote itself.

Why does this work?

The game is simple, but playing it well is not. To guess what comes after "The capital of Japan is", the model must know geography. After "2 + 3 =" it needs arithmetic; after "def add(a, b):" it needs to know how Python code is written. Making the next-token guess good on trillions of tokens forces the weights to soak up grammar, facts and styles — because they all help the guess. And one step at a time is enough to write long, sensible text, because every step reads everything written before, including the words the model itself has just chosen.

Three next-word guessers, side by side. You have now met three machines that play the same game.

n-gram (Unit 16)recurrent net (Unit 17)transformer LLM (Unit 18, this unit)
what it readsthe last one or two wordsone running note of everything so farevery earlier token, directly
the start of a long textforgotten after a few wordsfades as the note is rewrittenkept exactly, up to the context length
how it learnscountingone word after anotherevery position at once (the mask)
work for one new tokenone look-up in a tableone step of the cellattention over all earlier tokens (§6 makes it cheaper)

Rule of thumb. All three compute the same thing: a probability for the next token given the ones before. They differ only in how much of the past they can use. The transformer uses all of it, which is why it won.

Trap

The model does not plan the sentence and then type it out. There is no hidden finished answer waiting inside. Each token is chosen given only what is already written. When a reply starts well and then wanders, this is why: every new token becomes part of the "given" for all the tokens after it.

The realization

P(w1,…,wn)=∏t=1nP(wt∣w1,…,wt−1)\begin{gathered}P(w_1,\dots,w_n)\\ =\prod_{t=1}^{n}P\big(w_t\mid w_1,\dots,w_{t-1}\big)\end{gathered}

A language model is one function: "given the text so far, a probability for every possible next token". Run it, pick a token, append it, run it again. Multiply the guesses along the way and you get the probability of the whole text.

Pause & predict

In the worked example you add 5 to every score: (8, 7, 6, 4, 5)(8,\ 7,\ 6,\ 4,\ 5). What does the model do now?

Pause & predict

You ask a question 40 tokens long. The model writes a reply of 60 new tokens (the last one is the "end" token). How many times did the model run?

If you want the algebra · 1 proof, step by step
Prove it · the chain rule of probability

Claim. For any tokens w1,…,wnw_1,\dots,w_n, P(w1,…,wn)=∏t=1nP(wt∣w1,…,wt−1)P(w_1,\dots,w_n)=\prod_{t=1}^{n}P(w_t\mid w_1,\dots,w_{t-1}) (for t=1t=1 the "given" part is empty: P(w1)P(w_1)).

1
Conditional probability is defined by P(B∣A)=P(A,B)/P(A)P(B\mid A)=P(A,B)/P(A). Multiply both sides by P(A)P(A): P(A,B)=P(A) P(B∣A).\begin{gathered}P(A,B)\\ =P(A)\,P(B\mid A).\end{gathered} "A and then B" = "A" times "B, now that A has happened" (Unit 14).
2
Take A=(w1,…,wn−1)A=(w_1,\dots,w_{n-1}) — write it w<nw_{\lt n} — and B=wnB=w_n: P(w1,…,wn)=P(w1,…,wn−1)×P(wn∣w<n).\begin{aligned}&P(w_1,\dots,w_n)\\ &=P(w_1,\dots,w_{n-1})\\ &\quad\times P(w_n\mid w_{\lt n}).\end{aligned} The last guess peels off, given everything before it.
3
Do the same to P(w1,…,wn−1)P(w_1,\dots,w_{n-1}), and again, until only P(w1)P(w_1) is left. The factors that peel off are exactly P(wt∣w1,…,wt−1)P(w_t\mid w_1,\dots,w_{t-1}) for t=n,n−1,…,1t=n,n-1,\dots,1. ∎ No assumption was made about language. The n-gram's shortcut — "only the last two words matter" — is an extra assumption; the LLM does not make it.

The road ahead. The unit has four acts.

  1. A machine that guesses the next word (§1–§3): the loop, how a whole sentence is scored, and where the model's millions of numbers live.
  2. How it writes (§4–§7): picking the word — greedy, temperature, top-k and top-p — then the cache that makes writing fast and the clocks that make long texts possible.
  3. Training at scale (§8): how loss falls as models and data grow, and how to split a fixed budget.
  4. From guesser to assistant (§9–§13): copying good answers, fine-tuning with a thin low-rank patch, a reward learned from people's choices, the leash that keeps the model sensible, and a shortcut called DPO.

In one sentence: An LLM is your phone's autocomplete grown huge — read everything so far, give every token a probability, pick one, append it, repeat — and the chain rule multiplies those guesses into the probability of the whole text.

Free preview · Unit 19 of 20

That was section 1. The rest of the unit opens when you unlock it.

13 more sections and the practice arena — 13 widgets, 32 checks and 16 solved problems in the whole unit (this preview had 1 widget and 2 checks).

Unlock Unit 19

  1. 2

    Scoring a whole sentence: likelihood, loss and perplexity

  2. 3

    Inside the stack, by the numbers

  3. 4

    Picking the word: greedy and temperature

  4. 5

    Cutting the tail: top-k and top-p

  5. 6

    The KV cache: never recompute the past

  6. 7

    Longer contexts: rotary positions at full size

  7. 8

    Scaling laws: bigger, with more data, predictably better

  8. 9

    Teaching it to follow instructions

  9. 10

    LoRA: fine-tune a giant by changing a thin slice

  10. 11

    A reward from human choices

  11. 12

    Chasing the reward on a leash

  12. 13

    DPO: skip the reward model

  13. 14

    What to carry forward

  14. 15

    Practice arena — sixteen problems, solved in full

Unlock this unit for ₹299, or all seven paid units for ₹999 — one-time payment, full refund within 7 days. See pricing.