The Math Behind the Machine/ Unit 16 · Words as Vectors Checks 0/2 Sign in
Unit 16 of 20 · by Prof. Saurabh

Words as Vectors

A computer cannot read: to it, a word is just a label, like a code number in a shopkeeper's register. In this unit we give every word a place on a map of meaning — a short list of numbers — so that words used alike live close together. First we count: guess the next word from the words before it, then weigh and squeeze the company words keep. Then we stop counting and start predicting: word2vec's two games, CBOW and skip-gram, learn the map by filling in blanks and guessing neighbours — and turn out to be counting in disguise. At the end, meaning is a direction: king − man + woman lands next to queen.

≈ 150 min read + play · five acts 23 interactive widgets · 5 in 3D · an embedding explorer to come back to 43 inline checks 🧾 30 proofs, folded away — open "if you want the algebra" when you are ready ✍ 16 solved practice problems

← Unit 15 · The Network, Whole

a word is where it lives · drag to orbit
1

A word is just a label

How could a computer ever know that chai is like coffee?

Imagine this

A new shopkeeper opens a stock register. He gives every item a code number: 1 is chai, 2 is coffee, 3 is a cricket bat.

Is 2 "closer" to 1 than 3 is? No. The numbers are just labels. They say nothing about what the items are. Chai and coffee are both hot drinks, but the register cannot know that.

A computer sees words the same way. To a computer, "chai" is just a label. Our job in this unit is to turn every word into numbers that carry its meaning.

First, what counts as a word? Before a computer can count anything, it cuts the text into pieces called tokens. Often everything is lower-cased first, so that "Chai" and "chai" count as one word. Punctuation becomes a token of its own. And we add two markers to every sentence: ⟨s⟩\langle s\rangle at the start and ⟨/s⟩\langle/s\rangle at the end. They mean "a sentence starts here" and "a sentence stops here", and §2 will need them. So I drink chai. becomes

<s> I drink chai . </s>

(We keep "I" as a capital so the examples read naturally.) Modern language models go one step further and cut rare words into smaller pieces; §16 shows how.

Now the simplest way to turn a word into numbers: give each word its own slot. Take a tiny vocabulary of three words — chai, coffee, cricket. Each word becomes a list of three numbers with a single 1 in its own slot:

chai=(1,0,0),coffee=(0,1,0),cricket=(0,0,1).\begin{aligned}\text{chai}&=(1,0,0),\\ \text{coffee}&=(0,1,0),\\ \text{cricket}&=(0,0,1).\end{aligned}

A list of zeros with a single 1 is called a one-hot vector. With a real vocabulary of 50 000 words, each list is 50 000 numbers long: one 1 and 49 999 zeros.

Now measure how alike two words are, the way Unit 3 taught us: with the dot product. Multiply slot by slot and add:

chai⋅coffee=1⋅0+0⋅1+0⋅0=0.\text{chai}\cdot\text{coffee}=1\cdot0+0\cdot1+0\cdot0=0.

Chai with cricket? Also 0. Every pair of different one-hot words has dot product 0: they all stand at right angles to each other. And every pair is the same distance apart: 12+12=2≈1.414\sqrt{1^2+1^2}=\sqrt2\approx1.414. In one-hot land, chai is exactly as far from coffee as it is from cricket.

The picture to keep: a phone directory and a city map

A phone directory gives every person a line of their own. It tells you who exists, but not who lives near whom. One-hot lists are a directory: every word on its own line, every pair equally far apart, no neighbourhoods.

A city map is different. People who live near each other are drawn near each other, and a neighbourhood has a character — the market, the stadium, the station. We want a map of words: tea words in one neighbourhood, cricket words in another, so that on the map close means similar. This whole unit is about turning the directory into the map.

From a directory to a mapOne-hot land puts each word on its own axis, all at right angles. In "meaning" land the lists are free, so words that are used alike can lean together. (The "meaning" arrows here are hand-made, to show the idea.)

Try: Start in one-hot land and read the three distances: all the same, 1.414. Then press ▶ let meaning pull and watch chai and coffee lean together (cosine 0.91) while cricket stays apart. Drag the picture to look from the side.

drag the picture to orbit

Trap

Why not just number the words 1, 2, 3, like the shopkeeper? Because that is worse than one-hot: it invents an order that isn't there. The computer would believe coffee (2) sits halfway between chai (1) and cricket (3), and that chai + cricket = 2 × coffee. One-hot at least tells no lies — it just tells nothing.

The realization

ei⋅ej=0,∥ei−ej∥=2(i≠j)\begin{gathered}\mathbf e_i\cdot\mathbf e_j=0,\\ \lVert\mathbf e_i-\mathbf e_j\rVert=\sqrt2\quad(i\ne j)\end{gathered}

One-hot lists put every word at right angles to every other word, all the same distance apart. They tell words apart, but they cannot say "these two are alike". We want short lists where close means similar. The rest of this unit is about finding them.

Pause & predict

With a vocabulary of 10 000 words written one-hot, what is the dot product of "chai" and "coffee", and how far apart are they?

Pause & predict

The shopkeeper adds a fourth word, "tea", to the one-hot vocabulary. Which of the old words does tea land closest to?

If you want the algebra · 1 proof, step by step
Prove it · every pair of one-hot words is √2 apart

Claim. For one-hot vectors ei\mathbf e_i and ej\mathbf e_j with i≠ji\ne j: ei⋅ej=0\mathbf e_i\cdot\mathbf e_j=0 and ∥ei−ej∥=2\lVert\mathbf e_i-\mathbf e_j\rVert=\sqrt2, whatever the vocabulary size VV.

1
The dot product adds VV products, one per slot. In every slot at least one of the two lists has a 0, because their single 1s sit in different slots. So every product is 0 and the sum is 0. Right angles: the cosine is 0 too.
2
The difference ei−ej\mathbf e_i-\mathbf e_j has +1+1 in slot ii, −1-1 in slot jj and 0 elsewhere. So ∥ei−ej∥2=12+(−1)2=2.\begin{aligned}&\lVert\mathbf e_i-\mathbf e_j\rVert^2\\ &=1^2+(-1)^2=2.\end{aligned} ∎ Or use ∥a−b∥2=∥a∥2+∥b∥2−2 a⋅b=1+1−0\lVert\mathbf a-\mathbf b\rVert^2=\lVert\mathbf a\rVert^2+\lVert\mathbf b\rVert^2-2\,\mathbf a\cdot\mathbf b=1+1-0.

The road ahead. The unit has five acts.

  1. A word is just a label (§1) — the directory we start from.
  2. Guessing the next word by counting (§2–§4): a sentence as a chain of guesses, what to do about words never seen, and how to score a guesser by its surprise — its perplexity.
  3. Meaning from the company a word keeps (§5–§7): count the company, weigh it (TF-IDF and PMI), and squeeze it with the SVD from Unit 5.
  4. Stop counting, start predicting (§8–§14): word2vec's two games, CBOW and skip-gram; how to make them cheap; why counting and predicting arrive at the same answer; and a small neural language model.
  5. The geometry of meaning, and its limits (§15–§17): king − man + woman, words with two meanings, words cut into pieces, and the road to Units 17 and 18.

In one sentence: A one-hot list is a phone directory — every word on its own line, every pair at right angles and 2\sqrt2 apart — so it names words but carries no meaning; we want a city map, where close means similar.

Free preview · Unit 16 of 20

That was section 1. The rest of the unit opens when you unlock it.

16 more sections and the practice arena — 23 widgets, 43 checks and 16 solved problems in the whole unit (this preview had 1 widget and 2 checks).

Unlock Unit 16

  1. 2

    A sentence is a chain of guesses

  2. 3

    Never seen is not impossible: smoothing

  3. 4

    How surprised is the model? Perplexity

  4. 5

    You shall know a word by the company it keeps

  5. 6

    Not all company counts: TF-IDF and PMI

  6. 7

    Squeeze the table: the SVD finds friends of friends

  7. 8

    From counting to predicting: the word2vec idea

  8. 9

    CBOW: the committee fills in the blank

  9. 10

    Skip-gram: one word guesses its neighbours

  10. 11

    CBOW or skip-gram? Race them

  11. 12

    Making it cheap: negative sampling, hierarchical softmax, subsampling

  12. 13

    Why it works: counting and predicting meet

  13. 14

    A neural language model

  14. 15

    The geometry of meaning

  15. 16

    What one vector per word gets wrong — and how pieces help

  16. 17

    What to carry forward

  17. 18

    Practice arena — sixteen problems, solved in full

Unlock this unit for ₹299, or all seven paid units for ₹999 — one-time payment, full refund within 7 days. See pricing.