Machines with Memory

In Unit 16 every word became a list of numbers. So how do we turn a whole sentence into numbers? Here are three ideas. The first two fail, and how they fail tells us what we need.

Idea 1 · Throw the words in a bag. Add up the word vectors, or just count the words. People call this a bag of words. It is quick. But "dog bites man" and "man bites dog" have exactly the same bag: one "dog", one "bites", one "man". The sums are the same. The order is lost.

Idea 2 · Look at the last few words. Keep a small window, say the last 3 words. This is what the neural language model of Unit 16 did: it saw only a fixed window of the words just before. A window keeps local order, but it forgets everything older. Take the announcement "The train to Delhi from platform four is late". When "late" arrives, a 3-word window holds only "four is late". What is late? The word "train" has already dropped out.

Idea 3 · Read one word at a time and keep a running note. Think of a shopkeeper with a notebook. After each customer, he updates one line. But a plain running total does not care about order either: 5 + 3 is the same as 3 + 5. So we add one twist. Before writing in the new word, we first change the old note a little — turn it, shrink it — with a matrix. Then an old word has been turned many times and a new word only once. The order leaves its mark.

A tiny example. Give each word a toy vector (hand-made, not trained): $\text{dog}=(1,0)$, $\text{man}=(0,1)$, $\text{bites}=(1,1)$. The note starts at $(0,0)$. At each word we do two things: turn the note a quarter turn with $W=\begin{pmatrix}0&-1\\ 1&0\end{pmatrix}$, then add the word. A quarter turn sends $(a,b)$ to $(-b,a)$.

• dog bites man: after "dog" the note is $(1,0)$. Turn it: $(0,1)$; add "bites": $(1,2)$. Turn it: $(-2,1)$; add "man": $(-2,2)$.
• man bites dog: after "man", $(0,1)$. Turn: $(-1,0)$; add "bites": $(0,1)$. Turn: $(-1,0)$; add "dog": $(0,0)$.

Two different notes: $(-2,2)$ and $(0,0)$. Without the turn, both would end at the plain sum $(2,2)$. The turn is what makes the note remember order.

The three readers side by side.

Rule of thumb. If the question is "which words appear?" (is this review about food or about service?), a bag is often enough. If the question is "who did what to whom?", you need a reader that keeps order — a running note.

The road ahead. The unit has five acts.

1. Why a reader needs memory (§1–§3): order, the recurrent cell, and the same cell copied along time.
2. A cell that talks (§4–§5): the many jobs one cell can do, and a small language model that learns to write, one character at a time.
3. Blame through time (§6–§8): training sends blame back through every word; it is multiplied by the same matrix again and again, so eigenvalues decide whether memory fades or explodes; then first aid.
4. Gates, and reading both ways (§9–§11): the LSTM's express lane, the GRU's blend dial, and readers that read in both directions or in floors.
5. Translating through one summary (§12–§14): an encoder–decoder that turns English into Hindi, how it picks and grades its words, and the bottleneck that Unit 18 removes.