The Math Behind the Machine/ Unit 15 · The Network, Whole Checks 0/1 Sign in
Unit 15 of 20 · by Prof. Saurabh

The Network, Whole

For fourteen units we collected parts: matrices that move space, the chain rule, blame flowing backwards, a step downhill, softmax. Now we bolt them together into a real neural network — and we keep every number visible. A layer is just three things: multiply by a matrix, add a shift, then bend. Values go forward through the layers. Then the blame for the mistake comes back along the same wires. At the output, the whole blame is simply prediction minus truth. Everything before it is Unit 7's "local slope times incoming blame", done with matrices. At the end you build your own network and watch it learn.

≈ 120 min read + play 14 interactive widgets · 5 in 3D · a playground to build your own network 20 inline checks 🧾 15 proofs, folded away — open "if you want the algebra" when you are ready ✍ 14 solved practice problems

← Unit 14 · Thinking in Probabilities

values go right · blame comes back · drag to orbit
1

A network is a stack of simple steps

Imagine this

Think of a line of stations in a factory. A part comes in. The first station does one small job and passes it on. The next station does another small job. No station is clever. But at the end of the line, a whole scooter rolls out.

A neural network is the same kind of line. Each station is simple. The power comes from putting many of them in a row.

Over fourteen units we collected the parts. A matrix is a machine that moves space (Unit 1). The chain rule multiplies slopes along a chain (Unit 6). Blame can flow backwards through a graph (Unit 7). A step downhill makes a guess better (Unit 9). Softmax turns scores into probabilities (Unit 14). Now we bolt them together.

Here is the one station a network is built from. We call it a layer, and it does three things in order:

  1. Multiply by a matrix. Mix the incoming numbers, each with its own weight.
  2. Add a shift. Add one extra number to each result. This is the bias.
  3. Bend. Pass each result through a simple bent function, like "keep it if positive, else make it zero". This is the activation.

In symbols, layer number ll takes the list a(l−1)\mathbf a^{(l-1)} that arrives and makes two new lists:

z(l)=W(l)a(l−1)+b(l),a(l)=ReLU⁡(z(l)).\begin{gathered}\mathbf z^{(l)}=W^{(l)}\mathbf a^{(l-1)}+\mathbf b^{(l)},\\ \mathbf a^{(l)}=\operatorname{ReLU}\big(\mathbf z^{(l)}\big).\end{gathered}

The list z\mathbf z is the mix before the bend. The list a\mathbf a is what comes out after the bend. The superscript (l)(l) just says which layer. That is all a layer is.

Meet the salary network. We will use one small network for the whole unit. It guesses a person's salary (in lakhs) from two facts: their age and their years of experience. The two facts go in. A layer of 3 neurons works on them. A layer of 2 neurons works on that. One neuron at the end gives the guess, y^\hat y. We write this shape as 2 → 3 → 2 → 1.

How big is each matrix? There is one simple rule. A weight joins one neuron in this layer to one neuron in the layer before. So the matrix has one row for every neuron here and one column for every neuron before:

W(l) is (neurons in layer l)×(neurons in layer l−1).W^{(l)}\ \text{is}\ (\text{neurons in layer } l)\times(\text{neurons in layer } l-1).

So W(1)W^{(1)} is 3×23\times2, W(2)W^{(2)} is 2×32\times3 and W(3)W^{(3)} is 1×21\times2. Each bias has one entry per neuron: 3×13\times1, 2×12\times1, 1×11\times1. Add them up: 6+3+6+2+2+1=206+3+6+2+2+1=20 numbers. Those 20 numbers are everything the network knows. Training means changing them.

One network, three ways to draw itThe same 2 → 3 → 2 → 1 network as a graph of neurons, as six matrices, and as one chain of eight boxes. Point at anything and its twin lights up in the other picture.

Try: Point at the middle bundle of wires in the graph and watch W(2)W^{(2)} light up. Then switch to the chain and point at z(1)\mathbf z^{(1)}. Last, turn on the shape quiz and click each "?" until the shape is right.

Point at a wire, a neuron or a box.
The realization

z(l)=W(l)a(l−1)+b(l)a(l)=ReLU⁡(z(l))\begin{gathered}\mathbf z^{(l)}=W^{(l)}\mathbf a^{(l-1)}+\mathbf b^{(l)}\\ \mathbf a^{(l)}=\operatorname{ReLU}\big(\mathbf z^{(l)}\big)\end{gathered}

A network is not a mystery box. It is a chain of simple steps: multiply, shift, bend — again and again. The graph, the matrices and the chain are three drawings of the same thing. The rows of each matrix belong to where the wires go; the columns to where they come from.

Pause & predict

A network has shape 4 → 5 → 3. What is the shape of W(2)W^{(2)}, and how many numbers (weights and biases) does the whole network have?

The road ahead. The unit has four acts.

  1. What a network is made of (§1–§4): one neuron is a line and a bend; without the bend, layers collapse; with it, enough folds can draw any shape.
  2. The forward pass (§5–§6): push age 30 and experience 10 through the salary network by hand, then score the answer.
  3. The backward pass (§7–§9): two rules carry the blame back to every one of the 20 numbers; then one step makes the guess better.
  4. Build your own (§10–§12): when blame fades or dies, and a playground where you build a network and watch it learn.

In one sentence: A network is a line of simple layers, and each layer only multiplies by a matrix, adds a shift, and bends — the matrix having one row per neuron here and one column per neuron before.

Free preview · Unit 15 of 20

That was section 1. The rest of the unit opens when you unlock it.

11 more sections and the practice arena — 14 widgets, 20 checks and 14 solved problems in the whole unit (this preview had 1 widget and 1 check).

Unlock Unit 15

  1. 2

    One neuron is a line and a bend

  2. 3

    Why the bend matters: straight layers collapse

  3. 4

    Fold by fold: a network can draw any shape

  4. 5

    The forward pass, by hand

  5. 6

    What comes out, and how we score it

  6. 7

    Two kinds of arrow, two rules

  7. 8

    The backward pass, by hand

  8. 9

    Update, check, and why the step had to be tiny

  9. 10

    When the blame fades, explodes, or dies

  10. 11

    The playground: build a network

  11. 12

    What to carry forward

  12. 13

    Practice arena — fourteen problems, solved in full

Unlock this unit for ₹299, or all seven paid units for ₹999 — one-time payment, full refund within 7 days. See pricing.