Unit 7 · Backpropagation & Automatic Differentiation

1

The million-knob problem

Imagine this

You make chai for twenty guests. You taste it: too sweet, and a little weak. What do you change?

You have four knobs: sugar, tea leaves, milk and boiling time. With four knobs you can reason it out. Less sugar, more tea leaves, done.

Now imagine a recipe with a million knobs, and no idea what each one does. That is a neural network. This unit answers one question: how does each knob find out which way to turn?

First, the names. The network makes a guess. The loss is one number that says how bad the guess was, like "too sweet, by this much". Training means turning every knob a little, so that the loss goes down.

From Unit 6 you know what each knob needs. It needs the slope of the loss with respect to that knob — mathematicians call it the partial derivative. It says: "turn me up a little, and the loss changes by this much". Gradient descent then turns each knob a little bit downhill.

So the whole job is this: find a million slopes, quickly, at every training step.

Try doing it by hand for one small function from the lecture:

f(x)=\sqrt{x^2+e^{x^2}}+\cos\!\big(x^2+e^{x^2}\big)

Its slope, worked out with the chain rule, is already a mouthful:

\begin{aligned}\frac{df}{dx}&=\frac{2x+2xe^{x^2}}{2\sqrt{x^2+e^{x^2}}}\\[.3em]&\quad-\sin\!\big(x^2+e^{x^2}\big)\big(2x+2xe^{x^2}\big)\end{aligned}

That is for one input. A real network has millions of knobs, even billions. Writing a formula for each slope is hopeless.

The way out has two ideas. First, break the function into tiny steps. Second, send the blame backwards through those steps. That is backpropagation. The rest of this unit is these two ideas, slowly.

The realization

Training needs one slope per knob, at every step. So the real question is not "what is the slope?" but "how do we get a million slopes cheaply?"

Backpropagation is the answer: tiny steps, and blame sent backwards.

Pause & predict

A network has a million knobs. One way to find each knob's slope: nudge that knob, run the network again, and see how the loss moved. How much work is that per training step?

In one sentence: training needs the loss's slope for every knob at every step, and finding them one knob at a time would cost one run of the network per knob.

2

Baby steps — the computation graph

Imagine this

Your food order arrives 10 minutes late. The delivery app wants to know who caused the delay.

It looks at the chain: the kitchen cooked, the rider picked up, the rider drove to you. It starts at the end, "10 minutes late", and walks backwards along the chain. At each step it asks: how much of the delay came from you?

That backward walk is exactly what backpropagation does. First the order goes forward through the steps. Then the blame comes back through the same steps.

Let's do it with the scary function from Section 1. We will not differentiate it. We will just compute it, one tiny step at a time, and give each result a name:

a=x^2,\quad b=e^{a},\quad c=a+b,\quad d=\sqrt{c},\quad e=\cos(c),\quad f=d+e

Each step is easy: a square, an exponential, an addition. Draw each step as a dot, with an arrow from each input to the step that uses it. This picture is a computation graph. Every deep-learning library builds one behind the scenes.

A tiny example with $x=1$ : $a=1$ , $b=e\approx2.718$ , $c\approx3.718$ , $d\approx1.928$ , $e=\cos(3.718)\approx-0.838$ , so $f\approx1.090$ . That is the forward pass: values flow from left to right.

Now the backward pass. Every dot gets a blame: how much $f$ would change if that dot changed by a tiny bit. We write it with a bar, $\bar v$ (read it: "v-bar"), and it means $\partial f/\partial v$ . The last dot blames itself fully, so $\bar f=1$ . Then each dot hands blame to the dots that fed it, multiplied by its own small, easy slope.

Try: press ▶ forward. Then press step once again and again to walk the blame home. Pause at $c$ : two pieces of blame arrive there and add.

drag the graph to orbit it

input x1

—

Green: values going forward. Orange: blame coming back. The blame that reaches $x$ is the answer $df/dx$ , checked here against a nudge.

Errata — check this against your companion doc

The companion runs this graph at $x=2$ and reports $e=\cos(58.6)\approx0.83$ and $df/dx\approx-2.224$ . The slip: $58.6$ is in radians. In radians, $\cos(58.6)\approx-0.46$ and $\sin(58.6)\approx0.89$ .

So $\bar c=\tfrac{1}{2\sqrt c}-\sin(c)\approx0.065-0.888=-0.822$ (not $-0.01$ ), and the true answer is $df/dx\big|_{x=2}\approx\mathbf{-182.87}$ . The $e^{x^2}$ term makes it huge. The companion's method is right; only the arithmetic slipped. Set the slider to $x=2$ and see for yourself.

The realization

Any function, however scary, is a chain of easy steps. Forward, each step computes a value. Backward, each step receives a blame $\bar v=\partial f/\partial v$ , starting from $\bar f=1$ .

The blame that reaches $x$ at the far left is the answer $df/dx$ .

Pause & predict

In the graph, $a=x^2$ feeds two steps: $b=e^a$ and $c=a+b$ . When the blame walks back, what does $a$ receive?

In one sentence: write the function as a chain of easy steps; values flow forward, and blame $\bar v=\partial f/\partial v$ flows backward from $\bar f=1$ to every step.

3

The whole algorithm is two rules

Imagine this

Petrol goes up by ₹1 a litre. How much does your monthly budget go up?

One route. Each ₹1 on petrol adds ₹0.50 per km to auto fares. You ride 40 km a month. So your fares rise by 0.5 × 40 = ₹20. Along a chain of effects, the rates multiply.

A second route. Petrol also makes vegetables costlier to bring to market: ₹15 more on your monthly vegetable bill. Your total: 20 + 15 = ₹35 for every ₹1 on petrol. When two routes reach the same place, their effects add.

That is all of backpropagation. Two rules:

×Rule 1 — along a path, multiply. Each arrow has a small slope: how much the next step moves when this one moves by 1. Along a path, these slopes multiply, like the petrol rates.
+Rule 2 — where paths meet, add. If $x$ can reach $f$ by several routes, work out each route's product, then add them up.

Backprop simply organises these two rules so that no path is ever walked twice.

Here is the smallest graph with two routes, from the practice set: $f=uv$ , where $u=x^2+y$ and $v=x-y$ . At $(x,y)=(1,1)$ we get $u=2$ , $v=0$ and $f=0$ .

Route through $u$ : the slope from $x$ to $u$ is $2x=2$ , and from $u$ to $f$ it is $v=0$ . Product: $2\times0=0$ .

Route through $v$ : the slope from $x$ to $v$ is $1$ , and from $v$ to $f$ it is $u=2$ . Product: $1\times2=2$ . Now add the routes:

\frac{\partial f}{\partial x}=\underbrace{0}_{\text{via }u}+\underbrace{2}_{\text{via }v}=2

Check it the long way. $f=(x^2+y)(x-y)=x^3-x^2y+xy-y^2$ , so $\partial f/\partial x=3x^2-2xy+y=3-2+1=2$ . Same answer. Two tiny products beat one messy expansion, and the gap grows with every layer.

Try: press ▶ send a nudge and watch it travel both roads. Then pick one road at a time. Slide $x$ to $0$ : one road now carries nothing.

drag the picture to orbit

x1

y1

—

A road's number is its slope; along a road they multiply. The glass column at $f$ stacks what each road delivers, so its height is $\partial f/\partial x$ .

The realization

$\frac{\partial f}{\partial x}=\sum_{\text{routes}}\ \prod_{\text{arrows on the route}}\big(\text{small slope}\big)$

Multiply along each route. Add the routes. Backprop does exactly this, and shares the work so that no route is walked twice.

Pause & predict

Still at $(x,y)=(1,1)$ . What is $\partial f/\partial y$ ? (Route via $u$ : $\partial u/\partial y=1$ . Route via $v$ : $\partial v/\partial y=-1$ .)

In one sentence: backpropagation is two rules — multiply the small slopes along each path, add the paths — done so that no path is walked twice.

4

Why you can trust the two rules — the chain rule, proved

Imagine this

Open a map app and zoom into a winding mountain road. Zoom in more, and more. At some point, the road on your screen looks straight.

That is what a derivative really is. Zoom in close enough on a smooth curve, and it looks like a straight line. The slope of that line is the derivative. The small bend you can still see is the leftover, and it fades fast as you zoom in.

We have been leaning on "multiply along paths, add across paths". It had better be true. The proof needs just the zoom idea, written in symbols.

Take a function $g$ and a point $x_0$ . Nudge the input by a small amount $h$ . Saying " $g$ has slope $g'(x_0)$ " means:

g(x_0+h)=g(x_0)+g'(x_0)\,h+r(h),\qquad\text{where }\ \frac{r(h)}{h}\to 0\ \text{ as }h\to 0

In words: new value = old value + slope × nudge + a leftover. And the leftover $r(h)$ is tiny even compared with the tiny nudge $h$ . That last part is the zoom: the bend fades faster than you zoom in.

A small example. Suppose the leftover is $r(h)=0.35\,h^2$ :

nudge $h$	leftover $r(h)$	$r(h)/h$
$1$	$0.350$	$0.350$
$\tfrac12$	$0.088$	$0.175$
$\tfrac14$	$0.022$	$0.088$

Halve $h$ and the leftover drops to a quarter. So even $r(h)/h$ halves each time, on its way to 0.

Line plus dying leftover. Near

x_0

the curve

g

(blue) and its tangent (orange) differ by the red gap

r(h)

. Halve the nudge and the gap does not merely shrink — it shrinks faster than $h$ (here

r(h)=0.35\,h^2

), so

r(h)/h

itself goes to zero. That is the whole content of "has a derivative".

With this "line plus leftover" promise, both rules follow in a few lines each.

×Rule 1. Put two promises in a row, $x\to u\to y$ . The slopes multiply, and the two leftovers stay tiny.
+Rule 2. Let $x$ reach $f$ through several middle steps at once. Each route brings its own product, and they add.

If you want the algebra · both rules proved, step by step

Prove it · Rule 1 — along a path, multiply

Claim. If $u=g(x)$ and $y=f(u)$ , then at $x_0$ (writing $u_0=g(x_0)$ ): $\ \dfrac{dy}{dx}=f'(u_0)\,g'(x_0)$ — the two slopes multiply.

1

Nudge the input by

h

and ask how much

u

moves. That is

g

's own promise:

\Delta u \;=\; g(x_0+h)-g(x_0)\;=\;g'(x_0)\,h+r_g(h)

Line plus leftover, with

r_g(h)/h\to0

. Nothing new — just the definition, applied to

g

.

2

Feed that movement to

f

. That is

f

's promise, written at

u_0

with nudge

\Delta u

:

f(u_0+\Delta u)\;=\;f(u_0)+f'(u_0)\,\Delta u+r_f(\Delta u)

Same definition, applied to

f

. Note the nudge

f

receives is

\Delta u

, not

h

.

3

Substitute step 1 into step 2 and multiply out:

f\big(g(x_0+h)\big)=f(u_0)+\underbrace{f'(u_0)\,g'(x_0)}_{\text{the clean slope}}\,h\;+\;\underbrace{f'(u_0)\,r_g(h)+r_f(\Delta u)}_{\text{call it }R(h)}

One term proportional to

h

, and everything else swept into a single bracket

R(h)

.

4

Check that

R(h)

is a legal leftover — that

R(h)/h\to0

. Piece by piece:

\frac{f'(u_0)\,r_g(h)}{h}\to f'(u_0)\cdot 0=0,\qquad \frac{r_f(\Delta u)}{h}=\frac{r_f(\Delta u)}{\Delta u}\cdot\frac{\Delta u}{h}\to 0\cdot g'(x_0)=0

The first piece is a constant times something dying. The second:

\Delta u/h\to g'(x_0)

by step 1, while

r_f(\Delta u)/\Delta u\to0

because

\Delta u\to0

. (Fine point: if

\Delta u

happens to be exactly

0

, then

r_f(\Delta u)=r_f(0)=0

and the term is already zero — no division needed.)

5

Read off the conclusion. We have shown

f\big(g(x_0+h)\big)=f\big(g(x_0)\big)+\big[f'(u_0)\,g'(x_0)\big]\,h+R(h),\qquad \frac{R(h)}{h}\to0

— one clean slope, one dying leftover. That is exactly the shape of a derivative's promise, so the number in the bracket is the derivative of the composite. ∎ Two promises composed make a third promise, and slopes multiply while the leftovers stay harmless. "Multiply along a path" is now a theorem.

Now the second rule. Suppose $f$ depends on several middle values $u_1,\dots,u_m$ at once, and each $u_k$ depends on $x$ . That is the fork in the graph: one input, many routes.

Prove it · Rule 2 — across paths, add

Claim. $\ \dfrac{df}{dx}=\displaystyle\sum_{k=1}^{m}\frac{\partial f}{\partial u_k}\,\frac{du_k}{dx}$ — each route contributes its product, and the contributions add.

1

The many-input promise (Unit 6's total derivative): nudging all inputs at once,

f(u_1+\Delta u_1,\dots,u_m+\Delta u_m)=f(\mathbf u)+\sum_{k}\frac{\partial f}{\partial u_k}\,\Delta u_k+r,\qquad \frac{r}{\|\Delta \mathbf u\|}\to0

A multivariable derivative is a plane-promise: one slope per input, one shared leftover that dies faster than the total nudge. This is what "differentiable" means in several variables.

2

Each intermediate keeps its own one-variable promise:

\Delta u_k=\frac{du_k}{dx}\,h+r_k(h),\qquad \frac{r_k(h)}{h}\to0

Every route's first leg is an honest derivative —

m

copies of step 1 from Rule 1.

3

Substitute step 2 into step 1 and collect the terms proportional to

h

:

f(\cdots)=f(\mathbf u)+\Big[\sum_{k}\frac{\partial f}{\partial u_k}\frac{du_k}{dx}\Big]h+R(h),\qquad R(h)=\sum_k\frac{\partial f}{\partial u_k}\,r_k(h)+r

The bracket is the claimed slope. Everything not proportional to

h

— one leftover per route, plus the shared plane-leftover

r

— is swept into

R(h)

. Nothing has been dropped yet.

4

The route leftovers die: for each

k

,

\frac{1}{h}\cdot\frac{\partial f}{\partial u_k}\,r_k(h)=\frac{\partial f}{\partial u_k}\cdot\frac{r_k(h)}{h}\to\frac{\partial f}{\partial u_k}\cdot 0=0

A fixed number times something that dies — the same move as step 4 of Rule 1, once per route.

5

The shared leftover dies too. By step 2,

\|\Delta\mathbf u\|/h\to\|\mathbf u'(x)\|

, where

\mathbf u'=(du_1/dx,\dots,du_m/dx)

. So

\frac{r}{h}=\frac{r}{\|\Delta\mathbf u\|}\cdot\frac{\|\Delta\mathbf u\|}{h}\to 0\cdot\|\mathbf u'(x)\|=0

The constant is

\|\mathbf u'(x)\|

, the size of the combined nudge per unit

h

. Fine point: if

\Delta\mathbf u

is exactly

\mathbf 0

then

r=r(\mathbf 0)=0

and the term is already gone — no division needed. Hence

R(h)/h\to0

.

6

Same shape, same conclusion: the composite keeps a promise whose slope is the bracket — the sum over routes of (slope out) × (slope in). ∎ "Add across paths" is not a new fact about graphs. It is what a multi-input derivative means, read one route at a time.

The bridge

The two rules are matrix multiplication in disguise. Suppose a list of inputs $\mathbf x$ makes a list of middle values $\mathbf u$ , which makes a list of outputs $\mathbf y$ . Each stage has a table of slopes, its Jacobian (Unit 6). The chain rule says: multiply the tables, $J_{G\circ F}=J_G\,J_F$ .

Look at one entry of that product, using row-times-column:

\big(J_G J_F\big)_{ij}=\sum_{k}\big(J_G\big)_{ik}\big(J_F\big)_{kj}

Each term is one route: input $j$ moves middle value $k$ , which moves output $i$ . Multiply along the route (Rule 1). The sum adds one route per middle value (Rule 2). So row-times-column, which you have used since Unit 1, is the two rules of backprop.

One entry of a Jacobian product, drawn. The sensitivity of output

y_1

to input

x_2

is

\sum_k (J_G)_{1k}(J_F)_{k2}

. Each orange route

x_2\to u_k\to y_1

is one term: multiply its two edge slopes (Rule 1); the three routes add (Rule 2). The row-times-column rule is those two rules at once.

The realization

A derivative is a promise: "near here, the function is a straight line plus a leftover that fades fast". Chain two promises, and the slopes multiply. Let several routes run side by side, and their products add.

And one entry of a matrix product, $\sum_k (J_G)_{ik}(J_F)_{kj}$ , is both rules written at once.

Pause & predict

The promise asks for $r(h)/h\to0$ , not just $r(h)\to0$ . What would go wrong if we only asked for $r(h)\to0$ ?

Pause & predict

In $\big(J_G J_F\big)_{ij}=\sum_k (J_G)_{ik}(J_F)_{kj}$ , what is one single term $(J_G)_{ik}(J_F)_{kj}$ , in graph language?

In one sentence: a derivative is a "straight line plus fading leftover" promise; chaining promises multiplies slopes, side-by-side routes add, and row-times-column is both rules at once.

5

The recipe, written once and for all

Imagine this

Think of a company. Staff report to team leads, team leads report to managers, and the managers' work adds up to one number: this year's result.

The result comes in low. The CEO sends each manager a note: "your share of the problem is this much". Each manager passes the note down to each team lead, scaled by how much that lead's work affected the manager. A team lead who works for two managers gets two notes, and adds them.

One sweep from the top down, and everyone knows their share. That is the backprop recipe.

Now in symbols. Number every value in the graph. $x_1,\dots,x_d$ are the inputs, the middle steps come next, and the last one, $x_D$ , is the output $f$ . Each value is made from its parents (the values that feed it) by one easy function $g_i$ :

x_i = g_i\big(x_{\mathrm{Pa}(x_i)}\big)

Read $\mathrm{Pa}(x_i)$ as "the parents of $x_i$ ". Now walk backwards. The output blames itself fully. Every other value collects blame from its children — the values it feeds:

\frac{\partial f}{\partial x_D}=1,\qquad \frac{\partial f}{\partial x_i}=\sum_{j:\,x_i\in \mathrm{Pa}(x_j)}\frac{\partial f}{\partial x_j}\,\frac{\partial g_j}{\partial x_i}

Read the second formula like the company notes. Child $j$ already knows its blame $\partial f/\partial x_j$ . It sends each parent a copy, scaled by its own small slope $\partial g_j/\partial x_i$ . A parent with several children adds up its notes.

Why is this cheap? Two reasons. Each blame is worked out once, then shared by all the parents. And each small slope is easy, because each step is easy: the slope of $e^a$ or $\sqrt c$ , never of the whole scary function. So finding every slope costs about the same as computing the function once.

If you want the algebra · why the recipe always gives the right answer

Prove it · the recipe is correct

Claim. Process the nodes from right to left; then the recipe's $\bar x_i$ equals the true $\partial f/\partial x_i$ for every node in the graph.

1

Base case. The rightmost node:

\bar x_D=1=\partial f/\partial f

— the output's sensitivity to itself. Nudge

f

by

h

and

f

changes by

h

. Slope 1, no leftover at all.

2

The order does the bookkeeping. Working right to left means: by the time we reach node

x_i

, every child

x_j

(every node that uses

x_i

) has already been processed and — by induction — already holds its correct blame

\bar x_j=\partial f/\partial x_j

. Children sit to the right of their parents in the graph, so a right-to-left sweep always finds the mail ready before it must be sent.

3

Apply Rule 2 at node $x_i$ , with its children as the intermediates. Any nudge of

x_i

can reach

f

only through the children, so §4's theorem says

\frac{\partial f}{\partial x_i}=\sum_{j:\,x_i\in\mathrm{Pa}(x_j)}\frac{\partial f}{\partial x_j}\,\frac{\partial g_j}{\partial x_i}

— and every factor on the right is one the recipe possesses: the child's (already correct) blame, times the local slope of the child's baby step. So the value the recipe writes into

\bar x_i

is the true derivative. Right-to-left induction covers the whole graph. ∎ The recipe is not a new algorithm to take on trust — it is Rule 2, applied once per node, in an order that guarantees the mail is always ready.

The realization

Set $\bar f=1$ . Walk from right to left. At each value, add up (child's blame × child's small slope) over all its children.

One sweep, and every value in the graph — every input, every weight — holds its slope.

Pause & predict

In the recipe, a value's blame is a sum over its children (the values it feeds). Why children, and not parents?

In one sentence: set $\bar f=1$ , then let every value send each parent its own blame times a small slope — one right-to-left sweep prices every knob.

6

One neuron, trained by your own hands

Imagine this

You step into a hotel shower. You want the water just right. You turn the knob, feel the water, and notice: still too cold.

How far off you are tells you how much to turn. Which way you are off tells you the direction. Turn, feel, turn again. After a few goes, the water is perfect.

A neuron learns exactly like this. It has two knobs, a weight $w$ and a bias $b$ . Backprop "feels the water" and tells each knob how to turn.

The smallest network there is: one neuron. It takes an input $x$ , and it should output a target $y$ . Three easy steps lead from the knobs to the loss:

z=wx+b,\qquad a=\tanh(z),\qquad L=\tfrac12\,(a-y)^2

In words: mix the input with the knobs ( $z$ ). Squash it into the range −1 to 1 with $\tanh$ (that is $a$ , the output). Measure the miss ( $L$ , the loss: half the squared gap).

Now send the blame back. The key stop is $\delta$ (the Greek letter "delta"): the blame that reaches $z$ .

\delta=\frac{\partial L}{\partial z}=\underbrace{(a-y)}_{\text{the miss}}\;\underbrace{(1-a^2)}_{\text{slope of }\tanh}

Once you have $\delta$ , each knob reads its slope straight off it: $\partial L/\partial w=\delta\,x$ and $\partial L/\partial b=\delta$ . Compute $\delta$ once, hand out copies.

With the practice set's numbers, $w=0.5,\ b=0,\ x=1,\ y=1$ : $z=0.5$ , $a=\tanh 0.5\approx0.462$ and $L\approx0.145$ . The miss is $a-y\approx-0.538$ : too low. So $\delta\approx(-0.538)(0.786)\approx-0.423$ . Both slopes are negative, which says: make $w$ and $b$ bigger.

One training step then moves each knob a little downhill: $w\leftarrow w-\eta\,\partial L/\partial w$ . Here $\eta$ (read "eta") is the step size; the widget uses $\eta=0.5$ .

Try: before each step, guess the sign of $\delta$ . Then set the target to $y=-1$ and watch the same machine push $w$ the other way.

drag the landscape to orbit it

input x1

target y1

—

It starts at the practice set's numbers: $L=0.1447$ and $\delta=-0.4230$ . Both slopes are negative, so each step makes $w$ and $b$ bigger and the dot rolls downhill.

If you want the algebra · every number in the widget, from scratch

Prove it · every number in the widget, from scratch

Claim. $\delta=(a-y)(1-a^2)$ . Nothing to memorise — derive each local slope, then chain them with Rule 1.

1

The loss.

L(a)=\tfrac12(a-y)^2

. Nudge

a

by

\Delta

and expand the square:

\tfrac12(a+\Delta-y)^2=\tfrac12(a-y)^2+(a-y)\,\Delta+\tfrac12\Delta^2\quad\Rightarrow\quad \frac{\partial L}{\partial a}=a-y

The

\tfrac12\Delta^2

is the dying leftover. And now you can see why the

\tfrac12

was planted in the loss: it exists purely to eat the 2 that squaring brings down.

2

The activation.

a=\tanh z=\dfrac{s}{c}

with

s=e^z-e^{-z}

,

c=e^z+e^{-z}

. Notice the pair swaps under differentiation:

s'=c

and

c'=s

. Quotient rule:

\Big(\frac{s}{c}\Big)'=\frac{s'c-sc'}{c^2}=\frac{c^2-s^2}{c^2}=1-\Big(\frac{s}{c}\Big)^2=1-\tanh^2 z=1-a^2

The derivative comes out written in terms of the output

a

— which the forward pass already computed. So the backward pass buys

1-a^2

for one multiplication, no new function calls.

3

The pre-activation.

z=wx+b

is linear, and a linear function's slopes are just its coefficients:

\frac{\partial z}{\partial w}=x,\qquad \frac{\partial z}{\partial b}=1,\qquad \frac{\partial z}{\partial x}=w

Nudge

w

by

\Delta

and

z

moves by

x\Delta

exactly — not even a leftover this time.

4

Chain them (Rule 1, twice).

\delta=\frac{\partial L}{\partial z}=\frac{\partial L}{\partial a}\cdot\frac{da}{dz}=(a-y)(1-a^2)

With the widget's starting numbers,

a-y=0.4621-1=-0.5379

and

1-a^2=0.7864

:

\delta=(-0.5379)(0.7864)=-0.4230

then

\ \partial L/\partial w=\delta\,x,\ \ \partial L/\partial b=\delta,\ \ \partial L/\partial x=\delta\,w

. ∎ Every figure the widget displays is these four lines evaluated — press the buttons above and check any of them by hand.

The realization

$\delta=(a-y)(1-a^2)$ : the miss, times how steep the squash is. Every knob that feeds $z$ then gets $\delta$ times one number: $x$ for the weight, $1$ for the bias.

Step each knob against its slope, repeat, and the loss really falls. That is training, complete.

Pause & predict

At the start, $\partial L/\partial w=\delta\cdot x$ and $\partial L/\partial b=\delta\cdot 1$ , with $\delta=-0.423$ . What made them so cheap to get?

Pause & predict

Set the target to $y=-1$ instead. Before pressing anything: which way will the next step push $w$ ?

In one sentence: compute the blame $\delta=\partial L/\partial z$ once, and every knob feeding $z$ reads its slope off it — then step downhill and the loss really falls.

7

A whole layer at once — backprop with matrices

Imagine this

A talent show has two judges and a few contestants. Every judge watches every contestant, but each judge cares about each one differently. Those "how much I care" numbers are the weights.

After the show, the producer says: "Judge 1 scored too low, judge 2 too high." Who should change what?

1A judge's weight on a contestant should change a lot only if that judge was badly off and that contestant actually performed.
2Each contestant collects feedback from every judge who watched them.

Those two sentences are this whole section.

A real layer is many neurons side by side, all reading the same input list $\mathbf x$ . In symbols:

Z=A\mathbf{x}+\mathbf{b},\qquad \mathbf{a}=\sigma(Z)

$A$ is the table of weights: row $i$ holds neuron $i$ 's weights. $\mathbf b$ holds the biases. The sigmoid $\sigma(z)=\tfrac{1}{1+e^{-z}}$ squashes each number into the range 0 to 1, one neuron at a time.

From the layer above, blame arrives: $\boldsymbol\delta^{(a)}$ , one number per neuron. Each says how fast the loss would rise if that neuron's output rose a little. Passing it back through the layer takes four lines:

Step	In everyday words	In symbols
Through the squash	Each neuron's blame is scaled by how steep its own squash is. Neurons don't mix.	$\boldsymbol\delta^{(Z)}=\boldsymbol\delta^{(a)}\odot \mathbf a\odot(1-\mathbf a)$
A weight	Blame of its neuron × activity of its input. Big only when both are big.	$\dfrac{\partial L}{\partial A}=\boldsymbol\delta^{(Z)}\mathbf{x}^{\top}$
A bias	Gets its neuron's blame, unchanged.	$\dfrac{\partial L}{\partial \mathbf b}=\boldsymbol\delta^{(Z)}$
The input	Collects blame from every neuron it fed. This becomes the note for the layer below.	$\dfrac{\partial L}{\partial \mathbf x}=A^{\top}\boldsymbol\delta^{(Z)}$

Two symbols to read aloud. $\odot$ means "multiply entry by entry": first with first, second with second. And $\mathbf a\odot(1-\mathbf a)$ is the sigmoid's slope. It needs only the output $\mathbf a$ , which the forward pass already has.

A tiny example with the widget's numbers. Neuron 1 outputs $a_1=0.622$ , so its slope is $0.622\times0.378\approx0.235$ . Blame $0.6$ arrives, and $0.6\times0.235\approx0.141$ gets through. Neuron 2 outputs $a_2=0.953$ : almost maxed out, like a volume knob already at 95%. Its slope is only $0.045$ . Blame $-0.4$ arrives, and just $-0.018$ gets through.

The weights' slope $\boldsymbol\delta^{(Z)}\mathbf x^\top$ is an outer product: a grid whose row $i$ , column $j$ holds $\delta_i\,x_j$ . And $A^{\top}$ (read "A transpose": the table flipped over its diagonal) sends the blame back along the columns. Forward, each neuron reads its row. Backward, each input collects its column.

Try: press ▶ play to walk the four stages. Then slide $\delta^{(a)}_1$ : row 1 of the weight grid changes, row 2 does not.

δ⁽ᵃ⁾₁ from above0.6

—

A tiny layer so every number fits: $A=$ 1−120, $\mathbf b=(0,\,1)$ , $\mathbf x=(1,\,0.5)$ . From above, $\boldsymbol\delta^{(a)}=(0.6,\,-0.4)$ arrives.

Why the transpose. Forward, neuron

i

reads along row

i

— that is what

A\mathbf x

does. Backward, input

j

must collect blame from every neuron it fed, which means reading down column

j

— and reading columns of

A

is reading rows of

A^{\top}

. Same numbers, opposite direction.

If you want the algebra · the sigmoid's slope, the diagonal Jacobian, and the three matrix formulas

Prove it · where σ(1−σ) comes from

Claim. $\sigma(z)=\dfrac{1}{1+e^{-z}}$ satisfies $\sigma'=\sigma(1-\sigma)$ — so the backward pass needs no new function evaluations.

1

Differentiate with Rule 1:

\sigma=(1+e^{-z})^{-1}

, outer slope

-(1+e^{-z})^{-2}

, inner slope

-e^{-z}\!:

\sigma'=-(1+e^{-z})^{-2}\cdot(-e^{-z})=\frac{e^{-z}}{(1+e^{-z})^{2}}

The two minus signs cancel — sigmoid always rises.

2

Split the fraction into two factors you can recognise:

\frac{e^{-z}}{(1+e^{-z})^{2}}=\underbrace{\frac{1}{1+e^{-z}}}_{=\ \sigma}\cdot\underbrace{\frac{e^{-z}}{1+e^{-z}}}_{=\ 1-\sigma} \qquad\text{since}\quad 1-\sigma=\frac{(1+e^{-z})-1}{1+e^{-z}}=\frac{e^{-z}}{1+e^{-z}}

So

\sigma'=\sigma(1-\sigma)

. ∎ Output in, derivative out:

\mathbf a\odot(1-\mathbf a)

uses only the

\mathbf a

the forward pass cached. This is why frameworks store activations — the way back is pure arithmetic.

Prove it · why the sigmoid step is entrywise — the Jacobian is diagonal

Claim. $\dfrac{\partial\mathbf a}{\partial Z}=\operatorname{diag}\!\big(\mathbf a\odot(1-\mathbf a)\big)$ , and therefore $\boldsymbol\delta^{(Z)}=\boldsymbol\delta^{(a)}\odot\mathbf a\odot(1-\mathbf a)$ — a multiplication per neuron, no matrix at all.

1

Who depends on whom.

a_i=\sigma(Z_i)

reads exactly one input,

Z_i

. Nudge a different

Z_j

and

a_i

does not move:

\frac{\partial a_i}{\partial Z_j}=0\quad(i\neq j)

Every off-diagonal entry of the Jacobian is zero because the activation never mixes neurons — it is applied one entry at a time.

2

The diagonal. On its own input,

a_i

is the one-variable function

\sigma

, and the box above gives its slope:

\frac{\partial a_i}{\partial Z_i}=\sigma'(Z_i)=\sigma(Z_i)\big(1-\sigma(Z_i)\big)=a_i(1-a_i)

Steps 1 and 2 together: a matrix with

a_i(1-a_i)

down the diagonal and

0

everywhere else —

\operatorname{diag}(\mathbf a\odot(1-\mathbf a))

.

3

Chain it (Rule 1 in matrix form). The blame on

Z

is the blame on

\mathbf a

times this Jacobian; multiplying a row by a diagonal matrix scales each entry:

\delta^{(Z)}_i=\sum_j \delta^{(a)}_j\,\frac{\partial a_j}{\partial Z_i}=\delta^{(a)}_i\,a_i(1-a_i)

The sum over

j

is Rule 2, but only the

j=i

term survives (step 1) — so the "sum over routes" has one route per neuron. Written for all

i

at once, that is the

\odot

formula.

4

Check on the widget's numbers.

\mathbf a=(0.622,\,0.953)

gives

\mathbf a\odot(1-\mathbf a)=(0.235,\,0.045)

; with

\boldsymbol\delta^{(a)}=(0.6,\,-0.4)

,

\boldsymbol\delta^{(Z)}=(0.6\cdot0.235,\ \ -0.4\cdot0.045)=(0.141,\,-0.018)

— exactly what stage 3 below displays. ∎ Neuron 2 sits at

a=0.953

, nearly saturated, so its factor

0.045

throttles almost all of its blame. That is the vanishing-gradient mechanism, in one entry of a diagonal matrix.

Prove it · the three matrix formulas, entry by entry

Claim. The matrix formulas hide no new laws — they are Rule 1 and Rule 2 aimed at each entry, then reassembled into matrix shape. Write $\boldsymbol\delta=\boldsymbol\delta^{(Z)}$ for short.

1

A weight $A_{ij}$ . Scan the forward pass

Z_k=\sum_p A_{kp}x_p+b_k

: the entry

A_{ij}

appears in exactly one place — the row

k=i

, multiplied by

x_j

. One appearance means one path, so Rule 1 alone:

\frac{\partial L}{\partial A_{ij}}=\delta_i\cdot\frac{\partial Z_i}{\partial A_{ij}}=\delta_i\,x_j

"Blame of my row, times activity of my column" — now proved, not just recited.

2

Assemble. The matrix whose

(i,j)

entry is

\delta_i x_j

is, by definition, the outer product:

\frac{\partial L}{\partial A}=\boldsymbol\delta\,\mathbf x^{\top}

Same shape as

A

, one honest entry per weight, ready for

A\leftarrow A-\eta\,\partial L/\partial A

.

3

A bias $b_i$ . It appears only in

Z_i

, with slope

1

:

\ \partial L/\partial b_i=\delta_i\cdot1

, so

\partial L/\partial\mathbf b=\boldsymbol\delta

. The bias rides its own row and nothing else — blame arrives undiluted.

4

An input $x_j$ — the fork.

x_j

appears in every row:

Z_1,\dots,Z_m

all use it.

m

appearances,

m

paths — Rule 2:

\frac{\partial L}{\partial x_j}=\sum_{i=1}^{m}\delta_i\,\frac{\partial Z_i}{\partial x_j}=\sum_{i=1}^{m}\delta_i\,A_{ij}=\big(A^{\top}\boldsymbol\delta\big)_j

Summing

\delta_i A_{ij}

over

i

walks down column

j

of

A

— and reading columns is exactly what the transpose does. The mysterious

A^{\top}

is Rule 2 wearing matrix clothes.

5

All three formulas recovered from two rules and one scan of where each variable appears. ∎ This "find every appearance, add a term per appearance" habit is the entire skill of differentiating matrix expressions. The very next section runs on it.

The realization

Through the squash, scale each neuron's blame by its own slope. Then weights get $\boldsymbol\delta^{(Z)}\mathbf x^\top$ , biases get $\boldsymbol\delta^{(Z)}$ , and the layer below gets $A^\top\boldsymbol\delta^{(Z)}$ .

Repeat, layer after layer. That is the inner loop of all of deep learning.

Pause & predict

A layer has a weight table $A$ of size $100\times 300$ . What is the shape of $\partial L/\partial A=\boldsymbol\delta^{(Z)}\mathbf{x}^{\top}$ , and why must it be?

Pause & predict

Why does the blame go back through $A^{\top}$ (the flipped table) instead of $A$ itself?

In one sentence: scale the blame by each neuron's slope, then weights get $\boldsymbol\delta^{(Z)}\mathbf x^{\top}$ , biases get $\boldsymbol\delta^{(Z)}$ , and the layer below gets $A^{\top}\boldsymbol\delta^{(Z)}$ — four lines, the inner loop of deep learning.

8

The gradient cookbook — five rules you'll use forever

Imagine this

A good cook does not work out how to make dal from scratch every evening. She knows a few basic recipes by heart, and combines them.

Slopes of matrix formulas work the same way. Five basic recipes cover almost everything a real model needs. Learn them as sentences, and the symbols follow.

One quick example first. Take $\mathbf a=(2,3)$ and the dot product $\mathbf x^\top\mathbf a=2x_1+3x_2$ . Nudge $x_1$ by 1: the value goes up by 2. Nudge $x_2$ by 1: it goes up by 3. So the slope is the list $(2,3)$ — just $\mathbf a$ itself. That is recipe 1.

Here are all five. (Slopes are written as rows, as in Unit 6.)

Recipe	In everyday words
$\dfrac{\partial\, \mathbf{x}^{\top}\mathbf{a}}{\partial \mathbf{x}}=\mathbf{a}^{\top}$	The slope of a dot product is just the other vector.
$\dfrac{\partial\, \mathbf{a}^{\top}\mathbf{x}}{\partial \mathbf{x}}=\mathbf{a}^{\top}$	Same thing: a dot product doesn't care about order.
$\dfrac{\partial\, \mathbf{a}^{\top}X\mathbf{b}}{\partial X}=\mathbf{a}\mathbf{b}^{\top}$	Each entry $X_{ij}$ is multiplied by $a_i b_j$ , so the slope is that grid: an outer product.
$\dfrac{\partial\, \mathbf{x}^{\top}B\mathbf{x}}{\partial \mathbf{x}}=\mathbf{x}^{\top}\!\big(B+B^{\top}\big)$	$\mathbf x$ appears twice, so you get two terms: $B$ from one side, $B^\top$ from the other.
$\dfrac{\partial}{\partial \mathbf{s}}(\mathbf x-A\mathbf s)^{\top}W(\mathbf x-A\mathbf s)=-2(\mathbf x-A\mathbf s)^{\top}WA$	For symmetric $W$ : the least-squares slope. It is the chain rule through the leftover error $\mathbf x-A\mathbf s$ .

You have met two of these already. Recipe 3 is why the weights' slope in Section 7 came out as an outer product. Recipe 5 is the least-squares loss that trains linear models; it returns in Unit 9. The arena's Problem 7 checks two recipes with real numbers, and Problem 10 uses them on a real model.

One method proves them all: find every place the variable appears, and add one term per place.

If you want the algebra · the whole cookbook in four short derivations

Prove it · the whole cookbook, four short derivations

Claim. Every identity in the table is entry-wise differentiation, organised. The method is always the same: find every appearance of the variable, add one term per appearance.

1

Dot product.

\mathbf x^{\top}\mathbf a=\sum_i x_i a_i

. Differentiate with respect to one coordinate

x_k\!:

every term is constant except

x_k a_k

, whose slope is

a_k

. Collect the coordinates into a row:

\frac{\partial(\mathbf x^{\top}\mathbf a)}{\partial\mathbf x}=\mathbf a^{\top}

And

\mathbf a^{\top}\mathbf x

is the same scalar, so the table's second line comes free.

2

The sandwich.

\mathbf a^{\top}X\mathbf b=\sum_{i,j}a_i X_{ij}b_j

. The entry

X_{ij}

appears once, with coefficient

a_ib_j\!:

\frac{\partial(\mathbf a^{\top}X\mathbf b)}{\partial X_{ij}}=a_ib_j\quad\Rightarrow\quad \frac{\partial(\mathbf a^{\top}X\mathbf b)}{\partial X}=\mathbf a\mathbf b^{\top}

The same one-appearance argument that made

\partial L/\partial A

an outer product in the layer proof — it is literally the same fact.

3

The quadratic — where $B+B^{\top}$ is born.

\mathbf x^{\top}B\mathbf x=\sum_{i,j}B_{ij}\,x_i\,x_j

, and

\mathbf x

appears twice in each term. Product rule on

x_ix_j

, with respect to

x_k

: the left copy fires when

i=k

, contributing

\sum_j B_{kj}x_j=(B\mathbf x)_k

; the right copy fires when

j=k

, contributing

\sum_i B_{ik}x_i=(B^{\top}\mathbf x)_k

. Add the two:

\frac{\partial(\mathbf x^{\top}B\mathbf x)}{\partial x_k}=(B\mathbf x)_k+(B^{\top}\mathbf x)_k\quad\Rightarrow\quad \frac{\partial(\mathbf x^{\top}B\mathbf x)}{\partial\mathbf x}=\mathbf x^{\top}\big(B+B^{\top}\big)

Two appearances of

\mathbf x

, two donated terms — the "mysterious"

+B^{\top}

is Rule 2's fork, hiding in plain algebra. For symmetric

B

it collapses to

2\mathbf x^{\top}B

.

4

Least squares — a chain of the previous lines. Set the residual

\mathbf r=\mathbf x-A\mathbf s

and

E=\mathbf r^{\top}W\mathbf r

with

W

symmetric. Rule 1 with

\mathbf r

as the intermediate:

\begin{aligned}\frac{\partial E}{\partial\mathbf s}&=\frac{\partial E}{\partial\mathbf r}\,\frac{\partial\mathbf r}{\partial\mathbf s} =\underbrace{\mathbf r^{\top}(W+W^{\top})}_{\text{line 3}}\cdot\underbrace{(-A)}_{\mathbf r\text{ linear in }\mathbf s}\\[.3em] &=2\,\mathbf r^{\top}W\cdot(-A)=-2(\mathbf x-A\mathbf s)^{\top}WA\end{aligned}

The

2

came from

W+W^{\top}

under symmetry; the minus came from the

-A\mathbf s

inside the residual. Nothing to memorise once you can rebuild it in two lines. ∎

The realization

Dot product → the other vector. $\mathbf a^\top X\mathbf b$ → the outer product $\mathbf a\mathbf b^\top$ . $\mathbf x^\top B\mathbf x$ → $\mathbf x^\top(B+B^\top)$ , because $\mathbf x$ appears twice.

When in doubt, count where the variable appears, and add one term for each appearance.

Pause & predict

With $B=$ 1234 (not symmetric), what is the slope of $\mathbf x^{\top}B\mathbf x$ with respect to $\mathbf x=(x_1,x_2)$ ?

Pause & predict

In the least-squares slope $-2(\mathbf x-A\mathbf s)^{\top}WA$ , where exactly did the $2$ come from?

In one sentence: five recipes — dot product, outer product, the $B+B^\top$ doubling and least squares — give almost every small slope a real model needs.

9

Why it's so cheap — one sweep, everything reused

Imagine this

A metro line has ten stations. Every station wants to know its fare to the last stop.

The slow way: each station adds up every hop to the end, all by itself. Station 1 adds nine hops, station 2 adds eight, and so on. The same hops get added again and again.

The smart way: start at the end. The station next to the last stop is one hop away. The station before it asks its neighbour and adds one hop. Every hop is counted once.

Backprop is the smart way. (For slopes, the hops multiply instead of add, but the saving is the same.)

Stack $K$ layers, $f_i=\sigma_i(A_{i-1}f_{i-1}+b_{i-1})$ , and put a loss at the end: $L=\|\mathbf y-f_K\|^2$ . Each layer's knobs $\theta_i=\{A_i,b_i\}$ need a slope. The chain rule writes each one as a long product, reaching from the loss back to that layer:

\begin{aligned}\frac{\partial L}{\partial\theta_{K-1}}&=\frac{\partial L}{\partial f_K}\frac{\partial f_K}{\partial\theta_{K-1}}\\[.35em] \frac{\partial L}{\partial\theta_{K-2}}&=\frac{\partial L}{\partial f_K}\frac{\partial f_K}{\partial f_{K-1}}\frac{\partial f_{K-1}}{\partial\theta_{K-2}}\\[.35em] \frac{\partial L}{\partial\theta_{i}}&=\frac{\partial L}{\partial f_K}\frac{\partial f_K}{\partial f_{K-1}}\cdots\frac{\partial f_{i+2}}{\partial f_{i+1}}\frac{\partial f_{i+1}}{\partial\theta_{i}}\end{aligned}

Look at the pattern. Each line is the line above, plus one more factor. The start of every product is shared. That shared start is the running blame $\boldsymbol\delta$ that Section 7 passed down.

Count the work for $K=4$ layers. Building each product from scratch takes $2+3+4+5=14$ factors. Reusing the shared start takes just 2 new factors per layer: $8$ in all. For $K=8$ it is $44$ against $16$ . The slow count grows like $K^2/2$ ; the smart one grows like $K$ .

Try: press ▶ build the staircase. Then slide $K$ up to 8 and compare the grey pile (the slow way) with the lit blocks (backprop).

drag the picture to orbit

layers K4

—

One tower per layer, one block per chain-rule factor; the left tower sits next to the loss. Grey blocks are work the slow way repeats; lit blocks are all backprop does.

A second saving hides in the order. The loss is one number, so the first factor $\partial L/\partial f_K$ is a thin row. A row times a matrix is cheap, and gives another thin row. Starting from the input end instead would multiply big square matrices: for layers $m$ wide, about $m$ times more work.

So the sweep must start at the loss. That choice of order is called reverse-mode automatic differentiation.

If you want the algebra · the stacked formula, and why the sweep must start at the loss

Prove it · the stacked formula is the recipe of §5, unrolled

Claim. $\dfrac{\partial L}{\partial\theta_i}=\dfrac{\partial L}{\partial f_K}\,\dfrac{\partial f_K}{\partial f_{K-1}}\cdots\dfrac{\partial f_{i+2}}{\partial f_{i+1}}\,\dfrac{\partial f_{i+1}}{\partial\theta_i}$ — no new chain rule, just the postal recipe on a graph that happens to be a line.

1

Draw the graph.

f_{i+1}=\sigma_{i+1}(A_if_i+b_i)

means node

f_{i+1}

has exactly two parents,

f_i

and

\theta_i=\{A_i,b_i\}

, and node

f_i

has exactly one child,

f_{i+1}

. The loss

L

is the single child of

f_K

. A stack of layers is a chain: each activation feeds only the next. Forks exist only where a parameter joins in.

2

Recipe at an activation node. §5's rule sums over children;

f_j

has one, so the sum has one term:

\bar f_K=\frac{\partial L}{\partial f_K},\qquad \bar f_j=\bar f_{j+1}\,\frac{\partial f_{j+1}}{\partial f_j}\quad(j<K)

One child, one piece of mail. Rule 2's sum collapses to a single Rule 1 product.

3

Unroll. Apply step 2 repeatedly, from

K

down to

i+1

:

\bar f_{i+1}=\frac{\partial L}{\partial f_K}\,\frac{\partial f_K}{\partial f_{K-1}}\cdots\frac{\partial f_{i+2}}{\partial f_{i+1}}

Each application appends one more Jacobian on the right. This running product is the

\boldsymbol\delta^{(a)}

that §7 handed down to each layer.

4

Recipe at the parameter node.

\theta_i

has one child,

f_{i+1}

, so

\frac{\partial L}{\partial\theta_i}=\bar\theta_i=\bar f_{i+1}\,\frac{\partial f_{i+1}}{\partial\theta_i}

and substituting step 3 gives the claim. ∎ Every layer's gradient is the same prefix

\bar f_{i+1}

times one local factor — which is exactly why computing the prefixes once, from the loss end, prices the whole stack.

Prove it · why the sweep must start at the loss

Claim. The chain $\dfrac{\partial L}{\partial f_1}=\dfrac{\partial L}{\partial f_K}\,\dfrac{\partial f_K}{\partial f_{K-1}}\cdots\dfrac{\partial f_2}{\partial f_1}$ can be multiplied in either order. One order is about $m$ times cheaper — count it.

1

Shapes first (Unit 6's rule: outputs × inputs).

L

is a scalar, so

\partial L/\partial f_K

is a

1\times m

row. Each layer-to-layer Jacobian

\partial f_{i}/\partial f_{i-1}

is a full

m\times m

matrix, for layers of width

m

. One skinny row at the loss end; big square blocks everywhere else. That asymmetry is the whole story.

2

Multiply from the loss end.

(1\times m)\cdot(m\times m)

costs about

m^2

multiplications — and returns another

1\times m

row. Repeat through all

K

layers:

\text{cost}\approx K\,m^{2},\qquad\text{always carrying just a row}

That carried row is exactly the blame

\boldsymbol\delta

of the layer section — reverse mode never holds anything bigger than one layer's worth of numbers.

3

Multiply from the input end. The first product is

(m\times m)\cdot(m\times m)

: cost

m^3

, and it returns another

m\times m

matrix that must be carried the whole way:

\text{cost}\approx K\,m^{3}

For

m=1000

, that is a thousand times more work — from nothing but multiplying the same factors in the wrong order.

4

Same factors, same answer, wildly different bill. Keeping the scalar-loss end first means every step is a cheap row-times-matrix product. That choice of order is reverse-mode automatic differentiation. ∎ And it explains the fine print to come: reverse mode wins because training has ONE output (the loss) and millions of inputs. Flip that ratio and the other order — forward mode — wins instead.

The realization

Every layer's slope is the same shared start, times one new factor. Carry that start — the blame $\boldsymbol\delta$ — from the loss backwards, and each factor is used once.

So one backward sweep costs about as much as one forward pass, however many knobs there are.

Pause & predict

Rule of thumb: one backward pass costs about as much as one forward pass. What one fact makes that true?

In one sentence: all the layer slopes share one long start — carry it once, from the loss end, and the whole set costs a single sweep.

10

Linearization — the slope as a stand-in for the function

Imagine this

Stand in an open field. The ground looks flat. For walking to the gate, "the Earth is flat" is a perfect model.

For a flight from Delhi to London, it is badly wrong. The pilot must allow for the curve.

A slope gives you exactly this kind of "flat Earth" model of a function: excellent close by, worse and worse as you go further away.

Once you know the slopes at a point $\mathbf x_0$ , you have the best straight-line guess for the function near $\mathbf x_0$ :

f(\mathbf x)\;\approx\;f(\mathbf x_0)+\nabla f(\mathbf x_0)^{\top}(\mathbf x-\mathbf x_0)

In words: value here + slope × how far you moved. This is Unit 6's tangent line, for many inputs at once. ( $\nabla f$ , read "grad f", is the list of all the slopes.)

A tiny example from the companion: $f(x)=\sqrt{x^2+9}$ at $x_0=-4$ . Here $f(-4)=\sqrt{25}=5$ and the slope is $f'(-4)=-4/5=-0.8$ . So the stand-in line is $L(x)=5-0.8\,(x+4)$ .

At $x=-3.5$ , half a step away, the line guesses $5-0.8\times0.5=4.6$ . The truth is $4.610$ : very close. Go far away and the guess drifts off. Gradient descent trusts this line for one small step at a time, which is why its steps must be small.

Try: move the probe twice as far from the anchor. The error grows about four times.

✋ drag near the dot to move the anchor · elsewhere to move the probe

—

At anchor $-4$ the line is $L(x)=5-0.8(x+4)$ . Near the anchor it almost is the curve; far away the red error bar grows fast.

…and with two inputs: the stand-in planef(x, y) = eˣ cos y — the companion's second example; the glass sheet is the "flat Earth" at the anchor

drag the surface to orbit it

anchor x₀0

anchor y₀0

probe radius0.5

At the origin the slopes are $(1,0)$ , so the plane is $L=1+x$ : tilted along $x$ , flat along $y$ . Shrink the probe circle and the red gap shrinks like its radius squared.

If you want the algebra · the stand-in is the first two Taylor terms

Prove it · the stand-in is the first two Taylor terms

Claim. The linearization $L(x)=f(x_0)+f'(x_0)(x-x_0)$ is what is left of the Taylor series when every term with $(x-x_0)^2$ or higher is dropped — and the first dropped term tells you the error.

1

Taylor at $x_0$ (Unit 8's subject, used once here): with

h=x-x_0

,

f(x_0+h)=f(x_0)+f'(x_0)\,h+\tfrac12 f''(x_0)\,h^2+\tfrac16 f'''(x_0)\,h^3+\cdots

Keep the first two terms and you are holding

L(x)

exactly. Everything after them is §4's leftover

r(h)

, now written out.

2

The error is led by the $h^2$ term. For small

h

the

h^3

and later terms are smaller still, so

f(x)-L(x)\approx\tfrac12 f''(x_0)\,(x-x_0)^2

Twice the distance, four times the error — the same "leftover dies faster than

h

" you saw drawn in §4.

3

Check on the widget.

f=\sqrt{x^2+9}

, so

f''(x)=\dfrac{9}{(x^2+9)^{3/2}}

and

f''(-4)=\dfrac{9}{125}=0.072

. Probe at

x=-3.5

, i.e.

h=0.5

:

\tfrac12\cdot0.072\cdot0.5^2=0.009\qquad\text{vs. the widget's}\quad 4.6098-4.6=0.0098

The prediction is off only by the

h^3

crumbs. So a gradient does more than point downhill — with

f''

it also prices how far the linear stand-in can be trusted. ∎

The realization

$f(\mathbf x_0)+\nabla f(\mathbf x_0)^{\top}(\mathbf x-\mathbf x_0)$ is the "flat Earth" version of $f$ near $\mathbf x_0$ .

Its error grows like the square of the distance: twice as far, about four times the error. Every gradient-descent step leans on this promise.

Pause & predict

The companion also linearizes $f(x,y)=e^x\cos y$ at $(0,0)$ . There the value is $1$ and the slopes are $(1,0)$ . What is the stand-in plane?

In one sentence: $f(\mathbf x_0)+\nabla f(\mathbf x_0)^{\top}(\mathbf x-\mathbf x_0)$ is the best straight-line stand-in near $\mathbf x_0$ — the promise every gradient-descent step relies on.

11

Fine print worth knowing

Imagine this

You want the weight of one ₹1 coin, but your kitchen scale only shows whole grams. So you weigh your purse, take the coin out, and weigh it again. The difference is the coin: about 4 grams. That works.

Now try the same trick to weigh a single hair. The difference is far smaller than the scale's own error. You get noise.

Computers hit the same wall when they estimate slopes by nudging. That is one more reason backprop was needed.

Three ways to get a slope.

Way	How it works	The catch
Symbolic	Work out the formula, like on paper.	Formulas blow up in size for deep functions.
Numerical	Nudge the input, see how far the output moves, divide.	One run per knob, plus rounding noise (the hair problem).
Automatic (this unit)	Follow the program step by step, and apply the chain rule to each step.	Exact, and about as cheap as one run.

The hair problem, in numbers. The nudge estimate is

\frac{f(x+h)-f(x-h)}{2h}

It has two enemies. If $h$ is big, the straight line between the two points misses the curve's true slope. If $h$ is tiny, the computer subtracts two almost equal numbers, and the result drowns in rounding noise. Somewhere in between is a best $h$ . Find it:

Try: press ▶ shrink h, or slide it yourself. The error falls fast, then, near $10^{-6}$ , it turns and climbs again.

step h = 10^−3

—

The function is the one from Section 1, at $x=1$ , where the exact slope is $5.98308$ . Too big an $h$ bends the answer; too small drowns it in rounding noise.

If you want the algebra · the V-shape has a formula

Prove it · the V-shape has a formula

Claim. The checker's error falls like $h^2$ , rises like $1/h$ , and bottoms out at a best step $h_*$ of order $10^{-6}$ – $10^{-5}$ — all three facts derived, not observed.

1

Taylor both evaluations around

x

(Unit 6's tool, Unit 8's whole subject):

f(x\pm h)=f(x)\pm f'(x)h+\tfrac{f''(x)}{2}h^{2}\pm\tfrac{f'''(x)}{6}h^{3}+\cdots

Same expansion, two signs — writing them together is what makes the next line click.

2

Subtract — the even terms cancel.

f(x+h)-f(x-h)=2f'(x)h+\tfrac{f'''(x)}{3}h^{3}+\cdots

. Divide by

2h

:

\frac{f(x+h)-f(x-h)}{2h}=f'(x)+\frac{f'''(x)}{6}\,h^{2}+\cdots

The

h^2

term is the left wall of the V — slope 2 on log–log axes. (The one-sided version

\frac{f(x+h)-f(x)}{h}

keeps an

\frac{f''}{2}h

term — slope only 1. That is why the checker uses the centred difference.)

3

Rounding. The computer stores

f(x\pm h)

only to relative precision

u\approx10^{-16}

, so the subtraction of two nearly equal numbers carries an absolute error of about

u|f(x)|

; dividing by

2h

turns it into

\text{rounding error}\approx\frac{u\,|f(x)|}{2h}

The

1/h

is the right wall: shrink

h

and you divide the same noise by an ever smaller number.

4

Add the walls and set the slope to zero.

E(h)\approx\frac{|f'''|}{6}h^{2}+\frac{u|f|}{2h}

. Differentiate in

h

:

E'(h)=\frac{|f'''|}{3}\,h-\frac{u\,|f|}{2h^{2}}=0\quad\Longleftrightarrow\quad h^{3}=\frac{3\,u\,|f|}{2\,|f'''|}

A rising wall plus a falling wall has one lowest point, where the two slopes cancel. Multiplying through by

h^2

isolates

h^3

.

5

Read off the best step.

h_{*}=\Big(\frac{3\,u\,|f|}{2\,|f'''|}\Big)^{1/3}\sim u^{1/3}\approx 5\times10^{-6}\quad\text{when }|f'''|\approx|f|

For the monster at

x=1

,

|f|=1.09

but

|f'''|\approx153

, so

h_*\approx1\times10^{-6}

— the huge third derivative pushes the floor left, which is what the widget shows. ∎ And the best possible error is only

\sim u^{2/3}\approx10^{-11}

(about

10^{-10}

here) — while autodiff is exact to full precision. That is the entire case for automatic differentiation, compressed into one minimisation.

The realization

The nudge estimate's error is (curve error, shrinking like $h^2$ ) + (rounding noise, growing like $1/h$ ). So there is a best $h$ — about $10^{-6}$ here — and even there the answer is off by about $10^{-10}$ .

Autodiff has neither problem. Nudging survives only as a gradient checker: a quick test that a hand-written backward pass is right.

Pause & predict

In the gradient checker, why does making $h$ smaller and smaller eventually make the estimate worse?

The squash's slope is free. $\sigma'=\sigma(1-\sigma)$ needs only the output $\mathbf a$ , which the forward pass already has. The same trick works for $\tanh$ : its slope is $1-a^2$ . That is why libraries store every layer's output on the way forward. That store is the "memory cost" of training.

Vanishing gradients. Remember the party game where a message is whispered down a line of people, and it gets fainter each time? Each sigmoid layer passes on at most a quarter of the blame it receives, because $a(1-a)\le\tfrac14$ . With ten layers, the blame reaching layer 1 is at most $4^{-9}\approx4\times10^{-6}$ of what left the loss. The early layers hardly learn. This is why deep networks moved to ReLU, whose slope is 1 wherever it is switched on.

Vanishing gradients, drawn. The loss sits to the right of layer 10. Every sigmoid the blame crosses scales it by

a(1-a)\le\tfrac14

, so by layer 1 the strongest possible signal is

4^{-9}\approx4\times10^{-6}

of what left the loss — and saturated neurons (

a\approx0.95

, factor

0.045

) make it far worse. The early layers barely learn. ReLU's slope of

1

where active is the fix.

Backwards or forwards? Backprop starts at one output (the loss) and finds the slope for every input in one sweep. There is a mirror image, forward mode: it starts at one input and finds the slope of every output. Training has one loss and millions of knobs, so every deep-learning library runs backwards.

Which sweep to run. A sweep costs about one pass either way; the question is what one sweep buys. Reverse mode starts at one output and prices every input — millions of weights for one loss. Forward mode starts at one input and prices every output. Training has one output and millions of inputs, so every framework runs reverse.

Forward mode you can do by hand — dual numbers. Carry every number as a pair: its value and its slope. Then teach ordinary arithmetic to update both at once. For $f(x)=x\,e^x$ at $x=1$ : start with the pair $(1,\,1)$ . Applying $e^x$ gives $(e,\,e)$ . Multiplying by $x$ gives $(e,\,2e)$ . So $f(1)=e$ and $f'(1)=2e\approx5.437$ , and nobody wrote down a slope formula.

If you want the algebra · arithmetic that differentiates itself

Prove it · arithmetic that differentiates itself

Claim. Write each pair $(v,\dot v)$ as $v+\dot v\,\varepsilon$ , where $\varepsilon$ is a new symbol with one rule: $\varepsilon^{2}=0$ . Then ordinary algebra on these "dual numbers" performs exact differentiation.

1

Sums.

(a+\dot a\varepsilon)+(b+\dot b\varepsilon)=(a+b)+(\dot a+\dot b)\varepsilon

. Values add, derivatives add — the sum rule, with no one enforcing it.

2

Products. Multiply out and apply the one rule:

(a+\dot a\varepsilon)(b+\dot b\varepsilon)=ab+(a\dot b+\dot a b)\varepsilon+\dot a\dot b\,\varepsilon^{2}=ab+(a\dot b+\dot a b)\,\varepsilon

\varepsilon^2=0

killed the last term — it is the product of two tiny nudges, the very leftover that §4's promise says must die. What survives is exactly the product rule.

3

Any smooth function. Define

g(a+\dot a\varepsilon)=g(a)+g'(a)\,\dot a\,\varepsilon

— and this is forced, not chosen: Taylor gives

g(a+t)=g(a)+g'(a)t+\tfrac{g''(a)}{2}t^2+\cdots

, and with

t=\dot a\varepsilon

every term from

t^2

onwards contains

\varepsilon^2

and vanishes. The chain rule arrives baked in: whatever

\dot a

is carrying gets multiplied by the local slope

g'(a)

.

4

Run the program on pairs, seeding the input as

x=x_0+1\cdot\varepsilon

(its derivative with respect to itself is 1). Example —

f(x)=x\,e^{x}

at

x_0=1

:

x=(1,\,1)\ \xrightarrow{\ e^{x}\ }\ (e,\,e)\ \xrightarrow{\ \times\, x\ }\ \big(1\cdot e,\ \ 1\cdot e+1\cdot e\big)=(e,\,2e)

So

f(1)=e

and

f'(1)=2e\approx5.4366

— exact, and no derivative formula was ever written down. ∎ Check:

f'=e^{x}(1+x)

gives

2e

. ✓ This is forward-mode autodiff: one sweep prices all outputs of one input — the mirror image of backprop, and the winner when inputs are few and outputs many.

Pause & predict

Dual numbers use a new symbol $\varepsilon$ with one rule: $\varepsilon^{2}=0$ . That single rule makes the product rule appear by itself. What is $\varepsilon^{2}=0$ really saying?

In one sentence: autodiff is exact like the paper method and cheap like one run of the program, which is why nudging survives only as a checker.

12

What to carry forward

Imagine this

In PyTorch, training a network needs one line: loss.backward(). It can feel like magic.

It isn't. It is everything in this unit, run by a simple loop.

One picture carried the whole unit: a function drawn as a graph of tiny steps, with values flowing right and blame flowing left. Everything else was careful bookkeeping on that picture.

Idea	The one-line version	Where it returns
Computation graph	Any function as tiny steps joined by arrows	every library's autograd
Backward pass	$\bar f=1$ ; multiply along paths, add at forks	every training step you'll ever run
$\delta$ (the blame)	Computed once at each step, reused by every knob feeding it	Unit 9 · gradient descent at scale
The four layer formulas	$\delta^{(Z)}=\delta^{(a)}\!\odot a\odot(1{-}a)$ ; $\ \partial_A=\delta^{(Z)}x^{\top}$ ; $\ \partial_b=\delta^{(Z)}$ ; $\ \partial_x=A^{\top}\delta^{(Z)}$	the inner loop of deep learning
Gradient cookbook	Dot → $\mathbf a^\top$ ; $\ \mathbf a^\top X\mathbf b\to \mathbf a\mathbf b^\top$ ; $\ \mathbf x^\top B\mathbf x\to \mathbf x^\top(B{+}B^\top)$	Units 9–15, constantly
Cost of the slopes	All of them, for about the price of one forward pass	why billion-knob training is possible
Linearization	$f(\mathbf x_0)+\nabla f^{\top}(\mathbf x-\mathbf x_0)$ : the local "flat Earth"	Unit 8 · Taylor; Unit 9 · each descent step

The realization

Blame flows backwards through the same graph that computed the value. Multiply along paths, add at forks, and reuse every shared piece. Then a million slopes cost one sweep.

In one sentence: loss.backward() is this unit run by a for-loop — one backward sweep through the graph hands every knob its slope.

13

Practice arena — the unit's problem set, solved in full

Eleven problems. Three come from Part C of Prof. Saurabh's practice set (Parts A and B, on Taylor series and the Hessian, wait in Unit 8). Three come from his companion guide. Five are new: an identity drill (Problem 7), a full numeric backward pass (Problem 8), and three proof workouts (Problems 9–11). Every solution is machine-checked.

One habit does most of the work: draw the graph first, then walk it backwards. Multiply along paths, add at forks, and check every shape before trusting the algebra.

Problem 1medium

A one-layer network computes $\mathbf z=W\mathbf x$ and the loss $g(\mathbf z)=\tfrac12\|\mathbf z-\mathbf t\|^2$ against a target $\mathbf t$ , with $W=\begin{bmatrix}1&2\\0&1\\-1&1\end{bmatrix},\qquad \mathbf x=\begin{bmatrix}1\\1\end{bmatrix},\qquad \mathbf t=\begin{bmatrix}1\\0\\2\end{bmatrix}.$ (a) Compute $\nabla_{\mathbf z}\,g$ (with its dimension) and $J=\partial\mathbf z/\partial\mathbf x$ (with its dimension). (b) Combine them to get $\nabla_{\mathbf x}\,g$ , and give the loss value.

What this tests. One honest backward step through a linear layer: the upstream gradient is the residual, and it returns to the input through $W^{\top}$ .

Show the full solution

Step 1 — forward pass.

\mathbf z=W\mathbf x=(1{+}2,\ 0{+}1,\ -1{+}1)^{\top}=(3,1,0)^{\top}

. Residual

\mathbf r=\mathbf z-\mathbf t=(2,1,-2)^{\top}

, and the loss is

g=\tfrac12(2^2+1^2+(-2)^2)=\tfrac92=4.5

.

(a) Step 2 — gradient at the output. For

g=\tfrac12\|\mathbf z-\mathbf t\|^2

,

\ \nabla_{\mathbf z}\,g=\mathbf z-\mathbf t=\mathbf r=(2,1,-2)^{\top}

— dimension

3\times1

, one entry per output. The Jacobian of a linear map is the matrix itself:

J=\partial(W\mathbf x)/\partial\mathbf x=W

, dimension

3\times2

(outputs × inputs).

(b) Step 3 — chain them.

\nabla_{\mathbf x}\,g=J^{\top}\nabla_{\mathbf z}\,g=W^{\top}\mathbf r=\begin{bmatrix}1&0&-1\\2&1&1\end{bmatrix}\begin{bmatrix}2\\1\\-2\end{bmatrix}=\begin{bmatrix}1\cdot2+0\cdot1+(-1)(-2)\\2\cdot2+1\cdot1+1\cdot(-2)\end{bmatrix}=\begin{bmatrix}4\\3\end{bmatrix}

Dimension check:

\nabla_{\mathbf z}g

is

3\times1

,

W^{\top}

is

2\times3

, product

2\times1

— the shape of

\mathbf x

, as a gradient must be. ✓

$\nabla_{\mathbf z}g=(2,1,-2)^{\top}$ (3×1), $J=W$ (3×2), $\nabla_{\mathbf x}g=(4,3)^{\top}$ (2×1), loss $=4.5$ .

Remember

"Backprop through a linear layer" = "multiply the incoming gradient by $W^{\top}$ ". You can also check by substituting first: $g(\mathbf x)=\tfrac12\|W\mathbf x-\mathbf t\|^2$ has gradient $W^{\top}(W\mathbf x-\mathbf t)=W^{\top}\mathbf r$ — same answer, one line.

Problem 2medium

A neuron computes $z=wx+b,\ a=\tanh(z)$ , and the loss $L=\tfrac12(a-y)^2$ . Using $\tanh'(z)=1-\tanh^2(z)$ and the values $w=0.5,\ x=1,\ b=0,\ y=1$ : (a) do the forward pass $(z,a,L)$ ; (b) backpropagate to obtain $\partial L/\partial w,\ \partial L/\partial b,\ \partial L/\partial x$ .

What this tests. The full forward–backward cycle on a single neuron, and the shared blame $\delta$ that Section 6 trains with.

Show the full solution

(a) Forward.

z=0.5\cdot1+0=0.5

;

\ a=\tanh(0.5)=0.4621

;

\ L=\tfrac12(0.4621-1)^2=\tfrac12(-0.5379)^2=0.1447

.

(b) Step 1 — the shared blame. Walk

z\to a\to L

backwards:

\frac{\partial L}{\partial a}=a-y=-0.5379,\qquad \frac{da}{dz}=1-a^2=1-0.2136=0.7864

\delta\;=\;\frac{\partial L}{\partial z}=\frac{\partial L}{\partial a}\,\frac{da}{dz}=(-0.5379)(0.7864)=-0.4230

Step 2 — hand out copies.

z=wx+b

gives

\partial z/\partial w=x=1

,

\partial z/\partial b=1

,

\partial z/\partial x=w=0.5

:

\frac{\partial L}{\partial w}=\delta\cdot x=-0.4230,\qquad \frac{\partial L}{\partial b}=\delta=-0.4230,\qquad \frac{\partial L}{\partial x}=\delta\cdot w=-0.2115

$z=0.5,\ a=0.4621,\ L=0.1447$ ; $\ \partial L/\partial w=-0.4230,\ \partial L/\partial b=-0.4230,\ \partial L/\partial x=-0.2115$ .

Remember

$\delta$ is computed once and reused three times — the whole economy of backprop in miniature. Both parameter gradients are negative, so a descent step raises $w$ and $b$ , pushing $a$ toward the target $1$ . Section 6's widget runs exactly these numbers.

Problem 3easy

A scalar function is built from two intermediates: $f=uv$ , $\ u=x^2+y$ , $\ v=x-y$ . (a) Write $\partial f/\partial u,\ \partial f/\partial v$ and the four partials of $u,v$ with respect to $x,y$ . (b) Using the chain rule, assemble $\partial f/\partial x$ and $\partial f/\partial y$ , and evaluate both at $(x,y)=(1,1)$ .

What this tests. Multiply along paths, add across paths — the two rules of Section 3, by hand.

Show the full solution

(a) The local derivatives. Outer:

\partial f/\partial u=v

,

\ \partial f/\partial v=u

. Inner:

\partial u/\partial x=2x

,

\ \partial u/\partial y=1

,

\ \partial v/\partial x=1

,

\ \partial v/\partial y=-1

.

(b) Step 1 — two paths from each input.

\frac{\partial f}{\partial x}=\underbrace{\frac{\partial f}{\partial u}\frac{\partial u}{\partial x}}_{\text{via }u}+\underbrace{\frac{\partial f}{\partial v}\frac{\partial v}{\partial x}}_{\text{via }v}=v\,(2x)+u\,(1)=2x(x-y)+(x^2+y)

\frac{\partial f}{\partial y}=v\,(1)+u\,(-1)=(x-y)-(x^2+y)

Step 2 — evaluate at $(1,1)$ where

u=2,\ v=0

:

\frac{\partial f}{\partial x}\Big|_{(1,1)}=0\cdot2+2\cdot1=2,\qquad \frac{\partial f}{\partial y}\Big|_{(1,1)}=0\cdot1+2\cdot(-1)=-2

Step 3 — cross-check by expanding first.

f=(x^2+y)(x-y)=x^3-x^2y+xy-y^2

, so

\partial f/\partial x=3x^2-2xy+y=3-2+1=2

✓ and

\partial f/\partial y=-x^2+x-2y=-1+1-2=-2

✓.

$\partial f/\partial x\,(1,1)=2$ , $\ \partial f/\partial y\,(1,1)=-2$ — by paths and by expansion alike.

Problem 4hard

Define $g(\mathbf z,\boldsymbol\nu):=\log p(\mathbf x,\mathbf z)-\log q(\mathbf z,\boldsymbol\nu)$ with $\mathbf z:=t(\boldsymbol\epsilon,\boldsymbol\nu)$ , for differentiable functions $p,q,t$ and $\mathbf x\in\mathbb{R}^D,\ \mathbf z\in\mathbb{R}^E,\ \boldsymbol\nu\in\mathbb{R}^F,\ \boldsymbol\epsilon\in\mathbb{R}^G$ . Using the chain rule, compute the total derivative $\dfrac{d}{d\boldsymbol\nu}g(\mathbf z,\boldsymbol\nu)$ .

What this tests. Direct and indirect paths of influence — the multivariate chain rule when a variable appears both explicitly and through an intermediate. (This is the gradient inside variational autoencoders.)

Show the full solution

Step 1 — draw the graph.

\boldsymbol\nu

reaches

g

two ways: directly (it sits inside

\log q(\mathbf z,\boldsymbol\nu)

) and indirectly (it builds

\mathbf z=t(\boldsymbol\epsilon,\boldsymbol\nu)

, and

\mathbf z

feeds both

\log p

and

\log q

). Total derivative = sum over both routes.

Step 2 — the direct route. Hold

\mathbf z

fixed and differentiate

g

in the

\boldsymbol\nu

-slot only. The

\log p

term has no direct

\boldsymbol\nu

:

\frac{\partial g}{\partial\boldsymbol\nu}=-\frac{\partial \log q(\mathbf z,\boldsymbol\nu)}{\partial\boldsymbol\nu}\qquad(1\times F)

Step 3 — the indirect route, two links. First how

g

feels

\mathbf z

, then how

\mathbf z

feels

\boldsymbol\nu

:

\frac{\partial g}{\partial\mathbf z}=\frac{\partial \log p(\mathbf x,\mathbf z)}{\partial\mathbf z}-\frac{\partial \log q(\mathbf z,\boldsymbol\nu)}{\partial\mathbf z}\quad(1\times E),\qquad \frac{\partial\mathbf z}{\partial\boldsymbol\nu}=\frac{\partial t(\boldsymbol\epsilon,\boldsymbol\nu)}{\partial\boldsymbol\nu}\quad(E\times F)

Step 4 — add the routes.

\frac{d g}{d\boldsymbol\nu}=-\frac{\partial \log q}{\partial\boldsymbol\nu}\;+\;\left(\frac{\partial \log p}{\partial\mathbf z}-\frac{\partial \log q}{\partial\mathbf z}\right)\frac{\partial t}{\partial\boldsymbol\nu}

Dimension check:

(1\times F)+(1\times E)(E\times F)=(1\times F)

✓.

$\dfrac{dg}{d\boldsymbol\nu}=-\dfrac{\partial\log q}{\partial\boldsymbol\nu}+\Big(\dfrac{\partial\log p}{\partial\mathbf z}-\dfrac{\partial\log q}{\partial\mathbf z}\Big)\dfrac{\partial t}{\partial\boldsymbol\nu}$ .

Remember

Write $d/d\boldsymbol\nu$ (total) for "all routes", $\partial/\partial\boldsymbol\nu$ (partial) for "this slot only, everything else frozen". The whole problem is Section 3's fork rule wearing research-paper notation.

Problem 5medium

Compute $\dfrac{df}{d\mathbf x}$ for $f(z)=\log(1+z)$ , $\ z=\mathbf x^{\top}\mathbf x$ , $\ \mathbf x\in\mathbb{R}^D$ . State the dimension of every partial derivative involved.

What this tests. A two-step graph $\mathbf x\to z\to f$ where the middle is a scalar — plus the cookbook's dot-product identity applied to $\mathbf x^{\top}\mathbf x$ .

Show the full solution

Step 1 — the outer link.

\dfrac{df}{dz}=\dfrac{1}{1+z}

. Dimension

1\times1

.

Step 2 — the inner link.

z=\mathbf x^{\top}\mathbf x=\sum_i x_i^2

, so

\partial z/\partial x_k=2x_k

, assembled as the row

\frac{\partial z}{\partial\mathbf x}=2\mathbf x^{\top}\qquad(1\times D)

(Or by the cookbook:

\mathbf x^\top B\mathbf x

with

B=I

gives

\mathbf x^{\top}(I+I)=2\mathbf x^{\top}

.)

Step 3 — multiply the links.

\frac{df}{d\mathbf x}=\frac{df}{dz}\,\frac{\partial z}{\partial\mathbf x}=\frac{2\mathbf x^{\top}}{1+\mathbf x^{\top}\mathbf x}\qquad(1\times1)(1\times D)=(1\times D)\ \checkmark

Step 4 — sanity check with numbers. At

\mathbf x=(1,2)^{\top}

:

z=5

, so the formula gives

\tfrac{2}{6}(1,2)=(\tfrac13,\tfrac23)

— and differentiating

\log(1+x_1^2+x_2^2)

directly gives

\big(\tfrac{2x_1}{1+z},\tfrac{2x_2}{1+z}\big)=(\tfrac13,\tfrac23)

✓.

$\dfrac{df}{d\mathbf x}=\dfrac{2\mathbf x^{\top}}{1+\mathbf x^{\top}\mathbf x}$ , a $1\times D$ row.

Problem 6medium

Compute $\dfrac{d\mathbf f}{d\mathbf x}$ for $\mathbf f(\mathbf z)=\sin(\mathbf z)$ applied entrywise, $\ \mathbf z=A\mathbf x+\mathbf b$ , with $A\in\mathbb{R}^{E\times D},\ \mathbf x\in\mathbb{R}^{D},\ \mathbf b\in\mathbb{R}^{E}$ . State every dimension.

What this tests. The Jacobian chain rule with an entrywise nonlinearity — the exact structure of Section 7's layer, with $\sin$ in place of $\sigma$ .

Show the full solution

Step 1 — the outer Jacobian.

f_i=\sin(z_i)

touches only its own

z_i

, so off-diagonal partials vanish and

\frac{\partial\mathbf f}{\partial\mathbf z}=\operatorname{diag}\!\big(\cos(z_1),\dots,\cos(z_E)\big)=\operatorname{diag}\!\big(\cos(\mathbf z)\big)\qquad(E\times E)

— the same diagonal shape as Section 7's

\operatorname{diag}(\mathbf a\odot(1-\mathbf a))

, and for the same reason: entrywise functions have diagonal Jacobians.

Step 2 — the inner Jacobian.

\mathbf z=A\mathbf x+\mathbf b

is linear, so

\ \partial\mathbf z/\partial\mathbf x=A

(E\times D)

; the constant

\mathbf b

contributes nothing.

Step 3 — multiply, order matters.

\frac{d\mathbf f}{d\mathbf x}=\frac{\partial\mathbf f}{\partial\mathbf z}\,\frac{\partial\mathbf z}{\partial\mathbf x}=\operatorname{diag}\!\big(\cos(A\mathbf x+\mathbf b)\big)\,A\qquad(E\times E)(E\times D)=(E\times D)\ \checkmark

Concretely: row

i

of

A

, scaled by

\cos(z_i)

.

$\dfrac{d\mathbf f}{d\mathbf x}=\operatorname{diag}\!\big(\cos(A\mathbf x+\mathbf b)\big)A$ , an $E\times D$ Jacobian.

Problem 7medium

Verify two cookbook identities the honest way, entry by entry. (a) For $B=\begin{bmatrix}1&2\\3&4\end{bmatrix}$ , expand $\mathbf x^{\top}B\mathbf x$ and confirm $\partial(\mathbf x^{\top}B\mathbf x)/\partial\mathbf x=\mathbf x^{\top}(B+B^{\top})$ . (b) For $\mathbf a=(1,2)^{\top},\ \mathbf b=(3,1)^{\top}$ , expand $\mathbf a^{\top}X\mathbf b$ and confirm $\partial(\mathbf a^{\top}X\mathbf b)/\partial X=\mathbf a\mathbf b^{\top}$ .

What this tests. That the identities are nothing but organised entry-wise differentiation — once verified by hand, you may use them forever with a clear conscience.

Show the full solution

(a) Step 1 — expand the scalar.

\mathbf x^{\top}B\mathbf x=x_1^2+2x_1x_2+3x_2x_1+4x_2^2=x_1^2+5x_1x_2+4x_2^2

Step 2 — differentiate like any polynomial.

\partial/\partial x_1=2x_1+5x_2

,

\ \partial/\partial x_2=5x_1+8x_2

, so the gradient row is

[\,2x_1+5x_2,\ 5x_1+8x_2\,]

.

Step 3 — the identity's answer.

B+B^{\top}=\begin{bmatrix}2&5\\5&8\end{bmatrix}

, and

\mathbf x^{\top}(B+B^{\top})=[\,2x_1+5x_2,\ 5x_1+8x_2\,]

— identical. ✓ (Note

2B

would give

[2x_1{+}4x_2,\ 6x_1{+}8x_2]

— wrong unless

B

is symmetric.)

(b) Step 1 — expand. With

\mathbf a=(1,2)^{\top},\ \mathbf b=(3,1)^{\top}

:

\mathbf a^{\top}X\mathbf b=\sum_{i,j}a_i X_{ij} b_j=3X_{11}+X_{12}+6X_{21}+2X_{22}

Step 2 — differentiate per entry.

\partial/\partial X_{ij}=a_ib_j

: the four coefficients

3,1,6,2

arranged in

X

's own shape are

\frac{\partial(\mathbf a^{\top}X\mathbf b)}{\partial X}=\begin{bmatrix}3&1\\6&2\end{bmatrix}=\mathbf a\mathbf b^{\top}\ \checkmark

Both identities check entry by entry — and (b) is exactly why Section 7's $\partial\mathrm{Loss}/\partial A$ came out as the outer product $\boldsymbol\delta^{(Z)}\mathbf x^{\top}$ .

Problem 8hard

Run the full backward pass on the lecture's function $f(x)=\sqrt{x^2+e^{x^2}}+\cos(x^2+e^{x^2})$ at $x=1$ , using the graph $a=x^2,\ b=e^a,\ c=a+b,\ d=\sqrt c,\ e=\cos c,\ f=d+e$ . (a) Forward pass: all six values to 4 decimals. (b) Backward pass: all the blames $\bar f,\bar d,\bar e,\bar c,\bar b,\bar a,\bar x$ . (c) Confirm $\bar x$ against the closed-form derivative.

What this tests. The complete algorithm of Sections 2–5, executed by hand with real numbers — including the fork at $c$ and the fork at $a$ .

Show the full solution

(a) Forward, left to right.

a=1^2=1,\quad b=e^1=2.7183,\quad c=1+2.7183=3.7183

d=\sqrt{3.7183}=1.9283,\quad e=\cos(3.7183)=-0.8383,\quad f=1.9283-0.8383=1.0900

(Radians throughout —

c

is just a number, and

\cos

of it lives on the unit circle, not a protractor.)

(b) Backward, right to left. Start

\bar f=1

. Node

f=d+e

mails both parents a copy scaled by

1

:

\ \bar d=1,\ \bar e=1

.

The fork at $c$ .

c

feeds

d=\sqrt c

and

e=\cos c

, so its mail arrives in two pieces that add:

\bar c=\bar d\cdot\frac{1}{2\sqrt c}+\bar e\cdot(-\sin c)=\frac{1}{2(1.9283)}+\big(-\sin(3.7183)\big)=0.2593+0.5452=0.8045

(

\sin(3.7183)=-0.5452

— third quadrant, so the minus signs stack to a plus.)

Through $b$ , and the fork at $a$ .

\bar b=\bar c\cdot1=0.8045

. Then

a

feeds

b=e^a

and

c=a+b

:

\bar a=\bar b\cdot e^{a}+\bar c\cdot 1=0.8045\cdot2.7183+0.8045=2.1870+0.8045=2.9915

The last edge.

\ \bar x=\bar a\cdot 2x=2.9915\cdot2=5.9831

.

(c) The closed form agrees.

\frac{df}{dx}=\Big(\frac{1}{2\sqrt{x^2+e^{x^2}}}-\sin(x^2+e^{x^2})\Big)\big(2x+2xe^{x^2}\big)\Big|_{x=1}=0.8045\times7.4366=5.9831\ \checkmark

Notice the factor

0.8045

is

\bar c

, and

2x+2xe^{x^2}=7.4366

is what the graph assembled as

\bar a\cdot2x/\bar c

— the closed form and the graph are the same computation, differently organised.

Forward: $1,\ 2.7183,\ 3.7183,\ 1.9283,\ -0.8383,\ 1.0900$ . Backward: $\bar f=1,\ \bar d=\bar e=1,\ \bar c=\bar b=0.8045,\ \bar a=2.9915,\ \bar x=5.9831=df/dx$ . ✓

Watch out

Run the same pass at $x=2$ and $e^{x^2}=e^4=54.6$ dominates everything: $\bar c=-0.8223$ and $df/dx=-182.87$ . The companion doc's printed run at $x=2$ uses $\cos(58.6)\approx0.83$ , but the true value in radians is $-0.46$ — a calculator slip that flips the final answer completely (see the errata note in §2). The habit that catches such slips: always confirm a hand-computed derivative with a small numerical nudge.

Problem 9medium

The activation family, from first principles. (a) Show that the softplus $s(z)=\ln(1+e^{z})$ has derivative $s'(z)=\sigma(z)$ — the sigmoid is softplus's slope. (b) Prove the bridge $\tanh(z)=2\,\sigma(2z)-1$ . (c) Using (b) and $\sigma'=\sigma(1-\sigma)$ , derive $\tanh'=1-\tanh^{2}$ without a quotient rule.

What this tests. That the activation identities frameworks hard-code are a small connected family — one proof engine (find the appearances, chain the slopes) generates them all.

Show the full solution

(a) Chain rule through the logarithm. Outer slope

1/(1+e^{z})

, inner slope

e^{z}

:

s'(z)=\frac{e^{z}}{1+e^{z}}=\frac{1}{e^{-z}+1}=\sigma(z)\ \checkmark

(dividing top and bottom by

e^{z}

in the middle step). So the smooth ramp softplus steepens exactly at the sigmoid's rate — which is why

\sigma

is also called the logistic function.

(b) Build $2\sigma(2z)-1$ and simplify.

2\,\sigma(2z)-1=\frac{2}{1+e^{-2z}}-1=\frac{2-(1+e^{-2z})}{1+e^{-2z}}=\frac{1-e^{-2z}}{1+e^{-2z}}

Multiply top and bottom by

e^{z}

:

\frac{e^{z}-e^{-z}}{e^{z}+e^{-z}}=\tanh(z)\ \checkmark

In words: tanh is a sigmoid, stretched to run twice as fast and re-centred to fill

(-1,1)

.

(c) Differentiate the bridge with Rule 1. Write

t=\tanh z

and note from (b) that

\sigma(2z)=\tfrac{1+t}{2}

, hence

1-\sigma(2z)=\tfrac{1-t}{2}

. Then

\tanh'(z)=2\cdot\sigma'(2z)\cdot 2=4\,\sigma(2z)\big(1-\sigma(2z)\big)=4\cdot\frac{1+t}{2}\cdot\frac{1-t}{2}=(1+t)(1-t)=1-t^{2}\ \checkmark

— the factor

2

outside is the bridge's scaling, the factor

2

inside is the chain rule through

2z

.

$s'=\sigma$ ; $\ \tanh z=2\sigma(2z)-1$ ; $\ \tanh'=1-\tanh^{2}$ , derived purely from $\sigma'=\sigma(1-\sigma)$ .

Remember

All three facts are one family portrait: softplus's slope is $\sigma$ , $\tanh$ is a rescaled $\sigma$ , and both backward-pass identities ( $\sigma(1-\sigma)$ and $1-a^2$ ) are the same identity seen through the bridge. Knowing one buys you the others.

Problem 10hard

The cookbook earns its keep: ridge regression solved by pure algebra. Consider the regularised least-squares loss $E(\mathbf s)=\|\mathbf x-A\mathbf s\|^{2}+\lambda\|\mathbf s\|^{2}$ (a) Using the cookbook identities, derive $\nabla_{\mathbf s}E$ . (b) Set it to zero and show the minimiser solves $(A^{\top}A+\lambda I)\,\mathbf s=A^{\top}\mathbf x$ . (c) Solve numerically for $A=\begin{bmatrix}1&0\\2&1\\0&3\end{bmatrix},\qquad \mathbf x=\begin{bmatrix}1\\2\\0\end{bmatrix},\qquad \lambda=1.$

What this tests. That the identities you proved are tools, not trivia: two of them combine into the closed-form solution of ridge regression — the first machine-learning model this course can now train end-to-end by algebra alone.

Show the full solution

(a) Differentiate each piece with the cookbook. The first piece is the least-squares identity with

W=I

:

\frac{\partial}{\partial\mathbf s}\|\mathbf x-A\mathbf s\|^{2}=-2(\mathbf x-A\mathbf s)^{\top}A

The second is the quadratic identity with

B=I

(so

B+B^{\top}=2I

):

\frac{\partial}{\partial\mathbf s}\lambda\,\mathbf s^{\top}\mathbf s=2\lambda\,\mathbf s^{\top}

Sum:

\ \nabla_{\mathbf s}E=-2(\mathbf x-A\mathbf s)^{\top}A+2\lambda\,\mathbf s^{\top}

.

(b) Set the gradient to zero (and transpose the row into a column to read it as an equation in

\mathbf s

):

-2A^{\top}(\mathbf x-A\mathbf s)+2\lambda\mathbf s=\mathbf 0 \;\;\Longrightarrow\;\; A^{\top}A\,\mathbf s+\lambda\mathbf s=A^{\top}\mathbf x \;\;\Longrightarrow\;\; \big(A^{\top}A+\lambda I\big)\mathbf s=A^{\top}\mathbf x\ \checkmark

This is a stationary point, and it is the minimum:

E

is a sum of squares whose quadratic part

\mathbf s^{\top}(A^{\top}A+\lambda I)\mathbf s

is positive for every

\mathbf s\neq\mathbf 0

once

\lambda>0

— the bowl opens upward in every direction, and adding

\lambda I

even guarantees the system is invertible when

A^{\top}A

alone is not (Unit 5's nearly-singular warning, answered).

(c) Numbers.

A^{\top}A=\begin{bmatrix}1{+}4{+}0&0{+}2{+}0\\0{+}2{+}0&0{+}1{+}9\end{bmatrix}=\begin{bmatrix}5&2\\2&10\end{bmatrix},\qquad A^{\top}A+I=\begin{bmatrix}6&2\\2&11\end{bmatrix},\qquad A^{\top}\mathbf x=\begin{bmatrix}1{+}4{+}0\\0{+}2{+}0\end{bmatrix}=\begin{bmatrix}5\\2\end{bmatrix}

Solve

\begin{bmatrix}6&2\\2&11\end{bmatrix}\mathbf s=\begin{bmatrix}5\\2\end{bmatrix}

: determinant

66-4=62

, so by Cramer (or elimination)

s_1=\frac{5\cdot11-2\cdot2}{62}=\frac{51}{62}\approx0.8226,\qquad s_2=\frac{6\cdot2-2\cdot5}{62}=\frac{2}{62}=\frac{1}{31}\approx0.0323

$\nabla_{\mathbf s}E=-2(\mathbf x-A\mathbf s)^{\top}A+2\lambda\mathbf s^{\top}$ ; minimiser $(A^{\top}A+\lambda I)\mathbf s=A^{\top}\mathbf x$ ; numerically $\mathbf s\approx(0.8226,\ 0.0323)^{\top}$ .

Remember

Gradient $\to$ zero $\to$ linear system: for quadratic losses, "training" is one solve. Unit 9 begins exactly where this stops — with losses whose gradients cannot be set to zero in closed form, so we must walk downhill instead.

Problem 11medium

Differentiate a program with dual numbers (Section 11), seeding $x=x_0+1\cdot\varepsilon$ . (a) $f(x)=x^{2}e^{x}$ at $x_0=1$ . (b) $f(x)=\sqrt{x^{2}+9}$ at $x_0=-4$ — then compare with Section 10's stand-in line. (c) In one sentence: why are these results exact, where the gradient checker's finite differences never are?

What this tests. Forward-mode autodiff executed by hand: the $\varepsilon^{2}=0$ algebra runs the chain rule mechanically — you only ever do arithmetic on pairs.

Show the full solution

(a) Run the pairs. Seed

x=(1,\,1)

.

x^{2}=(1\cdot1,\ 1\cdot1+1\cdot1)=(1,\,2)\qquad e^{x}=(e,\ e\cdot1)=(e,\,e)

x^{2}\cdot e^{x}=(1\cdot e,\ \ 1\cdot e+2\cdot e)=(e,\ 3e)

So

f(1)=e\approx2.7183

and

f'(1)=3e\approx8.1548

. Check against the formula the program never used:

f'=(2x+x^{2})e^{x}\big|_{1}=3e

✓.

(b) A chain with a square root. Seed

x=(-4,\,1)

.

x^{2}=\big((-4)^2,\ 2\cdot(-4)\cdot1\big)=(16,\,-8)\qquad x^{2}+9=(25,\,-8)

\sqrt{\,\cdot\,}:\ \big(\sqrt{25},\ \tfrac{-8}{2\sqrt{25}}\big)=(5,\ -0.8)

So

f(-4)=5

and

f'(-4)=-0.8=-\tfrac45

— precisely the anchor value and slope of Section 10's stand-in line. The widget's tangent line and this pencil-and-paper pair computation are the same mathematics arriving by two roads.

(c) Why exact. No step ever subtracts two nearly equal numbers or divides by a small

h

— each rule (sum, product,

g'(a)\dot a

) is the corresponding calculus rule, applied with ordinary arithmetic — so the only error anywhere is standard floating-point rounding of the values themselves.

(a) $(e,\,3e)$ : $f'(1)=3e\approx8.1548$ . (b) $(5,\,-0.8)$ : $f'(-4)=-\tfrac45$ . (c) The algebra rules are the calculus rules — nothing is approximated, so nothing needs a step size.

Remember

Reverse mode (this unit's hero) and dual numbers (its mirror) bracket the field: backward for one output and many inputs, forward for one input and many outputs. Both differentiate the program, both are exact — and neither ever writes a derivative formula.

Next up

Unit 8 · Taylor & MacLaurin Series →

Linearization kept one derivative and got a line. Unit 8 keeps them all — and proves, from a single flat spot on a hiking trail, exactly how far each cut of the series can be trusted. Then it takes the second-order term into two variables, where it becomes the Hessian: the judge that tells a bowl from a dome from a saddle at the bottom of every loss valley.

← All units