Unit 6 · Differentiation — The Math Behind the Machine

1

You are standing in fog, and you want to get down

Imagine this

You are trekking in the hills near Munnar at dawn. The mist is so thick you cannot see your own shoes. You want to get down to the tea estate in the valley.

You cannot see the path. But you can feel one thing: the ground under your feet. Does it tilt down to the left or to the right? Is it steep or gentle?

So you take a small step towards "down". You feel again. You step again. Slowly, you reach the bottom, without ever seeing it.

This is exactly how a machine learns. Only the words change:

On the hill	Inside the machine
Your height	How wrong the model is right now. We call it the loss (or error).
Where you stand	The settings of the model's knobs, called its weights.
One small step	Nudging those knobs a little.
Reaching the valley floor	Training: making the loss as small as possible.

The machine is in the same fog as you. A real model has millions of knobs, so its "hill" has millions of directions. Nobody can draw it. The model can only feel the tilt where it stands.

That tilt has a name. With one knob it is the derivative. With many knobs it is the gradient. This whole unit is about measuring that tilt, for simple curves first and whole neural networks at the end.

Try: press ▶ take 10 steps and watch the hiker walk down using only the slope under its feet. Then drag it to the left of the hump and press again.

drag the hiker to move · drag the sky to orbit

position w1.5

fogon

—

Slide the fog away to see the whole valley. A real model never gets this view: the slope at its feet is all it has.

The realization

The slope tells you which way is up. So you step the other way. Slope positive? Step left. Slope negative? Step right.

$w_{\text{new}} = w - \eta\,L'(w)$

Here $L'(w)$ (read it: "L-prime of w") is the slope where you stand. $\eta$ (read it: "eta") is a small number that sets your step size. The minus sign is the "go the other way" part.

Pause & predict

Where you stand, the slope is $+3$ . Which way should you step to go down?

In one sentence: training is walking downhill in fog: you cannot see the valley, but you can always feel the slope under your feet, and that is enough.

2

The derivative, built from a shrinking triangle

Imagine this

You are in a cab. The driver's phone says 42 km/h. But speed is distance ÷ time. At one single instant you cover zero distance in zero time. So what does "42 right now" even mean?

The phone's trick: measure the distance over the last 1 second. Then over 0.1 second. Then 0.01 second. The answers get closer and closer to one number. That number is your speed right now.

The derivative is this "speed right now", for any curve. It answers: at this exact point, how fast is the output changing?

First, a quick experiment. Take any smooth curve and zoom in on one point. Keep zooming. The curve looks straighter and straighter. Zoom far enough and it is a straight line. The slope of that line is the derivative.

Try: press ▶ zoom in on sin x. Then pick |x|, set x₀ = 0, and zoom again.

point x₀0.6

zoom×1

—

Smooth curves turn into a straight line as you zoom; its slope is the derivative. |x| at x₀ = 0 keeps its corner forever: no line, so no derivative.

Pause & predict

A smooth curve straightens when you zoom in. The corner of $|x|$ at $0$ never does. What does that tell you?

Now the recipe for computing that slope. It is the cab trick, written with a curve:

①Stand at your point $x$ . Pick a second point a small gap $h$ to the right.
②Draw a straight line through both points. It is called a secant. Its slope is rise ÷ run.
③Shrink the gap $h$ . The second point slides towards the first. The line swings, then settles on the line that just touches the curve: the tangent.

The slope it settles on is the derivative:

\frac{df}{dx}=\lim_{h\to 0}\frac{f(x+h)-f(x)}{h}

Read it aloud: "the derivative of f is the number that rise ÷ run gets closer to, as the gap $h$ shrinks to zero". The word $\lim$ (read it: "the limit") means exactly "the number it settles on".

A tiny example. Take $f(x)=x^2$ at $x=1$ :

gap $h$	rise ÷ run	answer
$0.1$	$(1.21-1)/0.1$	$2.1$
$0.01$	$(1.0201-1)/0.01$	$2.01$
$0.001$	$(1.002001-1)/0.001$	$2.001$

The answers settle on $2$ . So the slope of $x^2$ at $x=1$ is $2$ . The sign also tells you direction: a positive derivative means uphill is to the right. (Section 6 turns this into a real compass.)

Try: press ▶ shrink h to 0. Watch the moving line settle onto the dashed tangent, and the number below settle too.

point x1

gap h1.2

—

The number settles without ever dividing by zero. That settled number is the derivative.

The realization

$\frac{df}{dx}=\lim_{h\to 0}\frac{f(x+h)-f(x)}{h}$

The derivative is rise ÷ run between two points, as the two points squeeze into one. It never divides by zero. It only asks what the ratio gets close to.

If you want the algebra · why the slope of

x^n

is

n x^{n-1}

Put $f(x)=x^n$ into the definition and expand the binomial:

\frac{df}{dx}=\lim_{h\to0}\frac{(x+h)^n-x^n}{h}=\lim_{h\to0}\frac{\sum_{i=0}^{n}\binom{n}{i}x^{n-i}h^{i}-x^n}{h}

The $i=0$ term of the sum is exactly $x^n$ , so it cancels the $-x^n$ , leaving a sum that starts at $i=1$ — and every surviving term carries at least one factor of $h$ , so the division by $h$ is legal:

\frac{df}{dx}=\lim_{h\to0}\sum_{i=1}^{n}\binom{n}{i}x^{n-i}h^{i-1}=\underbrace{\binom{n}{1}x^{n-1}}_{i=1,\ \text{no }h\text{ left}}+\lim_{h\to0}\underbrace{\sum_{i=2}^{n}\binom{n}{i}x^{n-i}h^{i-1}}_{\text{every term still has an }h\ \to\ 0}=n x^{n-1}

The whole trick is that exactly one term survives the shrinking: the one where the single factor of $h$ in the numerator was consumed by the $h$ in the denominator. Everything else still has an $h$ attached and vanishes.

Pause & predict

For $f(x)=x^2$ at $x=1$ , why does rise ÷ run settle on exactly $2$ ?

In one sentence: the derivative is the slope of a two-point line as the two points squeeze into one, and it exists exactly where the curve looks straight up close.

3

Four rules, and the only one that really matters

Imagine this

You are riding a geared bicycle. One turn of the pedals turns the back wheel 3 times. One turn of the back wheel moves you 2 metres.

How far do you go per pedal turn? $3\times 2 = 6$ metres. You multiplied the two rates. That is the whole chain rule.

Nobody works out a limit by hand every time. We learn a few ready-made slopes (like $x^n \to n x^{n-1}$ , $\sin x\to\cos x$ , $e^x\to e^x$ ). Then four rules combine them:

Rule	In everyday words	In symbols
Sum	Two things change together. Their changes add up.	$(f+g)'=f'+g'$
Product	Change one factor at a time, then add the two effects.	$(fg)'=f'g+fg'$
Quotient	The same idea for a fraction.	$\left(\frac{f}{g}\right)'=\frac{f'g-fg'}{g^2}$
Chain	One machine feeds another. Their rates multiply.	$\big(g(f(x))\big)'=g'(f(x))\cdot f'(x)$

The first three are handy tools. The fourth, the chain rule, is the reason deep learning works. So let us look at it slowly.

Picture two machines in a row. $x$ goes into machine $f$ . Its output $u=f(x)$ goes straight into machine $g$ . Out comes $g(u)$ .

Now nudge $x$ a tiny bit. Machine $f$ multiplies the nudge by its own rate, $f'(x)$ . Machine $g$ takes that bigger nudge and multiplies it again, by $g'(u)$ . The total rate is the product. Read the formula as: "g-prime at u, times f-prime at x".

A tiny example: $h(x)=(2x+1)^4$ at $x=0.5$ . The inner machine is $u=2x+1$ , so $u=2$ and its rate is $2$ . The outer machine is $u^4$ , with rate $4u^3=4\times 8=32$ . Total: $32\times 2=\mathbf{64}$ .

Top: the ratios along a chain multiply — 3 then 5 is 15, exactly $g'(f(x))\cdot f'(x)$ . Bottom: six stages that each pass on half of the motion leave $1/2^6=1/64$ of it; fifty stages at 0.9 would leave $0.9^{50}\approx0.005$ . Gear sizes are schematic — the printed ratios are what matter.

The picture also shows a danger. A deep network is like 50 machines in a row. If each one passes on only 0.9 of the nudge, then $0.9^{50}\approx 0.005$ reaches the first layer. Almost nothing. The first layers hardly learn. This is called the vanishing gradient problem.

Try: press ▶ send a nudge through and watch one small turn grow at each gear. Then press ▶ shrink the nudge: the measured total settles on the two rates multiplied.

drag the picture to orbit

input x0.5

nudge size0.1

—

Each glowing arc shows how far that gear turned. The arcs use a log scale: every extra 60° means ×10.

The realization

$\frac{d}{dx}\,g\big(f(x)\big)=g'\big(f(x)\big)\cdot f'(x)$

Machines in a row: their rates multiply. Work out the outer rate at the inner value, then multiply by the inner rate. A deep network is a long row of machines, so this one rule runs all of machine learning.

Worked example ·

h(x)=(2x+1)^4

, step by step

Name the two machines. Inner: $f(x)=2x+1$ . Outer: $g(u)=u^4$ . Their separate derivatives are easy: $f'(x)=2$ and $g'(u)=4u^3$ .

The chain rule says evaluate the outer derivative at the inner value, then multiply by the inner derivative:

h'(x)=g'(f(x))\cdot f'(x)=4(2x+1)^3\cdot 2 = 8(2x+1)^3

The classic slip is writing $4(2x+1)^3$ and stopping — forgetting to multiply by the inner derivative. The gear picture is the cure: you cannot report the final gear's speed without accounting for every gear it passed through.

If you want the algebra · why the product rule and the chain rule work

Why the product rule holds — derived

Claim. If $f$ and $g$ are differentiable at $x$ , then $(fg)'(x)=f'(x)\,g(x)+f(x)\,g'(x)$ .

1

Start from the definition applied to the product:

(fg)'(x)=\lim_{h\to0}\frac{f(x+h)\,g(x+h)-f(x)\,g(x)}{h}

Nothing has happened yet — this is just "nudge, subtract, divide, shrink" with

fg

in the slot.

2

Add and subtract the same quantity,

f(x+h)\,g(x)

, in the numerator:

f(x+h)g(x+h)-\underbrace{f(x+h)g(x)+f(x+h)g(x)}_{=\,0}-f(x)g(x)

Adding zero changes nothing, but it plants a "half-way house" between the two products: one factor updated, the other still old.

3

Group into two brackets, each of which changes only one factor:

=f(x+h)\big[g(x+h)-g(x)\big]+g(x)\big[f(x+h)-f(x)\big]

First bracket:

f(x+h)

is common to both terms, pull it out. Second bracket:

g(x)

is common. Check by expanding — all four original terms come back.

4

Divide by

h

and let

h\to0

, one piece at a time:

\underbrace{f(x+h)}_{\to\,f(x)}\cdot\underbrace{\frac{g(x+h)-g(x)}{h}}_{\to\,g'(x)}\;+\;g(x)\cdot\underbrace{\frac{f(x+h)-f(x)}{h}}_{\to\,f'(x)}

Each fraction is a difference quotient, so it tends to the corresponding derivative. The factor

f(x+h)\to f(x)

because a differentiable function is continuous. The limit of a sum of products is the sum of the products of the limits, when each limit exists.

5

Read off:

(fg)'=f(x)\,g'(x)+g(x)\,f'(x)

. Same as claimed. ∎ The whole proof is step 2: change one factor at a time, and the two single-factor changes add. Every product-type rule in matrix calculus repeats this move.

Why the chain rule multiplies — derived (the four-step version)

Claim. If $u=f(x)$ and $h=g(u)$ , then $\dfrac{dh}{dx}=g'(u)\cdot f'(x)$ — the two sensitivities multiply.

1

Nudge

x

by

\Delta x

. The first machine's own definition says how much

u

moves:

\Delta u=f(x+\Delta x)-f(x)=f'(x)\,\Delta x+\varepsilon_1\,\Delta x,\qquad \varepsilon_1\to0\ \text{as}\ \Delta x\to0

"Slope times nudge, plus a leftover that is small even compared with the nudge" — that is exactly what

f'(x)=\lim\Delta u/\Delta x

means, rewritten without the limit sign.

2

Feed

\Delta u

to the second machine and use its definition at the point

u

:

\Delta h=g(u+\Delta u)-g(u)=g'(u)\,\Delta u+\varepsilon_2\,\Delta u,\qquad \varepsilon_2\to0\ \text{as}\ \Delta u\to0

Same sentence, second machine. Note the nudge it receives is

\Delta u

, whatever size step 1 made it.

3

Substitute step 1 into step 2 and divide by

\Delta x

:

\frac{\Delta h}{\Delta x}=\big(g'(u)+\varepsilon_2\big)\frac{\Delta u}{\Delta x}=\big(g'(u)+\varepsilon_2\big)\big(f'(x)+\varepsilon_1\big)

Pure algebra: factor

\Delta u

out of step 2, then replace

\Delta u/\Delta x

by step 1 divided through by

\Delta x

.

4

Let

\Delta x\to0

. Then

\Delta u\to0

by step 1, so both

\varepsilon_1

and

\varepsilon_2

die:

\frac{dh}{dx}=\lim_{\Delta x\to0}\frac{\Delta h}{\Delta x}=g'(u)\cdot f'(x)

The ratios multiply. ∎ This is what the widget measures:

\Delta h/\Delta x

at finite nudges is

(g'+\varepsilon_2)(f'+\varepsilon_1)

, and the

\varepsilon

's are the "curvature gap" that shrinks as you shrink the nudge. (One fine point — what if

\Delta u

is exactly

0

? Unit 7 §4 handles it and gives the fully rigorous version.)

Pause & predict

In the product-rule proof (in the drawer above), we add and subtract $f(x+h)\,g(x)$ . Why?

Pause & predict

Use the chain rule. What is the slope of $\sin(x^2)$ ?

In one sentence: rates in a row multiply, and a deep network is a long row, which is why the chain rule runs machine learning.

4

Taylor — rebuilding a whole function out of its derivatives

Imagine this

A friend guides you on the phone to a café. "You're at the gate. Walk straight ahead." You can picture the road a little. "It bends slightly left." Better. "And the bend gets sharper after the temple." Better still.

Each extra sentence adds one more detail about the road at the spot where you stand. With enough of them, you can draw the road quite far ahead.

Taylor's idea is exactly this, for a function. Stand at one point $x_0$ (we call it the anchor). Then copy, one by one:

0where the function is (its value) — a flat line;
1which way it is heading (its slope) — the best straight line;
2how it bends (the second derivative) — the best parabola;
3how the bend changes (the third derivative) — the best cubic, and so on.

Stack $n$ of these copies and you get the Taylor polynomial:

T_n(x)=\sum_{k=0}^{n}\frac{f^{(k)}(x_0)}{k!}\,(x-x_0)^{k}

Read it: "add up, for $k$ from 0 to $n$ : the $k$ -th derivative at the anchor, divided by $k!$ , times the distance from the anchor to the power $k$ ". Here $f^{(k)}$ means "differentiate $k$ times", and $k!$ (read "k factorial") is $1\times2\times\cdots\times k$ . The $k!$ is a correction so that the copies don't spoil each other. The drawer below shows why.

A tiny example. For $e^x$ at $x_0=0$ , every derivative equals $1$ . So $T_3(x)=1+x+\frac{x^2}{2}+\frac{x^3}{6}$ . At $x=0.5$ this gives $1.6458$ . The true value $e^{0.5}$ is $1.6487$ . Four terms, and we are already within $0.003$ .

Let $n$ grow forever and you get the Taylor series. With the anchor at $0$ , it is called the Maclaurin series. This is how your calculator works out $\sin$ and $e^x$ . Unit 8 is all about it.

Try: press ▶ add terms one by one. The newest term glows in the formula and the bars. The thin purple curve is the error: it flattens near the anchor as terms are added.

degree n1

—

On x⁴ the copy becomes exact at degree 4. On ln(1+x) it is great near the anchor but falls apart past $x=1$ .

The realization

$T_n(x)=f(x_0)+f'(x_0)(x-x_0)+\frac{f''(x_0)}{2!}(x-x_0)^2+\cdots$

Each term copies one more derivative at the anchor: first the value, then the slope, then the bend. Near the anchor the copy is excellent. Far away it can fail, so Taylor is a local promise.

Worked examples ·

x^4

rebuilt exactly, and

\sin x+\cos x

Example 1 — a polynomial is reproduced exactly. Take $f(x)=x^4$ and anchor at $x_0=1$ . The derivatives at $1$ are

f(1)=1,\quad f'(1)=4,\quad f''(1)=12,\quad f'''(1)=24,\quad f^{(4)}(1)=24,\quad f^{(5)}(1)=f^{(6)}(1)=0

so, dividing each by $k!$ $(1,1,2,6,24)$ :

T_6(x)=1+4(x-1)+6(x-1)^2+4(x-1)^3+(x-1)^4 = x^4 = f(x)

Expanding really does collapse back to $x^4$ — a Taylor polynomial of degree $n$ reproduces any polynomial of degree $k\le n$ exactly, because all higher derivatives are identically zero. The approximation has nothing left to approximate.

Careful with the lecture's line here — the slide prints the raw derivatives without the $k!$ . See the errata note just below this fold-out.

Example 2 — a Maclaurin series that folds into two famous ones. For $f(x)=\sin x+\cos x$ at $x_0=0$ , the derivatives cycle with period 4: $f(0)=1,\ f'(0)=1,\ f''(0)=-1,\ f'''(0)=-1$ , then repeat. So

T_\infty(x)=1+x-\tfrac{1}{2!}x^2-\tfrac{1}{3!}x^3+\tfrac{1}{4!}x^4+\tfrac{1}{5!}x^5-\cdots

Sort the terms by parity and the two halves are recognisable on sight:

\underbrace{\sum_{k=0}^{\infty}\frac{(-1)^k}{(2k)!}x^{2k}}_{\cos x}+\underbrace{\sum_{k=0}^{\infty}\frac{(-1)^k}{(2k+1)!}x^{2k+1}}_{\sin x}=\cos x+\sin x

The series reassembles the function it came from — a satisfying closed loop, and a good sanity check on the machinery.

Errata — check this against your slides

On the Taylor-polynomial example ( $f(x)=x^4$ at $x_0=1$ ), the lecture slide lists the polynomial as $1+4(x-1)+12(x-1)^2+24(x-1)^3+24(x-1)^4$ . Those are the raw derivatives $f^{(k)}(1)$ — the division by $k!$ has not been applied. Expanding that expression gives $24x^4-72x^3+84x^2-44x+9$ , which is not $x^4$ ; at $x=2$ it returns $65$ instead of $16$ .

With the factorials divided out — $12/2!=6$ , $24/3!=4$ , $24/4!=1$ — the polynomial becomes $1+4(x-1)+6(x-1)^2+4(x-1)^3+(x-1)^4$ , whose coefficients are the binomial numbers $1,4,6,4,1$ . That really does collapse to $\big((x-1)+1\big)^4=x^4$ , which is the point the slide is making.

If you want the algebra · why the

k!

is there, and why

\sigma'=\sigma(1-\sigma)

Why the k! is there — derived

Claim. With $T_n(x)=\sum_{k=0}^{n}c_k\,(x-x_0)^k$ , the choice $c_k=f^{(k)}(x_0)/k!$ is the only one that makes every derivative of $T_n$ at $x_0$ agree with $f$ 's, up to order $n$ .

1

Differentiate one power term

k

times:

\frac{d^k}{dx^k}(x-x_0)^k=k\cdot(k-1)\cdots 2\cdot 1=k!

Each differentiation pulls the current exponent down as a factor and lowers it by one:

k

, then

k-1

, … down to

1

. The product of those factors is

k!

, and what is left is

(x-x_0)^0=1

.

2

Now differentiate a different power

(x-x_0)^j

k

times and evaluate at

x_0

:

\frac{d^k}{dx^k}(x-x_0)^j\Big|_{x_0}=\begin{cases}0 & j<k\ \ (\text{differentiated to nothing})\\[2pt] \frac{j!}{(j-k)!}(x-x_0)^{j-k}\big|_{x_0}=0 & j>k\ \ (\text{a positive power of }0)\\[2pt] k! & j=k\end{cases}

Lower powers are killed outright; higher powers keep a factor

(x-x_0)^{j-k}

that vanishes at the anchor. Only the matching power survives.

3

Apply

d^k/dx^k

to the whole sum and evaluate at

x_0

. By step 2, every term dies except

j=k

:

T_n^{(k)}(x_0)=c_k\cdot k!

The derivative of a sum is the sum of derivatives, so we may treat each term separately — and step 2 says just one of them is nonzero.

4

Demand

T_n^{(k)}(x_0)=f^{(k)}(x_0)

and solve:

c_k\,k!=f^{(k)}(x_0)\quad\Longrightarrow\quad c_k=\frac{f^{(k)}(x_0)}{k!}

One equation per

k

, one unknown per

k

— no other choice works. ∎ So the

k!

is not a convention: it is what cancels the

k!

that differentiation manufactures. Forget it and the "derivatives" of your polynomial are

k!

times too large — which is exactly the slip in the errata note above.

Why σ′ = σ(1 − σ) — derived

Claim. For the sigmoid $\sigma(z)=\dfrac{1}{1+e^{-z}}$ , the derivative is $\sigma'(z)=\sigma(z)\big(1-\sigma(z)\big)$ — no quotient rule needed at evaluation time.

1

Write

\sigma

as a power and use the chain rule with outer

u^{-1}

, inner

u=1+e^{-z}

:

\sigma'(z)=-\big(1+e^{-z}\big)^{-2}\cdot\frac{d}{dz}\big(1+e^{-z}\big)=-\big(1+e^{-z}\big)^{-2}\cdot(-e^{-z})=\frac{e^{-z}}{(1+e^{-z})^2}

Outer derivative

-u^{-2}

, inner derivative

-e^{-z}

(the chain rule again, on

e^{-z}

); the two minus signs cancel.

2

Split the fraction into a product of two factors:

\frac{e^{-z}}{(1+e^{-z})^2}=\underbrace{\frac{1}{1+e^{-z}}}_{\sigma(z)}\cdot\underbrace{\frac{e^{-z}}{1+e^{-z}}}_{\text{call it }w}

A square in the denominator is two copies of the same factor; hand one copy to each piece.

3

Recognise the second factor as

1-\sigma

:

1-\sigma(z)=\frac{1+e^{-z}}{1+e^{-z}}-\frac{1}{1+e^{-z}}=\frac{e^{-z}}{1+e^{-z}}=w

Put

1

over the common denominator and subtract — the numerator collapses to

e^{-z}

.

4

Combine:

\sigma'=\sigma\cdot(1-\sigma)

. Done. ∎ Why it matters: at

z=0

,

\sigma=\tfrac12

so

\sigma'=\tfrac14

— the largest slope the sigmoid ever has, and the "¼ per layer" that shrinks gradients in deep sigmoid networks. Problem 9 uses this identity to get the Taylor terms without a single quotient rule.

Pause & predict

In the widget, pick $x^4$ and raise the degree past 4. Why does nothing change?

Pause & predict

The sigmoid $\sigma(z)=\frac{1}{1+e^{-z}}$ has Maclaurin series $\tfrac12+\tfrac{z}{4}-\tfrac{z^3}{48}+\cdots$ . There is no $z^2$ term. What does that tell you?

In one sentence: a Taylor polynomial copies a function's value, slope, bend and more at one point, and the $k!$ keeps the copies from disturbing each other.

5

Many knobs at once — partial derivatives

Imagine this

You are making chai. The taste depends on two knobs: how much sugar and how long you boil it.

You want to know what sugar alone does. So you keep the boiling time fixed and change only the sugar. Then you do the opposite: fix the sugar and change the boiling time. Each test tells you about one knob, while the other stays frozen.

Real models have millions of knobs. So the function takes many numbers in and gives one score out. We write $f:\mathbb{R}^n\to\mathbb{R}$ (read it: "f takes $n$ numbers and returns one").

We use the chai trick. Change one input, freeze all the others, and measure the slope. That is a partial derivative. It is written with a curly $\partial$ (read it: "partial") to remind you the other inputs exist, but are frozen:

\frac{\partial f}{\partial x_1}=\lim_{h\to 0}\frac{f(x_1+h,\,x_2,\dots,x_n)-f(x_1,x_2,\dots,x_n)}{h}

Only the first input moves. Everything else is the same on both sides. In practice: when you differentiate with respect to $x$ , treat $y$ as if it were a fixed number, like 7. All your old rules still work.

A tiny example: $f(x,y)=x^2y$ . With $y$ frozen, $y$ is just a number in front, so $\partial f/\partial x=2xy$ . With $x$ frozen, $\partial f/\partial y=x^2$ . At $(3,2)$ : the $x$ -slope is $12$ , the $y$ -slope is $9$ .

Picture it. The graph $z=f(x,y)$ is a hilly landscape. Freezing $y$ is like slicing it with a vertical knife. The cut edge is an ordinary curve. The partial derivative is the slope of that curve.

Try: set x₀ and y₀ to 0. Both slopes become 0, yet one slice curves up and the other curves down. That shape is a saddle.

drag the picture to orbit

x₀0.8

y₀-0.6

—

The surface is $f(x,y)=x^2-y^2+\tfrac12 xy$ . The small panel lays each slice flat and shows its slope: that slope is the partial derivative.

The realization

$\frac{\partial f}{\partial x}=\text{the slope when only } x \text{ moves}$

Freeze every other input, then differentiate as usual. Each partial derivative is the slope of one slice through the landscape.

Worked examples · two partial derivatives from the lecture

Example 1: $f(x,y)=(x+2y^3)^2$ . This is a composition, so the chain rule applies inside each partial. Outer: square. Inner: $x+2y^3$ .

\frac{\partial f}{\partial x}=2(x+2y^3)\cdot\underbrace{\frac{\partial(x+2y^3)}{\partial x}}_{=\,1}=2(x+2y^3)

\frac{\partial f}{\partial y}=2(x+2y^3)\cdot\underbrace{\frac{\partial(x+2y^3)}{\partial y}}_{=\,6y^2}=12y^2(x+2y^3)

Same outer derivative both times; the two answers differ only in the inner derivative — the fingerprint of the chain rule.

Example 2: $f(x_1,x_2)=x_1^2x_2+x_1x_2^3$ . No composition, just the freeze-the-other rule applied term by term. Treating $x_2$ as a constant, $x_1^2x_2$ differentiates to $2x_1x_2$ and $x_1x_2^3$ to $x_2^3$ :

\frac{\partial f}{\partial x_1}=2x_1x_2+x_2^3,\qquad \frac{\partial f}{\partial x_2}=x_1^2+3x_1x_2^2

Collecting both into one row gives the gradient of the next section:

\frac{df}{d\mathbf{x}}=\big[\,2x_1x_2+x_2^3,\ \ x_1^2+3x_1x_2^2\,\big]\in\mathbb{R}^{1\times 2}

Pause & predict

For $f(x,y)=x^2y+3xy^2-y^3$ , what is $\frac{\partial f}{\partial x}$ ?

In one sentence: to take a partial derivative, freeze every other input and differentiate as usual.

6

The gradient — a compass that always points uphill

Imagine this

Back on the foggy hill. Now you may step in any direction, not just left or right. You test two directions:

One step east takes you 4 cm up. One step north takes you 3 cm up. Which way climbs fastest? Not east, not north, but somewhere in between. The pair $(4, 3)$ is an arrow pointing exactly at the steepest way up. Its length, $\sqrt{16+9}=5$ , is how steep that best way is: 5 cm per step.

Those two test numbers are partial derivatives. Put all of them in one list and you get the gradient. It is written $\nabla f$ (read it: "grad f"):

\nabla_{\mathbf x} f=\frac{df}{d\mathbf x}=\left[\frac{\partial f}{\partial x_1},\ \frac{\partial f}{\partial x_2},\ \dots,\ \frac{\partial f}{\partial x_n}\right]

The gradient is an arrow with two jobs:

➚Its direction is the steepest way uphill. Not just "some way up": the best way up.
↕Its length is how steep that is. A long arrow means a cliff. A near-zero arrow means flat ground: a valley floor, a hilltop, or a saddle.

Why the steepest? Suppose you step in some direction $\mathbf u$ (an arrow of length 1). Your climb rate is a dot product, which you met in Unit 3:

\text{climb rate}=\nabla f\cdot\mathbf u=\|\nabla f\|\cos\theta

Here $\theta$ (read it: "theta") is the angle between your step and the gradient. In our story: step north, and the rate is $(4,3)\cdot(0,1)=3$ . Step along the gradient ( $\theta=0^\circ$ ): $\cos\theta=1$ , rate $5$ , the best. Step at $90^\circ$ : rate $0$ , you stay level. Step backwards ( $180^\circ$ ): rate $-5$ , the steepest way down.

Try: aim the purple step along the dashed level line: the climb rate is 0. Swing it onto the orange arrow: the rate is the best possible. (On touch: tap to aim.)

✋ drag on the surface to aim your step · grab the base-camp dot to move it · drag the sky to orbit

aim θ149°

—

The curve below is climb rate against aim: $\|\nabla f\|\cos\theta$ . Top on the gradient, zero along the level line, bottom straight opposite.

Pause & predict

Here the gradient has length $\|\nabla f\| = 2$ . You must step at $60^\circ$ away from it. How fast do you climb?

One more fact, easy to see on a trekking map. A contour line joins points of equal height. Walk along one, and your height does not change. So all the change must happen across it. That is why the gradient always crosses contour lines at a right angle.

Try: switch on show contour ⟂ arrow and drag the point around. The little square marks a right angle, everywhere.

drag on the surface to move the point · drag the sky to orbit

—

The arrow always crosses the contour lines at a right angle. On the stretched valley it mostly ignores the long flat direction; later units fix that.

The realization

$\mathbf{w}_{\text{new}}=\mathbf{w}_{\text{old}}-\eta\,\nabla L(\mathbf{w})$

The gradient points steepest uphill, and its length says how steep. So minus the gradient is steepest downhill. This one line, repeated, is how almost every model in machine learning is trained.

If you want the algebra · steepest ascent, the right angle to contours, and

\nabla(\mathbf x^\top A\mathbf x)

Why the gradient is the steepest-ascent direction — derived

Claim. Among all unit directions $\mathbf u$ , the rate of change of $f$ is largest when $\mathbf u=\nabla f/\|\nabla f\|$ , and that largest rate is $\|\nabla f\|$ .

1

Walk from

\mathbf x

in direction

\mathbf u

(with

\|\mathbf u\|=1

) a distance

t

. The rate of change along that walk is the directional derivative:

D_{\mathbf u}f=\lim_{t\to0}\frac{f(\mathbf x+t\mathbf u)-f(\mathbf x)}{t}

Same "nudge, subtract, divide, shrink" as Section 2 — the nudge is now the vector

t\mathbf u

.

2

Use the first-order promise of a differentiable

f

:

f(\mathbf x+\mathbf d)=f(\mathbf x)+\nabla f\cdot\mathbf d+r(\mathbf d)

with

r(\mathbf d)/\|\mathbf d\|\to0

. Put

\mathbf d=t\mathbf u

:

\frac{f(\mathbf x+t\mathbf u)-f(\mathbf x)}{t}=\nabla f\cdot\mathbf u+\frac{r(t\mathbf u)}{t}\ \longrightarrow\ \nabla f\cdot\mathbf u

\|t\mathbf u\|=|t|

, so

r(t\mathbf u)/t\to0

is the promise itself. Hence

D_{\mathbf u}f=\nabla f\cdot\mathbf u

: every direction is priced by one dot product.

3

Bound the dot product with Cauchy–Schwarz (Unit 3):

\nabla f\cdot\mathbf u=\|\nabla f\|\,\|\mathbf u\|\cos\theta=\|\nabla f\|\cos\theta\le\|\nabla f\|

\theta

is the angle between

\mathbf u

and

\nabla f

;

\|\mathbf u\|=1

; and

\cos\theta\le1

. This is the cosine the widget's gauge draws.

4

Equality holds exactly when

\cos\theta=1

, i.e.

\theta=0

, i.e.

\mathbf u

points along

\nabla f

. Then the rate is

\|\nabla f\|

. Steepest ascent, proved. ∎ The mirror statement is free:

\cos\theta=-1

at

\theta=180^\circ

, so

-\nabla f

is the steepest descent direction with rate

-\|\nabla f\|

. That is the minus sign in gradient descent.

Why the gradient is perpendicular to contours — derived

Claim. At any point of a level curve $f(x,y)=c$ , the gradient $\nabla f$ is perpendicular to the curve's tangent.

1

Describe the contour as a path

\mathbf r(t)=(x(t),y(t))

that stays on the level set:

f\big(x(t),y(t)\big)=c\quad\text{for every }t

A contour is by definition a set where

f

does not change; walking along it, the value stays pinned at

c

.

2

Differentiate both sides with respect to

t

, using the chain rule with a vector in the middle (Section 10's row-times-column):

\frac{\partial f}{\partial x}\,x'(t)+\frac{\partial f}{\partial y}\,y'(t)=\frac{d}{dt}c=0

Left side: how

f

changes along the path — each input's rate of change weighted by

f

's sensitivity to it. Right side: the derivative of a constant is zero.

3

Recognise the left side as a dot product:

\nabla f\cdot\mathbf r'(t)=0

\nabla f=[\partial f/\partial x,\ \partial f/\partial y]

and

\mathbf r'(t)=(x'(t),y'(t))

is the tangent vector to the contour.

4

A zero dot product between two nonzero vectors means a right angle (Unit 3). So

\nabla f\perp

tangent. ∎ This is the "level walk" in the widget above: aim along the contour and the climb rate

\nabla f\cdot\mathbf u

reads exactly

0

. The same argument works in any dimension — level surfaces in 3-D, and so on.

Why ∇(xᵀAx) = xᵀ(A + Aᵀ) — derived

Claim. For any square matrix $A$ (symmetric or not), the gradient row of $f(\mathbf x)=\mathbf x^\top A\mathbf x$ is $\mathbf x^\top(A+A^\top)$ . Only when $A=A^\top$ does this collapse to $2\mathbf x^\top A$ .

1

Write the quadratic form as a double sum:

f(\mathbf x)=\mathbf x^\top A\mathbf x=\sum_{i}\sum_{j}A_{ij}\,x_i\,x_j

(A\mathbf x)_i=\sum_j A_{ij}x_j

, then

\mathbf x^\top(A\mathbf x)=\sum_i x_i(A\mathbf x)_i

. Every entry of

A

meets one pair of coordinates.

2

Differentiate with respect to one coordinate

x_k

. The variable

x_k

appears in a term when

i=k

, when

j=k

, or both:

\frac{\partial f}{\partial x_k}=\underbrace{\sum_{j}A_{kj}\,x_j}_{i=k\text{ terms}}+\underbrace{\sum_{i}A_{ik}\,x_i}_{j=k\text{ terms}}

Product rule on each

x_ix_j

: if only

i=k

, the derivative is

A_{kj}x_j

; if only

j=k

, it is

A_{ik}x_i

; the diagonal term

A_{kk}x_k^2

contributes

2A_{kk}x_k

, which is exactly what both sums give it together. Nothing is double-counted.

3

Read each sum as a matrix–vector entry:

\sum_{j}A_{kj}x_j=(A\mathbf x)_k,\qquad \sum_{i}A_{ik}x_i=(A^\top\mathbf x)_k

The first is row

k

of

A

dotted with

\mathbf x

. The second uses column

k

of

A

— which is row

k

of

A^\top

.

4

Assemble all

k

into one row:

\nabla f=\big[(A\mathbf x)+(A^\top\mathbf x)\big]^\top=\mathbf x^\top(A^\top+A)=\mathbf x^\top(A+A^\top)

Symmetric

A

gives

2\mathbf x^\top A

; nothing else does. ∎ Problem 8 in the arena runs this with

A=\begin{bmatrix}1&2\\0&3\end{bmatrix}

: the honest answer at

(1,1)

is

[3,7]

, while the "

2A

" shortcut gives

[1,9]

— a wrong answer you can check by expanding the components.

Pause & predict

At some point, the gradient of a loss is $\nabla L=[0,0]$ . What can you say?

In one sentence: the gradient puts all the partial derivatives in one arrow: it points steepest uphill, its length is the steepness, and minus it is the direction of training.

7

The five shapes a derivative can take

Imagine this

A class teacher fills in a mark sheet. There is one mark for every student in every subject. 30 students, 5 subjects: a table of $30\times5=150$ marks. One row per student, one column per subject.

Nobody memorises the size of that table. You just count.

A derivative is a mark sheet of sensitivities. It keeps one number for every pair (one output, one input). That number answers: "if I nudge this input, how much does this output move?"

So its shape is never a mystery. Count the outputs, count the inputs. Outputs make the rows. Inputs make the columns. Example: 3 outputs and 4 inputs give a $3\times4$ table, as in the picture.

The bookkeeping picture. Outputs index the rows, inputs index the columns, and the cell at (row $i$ , column $j$ ) holds $\partial f_i/\partial x_j$ . A row is one output's gradient; a column is how everything reacts to one input.

Try: pick vector in, vector out: the derivative is a grid with one row per output. Then pick scalar out and it shrinks to a single row.

input

output

—

Input	Output	Derivative is a…	Shape	Where you meet it
scalar	scalar	scalar	$1$	a single learning-rate schedule
vector $\mathbb{R}^n$	scalar	gradient	$n$	loss w.r.t. weights — the big one
matrix $\mathbb{R}^{m\times n}$	scalar	gradient matrix	$m\times n$	loss w.r.t. a CNN kernel
vector $\mathbb{R}^n$	vector $\mathbb{R}^m$	Jacobian	$m\times n$	one layer of a network
matrix	matrix	rank-4 tensor	$p\times q\times m\times n$	lurking inside every autodiff library

The realization

$\text{shape of the derivative}=(\text{number of outputs})\times(\text{number of inputs})$

Two rows of the table do almost all the work in machine learning. The gradient: the loss is one number and the weights are a list, so the derivative is a list shaped like the weights. That is why you can subtract it from them. The Jacobian: a layer turns a list into a list, so its derivative is a matrix.

Pause & predict

A network layer takes 100 numbers in and gives 50 numbers out ( $\mathbb{R}^{100}\to\mathbb{R}^{50}$ ). What shape is its derivative?

In one sentence: count before you compute: one sensitivity per (output, input) pair, with outputs as rows and inputs as columns.

8

The Jacobian — a local rulebook for a curved map

Imagine this

You are playing a car-racing game. The joystick has two directions. The car's speed has two parts: forward and sideways.

Right now, a tiny push forward on the stick changes the car's speed by "+5 forward, +1 sideways". That little rulebook works only at this moment. On an icy bend a second later, the same push does something else.

Now both ends are lists. We write $\mathbf{f}:\mathbb{R}^n\to\mathbb{R}^m$ (read it: "n numbers in, m numbers out"). A neural-network layer looks exactly like this.

Each output has its own gradient, one row of slopes. Stack the rows and you get the Jacobian, $\mathbf J$ :

\mathbf{J}=\frac{d\mathbf{f}(\mathbf{x})}{d\mathbf{x}}=\begin{bmatrix}\frac{\partial f_1}{\partial x_1}&\cdots&\frac{\partial f_1}{\partial x_n}\\ \vdots& &\vdots\\ \frac{\partial f_m}{\partial x_1}&\cdots&\frac{\partial f_m}{\partial x_n}\end{bmatrix}\in\mathbb{R}^{m\times n}

The entry in row $i$ , column $j$ says how much output $i$ moves when input $j$ is nudged. Read a row and you see how one output feels every input. Read a column and you see how all outputs react to one input.

A tiny example: $\mathbf f(x,y)=(x^2,\ xy)$ at $(1,2)$ . The rows are $[2x,\ 0]$ and $[y,\ x]$ , so $\mathbf J=\begin{bmatrix}2&0\\2&1\end{bmatrix}$ . Nudge $x$ by $0.01$ , and $\mathbf J$ predicts both outputs grow by about $0.02$ . Check: $\mathbf f(1.01,2)=(1.0201,\ 2.02)$ . It matches.

Here is the big idea. Zoom in on a curved map and it starts to look like a plain matrix map: straight grid lines stay straight. That matrix is $\mathbf J$ . It is the game's rulebook for this spot.

Try: press ▶ zoom until linear on the squaring map. Then pick the exponential map and press jump y by 2π: the input point moves, the output patch stays put.

x₀0.6

y₀0.5

window0.8

—

Shrink the window and the bent patch becomes the dashed parallelogram that $\mathbf J$ predicts. Looking straight up close is exactly what "differentiable" means.

The realization

$J_{ij}=\frac{\partial f_i}{\partial x_j}$

Row $i$ , column $j$ : how much output $i$ moves when input $j$ is nudged. Zoom in on any smooth map and it becomes this matrix.

Worked example · a matrix map is its own Jacobian

Take $\mathbf f(\mathbf x)=A\mathbf x$ with $A\in\mathbb{R}^{M\times N}$ . Component $i$ is the plain sum $f_i(\mathbf x)=\sum_{j=1}^{N}A_{ij}x_j$ . Differentiate it with respect to one particular $x_j$ : every term of that sum is a constant except the $j$ -th, which contributes its coefficient. So

\frac{\partial f_i}{\partial x_j}=A_{ij}\quad\Longrightarrow\quad \frac{d\mathbf f}{d\mathbf x}=\begin{bmatrix}A_{11}&\cdots&A_{1N}\\ \vdots& &\vdots\\ A_{M1}&\cdots&A_{MN}\end{bmatrix}=A

The Jacobian of a linear map is that map's own matrix — the exact analogue of $\frac{d}{dx}(ax)=a$ . It also explains why the "linear layer" of a network is the easy case in backpropagation: its local rulebook never changes, no matter where you evaluate it.

Pause & predict

For $\mathbf f(x,y)=\big(x^2-y^2,\ xe^y,\ \ln(1+x^2+y^2)\big)$ , what is $\mathbf J$ at $(1,0)$ ?

In one sentence: the Jacobian is the matrix a curved map turns into when you zoom in close.

9

The Jacobian determinant — area, invertibility, one warning

Imagine this

You print a photo on a rubber sheet, then pull the sheet. Some parts grow, some shrink. Before pulling, you draw a tiny 1 cm square on the sheet. After pulling, it has become a tiny slanted box.

How many times bigger is its area now? That single number is the Jacobian determinant, $\det\mathbf J$ .

In Unit 4, the determinant of a matrix was its area dial. Here it does the same job, but only near one point. At each point, $\det\mathbf J$ is the local area factor. Three cases:

$\det\mathbf J$	What happens to a tiny square
bigger than 0	Its area is multiplied by $\det\mathbf J$ .
smaller than 0	Its area is multiplied by $\|\det\mathbf J\|$ , and it is flipped, like a mirror image.
exactly 0	It is squashed flat onto a line. Information is lost. You cannot undo it.

Example: for $\mathbf J=\begin{bmatrix}2&1\\0.5&1.5\end{bmatrix}$ , $\det\mathbf J=2\times1.5-1\times0.5=2.5$ . A tiny square comes out $2.5$ times bigger.

Left: the unit square goes to the parallelogram spanned by $\mathbf J$ 's columns, of area $|\det\mathbf J|$ (here $2\cdot1.5-1\cdot0.5=2.5$ ); a negative determinant keeps the area but mirrors the picture. Right: the exponential map sends the strip $0\le y\le4\pi$ twice around the same ring — lap two lands exactly on lap one.

This gives a real theorem, the inverse function theorem. Wherever $\det\mathbf J\neq 0$ , you can undo the map near that point: nearby inputs give different outputs, so you can run the map backwards.

Now the warning. "Undoable near every point" does not mean "undoable everywhere at once".

Watch out

Think of a clock. 3 o'clock and 15 o'clock land on the same spot of the dial. Every small turn of the hand is easy to undo. Yet two different times share one position.

The exponential map $T(x,y)=\big(e^x\cos y,\ e^x\sin y\big)$ is that clock. Its $\det\mathbf J=e^{2x}$ is positive for every $x$ , so it is undoable near every point. But $\cos$ and $\sin$ repeat every $2\pi$ , so $T(x,y)=T(x,y+2\pi)$ . Many different inputs land on the same output. The picture above shows the plane wrapping round and round.

See it in the widget above: pick the exponential map and press jump y by 2π. The input point moves. The output patch does not move at all.

The realization

$\text{area after}\approx|\det\mathbf J|\times\text{area before}$

$\det\mathbf J$ is the local area factor. Not zero means you can undo the map near that point. But being undoable near every point still allows the map to wrap around and repeat.

If you want the algebra · why

\det\mathbf J

is the area factor

Why det J is the area factor — derived

Claim. Near a point, a differentiable map $\mathbf f:\mathbb R^2\to\mathbb R^2$ sends a tiny square of area $\varepsilon^2$ to a region of area $\approx|\det\mathbf J|\,\varepsilon^2$ , with a sign flip of orientation exactly when $\det\mathbf J<0$ .

1

Take the square with corner

\mathbf x_0

and sides

\varepsilon\mathbf e_1,\ \varepsilon\mathbf e_2

. Use the first-order promise on each side:

\mathbf f(\mathbf x_0+\varepsilon\mathbf e_1)-\mathbf f(\mathbf x_0)=\varepsilon\,\mathbf J\mathbf e_1+\mathbf r_1,\qquad \mathbf f(\mathbf x_0+\varepsilon\mathbf e_2)-\mathbf f(\mathbf x_0)=\varepsilon\,\mathbf J\mathbf e_2+\mathbf r_2

"Differentiable" means

\mathbf f(\mathbf x_0+\mathbf d)=\mathbf f(\mathbf x_0)+\mathbf J\mathbf d+\mathbf r(\mathbf d)

with

\|\mathbf r\|/\|\mathbf d\|\to0

. The leftovers

\mathbf r_1,\mathbf r_2

are small even compared with

\varepsilon

.

2

Read off

\mathbf J\mathbf e_1

and

\mathbf J\mathbf e_2

: they are the columns of

\mathbf J

. So, ignoring the leftovers, the image is the parallelogram spanned by

\varepsilon\mathbf c_1

and

\varepsilon\mathbf c_2

where

\mathbf c_1,\mathbf c_2

are those columns. Multiplying a matrix by

\mathbf e_1=(1,0)

picks out its first column; likewise

\mathbf e_2

picks the second.

3

Unit 4's fact: the parallelogram spanned by two vectors

\mathbf a=(a_1,a_2)

and

\mathbf b=(b_1,b_2)

has signed area

a_1b_2-a_2b_1=\det[\mathbf a\ \mathbf b]

. Apply it:

\text{signed area}=\det[\varepsilon\mathbf c_1\ \ \varepsilon\mathbf c_2]=\varepsilon^2\det[\mathbf c_1\ \mathbf c_2]=\varepsilon^2\det\mathbf J

Scaling each column by

\varepsilon

scales the determinant by

\varepsilon

twice. The sign is positive when

\mathbf c_1\to\mathbf c_2

turns counter-clockwise (like

\mathbf e_1\to\mathbf e_2

) and negative when the turn has been mirrored.

4

The leftovers change the area by an amount that is small compared with

\varepsilon^2

, so the ratio

\text{area out}/\text{area in}\to|\det\mathbf J|

as

\varepsilon\to0

. Area factor, proved. ∎ Each side is off by something

\ll\varepsilon

, so the area is off by something

\ll\varepsilon^2

. For the exponential map the columns are

e^x(\cos y,\sin y)

and

e^x(-\sin y,\cos y)

— perpendicular, each of length

e^x

— so the area factor is

e^{2x}

, matching Problem 7.

Pause & predict

At some point, a map $\mathbb R^2\to\mathbb R^2$ has $\det\mathbf J=-3$ . What happens to a tiny square there?

Pause & predict

A map $\mathbb{R}^2\to\mathbb{R}^2$ has $\det\mathbf J = 0$ along a whole curve. What happens there?

In one sentence: $\det\mathbf J$ is the local area factor: non-zero means you can undo the map nearby, but not always everywhere at once.

10

The chain rule becomes matrix multiplication — and that is backpropagation

Imagine this

A bakery makes cakes in three stations: mix, bake, ice. The taster says: "too sweet, by 2 points".

Who should change what? The icing station works out its share of the blame and passes the rest back to the baking station. The baker does the same and passes the rest back to the mixer. Blame flows backwards, one station at a time. No station needs to understand the whole bakery.

That is backpropagation. The "blame" is the gradient.

In Section 3 the chain rule multiplied two numbers. When the inputs and outputs are lists, each machine's rate is a matrix, its Jacobian. So the chain rule multiplies matrices:

\mathbf{x}\ \xrightarrow{\ \mathbf F\ }\ \mathbf{y}\ \xrightarrow{\ \mathbf G\ }\ \mathbf{z}\qquad\Longrightarrow\qquad \frac{d\mathbf z}{d\mathbf x}=\mathbf J_{\mathbf G}\,\mathbf J_{\mathbf F}

Read it: "the later machine's Jacobian on the left, times the earlier one's". Order matters, because matrix products do.

The shapes check themselves. Say $\mathbf F$ takes 4 numbers to 3, and $\mathbf G$ takes 3 to 2. Then $\mathbf J_{\mathbf G}$ is $2\times3$ , $\mathbf J_{\mathbf F}$ is $3\times4$ , and the product is $2\times4$ : 4 in, 2 out. If the inner sizes match, you almost surely have the order right.

Now the clever part. A network always ends in one number, the loss. So the first factor at the loss end is a thin row. And a row times a matrix is again a row. Start multiplying at the loss end, and you only ever carry one thin row backwards. That is the whole trick of backpropagation.

So what: going backwards costs about as much as going forwards. One training step costs roughly two forward passes, however many weights there are. Multiply in the other order and you build a full matrix at every layer.

The same three Jacobians multiplied in two orders. Starting from the loss keeps every intermediate a single row (top). Starting from the input builds a full matrix at every stage (bottom) — the widget below only ever does the top row.

Try: press ▶ advance one stage four times. Before each press, guess what fills in next. On the way back, watch the orange row: it stays two numbers wide the whole time.

drag the picture to orbit

x₁1

x₂1

—

The two gates are the Jacobians: $W$ and $\operatorname{diag}(1-a^2)$ . Each backward step is (row) × (gate) = a new row.

The realization

$\frac{\partial L}{\partial \mathbf x}=\underbrace{\frac{\partial L}{\partial \mathbf a}}_{\text{row}}\;\mathbf J_{\tanh}\;\mathbf J_{W}$

The chain rule for lists is matrix multiplication, later layer on the left. Backpropagation just multiplies from the loss end, so it only ever carries a thin row.

Worked example · a chain with a list in the middle

Let $f(\mathbf x)=\exp(x_1x_2^2)$ with $\mathbf x=g(t)=\big(t\cos t,\ t\sin t\big)$ . Here $g:\mathbb{R}\to\mathbb{R}^2$ and $f:\mathbb{R}^2\to\mathbb{R}$ , so the shapes are $\frac{\partial f}{\partial\mathbf x}\in\mathbb{R}^{1\times2}$ and $\frac{\partial\mathbf x}{\partial t}\in\mathbb{R}^{2\times1}$ — their product is $1\times1$ , a single number, as it must be.

\frac{\partial f}{\partial \mathbf x}=\Big[\exp(x_1x_2^2)\,x_2^2,\quad 2\exp(x_1x_2^2)\,x_1x_2\Big],\qquad \frac{\partial \mathbf x}{\partial t}=\begin{bmatrix}\cos t-t\sin t\\ \sin t+t\cos t\end{bmatrix}

Multiply the row by the column:

\frac{dh}{dt}=\exp(x_1x_2^2)\Big(x_2^2(\cos t-t\sin t)+2x_1x_2(\sin t+t\cos t)\Big),\quad x_1=t\cos t,\ x_2=t\sin t

A simpler version of the same pattern appears earlier in the lecture: for $f(x_1,x_2)=x_1^2+2x_2$ with $x_1=\sin t$ , $x_2=\cos t$ ,

\frac{df}{dt}=\frac{\partial f}{\partial x_1}\frac{\partial x_1}{\partial t}+\frac{\partial f}{\partial x_2}\frac{\partial x_2}{\partial t}=2\sin t\cos t-2\sin t=2\sin t(\cos t-1)

That sum of products is the row-times-column multiplication written out longhand. Every "sum over paths" you meet in calculus is a matrix product waiting to be recognised.

If you want the algebra · why the chain rule for lists is a matrix product

Why the vector chain rule is a matrix product — derived entry by entry

Claim. If $\mathbf y=\mathbf F(\mathbf x)$ and $\mathbf z=\mathbf G(\mathbf y)$ , then the Jacobian of the composite has entries $\big(\mathbf J_{\mathbf z\leftarrow\mathbf x}\big)_{ij}=\sum_k (\mathbf J_{\mathbf G})_{ik}(\mathbf J_{\mathbf F})_{kj}$ — which is precisely the $(i,j)$ entry of $\mathbf J_{\mathbf G}\mathbf J_{\mathbf F}$ .

1

Fix one input coordinate

x_j

and one output coordinate

z_i

. Output

z_i

depends on

x_j

only through the intermediates

y_1,\dots,y_p

:

z_i=G_i\big(y_1(\mathbf x),\dots,y_p(\mathbf x)\big)

Every route from

x_j

to

z_i

passes through some

y_k

— there is no direct wire.

2

Nudge

x_j

by

h

, holding the other inputs fixed. Each intermediate moves by its own partial (first-order promise of

\mathbf F

):

\Delta y_k=\frac{\partial y_k}{\partial x_j}\,h+(\text{small})\quad\text{for each }k

This is the 1-D chain-rule step 1 from Section 3, done once per intermediate.

3

Feed all

p

nudges into

G_i

at once. The many-input promise of

G_i

(Section 6, step 2 of the steepest-ascent derivation) adds the effects:

\Delta z_i=\sum_{k=1}^{p}\frac{\partial z_i}{\partial y_k}\,\Delta y_k+(\text{small})=\sum_{k}\frac{\partial z_i}{\partial y_k}\frac{\partial y_k}{\partial x_j}\,h+(\text{small})

A differentiable function of several inputs responds to simultaneous nudges by summing "sensitivity × nudge" over the inputs; then substitute step 2 for each

\Delta y_k

.

4

Divide by

h

, let

h\to0

:

\frac{\partial z_i}{\partial x_j}=\sum_{k=1}^{p}\frac{\partial z_i}{\partial y_k}\,\frac{\partial y_k}{\partial x_j}=\sum_k(\mathbf J_{\mathbf G})_{ik}\,(\mathbf J_{\mathbf F})_{kj}

Row

i

of

\mathbf J_{\mathbf G}

dotted with column

j

of

\mathbf J_{\mathbf F}

: a matrix product. ∎ "Sum over all routes

x_j\to y_k\to z_i

, multiplying along each route" is the definition of matrix multiplication. That is why the order is

\mathbf J_{\mathbf G}\mathbf J_{\mathbf F}

, later layer on the left, and why the shapes

(m\times p)(p\times n)

line up automatically.

Pause & predict

Layer $\mathbf F$ takes 4 numbers to 3 ( $\mathbb{R}^{4}\to\mathbb{R}^{3}$ ). Then $\mathbf G$ takes 3 to 2. Which product is the Jacobian of the whole chain?

In one sentence: backpropagation is the chain rule as matrix multiplication, done from the loss end backwards so that nothing bigger than a row is ever built.

11

Walking down the valley

Imagine this

You are setting the temperature of a hotel shower. It's too cold, so you give the knob a big turn. Now it's scalding. You swing it back hard. Freezing again. Each swing overshoots more.

Small turns are slower, but they get you there. How much you turn per try matters as much as which way you turn.

You now have everything for the walk down. Measure the gradient, step against it, repeat:

\mathbf{w}\ \leftarrow\ \mathbf{w}-\eta\,\nabla L(\mathbf{w})

The gradient picks the direction. The learning rate $\eta$ (read it: "eta") picks the step size. That is the shower knob.

Why can a big step fail? The gradient only describes the ground right where you measured it. Take a small step and that holds. Take a huge step and you jump past the valley floor, landing higher on the far side. The gradient there is steeper, so the next jump is even bigger.

A tiny example: $L(w)=w^2$ , whose slope is $2w$ . One step gives $w-\eta\cdot 2w=(1-2\eta)\,w$ . With $\eta=0.1$ , each step multiplies $w$ by $0.8$ : it shrinks towards $0$ . With $\eta=1.05$ , each step multiplies $w$ by $-1.1$ : it flips side and grows. The walk explodes.

Try: press the four η buttons. 0.05 crawls, 0.5 is quick, 0.95 zig-zags but gets there, 1.05 explodes. The small chart tracks the loss step by step.

drag on the surface to move the start · drag the sky to orbit

rate η0.18

steps18

loss after each step (log scale)

—

The valley is 8 times steeper one way than the other. Even a good run zig-zags across the narrow way; later units fix that.

The realization

$\mathbf w\leftarrow\mathbf w-\eta\,\nabla L(\mathbf w)$

The gradient says which way. The learning rate $\eta$ says how far to trust it. Too small and you crawl. Too big and you overshoot, again and again.

Pause & predict

Your training loss swings up and down and keeps growing. Which knob do you reach for first?

In one sentence: gradient descent is $\mathbf w \leftarrow \mathbf w-\eta\nabla L$ : the gradient picks the direction, and $\eta$ decides how far to trust it.

12

Fine print worth knowing

Imagine this

Two recipes for the same cake. One measures in cups, the other in grams. Both are correct. But mix the two in one cake and it's ruined.

Maths meets code in the same way. The four notes below are small, but each one has caused real bugs.

1 · Row or column? This unit writes a gradient as a row, like the lecture. Then the chain rule reads left to right, with no flipping. Most code writes it as a column, shaped like the weights. Both are fine. Mixing them is not. If a formula looks flipped (transposed), check the habit before the maths.

Left: the two conventions carry the same numbers; the chain rule reads $\mathbf J_{\mathbf G}\mathbf J_{\mathbf F}$ in rows and $\mathbf J_{\mathbf F}^\top\mathbf J_{\mathbf G}^\top$ in columns. Right: at ReLU's kink no single tangent exists, but every slope between the two one-sided slopes $0$ and $1$ is a valid subgradient — the fan of dashed lines. Frameworks pick one.

2 · Order matters now. With plain numbers, $3\times5=5\times3$ . With matrices, the order changes the answer. Keep the factors in the order the chain rule gives them.

3 · Some functions have corners. $|x|$ has no slope at $0$ . From the left the slope is $-1$ . From the right it is $+1$ . No single number fits both. ReLU, the most common switch in neural networks, has the same corner. Software simply picks a value there and moves on. The widget below shows the two sides disagreeing.

4 · The table nobody builds. Matrix in, matrix out gives a four-way table of slopes. For a $100\times100$ layer that is $10^8$ numbers. PyTorch and TensorFlow never store it. Each step only passes the incoming gradient along to the next one, as the backprop widget did.

Try: pick √(x² + 0.01) and press ▶ squeeze h → 0: both side lines swing round to slope 0. Now try |x|: they never agree.

gap h1.2

—

A derivative exists only when the left and right slopes settle on the same number. On $|x|$ and ReLU they settle on two different numbers, so there is no slope at 0.

The realization

$\text{slope from the left}=\text{slope from the right}\ \Longleftrightarrow\ \text{a derivative exists}$

At a corner the two sides disagree, so there is no derivative. Software just picks a value in between. Rows versus columns is only a habit, but the order of matrix factors is a rule.

Pause & predict

PyTorch says the slope of ReLU at 0 is 0. A friend worries this "wrong" slope will break training. What do you tell them?

In one sentence: rows versus columns is a habit, the order of matrix factors is a rule, a corner has no derivative, and the giant four-way table is used but never stored.

13

What to carry forward

One idea built this whole unit: nudge the input a little, see how much the output moves, and take the ratio as the nudge shrinks to nothing. Everything else was the same idea for bigger shapes.

Idea	In one line	Where it comes back
Derivative	Rise ÷ run as the gap shrinks to zero; its sign says which way is up	everywhere below
Chain rule	Machines in a row: their rates multiply	Unit 7 · backpropagation
Taylor	Copy the value, slope, bend… at one point; $k!$ keeps the books	Unit 8 · Taylor & Maclaurin
Partial derivative	Freeze every other input, then use the usual rules	every gradient you ever compute
Gradient	All partials in one arrow: direction = steepest up, length = steepness	Unit 9 · gradient descent
Jacobian	One row per output; the matrix a curved map becomes up close	Unit 7 · layers; later, PCA
$\det \mathbf J$	Local area factor; non-zero means undoable nearby	change of variables, normalising flows
Chain rule for lists	Matrix multiplication, later layer on the left	Unit 7 · the training loop

The realization

If you keep one habit from this unit, make it counting shapes before computing anything. Outputs give rows, inputs give columns. A gradient has the same shape as the thing you differentiate by. In a product, the inner sizes must match. Most matrix-calculus mistakes are shape mistakes, and counting catches them before any algebra.

In one sentence: nudge, measure, divide, shrink: that one move, kept tidy with shapes, is all of differentiation for machine learning.

14

Practice arena — the unit's problem set, solved in full

Here are eight problems from Prof. Saurabh's practice set, plus two extra ones on Taylor polynomials and the chain rule for lists. Every solution was checked by computer algebra.

One rule does most of the work: to find $\partial f/\partial x$ , treat every other variable as a fixed number. Try each problem on paper first. Then open the solution to see every step.

Problem 1easy

Let $f(x,y)=x^2y+3xy^2-y^3$ . (a) Compute $\partial f/\partial x$ and $\partial f/\partial y$ and write $\nabla f$ as a row vector. (b) Evaluate $\nabla f$ at $(2,-1)$ . (c) Verify $\partial f/\partial x$ at $(2,-1)$ directly from the limit definition.

What this tests. The freeze-the-others rule, and the fact that the rules you use are not a separate system from the limit definition — they are shortcuts for it.

Show the full solution

(a) Step 1 — $\partial f/\partial x$ , holding $y$ fixed. Take the three terms one at a time, treating

y

as a number:

\frac{\partial}{\partial x}(x^2y)=y\cdot 2x=2xy,\qquad \frac{\partial}{\partial x}(3xy^2)=3y^2,\qquad \frac{\partial}{\partial x}(-y^3)=0

The last term dies completely because it contains no

x

. Adding:

\ \partial f/\partial x = 2xy+3y^2

.

Step 2 — $\partial f/\partial y$ , holding $x$ fixed.

\frac{\partial}{\partial y}(x^2y)=x^2,\qquad \frac{\partial}{\partial y}(3xy^2)=3x\cdot 2y=6xy,\qquad \frac{\partial}{\partial y}(-y^3)=-3y^2

Adding:

\ \partial f/\partial y=x^2+6xy-3y^2

.

Step 3 — assemble.

\nabla f(x,y)=\big[\,2xy+3y^2,\ \ x^2+6xy-3y^2\,\big]

.

(b) Substitute $x=2,\ y=-1$ (note

y^2=1

,

y^3=-1

):

\frac{\partial f}{\partial x}\Big|_{(2,-1)}=2(2)(-1)+3(1)=-4+3=-1

\frac{\partial f}{\partial y}\Big|_{(2,-1)}=(2)^2+6(2)(-1)-3(1)=4-12-3=-11

(c) Step 1 — collapse to one variable. Fix

y=-1

:

f(x,-1)=x^2(-1)+3x(1)-(-1)=-x^2+3x+1

.

Step 2 — form $f(2+h,-1)$ .

-(2+h)^2+3(2+h)+1=-(4+4h+h^2)+6+3h+1=3-h-h^2

and

f(2,-1)=-4+6+1=3

.

Step 3 — difference quotient and limit.

\frac{(3-h-h^2)-3}{h}=\frac{-h-h^2}{h}=-1-h\ \xrightarrow[h\to0]{}\ -1

matching part (b) exactly.

$\nabla f=[\,2xy+3y^2,\ x^2+6xy-3y^2\,]$ , and $\nabla f(2,-1)=[-1,\,-11]$ .

Remember

Notice the cancellation in step 3: the $h$ in the denominator is consumed before the limit is taken. That is the whole reason the definition never divides by zero.

Problem 2medium

Let $f(x,y)=\dfrac{x^2y}{x+2y}$ with $x+2y\neq0$ . Without using the quotient rule — rewrite it as $f=x^2y(x+2y)^{-1}$ and use the product rule together with $\frac{d}{dt}t^{-1}=-t^{-2}$ . (a) Compute $\partial f/\partial x$ . (b) Compute $\partial f/\partial y$ . (c) Write $\nabla f$ and evaluate at $(1,1)$ .

What this tests. Product plus chain rule in combination, and the discipline of putting a fractional answer over a common denominator so it can be checked.

Show the full solution

(a) Step 1 — name the factors and their $x$ -derivatives. With

y

held fixed, write

f=g\cdot w

where

g=x^2y\ \Rightarrow\ g_x=2xy,\qquad w=(x+2y)^{-1}\ \Rightarrow\ w_x=-(x+2y)^{-2}\cdot\underbrace{1}_{\partial(x+2y)/\partial x}

Step 2 — product rule $(gw)_x=g_xw+gw_x$ .

\frac{\partial f}{\partial x}=\frac{2xy}{x+2y}-\frac{x^2y}{(x+2y)^2}

Step 3 — common denominator $(x+2y)^2$ .

\frac{\partial f}{\partial x}=\frac{2xy(x+2y)-x^2y}{(x+2y)^2}=\frac{2x^2y+4xy^2-x^2y}{(x+2y)^2}=\frac{x^2y+4xy^2}{(x+2y)^2}=\frac{xy(x+4y)}{(x+2y)^2}

(b) Step 1 — now hold $x$ fixed.

g_y=x^2

, and

w_y=-(x+2y)^{-2}\cdot 2=-2(x+2y)^{-2}

(the inner derivative is now

2

, not

1

— this is where the two parts differ).

Step 2 — product rule, then combine.

\frac{\partial f}{\partial y}=\frac{x^2}{x+2y}-\frac{2x^2y}{(x+2y)^2}=\frac{x^2(x+2y)-2x^2y}{(x+2y)^2}=\frac{x^3+2x^2y-2x^2y}{(x+2y)^2}=\frac{x^3}{(x+2y)^2}

The

2x^2y

terms cancel exactly — a good sign the algebra is right.

(c) Evaluate at $(1,1)$ , where

x+2y=3

so

(x+2y)^2=9

:

\nabla f=\left[\frac{xy(x+4y)}{(x+2y)^2},\ \frac{x^3}{(x+2y)^2}\right],\qquad \nabla f(1,1)=\left[\frac{1\cdot1\cdot5}{9},\ \frac{1}{9}\right]=\left[\tfrac59,\ \tfrac19\right]

Independent check of $\partial f/\partial y(1,1)$ from the definition. Fix

x=1

:

f(1,y)=\dfrac{y}{1+2y}

, so

f(1,1)=\tfrac13

and

\frac{1}{h}\left(\frac{1+h}{3+2h}-\frac13\right)=\frac{3(1+h)-(3+2h)}{3h(3+2h)}=\frac{h}{3h(3+2h)}=\frac{1}{3(3+2h)}\ \xrightarrow[h\to0]{}\ \frac19\ \checkmark

$\nabla f=\left[\dfrac{xy(x+4y)}{(x+2y)^2},\ \dfrac{x^3}{(x+2y)^2}\right]$ , and $\nabla f(1,1)=\left[\tfrac59,\ \tfrac19\right]$ .

Watch out

The two parts share the same product-rule skeleton and differ only in one number — the inner derivative of $x+2y$ , which is $1$ for $x$ and $2$ for $y$ . Dropping that $2$ is the most common slip here.

Problem 3easy

Let $f(x,y,z)=x^2e^y+y\cos z$ . (a) Compute the three partials. (b) Write $\nabla f$ and evaluate at $(1,0,0)$ .

What this tests. Three variables instead of two, with exponential and trigonometric pieces — and the habit of noticing which whole terms vanish.

Show the full solution

Step 1 — $\partial f/\partial x$ . Here

e^y

is a constant multiplier and

y\cos z

contains no

x

at all, so it dies:

\frac{\partial f}{\partial x}=e^y\cdot 2x+0=2xe^y

Step 2 — $\partial f/\partial y$ . Now

x^2

and

\cos z

are constants; use

\frac{d}{dy}e^y=e^y

on the first term and

\frac{d}{dy}y=1

on the second:

\frac{\partial f}{\partial y}=x^2e^y+\cos z

Step 3 — $\partial f/\partial z$ . The first term has no

z

; in the second,

y

is a constant multiplier on

\cos z

:

\frac{\partial f}{\partial z}=0+y(-\sin z)=-y\sin z

Evaluate at $(1,0,0)$ , using

e^0=1

,

\cos 0=1

,

\sin 0=0

:

\nabla f=\big[2xe^y,\ x^2e^y+\cos z,\ -y\sin z\big],\quad \nabla f(1,0,0)=\big[2,\ 1+1,\ 0\big]=[\,2,\ 2,\ 0\,]

$\nabla f=[\,2xe^y,\ x^2e^y+\cos z,\ -y\sin z\,]$ , and $\nabla f(1,0,0)=[\,2,\,2,\,0\,]$ .

Aha

The third component is zero not because $z$ is irrelevant to $f$ , but because $y=0$ at this particular point switches off the only term containing $z$ . Move to $y\neq0$ and $z$ matters again — gradients are local statements.

Problem 4medium

(a) For $f(x,y,z)=\ln(x^2+y^2+z^2)$ , compute $\nabla f$ and evaluate at $(1,2,2)$ . (b) For $g(\mathbf x)=\lVert\mathbf x\rVert=\sqrt{x^2+y^2+z^2}$ , compute $\nabla g$ and evaluate at $(1,2,2)$ . (c) What is special about the direction and length of $\nabla g$ at any nonzero point?

What this tests. The chain rule on radially symmetric functions, and a geometric fact used constantly in ML: the gradient of the norm is the unit radial direction.

Show the full solution

(a) Step 1 — set the inner function. Let

u=x^2+y^2+z^2

, so

f=\ln u

and

\frac{\partial}{\partial x}\ln u=\frac1u\frac{\partial u}{\partial x}

. The inner partials are

u_x=2x,\ u_y=2y,\ u_z=2z

.

Step 2 — apply the chain rule to each variable.

\frac{\partial f}{\partial x}=\frac{2x}{x^2+y^2+z^2},\qquad \frac{\partial f}{\partial y}=\frac{2y}{u},\qquad \frac{\partial f}{\partial z}=\frac{2z}{u}

Step 3 — assemble and evaluate.

\nabla f=\dfrac{2}{x^2+y^2+z^2}[x,y,z]

. At

(1,2,2)

,

u=1+4+4=9

, so

\nabla f(1,2,2)=\tfrac29[1,2,2]=\left[\tfrac29,\ \tfrac49,\ \tfrac49\right]

(b) Step 1 — same inner function, outer is a square root.

g=u^{1/2}

, and

\frac{\partial}{\partial x}u^{1/2}=\tfrac12u^{-1/2}\cdot 2x=\dfrac{x}{\sqrt u}

, similarly for

y

and

z

.

Step 2 — assemble and evaluate.

\nabla g=\dfrac{1}{\sqrt u}[x,y,z]=\dfrac{\mathbf x^\top}{\lVert\mathbf x\rVert}

. At

(1,2,2)

,

\sqrt u=3

:

\nabla g(1,2,2)=\tfrac13[1,2,2]=\left[\tfrac13,\ \tfrac23,\ \tfrac23\right]

(c)

\nabla g=\mathbf x/\lVert\mathbf x\rVert

is the unit vector pointing radially outward. Its length is

1

everywhere:

\sqrt{\tfrac19+\tfrac49+\tfrac49}=\sqrt{\tfrac99}=1

So distance-from-the-origin grows fastest straight away from the origin, at unit rate — exactly what you would expect, now derived rather than assumed.

$\nabla f(1,2,2)=\tfrac29[1,2,2]$ ; $\ \nabla g=\mathbf x^\top/\lVert\mathbf x\rVert$ with $\nabla g(1,2,2)=\tfrac13[1,2,2]$ , a unit vector.

Aha

Both gradients point along $[1,2,2]$ — the same direction, different lengths. Any radially symmetric function has a gradient along the radius; only the magnitude carries information about which function it was.

Problem 5easy

Let $f(x,y)=\big(2x^2-y\big)^3$ . (a) Compute $\partial f/\partial x$ and $\partial f/\partial y$ . (b) Write $\nabla f$ and evaluate at $(1,1)$ .

What this tests. The chain rule in its purest multivariable form: one outer power, two different inner derivatives.

Show the full solution

Step 1 — spot the structure. The function is

(\text{inner})^3

with

\text{inner}=2x^2-y

, so

\frac{\partial}{\partial\bullet}(\text{inner})^3=3(\text{inner})^2\cdot\frac{\partial(\text{inner})}{\partial\bullet}

.

Step 2 — $\partial f/\partial x$ . Hold

y

fixed; the inner derivative is

4x

:

\frac{\partial f}{\partial x}=3(2x^2-y)^2\cdot 4x=12x(2x^2-y)^2

Step 3 — $\partial f/\partial y$ . Hold

x

fixed; the inner derivative is

-1

:

\frac{\partial f}{\partial y}=3(2x^2-y)^2\cdot(-1)=-3(2x^2-y)^2

(b) Evaluate at $(1,1)$ , where the inner is

2(1)^2-1=1

so

(2x^2-y)^2=1

:

\nabla f(1,1)=\big[12(1)(1),\ -3(1)\big]=[\,12,\ -3\,]

$\nabla f=\big[12x(2x^2-y)^2,\ -3(2x^2-y)^2\big]$ , and $\nabla f(1,1)=[\,12,\,-3\,]$ .

Aha

Factor out the shared piece and the structure appears: $\nabla f=3(2x^2-y)^2\,[\,4x,\,-1\,]=3(\text{inner})^2\,\nabla(\text{inner})$ . The chain rule in vector form — outer derivative times the gradient of the inner. No cube ever has to be expanded.

Problem 6medium

Consider $\mathbf f:\mathbb{R}^2\to\mathbb{R}^3$ , $\ \mathbf f(x,y)=\big(x^2-y^2,\ \ xe^y,\ \ \ln(1+x^2+y^2)\big)$ . (a) Compute the Jacobian and state its dimensions. (b) Evaluate at $(1,0)$ . (c) Say in words what the rows and columns represent.

What this tests. Assembling a non-square Jacobian, and reading it in both directions — per output and per input.

Show the full solution

Step 1 — fix the shape before differentiating. Three outputs, two inputs, and

J_{ij}=\partial f_i/\partial x_j

, so the answer is

3\times2

. Six partials to compute.

Step 2 — row 1: $f_1=x^2-y^2$ .

\ \partial f_1/\partial x=2x

,

\ \partial f_1/\partial y=-2y

.

Step 3 — row 2: $f_2=xe^y$ . Holding

y

fixed,

e^y

is a constant multiplier so

\partial f_2/\partial x=e^y

. Holding

x

fixed,

\frac{d}{dy}e^y=e^y

so

\partial f_2/\partial y=xe^y

.

Step 4 — row 3: $f_3=\ln(1+x^2+y^2)$ . With

u=1+x^2+y^2

and

\frac{\partial}{\partial x}\ln u=\frac1u\frac{\partial u}{\partial x}

:

\frac{\partial f_3}{\partial x}=\frac{2x}{1+x^2+y^2},\qquad \frac{\partial f_3}{\partial y}=\frac{2y}{1+x^2+y^2}

Step 5 — assemble.

\mathbf J_{\mathbf f}(x,y)=\begin{bmatrix}2x&-2y\\ e^y&xe^y\\ \dfrac{2x}{1+x^2+y^2}&\dfrac{2y}{1+x^2+y^2}\end{bmatrix}\in\mathbb{R}^{3\times2}

(b) Substitute $x=1,\ y=0$ , so

e^0=1

and

1+x^2+y^2=2

:

\mathbf J_{\mathbf f}(1,0)=\begin{bmatrix}2&0\\ 1&1\\ 2/2&0/2\end{bmatrix}=\begin{bmatrix}2&0\\1&1\\1&0\end{bmatrix}

(c) Row $i$ is the gradient

\nabla f_i

— a

1\times2

row saying how output

i

responds to each input. Column $j$ is

\partial\mathbf f/\partial x_j

— how the whole output vector moves when input

j

alone changes. Here column 1 is

\partial\mathbf f/\partial x

and column 2 is

\partial\mathbf f/\partial y

.

$\mathbf J\in\mathbb{R}^{3\times2}$ as above, with $\mathbf J(1,0)=\begin{bmatrix}2&0\\1&1\\1&0\end{bmatrix}$ .

Remember

Building the matrix as a stack of gradients — compute $\nabla f_1,\nabla f_2,\nabla f_3$ separately, then stack — gives the same answer and is much harder to get wrong than filling in six entries by position.

Problem 7hard

Consider $T:\mathbb{R}^2\to\mathbb{R}^2$ , $\ T(x,y)=\big(e^x\cos y,\ e^x\sin y\big)$ . (a) Compute $\mathbf J_T$ . (b) Evaluate $\det\mathbf J_T$ . (c) Where is $T$ not locally invertible? Is $T$ globally one-to-one?

What this tests. The determinant as a local area factor, the inverse function theorem, and the gap between local and global statements.

Show the full solution

(a) Step 1 — partials of $u=e^x\cos y$ . With respect to

x

,

\cos y

is a constant:

u_x=e^x\cos y

. With respect to

y

,

e^x

is a constant:

u_y=-e^x\sin y

.

Step 2 — partials of $v=e^x\sin y$ .

\ v_x=e^x\sin y

,

\ v_y=e^x\cos y

.

Step 3 — assemble.

\mathbf J_T=\begin{bmatrix}\partial u/\partial x&\partial u/\partial y\\ \partial v/\partial x&\partial v/\partial y\end{bmatrix}=\begin{bmatrix}e^x\cos y&-e^x\sin y\\ e^x\sin y&e^x\cos y\end{bmatrix}

(b) Determinant via

ad-bc

:

\det\mathbf J_T=(e^x\cos y)(e^x\cos y)-(-e^x\sin y)(e^x\sin y)=e^{2x}\big(\cos^2y+\sin^2y\big)=e^{2x}

(c) Step 1 — local invertibility. The inverse function theorem guarantees local invertibility wherever

\det\mathbf J\neq0

. Since

e^{2x}>0

for every real

x

, the determinant is never zero:

T

is locally invertible everywhere.

Step 2 — global injectivity fails anyway. Because

\cos

and

\sin

have period

2\pi

,

T(x,y+2\pi)=\big(e^x\cos(y+2\pi),\ e^x\sin(y+2\pi)\big)=\big(e^x\cos y,\ e^x\sin y\big)=T(x,y)

Infinitely many distinct inputs share an output, so

T

is not globally one-to-one. "Locally invertible everywhere" and "globally invertible" are genuinely different claims.

$\mathbf J_T=\begin{bmatrix}e^x\cos y&-e^x\sin y\\ e^x\sin y&e^x\cos y\end{bmatrix}$ , $\ \det\mathbf J_T=e^{2x}>0$ always — locally invertible everywhere, but not globally one-to-one.

Aha

Write $z=x+iy$ and the map is simply $w=e^{z}$ . For a holomorphic map the real Jacobian determinant equals $\lvert dw/dz\rvert^2=\lvert e^z\rvert^2=e^{2x}$ — the same answer, arrived at with no partial derivatives at all.

Problem 8hard

Three identities are worth memorising: $\ \nabla(\mathbf b^\top\mathbf x)=\mathbf b^\top$ , $\ \nabla(\mathbf x^\top\mathbf x)=2\mathbf x^\top$ , $\ \nabla(\mathbf x^\top A\mathbf x)=\mathbf x^\top(A+A^\top)$ . Let $f(\mathbf x)=\mathbf x^\top A\mathbf x-\mathbf b^\top\mathbf x$ with $A=\begin{bmatrix}1&2\\0&3\end{bmatrix}$ (not symmetric) and $\mathbf b=\begin{bmatrix}1\\1\end{bmatrix}$ . (a) Write $f$ in components. (b) Apply the identities to get $\nabla f$ — why does $A+A^\top$ appear rather than $2A$ ? (c) Evaluate at $(1,1)$ .

What this tests. Matrix-calculus identities, and the specific trap of assuming symmetry when none was given.

Show the full solution

(a) Step 1 — compute $A\mathbf x$ .

A\mathbf x=\begin{bmatrix}1&2\\0&3\end{bmatrix}\begin{bmatrix}x_1\\x_2\end{bmatrix}=\begin{bmatrix}x_1+2x_2\\3x_2\end{bmatrix}

Step 2 — then $\mathbf x^\top(A\mathbf x)$ .

[x_1,x_2]\begin{bmatrix}x_1+2x_2\\3x_2\end{bmatrix}=x_1(x_1+2x_2)+x_2(3x_2)=x_1^2+2x_1x_2+3x_2^2

Step 3 — subtract $\mathbf b^\top\mathbf x=x_1+x_2$ .

f(x_1,x_2)=x_1^2+2x_1x_2+3x_2^2-x_1-x_2

(b) Step 1 — form $A+A^\top$ .

A^\top=\begin{bmatrix}1&0\\2&3\end{bmatrix}

, so

A+A^\top=\begin{bmatrix}2&2\\2&6\end{bmatrix}

.

Step 2 — apply the identities.

\nabla f=\mathbf x^\top(A+A^\top)-\mathbf b^\top=[x_1,x_2]\begin{bmatrix}2&2\\2&6\end{bmatrix}-[1,1]=\big[\,2x_1+2x_2-1,\ \ 2x_1+6x_2-1\,\big]

Step 3 — why $A+A^\top$ ? Write

\mathbf x^\top A\mathbf x=\sum_{i,j}A_{ij}x_ix_j

and differentiate with respect to

x_k

. The variable

x_k

appears twice in that double sum: once as the

i=k

factor (contributing

\sum_j A_{kj}x_j

) and once as the

j=k

factor (contributing

\sum_i A_{ik}x_i

). Adding the two contributions gives the

k

-th entry of

\mathbf x^\top(A+A^\top)

. Only when

A

is symmetric does

A+A^\top

collapse to

2A

.

(c) Substitute $x_1=x_2=1$ :

\nabla f(1,1)=[\,2+2-1,\ 2+6-1\,]=[\,3,\ 7\,]

$\nabla f=[\,2x_1+2x_2-1,\ 2x_1+6x_2-1\,]$ , and $\nabla f(1,1)=[\,3,\,7\,]$ .

Watch out

Had you used $2A$ instead, you would get $\nabla f(1,1)=[1,9]$ — visibly different, so this is a trap with teeth. The safest check is part (a): differentiate the component form directly, $\partial f/\partial x_1=2x_1+2x_2-1$ , and confirm. Expanding the cross term $2x_1x_2$ by hand automatically produces the symmetric off-diagonal $2$ s.

Problem 9mediumadded — covers Taylor

The practice set above never touches Taylor polynomials, so here they are. (a) Find the degree-4 Maclaurin polynomial of $\ln(1+x)$ and use it to estimate $\ln 1.5$ . (b) The sigmoid is $\sigma(z)=\frac{1}{1+e^{-z}}$ , with the useful identity $\sigma'=\sigma(1-\sigma)$ . Find its degree-3 Maclaurin polynomial. (c) What does the result in (b) say about logistic regression near $z=0$ ?

What this tests. Building a Taylor polynomial from repeated differentiation, keeping the $k!$ straight, and reading a vanishing coefficient as a statement about shape.

Show the full solution

(a) Step 1 — differentiate repeatedly at $x_0=0$ . With

f=\ln(1+x)

:

f'=\frac{1}{1+x},\quad f''=\frac{-1}{(1+x)^2},\quad f'''=\frac{2}{(1+x)^3},\quad f^{(4)}=\frac{-6}{(1+x)^4}

At

x=0

:

f(0)=0,\ f'(0)=1,\ f''(0)=-1,\ f'''(0)=2,\ f^{(4)}(0)=-6

.

Step 2 — divide each by $k!$ . The coefficients are

1/1!=1

,

-1/2!=-\tfrac12

,

2/3!=\tfrac13

,

-6/4!=-\tfrac14

:

T_4(x)=x-\frac{x^2}{2}+\frac{x^3}{3}-\frac{x^4}{4}

The pattern

\sum(-1)^{k+1}x^k/k

is worth recognising on sight.

Step 3 — estimate $\ln 1.5$ by setting $x=0.5$ .

T_4(0.5)=\tfrac12-\tfrac18+\tfrac1{24}-\tfrac1{64}=\tfrac{77}{192}\approx 0.401042

The true value is

\ln 1.5\approx0.405465

, so the error is about

0.0044

— roughly the size of the first omitted term,

x^5/5=1/160\approx0.00625

, as Taylor's remainder predicts.

(b) Step 1 — use the identity to avoid quotient-rule misery. From

\sigma'=\sigma(1-\sigma)

, differentiate again with the product rule:

\sigma''=\sigma'(1-\sigma)+\sigma(-\sigma')=\sigma'(1-2\sigma)

\sigma'''=\sigma''(1-2\sigma)+\sigma'(-2\sigma')=\sigma''(1-2\sigma)-2(\sigma')^2

Step 2 — evaluate at $z=0$ , where

\sigma(0)=\tfrac12

:

\sigma'(0)=\tfrac12\cdot\tfrac12=\tfrac14,\qquad \sigma''(0)=\tfrac14\cdot\underbrace{(1-2\cdot\tfrac12)}_{=\,0}=0,\qquad \sigma'''(0)=0-2\left(\tfrac14\right)^2=-\tfrac18

Step 3 — divide by $k!$ and assemble. Coefficients:

\tfrac12

,

\tfrac14

,

0/2!=0

,

-\tfrac18/3!=-\tfrac1{48}

:

T_3(z)=\frac12+\frac{z}{4}-\frac{z^3}{48}

(c) The quadratic term is exactly zero, so near

z=0

the sigmoid is

\tfrac12+\tfrac z4

up to a cubic correction — it is locally a straight line with slope

\tfrac14

. That is why a logistic model behaves almost linearly for small activations, why

\tfrac14

is the largest gradient the sigmoid can ever pass backwards (the source of vanishing gradients in deep sigmoid networks), and why keeping activations near zero — through normalisation — keeps a network in its responsive band.

(a) $T_4(x)=x-\tfrac{x^2}2+\tfrac{x^3}3-\tfrac{x^4}4$ , giving $\ln1.5\approx\tfrac{77}{192}\approx0.4010$ . (b) $T_3(z)=\tfrac12+\tfrac z4-\tfrac{z^3}{48}$ . (c) The sigmoid is locally linear at the origin, with maximum slope $\tfrac14$ .

Aha

The missing $z^2$ is not an accident of arithmetic — it is symmetry. The sigmoid has rotational symmetry about the point $(0,\tfrac12)$ , and odd symmetry kills every even-order term beyond the constant.

Problem 10hardadded — covers the vector chain rule

The set above computes Jacobians but never multiplies them, so here is the backpropagation step itself. (a) Let $\mathbf g(\mathbf x)=\big(x_1^2x_2,\ x_1+x_2^2\big)$ and $f(\mathbf u)=u_1u_2$ . Compute $\nabla(f\circ\mathbf g)$ as a product of Jacobians, and verify by substituting first. Evaluate at $(1,2)$ . (b) A linear layer computes $\mathbf z=W\mathbf x$ and the loss is $L=\tfrac12\lVert\mathbf z-\mathbf y\rVert^2$ . Show $\dfrac{\partial L}{\partial\mathbf x}=(\mathbf z-\mathbf y)^\top W$ , then evaluate for $W=\begin{bmatrix}2&-1\\1&3\end{bmatrix}$ , $\mathbf y=\begin{bmatrix}1\\2\end{bmatrix}$ , $\mathbf x=\begin{bmatrix}1\\1\end{bmatrix}$ .

What this tests. That the vector chain rule really is matrix multiplication, that the shapes police the order, and the exact gradient a linear layer sends backwards.

Show the full solution

(a) Step 1 — the two Jacobians, with shapes first.

\mathbf g:\mathbb{R}^2\to\mathbb{R}^2

gives a

2\times2

;

f:\mathbb{R}^2\to\mathbb{R}

gives a

1\times2

. The product must be

(1\times2)(2\times2)=1\times2

— a gradient, as expected.

\mathbf J_{\mathbf g}=\begin{bmatrix}2x_1x_2&x_1^2\\ 1&2x_2\end{bmatrix},\qquad \mathbf J_f=[\,u_2,\ u_1\,]=\big[\,x_1+x_2^2,\ \ x_1^2x_2\,\big]

Note

\mathbf J_f

must be evaluated at

\mathbf u=\mathbf g(\mathbf x)

— that substitution is the step people forget.

Step 2 — multiply, later stage on the left.

\nabla(f\circ\mathbf g)=\mathbf J_f\,\mathbf J_{\mathbf g}=\big[x_1+x_2^2,\ x_1^2x_2\big]\begin{bmatrix}2x_1x_2&x_1^2\\ 1&2x_2\end{bmatrix}

First entry:

(x_1+x_2^2)(2x_1x_2)+(x_1^2x_2)(1)=3x_1^2x_2+2x_1x_2^3

.
Second entry:

(x_1+x_2^2)(x_1^2)+(x_1^2x_2)(2x_2)=x_1^3+3x_1^2x_2^2

.

Step 3 — verify by composing first.

f(\mathbf g(\mathbf x))=(x_1^2x_2)(x_1+x_2^2)=x_1^3x_2+x_1^2x_2^3

. Differentiating directly:

\frac{\partial}{\partial x_1}=3x_1^2x_2+2x_1x_2^3,\qquad \frac{\partial}{\partial x_2}=x_1^3+3x_1^2x_2^2

Identical to step 2 — the matrix product was not a shortcut with a catch, it is the same derivative.

Step 4 — evaluate at $(1,2)$ .

3(1)(2)+2(1)(8)=6+16=22

;

\ 1+3(1)(4)=13

. So

\nabla(f\circ\mathbf g)(1,2)=[\,22,\ 13\,]

.

(b) Step 1 — split into two stages and take each Jacobian. Write

\mathbf r=\mathbf z-\mathbf y

and

L=\tfrac12\mathbf r^\top\mathbf r

. Then

\frac{\partial L}{\partial\mathbf z}=\mathbf r^\top=(\mathbf z-\mathbf y)^\top\ \ (1\times n),\qquad \frac{\partial \mathbf z}{\partial\mathbf x}=W\ \ (n\times n)

using

\nabla(\mathbf r^\top\mathbf r)=2\mathbf r^\top

from Problem 8 (the

\tfrac12

is chosen precisely to cancel that 2), and the fact that the Jacobian of a linear map is the map's own matrix.

Step 2 — chain them, later stage on the left.

\frac{\partial L}{\partial\mathbf x}=\frac{\partial L}{\partial\mathbf z}\cdot\frac{\partial\mathbf z}{\partial\mathbf x}=(\mathbf z-\mathbf y)^\top W

Shapes:

(1\times n)(n\times n)=1\times n

✓. This is the backward pass of a linear layer: the incoming gradient row gets multiplied by $W$ on its way back.

Step 3 — forward pass with the numbers.

\mathbf z=W\mathbf x=\begin{bmatrix}2&-1\\1&3\end{bmatrix}\begin{bmatrix}1\\1\end{bmatrix}=\begin{bmatrix}1\\4\end{bmatrix},\qquad \mathbf z-\mathbf y=\begin{bmatrix}1\\4\end{bmatrix}-\begin{bmatrix}1\\2\end{bmatrix}=\begin{bmatrix}0\\2\end{bmatrix}

so

L=\tfrac12(0^2+2^2)=2

.

Step 4 — backward pass.

\frac{\partial L}{\partial\mathbf x}=[\,0,\ 2\,]\begin{bmatrix}2&-1\\1&3\end{bmatrix}=[\,0\cdot2+2\cdot1,\ \ 0\cdot(-1)+2\cdot3\,]=[\,2,\ 6\,]

(a) $\nabla(f\circ\mathbf g)=\big[3x_1^2x_2+2x_1x_2^3,\ \ x_1^3+3x_1^2x_2^2\big]$ , equal to $[\,22,\,13\,]$ at $(1,2)$ . (b) $\dfrac{\partial L}{\partial\mathbf x}=(\mathbf z-\mathbf y)^\top W=[\,2,\ 6\,]$ , with $L=2$ .

Remember

In part (b) the first output was already perfect ( $z_1-y_1=0$ ) yet $\partial L/\partial x_1=2$ is not zero. The error in output 2 still reaches input 1, because $W$ couples them. Gradients flow along every path the weights create — which is exactly what makes a network trainable and its behaviour hard to reason about locally.

Next up

Unit 7 · Backpropagation & Automatic Differentiation →

You now hold every piece: the chain rule, Jacobians, and the insight that composing layers means multiplying matrices. Unit 7 turns that into an algorithm — how a network computes a million derivatives for roughly the price of two forward passes, and why it must run backwards to do it.

← All units