Unit 8 · Taylor & MacLaurin Series — The Math Behind the Machine

1

Every smooth curve is secretly a polynomial — up close

Imagine this

Open a maps app and zoom right in on a winding ghat road. At full zoom, the road looks perfectly straight.

Zoom out a little. Now you see it bend, gently, like a smile. Zoom out more, and the bend itself starts to change: it tightens, then loosens.

Every smooth curve behaves like this. Up close, it is a straight line. A bit further out, it is a gentle bend. Further still, you need more detail.

That is the idea behind a Taylor polynomial. Pick one spot on the curve, $x=a$ . We call it the anchor. Then copy what the curve is doing there, one fact at a time:

①its height — the value $f(a)$ ;
②its slope, how steep it is — the derivative $f'(a)$ ;
③its bend, how fast the slope is changing — the second derivative $f''(a)$ ;
④and so on: each new fact is "how fast the previous one changes".

Glue the facts together, dividing the $k$ -th one by $k!$ (read it "k factorial": $1\times2\times\cdots\times k$ ). Out comes the polynomial copy you built in Unit 6:

P_{n}(x)=f(a)+f'(a)(x-a)+\frac{f''(a)}{2!}(x-a)^2+\cdots+\frac{f^{(n)}(a)}{n!}(x-a)^n

A tiny example. Take $f(x)=\sqrt{x}$ with the anchor at $4$ . The height there is $2$ and the slope is $\tfrac14$ . So the straight-line copy says $\sqrt{4.2}\approx 2+\tfrac14(0.2)=2.05$ . The true value is $2.04939$ . Very close, but not exact.

That leaves two questions, and they are the whole of this unit:

?Why does it work? Why should a few facts at one spot give a good copy nearby?
?How wrong can it be? Not a hope, but a guarantee you can compute.

Both answers grow from one simple fact about a hiking trail. After that, we take the idea to two variables. There the "bend" becomes a small table of numbers called the Hessian, and that table tells a valley from a peak.

The realization

A Taylor polynomial is a copy built from a few facts at one spot: height, slope, bend, and so on. Each extra fact keeps the copy close for longer. What we still need is a promise about how close.

Pause & predict

The straight-line copy says $\sqrt{4.2}\approx2.05$ . The truth is $2.0494$ , a little smaller. The next piece of the copy uses the bend. Which way should it push the estimate?

In one sentence: a Taylor polynomial copies a curve's height, slope and bend at one spot, and this unit proves why that copy works and how far it can be trusted.

2

The flat spot — Rolle's theorem

Imagine this

You go on a trek. The first signboard says 1,000 m. After a day of ups and downs, the last signboard also says 1,000 m.

Somewhere on the way, the ground under your boots was perfectly level. Maybe at the top of a climb. Maybe at the bottom of a dip.

You can't be sure where. But you are sure it happened. You cannot go up and come back down without, for one moment, going neither up nor down.

This common-sense fact is called Rolle's theorem. The trail is the graph of a function $f$ , from $x=a$ to $x=b$ . A level spot is a point $c$ where the slope is zero: $f'(c)=0$ (read it: "the slope at c is zero").

The theorem needs three things. Each one is something any real trail has:

It needs	In everyday words	In symbols
No gaps	The trail never jumps. You walk every step of it.	$f$ continuous on $[a,b]$
No sharp corners	The trail never turns at a pointed tip.	$f$ differentiable on $(a,b)$
Same height at both ends	The first and last signboards match.	$f(a)=f(b)$

Then it promises: there is a $c$ strictly between $a$ and $b$ with $f'(c)=0$ .

A tiny example. Take $f(x)=x^2-2x$ from $0$ to $2$ . Both ends have height $0$ . The slope is $f'(x)=2x-2$ , and it is zero at $c=1$ : the bottom of the dip.

Why "no sharp corners"? Picture a trail shaped like a V, such as $|x|$ . It goes down, then up, and ends at the same height. But at the bottom there is a sharp tip, and the slope jumps from $-1$ straight to $+1$ . No step is ever level. Try it in the broken tab below.

Try: Press ▶ hunt for c. The plank glows green where it runs parallel to the rope. Then open broken: same heights at both ends, but no level spot anywhere.

drag the picture to orbit

left end a-1.6

right end b1.8

—

The rope's slope is the average slope. The plank's slope is the slope right here. The theorem says they match somewhere strictly between the flags.

The realization

If a smooth trail starts and ends at the same height, it has a level spot in between: $f'(c)=0$ . All it needs is no gaps and no sharp corners. This one small fact will build the whole unit.

Pause & predict

The V-shaped trail $f(x)=|x|$ from $-1$ to $1$ has no gaps, both ends at height $1$ , and a lowest point at $0$ . So why does Rolle's theorem not give a $c$ with $f'(c)=0$ ?

In one sentence: a smooth trail with the same height at both ends must be level somewhere in between, and a sharp corner cancels the promise.

3

Average equals instantaneous — the mean value theorem

Imagine this

You drive 200 km on the expressway in exactly 2 hours. Your average speed is 100 km/h.

Did your speedometer ever show exactly 100? It must have. If you were always below 100, you could not cover 200 km in time. If you were always above 100, you would have covered more.

So at some moment, your speed right then matched your average speed for the whole trip.

This is the mean value theorem ("mean" just means average). On a graph, the average slope over a stretch is the slope of the straight line joining its two ends. That line is called the chord.

The theorem says: somewhere in between, the curve's own slope equals the chord's slope.

f'(c)=\frac{f(b)-f(a)}{b-a}

Read it: "the slope at some point $c$ equals the rise divided by the run". It needs the same "no gaps, no sharp corners" as Rolle, but the ends may be at any heights.

A tiny example. Take $f(x)=x^2$ from $1$ to $3$ . The rise is $9-1=8$ and the run is $2$ , so the average slope is $4$ . The slope of $x^2$ is $2x$ , and $2c=4$ gives $c=2$ .

Rolle is the special case where both ends are at the same height, so the average slope is $0$ . And the proof runs the other way too. Tilt your head until the chord looks level, and the mean value theorem becomes Rolle. The middle tab of the flat-spot hunter above shows exactly this.

If you want the algebra · the mean value theorem, from Rolle

Prove it · the mean value theorem, from Rolle

Claim. If $f$ is continuous on $[a,b]$ and differentiable on $(a,b)$ , some $c\in(a,b)$ has $f'(c)=\dfrac{f(b)-f(a)}{b-a}$ .

1

Name the average slope and build the chord — the straight line through the two endpoints:

m=\frac{f(b)-f(a)}{b-a},\qquad \ell(x)=f(a)+m\,(x-a)

\ell

is a straight line with slope

m

that passes through

(a,f(a))

; by the choice of

m

it also passes through

(b,f(b))

.

2

Subtract the chord from the function:

g(x)=f(x)-\ell(x)=f(x)-f(a)-m\,(x-a)

g

is "how far

f

sits above its own chord". It inherits continuity and differentiability from

f

, because

\ell

is as smooth as anything can be.

3

Check the two ends:

\begin{aligned}g(a)&=f(a)-f(a)-0=0,\\ g(b)&=f(b)-f(a)-m\,(b-a)\\ &=f(b)-f(a)-\big(f(b)-f(a)\big)=0\end{aligned}

The chord meets the curve at both ends, so the gap is zero at both ends. That is exactly Rolle's "same height" hypothesis — for

g

.

4

Apply Rolle to

g

: some

c\in(a,b)

has

g'(c)=0

. But

g'(x)=f'(x)-m

, so

0=g'(c)=f'(c)-m\quad\Longrightarrow\quad f'(c)=m=\frac{f(b)-f(a)}{b-a}.

That is the claim. ∎ Tilting the picture by a straight line does not change where tangents are parallel to it. So the general theorem is the flat one in disguise.

Now rearrange the formula. It says something surprising:

f(b)=f(a)+f'(c)\,(b-a)

In words: height at $b$ = height at $a$ + slope × distance. That is the straight-line copy from Section 1. But here it is exact, as long as you use the slope at the right secret spot $c$ , not at $a$ .

Taylor's theorem is the same idea with more facts copied. Copy the slope, the bend and more at $a$ , and let one secret point $c$ soak up everything you left out.

The realization

Somewhere, the slope equals the average slope: $f'(c)=\dfrac{f(b)-f(a)}{b-a}$ . Rearranged, $f(b)=f(a)+f'(c)(b-a)$ . That is the straight-line copy, made exact by one secret point.

Pause & predict

Take $f(x)=x^3$ from $0$ to $2$ . The average slope is $\dfrac{8-0}{2-0}=4$ . Where is the point $c$ with $f'(c)=4$ ?

In one sentence: somewhere on the way your instant speed equals your average speed, and written as a formula this is already a straight-line copy with no error.

4

The Rolle ladder — one flat spot per rung

Imagine this

Picture a relay race down a staircase, one runner per step. A runner may only run when she holds two tokens.

Every runner gets one token free, at the start line. The top runner also finds a second token at the finish line. So she runs, and drops a new token somewhere in between, for the runner one step below.

Now he has two tokens as well, so he runs and drops one for the next. Step by step, the dropped tokens creep toward the start.

In the maths, the runners are a function and its derivatives: $F$ , then $F'$ , then $F''$ , and so on. A token is a zero, a place where that function equals $0$ . And "running" is Rolle's theorem: two zeros of a function give a zero of its slope in between.

The free tokens at the start line mean this: at $a$ , the function is zero and very flat. Its value and its first $n-1$ derivatives are all zero there. The token at the finish means $F(b)=0$ :

F(a)=F'(a)=\cdots=F^{(n-1)}(a)=0,\qquad F(b)=0.

Then the tokens pass all the way down. The last runner, $F^{(n)}$ , gets one too: there is a $c$ between $a$ and $b$ with $F^{(n)}(c)=0$ . The lecture calls this Proposition P.

A tiny example with $n=2$ . Take $F(x)=x^2(x-1)$ from $0$ to $1$ . Here $F(0)=F'(0)=0$ and $F(1)=0$ . Rung 1: $F'(x)=3x^2-2x$ is zero at $c_1=\tfrac23$ . Rung 2: $F''(x)=6x-2$ is zero at $c_2=\tfrac13$ , between $0$ and $c_1$ .

Try: Press ▶ climb and watch the zeros creep toward $a$ : $a<c_3<c_2<c_1<b$ . Then drag $b$ closer to $a$ and watch every $c_k$ squeeze in.

target b2

rungs n3

—

Here $F=f-P$ is the gap between $f$ and a copy that is flat at $a=0$ and zero at $b$ . The bottom rung's zero is where the leftover formula comes from.

If you want the algebra · Proposition P, one rung at a time

Prove it · Proposition P — n flat spots, nested

Claim. If $F(a)=F'(a)=\cdots=F^{(n-1)}(a)=0$ and $F(b)=0$ , with $F$ $n$ -times differentiable on $(a,b)$ and the lower derivatives continuous on $[a,b]$ , then some $c\in(a,b)$ has $F^{(n)}(c)=0$ .

1

Rung 1.

F(a)=0=F(b)

: Rolle gives

c_1\in(a,b)

with

F'(c_1)=0

. Two zeros of

F

at the ends, one flat spot of

F

in between. Standard Rolle.

2

Rung 2. Now look at

F'

. It is zero at

a

(given) and at

c_1

(rung 1). Rolle on

[a,c_1]

gives

c_2\in(a,c_1)

with

F''(c_2)=0

. The zero handed down from rung 1 plus the zero we were given at

a

are the two ends Rolle needs. The interval has shrunk:

c_2<c_1

.

3

Rung $k$ , in general. Suppose

F^{(k-1)}(c_{k-1})=0

with

c_{k-1}\in(a,b)

, and

k-1\le n-1

so that

F^{(k-1)}(a)=0

too. Rolle on

[a,c_{k-1}]

gives

c_k\in(a,c_{k-1})\quad\text{with}\quad F^{(k)}(c_k)=0.

Same move each time: the zero at

a

is free (it was assumed), the other zero is inherited from the rung above. That is why we needed

n-1

zero derivatives at

a

— one for each rung after the first.

4

After

n

rungs:

a<c_n<c_{n-1}<\cdots<c_1<b,\qquad F^{(n)}(c_n)=0.

Set

c=c_n

. Done. ∎ Each rung uses one Rolle and shrinks the interval toward

a

. The last rung needs no zero at

a

for

F^{(n)}

— it only needs the inherited

c_{n-1}

and the free zero of

F^{(n-1)}

at

a

.

The realization

Each Rolle step turns two zeros into one new zero, one derivative lower. If $F$ is zero at $b$ and flat to order $n-1$ at $a$ , the zeros pass all the way down: $F^{(n)}(c)=0$ for some $c$ in between.

Pause & predict

Why must $F$ 's derivatives be zero at $a$ up to order $n-1$ ? Why is $F(a)=0$ alone not enough?

In one sentence: a function that is very flat at $a$ and zero at $b$ passes one zero down per derivative, until its $n$ -th derivative is zero somewhere in between.

5

Taylor's theorem, built by hand

Imagine this

At 9 a.m. your phone battery is at 80%. It is dropping 6% an hour, and the drop is getting faster by about 2% an hour, every hour.

Forecast 1 p.m. with just these three facts: $80-6\times4-\tfrac12\times2\times4^2=40\%$ .

By 1 p.m. the real battery will differ a little. By how much? Taylor's theorem gives an exact answer: the error is what the next fact would add, if you measured it at the right moment during the morning.

Here is the plan, in plain words. We have a function $f$ and a polynomial copy $P$ . Its numbers (the coefficients) are not chosen yet.

1Measure the gap. Let $F(x)=f(x)-P(x)$ : how far the real curve is from the copy.
2Make the gap flat at $a$ . Choose the first numbers so the gap, its slope, its bend… are all zero at $a$ . This forces the famous Taylor numbers $f^{(k)}(a)/k!$ .
3Close the gap at $b$ . Choose the last number so the gap is zero at $b$ .
4Run the ladder. Now the gap fits Proposition P, so its $n$ -th derivative is zero at some $c$ . That one fact says the last number is $f^{(n)}(c)/n!$ .

Out comes Taylor's theorem. For some $c$ strictly between $a$ and $b$ :

f(b)=\underbrace{\sum_{k=0}^{n-1}\frac{f^{(k)}(a)}{k!}(b-a)^k}_{\text{the copy}}+\underbrace{\frac{f^{(n)}(c)}{n!}(b-a)^n}_{\text{the leftover}}

Read the leftover slowly. It looks exactly like the next term of the copy. The only change: the derivative is taken at a secret point $c$ , not at $a$ .

A tiny example. Copy $e^x$ at $0$ with two terms: $1+x$ . At $x=1$ the copy gives $2$ , and the truth is $e=2.71828$ . The leftover is $\tfrac{e^c}{2}\cdot1^2=0.71828$ , so $e^c=1.43656$ and $c=0.3623$ . A real point, between $0$ and $1$ , just as promised.

The gap function, pinned. For

f=e^x

,

a=0

,

n=3

,

b=2

: the gap

F=f-P

leaves

0

perfectly flat (value, slope and bend all zero), dips, and returns to zero at

b

. The three nested flat spots are exactly where the ladder puts them.

If you want the algebra · the three moves, step by step

Prove it · move 1 — flattening the gap at a fixes the first n coefficients

Claim. $F^{(k)}(a)=f^{(k)}(a)-k!\,a_k$ for $k=0,1,\dots,n$ . So $F^{(k)}(a)=0$ exactly when $a_k=\dfrac{f^{(k)}(a)}{k!}$ .

1

Differentiate one power

j

times and evaluate at

a

:

\frac{d^{\,j}}{dx^{j}}(x-a)^k\Big|_{x=a}=\begin{cases}0 & j<k\\[.2em] k! & j=k\\[.2em] 0 & j>k\end{cases}

Fewer differentiations than the power (

j<k

): a positive power of

(x-a)

survives, and it is

0

at

a

. Exactly

k

differentiations: the power is used up and leaves

k(k-1)\cdots1=k!

. More than

k

: a constant differentiated again is

0

. Only the term whose power matches the number of differentiations survives at

a

.

2

So the

k

-th derivative of

P

at

a

picks out one term:

P^{(k)}(a)=\sum_{j=0}^{n}a_j\,\frac{d^{\,k}}{dx^{k}}(x-a)^j\Big|_{a}=a_k\cdot k!

Every other term in the sum is killed by step 1 — either it still has a factor

(x-a)

, or it has already been differentiated to zero.

3

Therefore

F^{(k)}(a)=f^{(k)}(a)-P^{(k)}(a)=f^{(k)}(a)-k!\,a_k

, and demanding

F^{(k)}(a)=0

for

k=0,\dots,n-1

forces

a_k=\frac{f^{(k)}(a)}{k!},\qquad k=0,1,\dots,n-1.\

That is the claim. ∎ This is where the Taylor coefficients come from — not from a formula handed down, but from the demand that the gap be flat. The

k!

is the factorial left behind by differentiating

(x-a)^k

k

times.

Prove it · move 2 — the last coefficient makes the gap zero at b

Claim. With $a_0,\dots,a_{n-1}$ fixed by move 1, $F(b)=0$ holds exactly when $a_n=\dfrac{1}{(b-a)^n}\Big(f(b)-\displaystyle\sum_{k=0}^{n-1}\frac{f^{(k)}(a)}{k!}(b-a)^k\Big).$

1

Write out

F(b)

, separating the last term:

F(b)=f(b)-\sum_{k=0}^{n-1}\frac{f^{(k)}(a)}{k!}(b-a)^k-a_n(b-a)^n

The first

n

coefficients are already known from move 1; only

a_n

is free.

2

Set

F(b)=0

and solve for

a_n

:

\begin{aligned}a_n(b-a)^n&=f(b)-\sum_{k=0}^{n-1}\frac{f^{(k)}(a)}{k!}(b-a)^k\\[.35em] a_n&=\frac{1}{(b-a)^n}\Big(f(b)-\sum_{k=0}^{n-1}\frac{f^{(k)}(a)}{k!}(b-a)^k\Big).\end{aligned}

That is the claim. ∎ Divide by

(b-a)^n

, which is not zero because

b\ne a

. Read it in words:

a_n

is "the gap left by the first

n

terms, per unit of

(b-a)^n

". We do not yet know what

a_n

has to do with

f

's derivatives — that is move 3.

Prove it · move 3 — Taylor's theorem with the remainder

Claim. There is a $c$ strictly between $a$ and $b$ such that $f(b)=\sum_{k=0}^{n-1}\frac{f^{(k)}(a)}{k!}(b-a)^k+\frac{f^{(n)}(c)}{n!}(b-a)^n.$

1

Differentiate

F=f-P

exactly

n

times. By step 1 of move 1, the

n

-th derivative of

P

is the constant

n!\,a_n

(every lower power has been differentiated away, and there are no higher ones):

F^{(n)}(x)=f^{(n)}(x)-n!\,a_n\qquad\text{for every }x.

A degree-

n

polynomial differentiated

n

times is a constant: its leading coefficient times

n!

.

2

Proposition P gives

c\in(a,b)

with

F^{(n)}(c)=0

. At that

c

:

f^{(n)}(c)-n!\,a_n=0\quad\Longrightarrow\quad a_n=\frac{f^{(n)}(c)}{n!}.

The last coefficient — which move 2 defined as "the gap per unit

(b-a)^n

" — is revealed to be the

n

-th derivative at a secret point, divided by

n!

. It has the same shape as every other coefficient; only the point where the derivative is taken has moved from

a

to

c

.

3

Substitute this

a_n

into move 2's equation

F(b)=0

, i.e.

f(b)=\sum_{k<n}\frac{f^{(k)}(a)}{k!}(b-a)^k+a_n(b-a)^n

:

f(b)=\underbrace{\sum_{k=0}^{n-1}\frac{f^{(k)}(a)}{k!}(b-a)^k}_{\text{Taylor polynomial }P_{n-1}(b)}+\underbrace{\frac{f^{(n)}(c)}{n!}(b-a)^n}_{\text{remainder }R_n}.

That is the claim. ∎ The polynomial part uses derivatives at

a

— things you can compute. The remainder uses one derivative at an unknown

c

— which you cannot compute, but can bound. For

n=1

this is the mean value theorem,

f(b)=f(a)+f'(c)(b-a)

; the whole construction is that theorem, repeated up a ladder.

Why the leftover gets small

Replace $b$ by any $x$ : $f(x)=$ copy $+\ \dfrac{f^{(n)}(c)}{n!}(x-a)^n$ . The leftover has two parts you can see, and both help.

$(x-a)^n$ is tiny when $x$ is close to the anchor. $1/n!$ shrinks very fast as you keep more terms. The part you can't see, $f^{(n)}(c)$ , is stuck between the smallest and largest values of $f^{(n)}$ on the stretch. The next section turns that into a number.

The realization

Every smooth function is its copy plus one leftover term: $f(x)=P_{n-1}(x)+\dfrac{f^{(n)}(c)}{n!}(x-a)^n$ , exactly. The leftover looks like the next term, measured at a secret point. And the whole proof is Rolle, used again and again.

Pause & predict

In the proof, the last number was first chosen to close the gap at $b$ . Then it turned out to equal $f^{(n)}(c)/n!$ . What did the theorem actually prove?

Pause & predict

Take $f(x)=x^3$ , anchor $a=0$ , $n=2$ , and $b=2$ . The copy $f(0)+f'(0)x$ is just $0$ , so all of $f(2)=8$ is leftover: $8=\dfrac{f''(c)}{2!}\cdot2^2$ . What is $c$ ?

In one sentence: make the gap flat at $a$ and zero at $b$ , run the Rolle ladder, and the leftover turns out to be the next term measured at a secret point.

6

How big is the error? — the remainder in your hands

Imagine this

Your cab app can't know exactly where the traffic jams will be. But suppose it knows one thing: on this road, you never crawl slower than 20 km/h.

Then for a 10 km trip it can promise: at most 30 minutes. It never needed to know where the jams are. A worst case was enough for a guarantee.

The leftover has the same problem. We never find out the secret point $c$ . But suppose we know the worst case: $f^{(n)}$ never gets bigger than some number $M$ on the stretch. Then, wherever $c$ is:

|R_n(x)|\ \le\ \frac{M}{n!}\,|x-a|^n

Read it: "the error is at most $M$ , divided by $n$ factorial, times the distance to the power $n$ ". You can work it out before you trust the copy.

A tiny example. Copy $e^{0.1}$ with two terms: $1+0.1=1.1$ . On $[0,0.1]$ , the second derivative $e^x$ is at most $e^{0.1}<1.2$ , so take $M=1.2$ . The promise: error $\le\frac{1.2}{2}\times0.1^2=0.006$ . The true error is $1.10517-1.1=0.00517$ . Under the promise, as it must be.

Try: Load √4.2 (Problem A3). Slide the order from 2 to 3 and watch the error drop from $0.0006$ to $0.000015$ . Then drag $x$ far from the anchor and watch the shaded band grow.

order n2

evaluate at x1

—

The shaded band is the guarantee: the worst $|f^{(n)}|$ times $|x-a|^n/n!$ . The gold dot is the secret $c$ that makes the leftover formula exact.

If you want the algebra · the error bound in two lines

Prove it · the remainder bound

Claim. If $|f^{(n)}(t)|\le M$ for every $t$ between $a$ and $x$ , then $|f(x)-P_{n-1}(x)|\le\dfrac{M}{n!}|x-a|^n$ .

1

Taylor's theorem (with

b=x

) gives a

c

between

a

and

x

with

f(x)-P_{n-1}(x)=\frac{f^{(n)}(c)}{n!}(x-a)^n.

This is an equality, not an estimate. Everything uncertain is inside

f^{(n)}(c)

.

2

Take absolute values;

c

is between

a

and

x

, so the hypothesis applies to it:

|f(x)-P_{n-1}(x)|=\frac{|f^{(n)}(c)|}{n!}|x-a|^n\le\frac{M}{n!}|x-a|^n.

That is the claim. ∎ The only information used about

c

is where it lives. Whatever it is, the derivative there is at most

M

in size.

Two examples worth remembering

Small angles. $\cos x\approx1-\dfrac{x^2}{2}$ . Every derivative of $\cos$ is at most $1$ in size, and the first missing term is order $4$ (the $x^3$ term of $\cos$ is zero). So the error is at most $\dfrac{x^4}{24}$ . At $x=0.5$ that is $0.0026$ ; the actual error is $0.00258$ .

The number $e$ . $e\approx1+1+\tfrac1{2!}+\cdots+\tfrac1{9!}$ (ten terms). The error is at most $\dfrac{3}{10!}\approx8\times10^{-7}$ . Six correct decimals, from a sum you could do by hand. Practice Problem 4 works both in full.

The realization

You never need the secret point. If $|f^{(n)}|\le M$ on the stretch, then $|R_n|\le\dfrac{M}{n!}|x-a|^n$ . A worst case turns the exact leftover into a guarantee you can compute.

Pause & predict

You copy $\sin x$ by just $x$ . (The $x^2$ term of $\sin$ is zero, so this copy already includes it.) At $x=0.3$ , what is the best guarantee?

In one sentence: you never learn $c$ , but a worst-case size $M$ for the next derivative turns the leftover into a firm promise, $|R_n|\le M|x-a|^n/n!$ .

7

MacLaurin's series — and why the factorial always wins

Imagine this

Rahul and Priya each start a game with ₹1. Rahul's money is multiplied by 5 every day. That multiplier never changes.

Priya's money is multiplied by the day number: ×1 on day 1, ×2 on day 2, ×3 on day 3, and so on. For the first few days Rahul's multiplier is bigger. From day 6 on, Priya's is bigger, and it keeps growing.

After $n$ days Rahul has $5^n$ , a power. Priya has $1\times2\times\cdots\times n=n!$ , a factorial. Rahul leads early, but in the long run the factorial always wins.

A Taylor series anchored at $0$ has its own name: a MacLaurin series. Two famous ones. For $e^x$ , every derivative is $e^x$ again, so every derivative at $0$ is $1$ :

e^x=1+x+\frac{x^2}{2!}+\frac{x^3}{3!}+\cdots

For $\cos x$ , the derivatives at $0$ go round in a cycle: $1,0,-1,0,1,\dots$ . Only even powers survive, with signs taking turns:

\cos x=1-\frac{x^2}{2!}+\frac{x^4}{4!}-\frac{x^6}{6!}+\cdots

A tiny example: $e\approx1+1+\tfrac12+\tfrac16=2.667$ . The true value is $2.718$ . Add more terms and you get closer.

But does the endless sum really reach the function? The leftover answers it. For $e^x$ the leftover is at most $e^{|x|}\,\dfrac{|x|^n}{n!}$ . So everything rests on $\dfrac{|x|^n}{n!}$ : Rahul's money divided by Priya's. At $x=5$ the terms grow for a while ( $1, 5, 12.5, 20.8, 26.0$ ), then the factorial takes over and they crash.

Try: Press ▶ race: each tower is the last one times $|x|/(n+1)$ . Push $x$ to 5 for a bigger hump. Then try ln(1 + x) with $x>1$ : the towers never come down.

drag the picture to orbit

x2

terms kept4

—

Bright towers are the terms you kept. The orange beads show the error still left. Once $n+1$ passes $|x|$ , every tower is shorter than the one before.

If you want the algebra · why the factorial beats every power

Prove it · the factorial beats every power

Claim. For any fixed $x$ , $\dfrac{|x|^n}{n!}\to0$ as $n\to\infty$ . Hence the MacLaurin series of $e^x$ , $\sin x$ and $\cos x$ converge to the function at every $x$ .

1

Look at how each term grows into the next:

\frac{|x|^{n+1}/(n+1)!}{|x|^n/n!}=\frac{|x|}{n+1}.

Going from term

n

to term

n+1

multiplies the power by

|x|

(the same number every time) and the factorial by

n+1

(a number that keeps growing).

2

Once

n+1\ge2|x|

, each step multiplies the term by at most

\tfrac12

. So beyond that point the terms shrink at least as fast as

\tfrac12,\tfrac14,\tfrac18,\dots

times a fixed number — and that goes to zero. Before the crossover the terms may grow (the "hump" in the widget: for

x=5

the largest term is at

n=4

or

5

); after it they are halved, at least, every step.

3

For

e^x

:

|R_n|\le e^{|x|}|x|^n/n!\to0

. For

\sin x

and

\cos x

: every derivative is bounded by

1

, so

|R_n|\le|x|^n/n!\to0

. In each case

P_{n-1}(x)\to f(x)

: the series converges to the function everywhere. ∎ Contrast

\ln(1+x)=x-\tfrac{x^2}{2}+\tfrac{x^3}{3}-\cdots

: its terms shrink only like

|x|^n/n

— no factorial in the denominator — and for

|x|>1

they grow. Taylor's remainder is what tells you the difference between "trust it everywhere" and "trust it only near the anchor". Unit 6 saw the cliff at

x=1

; now you know why it is there.

The realization

Each new term is the previous one times $\dfrac{|x|}{n+1}$ . Once $n+1$ is bigger than $|x|$ , every step shrinks the term. So the series for $e^x$ , $\sin x$ and $\cos x$ can be trusted at every $x$ . For $\ln(1+x)$ there is no factorial, and past $x=1$ it fails.

Pause & predict

For $e^x$ at $x=5$ , the terms $5^n/n!$ first grow, then shrink. Which term is the largest?

In one sentence: a MacLaurin series is Taylor at $0$ , and the series for $e^x$ , $\sin x$ and $\cos x$ always settle down because $n!$ eventually grows faster than any $|x|^n$ .

8

Two variables — walk along a line and it is one variable again

Imagine this

When you plan a walk or a bike ride on a maps app, it can show an elevation profile: a simple graph of height against distance along your route.

The land itself is two-dimensional and full of hills. But along one route, it becomes a single curve, and you can read it like any school graph.

That is the whole trick for two variables. Now the function is a landscape: $z=f(x,y)$ is the height at the point $(x,y)$ .

Stand at $P=(a,b)$ . You want the height at a nearby point $S=(a+h,\,b+k)$ : $h$ steps east, $k$ steps north. Walk from $P$ to $S$ in a straight line, with a clock $t$ that reads $0$ at $P$ and $1$ at $S$ . Your height along the walk is:

F(t)=f(a+th,\;b+tk)

This $F$ is an ordinary one-variable function: the elevation profile. So everything from the first half of the unit works on it. The chain rule gives its slope and its bend (the proofs are in the drawer below):

F'=h\,f_x+k\,f_y,\qquad F''=h^2f_{xx}+2hk\,f_{xy}+k^2f_{yy}

Here $f_x$ (read: "the slope in the $x$ -direction") and $f_y$ are the two slopes. $f_{xx}$ , $f_{xy}$ and $f_{yy}$ say how those slopes change: the landscape's bend.

Put these into one-variable Taylor with two terms plus the bend, and you get the landscape's copy, the quadratic clone. Everything on the right is measured at $P$ :

f(a+h,b+k)\approx f+h\,f_x+k\,f_y+\tfrac12\big(h^2f_{xx}+2hk\,f_{xy}+k^2f_{yy}\big)

A tiny example: $f=e^x\sin y$ at $P=(0,0)$ . There $f=0$ , $f_x=0$ , $f_y=1$ , $f_{xx}=0$ , $f_{xy}=1$ , $f_{yy}=0$ . Step $(h,k)=(0.1,0.2)$ : the clone gives $0.2+\tfrac12(2\times0.1\times0.2)=0.22$ . The truth is $0.21956$ .

Try: Load Problem A2: the clone misses the truth by only $0.00044$ . Then press double the step: the gap grows about tenfold.

drag the picture to orbit

base a0

base b0

step h0.5

step k0.6

—

The surface is $f(x,y)=e^x\sin y$ . The glowing curve is your walk from $P$ to $S$ , and the small chart unrolls it into the one-variable $F(t)$ . The glass sheet is the quadratic clone.

If you want the algebra · the walk's slope, its bend, and Taylor in two variables

Prove it · the first derivative along the walk

Claim. $F'(t)=h\,f_x+k\,f_y$ , the partials evaluated at the current point $(a+th,\,b+tk)$ .

1

F

is

f

composed with the straight-line map

t\mapsto(x(t),y(t))

. Unit 6's chain rule for a path says

F'(t)=f_x\,\frac{dx}{dt}+f_y\,\frac{dy}{dt}.

Two routes from

t

to

F

— through

x

and through

y

— multiply along each route and add across them (Unit 7's two rules).

2

On a straight line the velocities are constants:

dx/dt=h

,

dy/dt=k

. So

F'(t)=h\,f_x(a+th,\,b+tk)+k\,f_y(a+th,\,b+tk).

That is the claim. ∎ This is the directional derivative of Unit 6 in the direction

(h,k)

: steepness of the landscape as felt by a walker moving with velocity

(h,k)

.

Prove it · the second derivative along the walk

Claim. $F''(t)=h^2f_{xx}+2hk\,f_{xy}+k^2f_{yy}$ , again at the current point.

1

F'=hf_x+kf_y

is itself a function of the position on the walk, so apply the same chain rule to it:

F''(t)=\frac{\partial F'}{\partial x}\frac{dx}{dt}+\frac{\partial F'}{\partial y}\frac{dy}{dt}=\frac{\partial(hf_x+kf_y)}{\partial x}\,h+\frac{\partial(hf_x+kf_y)}{\partial y}\,k.

Same two routes, same two constant velocities

h

and

k

.

2

Differentiate inside the brackets,

h

and

k

being constants:

\frac{\partial(hf_x+kf_y)}{\partial x}=hf_{xx}+kf_{yx},\qquad \frac{\partial(hf_x+kf_y)}{\partial y}=hf_{xy}+kf_{yy}.

f_{yx}

means "first

y

, then

x

". For a twice continuously differentiable

f

, Clairaut's theorem says the order does not matter:

f_{yx}=f_{xy}

.

3

Multiply out and collect, using

f_{yx}=f_{xy}

:

F''(t)=h\,(hf_{xx}+kf_{xy})+k\,(hf_{xy}+kf_{yy})=h^2f_{xx}+2hk\,f_{xy}+k^2f_{yy}.

That is the claim. ∎ A quadratic form in

(h,k)

: the three second partials are its coefficients. This single line is the birthplace of the Hessian.

Prove it · Taylor in two variables, with its remainder

Claim. For some $c\in(0,1)$ , $\begin{aligned}f(a+h,\,b+k)&=f(a,b)+h\,f_x(a,b)+k\,f_y(a,b)\\ &\quad+\tfrac12\big(h^2f_{xx}+2hkf_{xy}+k^2f_{yy}\big)\Big|_{(a+ch,\ b+ck)}.\end{aligned}$

1

One-variable Taylor for

F

on

[0,1]

with

n=2

:

F(1)=F(0)+F'(0)\,(1-0)+\frac{F''(c)}{2!}\,(1-0)^2,\qquad c\in(0,1).

The theorem of the previous sections, applied to the walk. The remainder is evaluated at a secret clock-time

c

— i.e. at a secret point on the segment.

2

Substitute the three ingredients:

F(1)=f(a+h,b+k)

,

F(0)=f(a,b)

,

F'(0)=hf_x(a,b)+kf_y(a,b)

, and

F''(c)=(h^2f_{xx}+2hkf_{xy}+k^2f_{yy})

evaluated at

(a+ch,\,b+ck)

. That is the claim. ∎ The first-order part is Unit 6's tangent plane. The second-order part is new: the landscape's curvature, packed into three numbers and one quadratic form. Replace the secret point by

(a,b)

and you get the quadratic clone

Q

— the glass surface in the widget — with a third-order leftover, exactly as in one variable.

The realization

Walk in a straight line and a landscape becomes a one-variable curve $F(t)$ . Its bend is $h^2f_{xx}+2hk\,f_{xy}+k^2f_{yy}$ . Those three second slopes are the landscape's curvature, and they are where the Hessian is born.

Pause & predict

On $f=e^x\sin y$ , from $P=(0,0)$ , the step $(0.1,0.2)$ gives clone $0.22$ and truth $0.21956$ : a gap of $0.00044$ . Now double the step to $(0.2,0.4)$ . Roughly what happens to the gap?

Pause & predict

Why does the middle term appear as $2hk\,f_{xy}$ , with a $2$ , while the others are just $h^2f_{xx}$ and $k^2f_{yy}$ ?

In one sentence: along a straight walk the landscape is a one-variable curve, and its bend $h^2f_{xx}+2hkf_{xy}+k^2f_{yy}$ is where the Hessian comes from.

9

At a flat point, the second-order term decides

Imagine this

Open the spirit-level app on your phone and put it down in three places: at the bottom of a bowl, on top of a rounded hill, and in the middle of a mountain pass.

All three times it says the same thing: perfectly level.

So "level" alone can't tell a valley from a peak from a pass. To tell them apart, you need to know how the ground curves around you.

A critical point is a spot where the ground is level: both slopes are zero, $f_x=f_y=0$ . At such a spot, the slope part of the clone is zero. Only the bend part is left, and it decides everything:

f(a+h,\,b+k)-f(a,b)=\tfrac12\,Q,\qquad Q=h^2f_{xx}+2hk\,f_{xy}+k^2f_{yy}

The left side is "how much higher you are after the step". So:

⌣ $Q>0$ for every step: every step goes up. You are at the bottom of a bowl, a local minimum.
⌢ $Q<0$ for every step: every step goes down. You are on a hilltop, a local maximum.
⤨ $Q>0$ for some steps and $Q<0$ for others: a mountain pass, a saddle.

A tiny example: $f=x^2-y^2$ at $(0,0)$ . Here $f_{xx}=2$ , $f_{xy}=0$ , $f_{yy}=-2$ , so $Q=2h^2-2k^2$ . A step east $(1,0)$ gives $+2$ : up. A step north $(0,1)$ gives $-2$ : down. A saddle.

The step, seen from above. From a flat point

P

, a step

(h,k)

to

S

changes the height by

\tfrac12Q

, where the quadratic form is evaluated at a secret point on the segment. Whether that is up or down is the question the Hessian will answer.

One small worry. Strictly, $Q$ is measured at a secret point on the way, not at $P$ itself. For small steps it doesn't matter. The secret point is so close to $P$ that the sign stays the same. So the three numbers at the critical point decide.

If you want the algebra · why the secret point cannot flip the sign

Prove it · the secret point cannot flip the sign

Claim. Suppose the second partials are continuous and $Q_0(h,k)=h^2f_{xx}(a,b)+2hkf_{xy}(a,b)+k^2f_{yy}(a,b)$ is positive for every direction $(h,k)\ne(0,0)$ . Then for all sufficiently small steps, $f(a+h,b+k)-f(a,b)>0$ . (The negative case is identical with signs flipped.)

1

Scale out the step size. Write

(h,k)=r(u,v)

with

u^2+v^2=1

and

r>0

. Then

Q_0(h,k)=r^2\,Q_0(u,v)

, and

Q_0(u,v)

is a continuous function on the unit circle, which is closed and bounded, so it attains a minimum

m

. Because

Q_0>0

at every point of the circle,

m>0

. A quadratic form's sign does not depend on the length of the step, only on its direction. The minimum over directions exists and is positive — that is the number the argument hangs on.

2

Compare the secret point with the anchor. The coefficients

f_{xx},f_{xy},f_{yy}

are continuous, so there is a radius

\rho

such that within distance

\rho

of

(a,b)

each coefficient differs from its value at

(a,b)

by less than

m/4

. For the form at the secret point this gives, for any unit direction,

\big|Q_{c}(u,v)-Q_0(u,v)\big|\le|u^2|\tfrac m4+|2uv|\tfrac m4+|v^2|\tfrac m4\le\tfrac m4(1+1+1)<m.

Using

u^2\le1

,

v^2\le1

,

|2uv|\le u^2+v^2=1

. The secret point lies on the segment, hence within

r<\rho

of the anchor, so the estimate applies to it.

3

Hence

Q_c(u,v)>Q_0(u,v)-m\ge m-m=0

for every direction, and so

f(a+h,b+k)-f(a,b)=\tfrac12\,r^2\,Q_c(u,v)>0\qquad\text{whenever }0<r<\rho.

That is the claim. ∎ "Sufficiently small" means

r<\rho

, a definite radius that depends only on how fast the second partials change. Inside that radius, the anchor's three numbers rule.

The realization

At a level spot, the height change after a small step is $\tfrac12Q$ . If $Q$ is always positive, it's a valley; always negative, a peak; both, a pass. The three second slopes at the spot decide which.

Pause & predict

At a critical point, why can the slope part never help you tell a minimum from a maximum?

In one sentence: at a level spot the slope part is zero, so the bend $Q$ alone decides whether you are in a valley, on a peak, or at a pass.

10

Taylor in matrix form — and how to remember it

Imagine this

Think of a sandwich, or a vada pav: bread, filling, bread.

The formula in this section has exactly that shape. The step you take is the bread, on both sides. A small table of curvatures is the filling. Bite through all three, and you get one number: how much the ground bends along your step.

Three numbers, $f_{xx}$ , $f_{xy}$ and $f_{yy}$ , decided everything in the last section. Let's give them a home: a 2×2 table called the Hessian, $H$ :

H=\begin{bmatrix}f_{xx} & f_{xy}\\ f_{xy} & f_{yy}\end{bmatrix}

It is symmetric: $f_{xy}$ sits in both corners, because the order of the two derivatives doesn't matter.

Now make the sandwich. Put the step $\mathbf h=(h,k)$ on both sides of $H$ (the little ${}^{\top}$ means "lay the column on its side as a row"):

\mathbf h^{\top}H\,\mathbf h=\begin{bmatrix}h & k\end{bmatrix}\begin{bmatrix}f_{xx} & f_{xy}\\ f_{xy} & f_{yy}\end{bmatrix}\begin{bmatrix}h\\ k\end{bmatrix}=h^2f_{xx}+2hk\,f_{xy}+k^2f_{yy}

That is exactly the walk's bend $Q$ . The two corner entries each give $hk\,f_{xy}$ , and together they make the $2hk\,f_{xy}$ .

A tiny example: $H=\begin{bmatrix}2&1\\1&2\end{bmatrix}$ and $\mathbf h=(1,-1)$ . Then $\mathbf h^{\top}H\mathbf h=2(1)^2+2(1)(-1)(1)+2(-1)^2=2-2+2=2$ .

With the Hessian, the whole Taylor formula fits on one line, for any number of variables. It looks just like the one-variable version. Here $\nabla f$ (read: "grad f") is the gradient, the list of all the slopes:

f(\mathbf x_0+\mathbf h)\approx f(\mathbf x_0)+\nabla f^{\top}\mathbf h+\tfrac12\,\mathbf h^{\top}H\,\mathbf h

How to remember it: move each piece up one level, and feed it one copy of the step for each level.

Piece	One variable	Many variables	Copies of the step
height	a number, $f(a)$	a number, $f(\mathbf x_0)$	none
slope	a number, $f'(a)$	a list, the gradient $\nabla f$	one: $\nabla f^{\top}\mathbf h$
bend	a number, $f''(a)$	a table, the Hessian $H$	two: $\mathbf h^{\top}H\mathbf h$

Rule of thumb: every term must end up as one plain number. A list needs one copy of the step to become a number. A table needs two, one on each side. The $\tfrac12$ is $1/2!$ , exactly as before.

The rank ladder. Promote each derivative by one rank — number, list, table — and feed it as many copies of the step

\mathbf h

as it has indices. The shapes do the remembering for you.

Try: Hover the two corner entries $f_{12}$ and $f_{21}$ : both make the same product $h_1h_2$ . That is the $2$ in $2hk\,f_{xy}$ . Then try three variables.

—

Each entry of $H$ is multiplied by the step part of its row and of its column. Mirror-image entries give the same product, so they merge.

If you want the algebra · the matrix form, for any number of variables

Prove it · the matrix form, from the line trick, for any number of variables

Claim. For $F(t)=f(\mathbf x_0+t\mathbf h)$ : $\ F'(t)=\nabla f^{\top}\mathbf h$ and $F''(t)=\mathbf h^{\top}H\mathbf h$ , both evaluated at $\mathbf x_0+t\mathbf h$ . Hence $f(\mathbf x_0+\mathbf h)=f(\mathbf x_0)+\nabla f(\mathbf x_0)^{\top}\mathbf h+\tfrac12\mathbf h^{\top}H(\mathbf x_0+c\mathbf h)\mathbf h$ for some $c\in(0,1)$ .

1

Along the walk the

i

-th coordinate is

x_i(t)=x_{0,i}+t\,h_i

, so

dx_i/dt=h_i

. The chain rule with

n

routes (Unit 7's add-across-paths, one path per coordinate):

F'(t)=\sum_{i=1}^{n}\frac{\partial f}{\partial x_i}\,\frac{dx_i}{dt}=\sum_{i=1}^{n}f_i\,h_i=\nabla f^{\top}\mathbf h.

A sum of (partial) × (step component) over one index is exactly a dot product — a row times a column.

2

Differentiate again. Each

f_i

is itself a function of position on the walk, so it gets its own chain rule with the same constant velocities:

F''(t)=\sum_{i=1}^{n}h_i\,\frac{d}{dt}f_i=\sum_{i=1}^{n}h_i\sum_{j=1}^{n}\frac{\partial f_i}{\partial x_j}\,h_j=\sum_{i=1}^{n}\sum_{j=1}^{n}h_i\,H_{ij}\,h_j=\mathbf h^{\top}H\,\mathbf h.

A double sum of (entry) × (left step component) × (right step component) is exactly a row × matrix × column. Here

H_{ij}=\partial^2 f/\partial x_j\partial x_i=\partial^2 f/\partial x_i\partial x_j

by Clairaut, which is why

H

is symmetric and why the two cross terms

h_1h_2H_{12}

and

h_2h_1H_{21}

merge into

2h_1h_2H_{12}

in two variables.

3

Apply one-variable Taylor with

n=2

to

F

on

[0,1]

:

F(1)=F(0)+F'(0)+\tfrac12F''(c)

for some

c\in(0,1)

. Substitute steps 1 and 2:

f(\mathbf x_0+\mathbf h)=f(\mathbf x_0)+\nabla f(\mathbf x_0)^{\top}\mathbf h+\tfrac12\,\mathbf h^{\top}H(\mathbf x_0+c\mathbf h)\,\mathbf h.

That is the claim. ∎ Nothing about the argument used

n=2

. The matrix form is not a new theorem — it is the one-variable theorem read along a line, with the chain rule turning derivatives into a gradient and a Hessian. Keep going and the third derivative would be an

n\times n\times n

cube of numbers eating three copies of

\mathbf h

; linear algebra stops being comfortable there, which is exactly why "second-order" is where most of optimisation stops too.

The realization

$f(\mathbf x_0+\mathbf h)\approx f(\mathbf x_0)+\nabla f^{\top}\mathbf h+\tfrac12\mathbf h^{\top}H\mathbf h$ . It is the one-variable formula with each piece moved up a level: number, list, table. The step goes in once for the list and twice, like bread, for the table.

Pause & predict

You remember $f(a)+f'(a)h+\tfrac12f''(a)h^2$ . Which is its many-variable version?

In one sentence: the Hessian packs the bends into a table, and the step sandwiches it, so many-variable Taylor reads exactly like the one-variable formula.

11

The judge of extrema — the Hessian

Imagine this

Put a marble on three shapes: a bowl, an upside-down bowl, and a potato chip shaped like a horse saddle.

In the bowl, any nudge rolls it back: a minimum. On the upside-down bowl, any nudge sends it rolling off: a maximum.

On the chip, it depends on the direction. Nudge it along the chip and it rolls back. Nudge it across and it falls off. That is a saddle.

At a level spot, the height change after a step is $\tfrac12\,\mathbf h^{\top}H\mathbf h$ . So "bowl, upside-down bowl, or chip?" becomes one question: is $\mathbf h^{\top}H\mathbf h$ positive for every step, negative for every step, or both?

In Unit 4's words: is $H$ positive-definite, negative-definite, or indefinite? A small table of numbers has become a judge.

For a 2×2 Hessian there is a quick test. Work out one number, the discriminant $D=f_{xx}f_{yy}-f_{xy}^2$ . (It is the determinant of $H$ .) Then read the table:

$D$	$f_{xx}$	Shape	Verdict
$>0$	$>0$	bowl	local minimum
$>0$	$<0$	upside-down bowl	local maximum
$<0$	any	saddle (the chip)	saddle point
$=0$	any	too flat to tell	no verdict

A tiny example: $f=x^2+xy+y^2$ at $(0,0)$ . Here $f_{xx}=2$ , $f_{xy}=1$ , $f_{yy}=2$ . So $D=2\times2-1^2=3>0$ and $f_{xx}>0$ : a bowl, a local minimum.

Try: Start from the bowl and slowly raise $f_{xy}$ . The bowl stretches and tilts. The moment $f_{xy}^2$ passes $f_{xx}f_{yy}$ , it tears open into a saddle.

drag the picture to orbit

f_xx2

f_xy0

f_yy2

—

The surface is just the bend part at a level spot: $z=\tfrac12(f_{xx}h^2+2f_{xy}hk+f_{yy}k^2)$ . The three tabs read the same table three ways, and they always agree.

Why does $D$ work? A bit of completing the square shows that $f_{xx}\,Q$ is a square plus $D$ times another square. Squares are never negative, so the sign of $D$ is in charge. The drawer has the details.

If you want the algebra · the judge's identity and the three verdicts

Prove it · the judge's identity

Claim. $f_{xx}\,Q=(h\,f_{xx}+k\,f_{xy})^2+\big(f_{xx}f_{yy}-f_{xy}^2\big)\,k^2$ . Writing $D=f_{xx}f_{yy}-f_{xy}^2$ (the discriminant, which is $\det H$ ): $\ f_{xx}Q=(hf_{xx}+kf_{xy})^2+Dk^2$ .

1

Multiply the form by

f_{xx}

:

f_{xx}Q=h^2f_{xx}^2+2hk\,f_{xx}f_{xy}+k^2f_{xx}f_{yy}.

Every term picks up one factor

f_{xx}

. The first two now look like the start of

(hf_{xx}+kf_{xy})^2

.

2

Expand the square that those two terms suggest:

(hf_{xx}+kf_{xy})^2=h^2f_{xx}^2+2hk\,f_{xx}f_{xy}+k^2f_{xy}^2.

It matches step 1 in the first two terms; the third term is

k^2f_{xy}^2

instead of

k^2f_{xx}f_{yy}

.

3

Subtract step 2 from step 1; only the

k^2

terms survive:

f_{xx}Q-(hf_{xx}+kf_{xy})^2=k^2f_{xx}f_{yy}-k^2f_{xy}^2=\big(f_{xx}f_{yy}-f_{xy}^2\big)k^2=Dk^2.

That is the claim. ∎ The right side is a square plus

D

times another square. Squares are never negative — so the sign of

f_{xx}Q

is controlled by the sign of

D

.

Prove it · the three verdicts

Claim. At a critical point: $D>0,\ f_{xx}>0\Rightarrow$ local minimum; $D>0,\ f_{xx}<0\Rightarrow$ local maximum; $D<0\Rightarrow$ saddle. $D=0$ : no verdict.

1

$D>0$ . Then

f_{xx}\ne0

(otherwise

D=-f_{xy}^2\le0

). By the identity,

f_{xx}Q=(\text{square})+D\,k^2\ge0

, with equality only if both

k=0

and

hf_{xx}=0

, i.e.

h=k=0

. So for every non-zero step,

f_{xx}Q>0

:

Q

has the sign of

f_{xx}

.

f_{xx}>0

:

Q>0

for all steps — every small step goes up — a bowl, a local minimum.

f_{xx}<0

:

Q<0

for all steps — a dome, a local maximum.

2

$D<0$ , with $f_{xx}\ne0$ . Step along the

x

-axis,

(h,k)=(1,0)

:

Q=f_{xx}

. Step along

(h,k)=(-f_{xy},\,f_{xx})

: the square in the identity is zero (

-f_{xy}f_{xx}+f_{xx}f_{xy}=0

), leaving

f_{xx}Q=Df_{xx}^2<0

, so

Q

has the sign opposite to

f_{xx}

. Two directions, two opposite signs: up one way, down another. That is a saddle. (If

f_{xx}=0

but

f_{yy}\ne0

, run the same argument with the roles of

h

and

k

swapped; if both vanish,

D=-f_{xy}^2<0

means

f_{xy}\ne0

and

Q=2hkf_{xy}

already changes sign between

(1,1)

and

(1,-1)

.)

3

$D=0$ . The identity reads

f_{xx}Q=(hf_{xx}+kf_{xy})^2\ge0

:

Q

never changes sign, but it is zero along a whole direction, and there the second-order term says nothing — the decision passes to third- and fourth-order terms, which this test does not see. Practice Problem 8 shows

x^4+y^4

(a minimum) and

x^4-y^4

(a saddle) with the same zero Hessian. ∎ "No verdict" is a fair answer, not a failure: the judge only hears second-order evidence.

Pause & predict

At a critical point you find $f_{xx}=-2$ , $f_{yy}=-2$ , $f_{xy}=1$ . What kind of point is it?

$D$ is one way to read the table. Unit 4 gave you two more, and they always agree:

①Sylvester's test. Look at the top-left corners: first $f_{xx}$ alone, then the whole determinant $D$ . Both positive means a bowl. For a 2×2 table, that is the $D$ -test above.
②Eigenvalues. Turn slowly on the spot and feel how sharply the ground bends in each direction. The strongest bend and the weakest bend are the two eigenvalues $\lambda_1\ge\lambda_2$ of $H$ . Both positive: bowl. Both negative: upside-down bowl. One of each: saddle.

And they fit together: $D=\lambda_1\lambda_2$ . Two positives or two negatives give $D>0$ ; one of each gives $D<0$ . The directions where the bend is strongest and weakest are the eigenvectors. The compass below lets you turn and feel it.

Try: Press ▶ sweep. On a saddle the needle crosses zero four times per turn; on a bowl, never. Then drag $f_{xy}$ and watch the longest and shortest spokes turn.

f_xx2

f_xy0

f_yy-2

direction θ30°

—

Green petals: the ground bends up in that direction. Red: it bends down. The longest and shortest spokes are the eigenvalues, pointing along the eigenvectors.

If you want the algebra · eigenvalues are the extreme bends, and the "upside-down" signs

Prove it · eigenvalues are the extreme curvatures, and their signs are the verdict

Claim. Let $H$ be symmetric with eigenvalues $\lambda_1\ge\lambda_2$ and orthonormal eigenvectors $\mathbf e_1,\mathbf e_2$ . Then for every unit direction $\mathbf u$ , $\ \lambda_2\le\mathbf u^{\top}H\mathbf u\le\lambda_1$ , with equality along $\mathbf e_2$ and $\mathbf e_1$ . Consequently $H$ is positive-definite $\iff\lambda_1,\lambda_2>0$ , negative-definite $\iff$ both negative, indefinite $\iff$ opposite signs; and $D=\det H=\lambda_1\lambda_2$ , $f_{xx}+f_{yy}=\lambda_1+\lambda_2$ .

1

Unit 4's spectral theorem:

H=Q\Lambda Q^{\top}

with

Q=[\mathbf e_1\ \mathbf e_2]

orthogonal and

\Lambda=\mathrm{diag}(\lambda_1,\lambda_2)

. Write any direction in the eigenbasis,

\mathbf u=\alpha\mathbf e_1+\beta\mathbf e_2

, i.e.

Q^{\top}\mathbf u=(\alpha,\beta)^{\top}

. Then

\mathbf u^{\top}H\mathbf u=(Q^{\top}\mathbf u)^{\top}\Lambda(Q^{\top}\mathbf u)=\lambda_1\alpha^2+\lambda_2\beta^2.

In the eigenbasis the cross term disappears: the quadratic form is just two squares, each weighted by an eigenvalue.

2

For a unit direction

\alpha^2+\beta^2=1

, so

\lambda_1\alpha^2+\lambda_2\beta^2

is a weighted average of

\lambda_1

and

\lambda_2

with non-negative weights summing to

1

. It therefore lies between them:

\lambda_2\le\mathbf u^{\top}H\mathbf u\le\lambda_1,

with

\lambda_1

reached at

\alpha=1

(direction

\mathbf e_1

) and

\lambda_2

at

\beta=1

(direction

\mathbf e_2

). The compass rose is exactly this: the needle's reading swings between the two eigenvalues, touching them along the eigenvectors.

3

Signs.

Q>0

for every direction

\iff

the minimum reading

\lambda_2>0

\iff

both eigenvalues positive.

Q<0

for every direction

\iff\lambda_1<0

. Opposite signs

\iff

the needle crosses zero

\iff

saddle. Definiteness is a statement about the smallest (or largest) curvature, and those are eigenvalues.

4

Bridge to the discriminant:

\det H=\det(Q\Lambda Q^{\top})=\det\Lambda=\lambda_1\lambda_2=D

, and

\operatorname{tr}H=f_{xx}+f_{yy}=\lambda_1+\lambda_2

. So

D<0\iff

opposite signs

\iff

saddle; and

D>0

with

f_{xx}>0

forces

f_{yy}>f_{xy}^2/f_{xx}\ge0

, hence

\lambda_1+\lambda_2>0

with

\lambda_1\lambda_2>0

: both positive. The three readings of

H

— discriminant, Sylvester, eigenvalues — are one verdict. ∎ Which reading to use is a matter of convenience:

D

is fastest by hand; eigenvalues also tell you how bowl-shaped (their ratio is the condition number that will govern gradient descent in Unit 9); Sylvester scales to

n\times n

Hessians where "

D

" no longer exists.

Prove it · negative-definite means the minors alternate

Claim. $H$ is negative-definite $\iff$ $f_{xx}<0$ and $D>0$ — the leading minors alternate in sign, starting negative.

1

H

is negative-definite exactly when

-H

is positive-definite (multiply

\mathbf h^{\top}H\mathbf h<0

by

-1

). A dome flipped upside down is a bowl.

2

Apply Sylvester to

-H=\begin{bmatrix}-f_{xx}&-f_{xy}\\-f_{xy}&-f_{yy}\end{bmatrix}

: its minors are

-f_{xx}

and

\det(-H)=(-1)^2\det H=D

. Both must be positive:

f_{xx}<0

and

D>0

. That is the claim. ∎ Flipping the sign of a

k\times k

block multiplies its determinant by

(-1)^k

— that is why, for a larger Hessian, the minors of a negative-definite matrix alternate: negative for odd

k

, positive for even

k

. The eigenvalues are all negative, as the lecture says.

The realization

The Hessian is the judge. Work out $D=f_{xx}f_{yy}-f_{xy}^2$ : positive with $f_{xx}>0$ is a bowl, positive with $f_{xx}<0$ is an upside-down bowl, negative is a saddle. The eigenvalues say the same thing: they are the strongest and weakest bends, and $D=\lambda_1\lambda_2$ .

Pause & predict

A Hessian has eigenvalues $\lambda_1=7.24$ and $\lambda_2=2.76$ at a critical point (Practice Problem 7). Without working out anything else, what do you know?

Pause & predict

For $f=e^x\sin y$ at $(0,0)$ , the Hessian is $\begin{bmatrix}0&1\\1&0\end{bmatrix}$ : both pure bends are zero. Is the ground flat there, or a saddle?

In one sentence: the Hessian's discriminant, its corner determinants and its eigenvalues are three ways to read one verdict — bowl, upside-down bowl, or saddle.

12

The judge at work — six landscapes

Imagine this

A trekking map of a hill station marks the special spots: peaks, lake beds and passes.

The map-maker finds them all the same way. First she finds every level spot. Then she checks how the ground curves around each one, one spot at a time.

That is the whole recipe. Three moves, every time:

1Find the level spots. Solve $f_x=0$ and $f_y=0$ together.
2Work out $f_{xx}$ , $f_{xy}$ , $f_{yy}$ at each spot separately. For anything beyond a simple bowl, they change from spot to spot.
3Read $D$ , then $f_{xx}$ . Give the verdict.

The widget has six landscapes: the companion's two worked examples, the practice set's landscapes, and two where the judge must stay silent.

Try: Load x³ + y³ − 3xy + 1: two level spots, one saddle and one minimum. Click each dot and compare. Then load the two $x^4$ landscapes, where the Hessian is all zeros and stays silent.

drag the picture to orbit · tap a dot to select it

—

For each landscape the widget solves $\nabla f=\mathbf 0$ , then judges every level spot with its own Hessian. The glass shell is the quadratic clone at the chosen spot.

One formula, two verdicts

Take $f=x^3+y^3-3xy+1$ . Move 1: $f_x=3x^2-3y=0$ and $f_y=3y^2-3x=0$ give $y=x^2$ and $x=y^2$ . So $x=x^4$ , and the level spots are $(0,0)$ and $(1,1)$ .

Moves 2 and 3: here $f_{xx}=6x$ , $f_{xy}=-3$ , $f_{yy}=6y$ . At $(0,0)$ : $D=0-9=-9$ , a saddle. At $(1,1)$ : $D=36-9=27$ with $f_{xx}=6>0$ , a local minimum, with height $0$ .

Only local, though: at $(-2,0)$ the height is $-7$ , lower still. The judge only rules on the spot's close neighbourhood.

Now the two landscapes where the judge must stay silent: $g=x^4+y^4$ and $h=x^4-y^4$ . Both have one level spot, the origin, and there every second slope is zero, so $D=0$ .

Yet they are different shapes. $g$ is never negative and is zero only at the origin: a minimum. $h$ goes up along the $x$ -axis and down along the $y$ -axis: a saddle. The difference is in the fourth-order terms, which the Hessian cannot see. When $D=0$ , you need another test: look at higher terms, or reason about the function's shape directly.

The realization

Find every level spot. Work out the Hessian at each one, because it changes from place to place. Read $D$ , then $f_{xx}$ . And when $D=0$ , the bends have run out of evidence, so look further.

Pause & predict

For $f=x^3-3xy+3y^2$ (Practice Problem 6), the level spots are $(0,0)$ and $(\tfrac12,\tfrac14)$ , with $f_{xx}=6x$ , $f_{xy}=-3$ , $f_{yy}=6$ . Which verdict goes where?

In one sentence: find every level spot, judge each one with its own Hessian, and when $D=0$ admit that the bends cannot decide.

13

Why a learning machine cares — curvature, conditioning, and the saddle problem

Imagine this

You are walking down into a long, narrow valley in thick fog. You can only feel the slope under your feet.

The side walls are steep, but the valley floor slopes gently. So each downhill step points mostly at the nearest wall. You bounce from wall to wall, and crawl along the valley.

This is how a learning machine trains.

Training is walking downhill. The machine's error, called the loss $L(\mathbf w)$ , depends on millions of weights $\mathbf w$ . Every step of gradient descent (Unit 9) is a step down that landscape. At the bottom the slopes are zero, and the Hessian is the judge from this unit. All eigenvalues positive: a true minimum. Mixed signs: a saddle you can still slide off.

The bends set the speed limit. Near the bottom, the loss is its own quadratic clone. The steepest bend $\lambda_{\max}$ forces small steps, or you overshoot across the valley. The gentlest bend $\lambda_{\min}$ then makes progress along the floor painfully slow. The ratio $\lambda_{\max}/\lambda_{\min}$ , the condition number, says how narrow the valley is. Momentum, RMSProp and Adam (Unit 11) are all ways to cope with a bad Hessian without ever computing it.

The clone suggests a jump. If you trusted the quadratic clone completely, you would not creep downhill. You would jump straight to the clone's lowest point. That is Newton's method, and its step is:

\mathbf h^{\star}=-H^{-1}\nabla f

Read it: "undo the bends, then go against the slope". Where the valley is steep, the step shrinks. Where it is flat, the step grows. For a perfect bowl, one jump lands exactly at the bottom.

If you want the algebra · Newton's step is the bottom of the clone

Prove it · Newton's step is the bottom of the quadratic clone

Claim. The quadratic clone of $f$ at $\mathbf x_0$ , $\ Q(\mathbf h)=f(\mathbf x_0)+\nabla f^{\top}\mathbf h+\tfrac12\mathbf h^{\top}H\mathbf h$ , is minimised (when $H$ is positive-definite) at $\mathbf h^{\star}=-H^{-1}\nabla f$ .

1

Differentiate

Q

with respect to

\mathbf h

, using Unit 7's cookbook (

\nabla_{\mathbf h}(\mathbf a^{\top}\mathbf h)=\mathbf a

,

\ \nabla_{\mathbf h}(\tfrac12\mathbf h^{\top}H\mathbf h)=H\mathbf h

for symmetric

H

):

\nabla_{\mathbf h}Q=\nabla f+H\mathbf h.

The clone is a quadratic, so its gradient is linear in

\mathbf h

.

2

Set it to zero and solve:

H\mathbf h^{\star}=-\nabla f

, so

\mathbf h^{\star}=-H^{-1}\nabla f

. The clone's Hessian is

H

itself, positive-definite, so this flat point of

Q

is its minimum — by this unit's own verdict. ∎ Gradient descent moves along

-\nabla f

, scaled by a step size you must guess. Newton moves along

-H^{-1}\nabla f

: the curvature rescales every direction — big steps where the valley is flat, small where it is steep. For a true quadratic it lands on the minimum in one jump. The price is

H

: for a million weights it is unaffordable, which is why the whole optimiser zoo exists.

Two ways down an elongated bowl. Contours of

\tfrac12(9u^2+v^2)

, condition number

\lambda_1/\lambda_2=9

. Gradient descent (orange, step size

0.2

) must keep its steps short for the steep axis, so it bounces between the walls and crawls along the valley; Newton's step (green) uses

H^{-1}

to rescale every direction and lands on the minimum at once. Unit 9 begins where the orange path does.

The realization

Near its bottom, a loss is its own quadratic clone. The Hessian's eigenvalues say how narrow the valley is, and so how hard plain descent will be. Its inverse turns the clone into Newton's one jump, $-H^{-1}\nabla f$ , which is powerful but costly for millions of weights.

Pause & predict

Gradient descent steps by $-\eta\nabla f$ ( $\eta$ , read "eta", is a fixed step size). Newton steps by $-H^{-1}\nabla f$ . In a stretched bowl with $\lambda_1$ much bigger than $\lambda_2$ , why doesn't Newton zig-zag?

In one sentence: near its bottom a loss is its own quadratic clone, so the Hessian decides how hard training is, and its inverse gives Newton's one-jump step.

14

What to carry forward

One level spot on a hiking trail built this whole unit. Everything else was the same move, Rolle, used again: tilted, stacked into a ladder, then walked along a line.

Idea	The one-line version	Where it returns
Rolle	Same height at both ends + smooth ⇒ a flat spot between	every existence argument in calculus
Mean value theorem	$f(b)=f(a)+f'(c)(b-a)$ — Rolle, tilted	Unit 9 · why a descent step works
Taylor with remainder	$f(x)=P_{n-1}(x)+\dfrac{f^{(n)}(c)}{n!}(x-a)^n$ , $c$ between $a$ and $x$	every "local approximation" from here on
The bound	$\|R_n\|\le M\|x-a\|^n/n!$ — you never need $c$	numerical methods, error budgets
MacLaurin	Taylor at $0$ ; $e^x,\sin,\cos$ converge everywhere because $n!$ wins	how your computer evaluates functions
The line trick	$F(t)=f(a+th,b+tk)$ : two variables become one	directional derivatives; convexity proofs (Unit 14)
The Hessian	$H=\begin{bmatrix}f_{xx}&f_{xy}\\f_{xy}&f_{yy}\end{bmatrix}$ ; $Q=\mathbf h^{\top}H\mathbf h$	Units 9–11 · the shape of every loss valley
The verdict	$D>0,f_{xx}>0$ min · $D>0,f_{xx}<0$ max · $D<0$ saddle · $D=0$ silent	classifying critical points, always
Three readings	$D=\det H=\lambda_1\lambda_2$ ; Sylvester's minors; eigenvalue signs	Unit 14 · positive-definite kernels and convexity
Newton's step	$-H^{-1}\nabla f$ : jump to the clone's bottom	Units 10–11 · second-order methods and why we avoid them

The realization

Every smooth function is, up close, a polynomial, and Taylor's theorem tells you exactly how far "up close" reaches. The leftover $f^{(n)}(c)(x-a)^n/n!$ came from a ladder of level spots. In two variables, the bend part is a table, the Hessian, whose signs say whether a level spot is the bottom of a bowl. When a training run says it has found a minimum, this unit is how you can check.

In one sentence: Taylor copies a function near a point with a guaranteed error, and the Hessian in that copy judges every valley a learning machine finds.

15

Practice arena — the unit's problem set, solved in full

Twelve problems, each solved in full. Six come from the practice set (three on Taylor, three on the Hessian), three from the companion guide, and three are new: an error-bound workout (Problem 4), a hunt for the secret point $c$ (Problem 5), and a proof workout for the judge's identity (Problem 12). Every number is machine-checked.

Two habits do most of the work. For a Taylor polynomial, list the derivatives at the anchor first, and keep the $k!$ . For a critical point, work out the Hessian at each point separately, then read $D$ first and $f_{xx}$ second.

Problem 1easy

Let $f(x)=\ln x$ . (a) Compute $f$ and its first four derivatives at $x=1$ . (b) Write the fourth-order Taylor polynomial $P_4(x)$ of $f$ about the point $a=1$ . (c) Use $P_4$ to estimate $\ln 1.1$ and compare with the true value.

What this tests. The recipe itself — differentiate, evaluate at the centre, divide by $k!$ , attach $(x-a)^k$ . Plan. Make a small table of derivatives first and evaluate every one at the centre before assembling anything; then look for the pattern in the coefficients, and finish by plugging in a nearby $x$ to see the polynomial earn its keep.

Show the full solution

(a) Step 1 — differentiate four times. Each derivative of

\ln x

is a power of

x

, so the pattern is easy to carry:

\begin{aligned} f&=\ln x, & f'&=x^{-1}, & f''&=-x^{-2},\\ f'''&=2x^{-3}, & f^{(4)}&=-6x^{-4}. \end{aligned}

At the centre

x=1

every power of

x

is

1

:

f(1)=0,\quad f'(1)=1,\quad f''(1)=-1,\quad f'''(1)=2,\quad f^{(4)}(1)=-6.

(b) Step 2 — divide by $k!$ and attach the powers of $(x-1)$ . The coefficient of

(x-1)^k

is

f^{(k)}(1)/k!

:

\begin{aligned} \frac{f'(1)}{1!}&=1, & \frac{f''(1)}{2!}&=-\frac12, & \frac{f'''(1)}{3!}&=\frac26=\frac13, & \frac{f^{(4)}(1)}{4!}&=-\frac{6}{24}=-\frac14, \end{aligned}

so

P_4(x)=(x-1)-\tfrac12(x-1)^2+\tfrac13(x-1)^3-\tfrac14(x-1)^4.

The coefficients are

(-1)^{k+1}/k

— the

k!

in the denominator has eaten the

(k-1)!

that the derivatives produced.

Step 3 — the quicker route. Put

u=x-1

. Then

\ln x=\ln(1+u)

, whose MacLaurin series is a standard one:

\ln(1+u)=u-\frac{u^2}{2}+\frac{u^3}{3}-\frac{u^4}{4}+\cdots

Truncate at degree 4 and write

u=x-1

back: the same

P_4

, in one line, with no differentiation at all. Expanding about

a

is always the same as expanding about

0

in the shifted variable

x-a

.

(c) Step 4 — check at $x=1.1$ , where

u=0.1

:

\begin{aligned} P_4(1.1)&=0.1-\frac{0.01}{2}+\frac{0.001}{3}-\frac{0.0001}{4}\\ &=0.1-0.005+0.000333-0.000025=0.095308. \end{aligned}

The true value is

\ln 1.1=0.095310

; the error is

1.8\times10^{-6}

, about the size of the first term we dropped,

u^5/5=2.0\times10^{-6}

. Four terms, six correct decimals.

(a) $f(1)=0,\ f'(1)=1,\ f''(1)=-1,\ f'''(1)=2,\ f^{(4)}(1)=-6$ . (b) $P_4(x)=(x-1)-\tfrac12(x-1)^2+\tfrac13(x-1)^3-\tfrac14(x-1)^4$ . (c) $P_4(1.1)=0.095308$ against $\ln1.1=0.095310$ , error $1.8\times10^{-6}$ .

Remember

The coefficient is $f^{(k)}(a)/k!$ , never $f^{(k)}(a)$ alone, and the variable is $(x-a)$ , never $x$ . Those two slips lose most of the marks on this type. When the function is a shifted standard one — $\ln(1+u)$ , $e^{u}$ , $\sqrt{1+u}$ — substitute first and copy the known series.

Problem 2easy

Let $f(x)=\sqrt{x}$ , expanded about the point $a=4$ . (a) Find the first-order (linear) polynomial $P_1(x)$ and the second-order (quadratic) polynomial $P_2(x)$ . (b) Use each to estimate $\sqrt{4.2}$ . (c) Explain why the quadratic estimate is the better one.

What this tests. That $P_1$ is the tangent line, that $P_2$ adds the curvature, and that the sign of $f''$ tells you on which side of the curve the tangent sits. Plan. Compute $f,f',f''$ at the centre; halve $f''$ when it becomes a coefficient; plug in the increment $x-4=0.2$ rather than $x$ itself; compare both estimates with the true value at the end.

Show the full solution

(a) Step 1 — derivatives at the centre. With

f=x^{1/2}

:

\begin{aligned} f(4)&=2, & f'(x)&=\tfrac12x^{-1/2} &&\Rightarrow f'(4)=\tfrac14,\\ && f''(x)&=-\tfrac14x^{-3/2} &&\Rightarrow f''(4)=-\tfrac14\cdot\tfrac18=-\tfrac1{32}. \end{aligned}

Step 2 — assemble. The quadratic coefficient is

f''(4)/2!=-\tfrac1{64}

, so

P_1(x)=2+\tfrac14(x-4),\qquad P_2(x)=2+\tfrac14(x-4)-\tfrac1{64}(x-4)^2.

(b) Step 3 — plug in the increment $h=x-4=0.2$ .

\begin{aligned} P_1(4.2)&=2+\tfrac14(0.2)=2.05,\\ P_2(4.2)&=2.05-\tfrac{1}{64}(0.2)^2=2.05-0.000625=2.049375. \end{aligned}

The true value is

\sqrt{4.2}=2.0493902

. So

P_1

is off by

+6.1\times10^{-4}

and

P_2

by

-1.5\times10^{-5}

— about forty times closer.

(c) Step 4 — why the quadratic wins.

P_1

is the tangent line at

x=4

: it matches the height and the slope there but knows nothing about bending. Since

f''(4)=-\tfrac1{32}<0

, the curve bends downward, so the tangent rides above it and drifts away — that is why

P_1

's error is positive. The remainder formula says how far: with

n=2

the error is

\tfrac12 f''(c)\,h^2

for some

c

between

4

and

4.2

, and

\tfrac12\cdot\tfrac1{32}\cdot0.04=6.25\times10^{-4}

— almost exactly the

6.1\times10^{-4}

we found.

P_2

puts that missing bend back in with the term

-\tfrac1{64}h^2

, so what remains is the third-order piece

\frac{f'''(4)}{3!}h^3=\frac{3/256}{6}\,(0.008)=\frac{0.008}{512}=1.56\times10^{-5},

which matches

P_2

's leftover error to two figures (and its sign explains why

P_2

now sits just below the curve).

Step 5 — the binomial shortcut. Factor out the centre and use

(1+t)^{1/2}=1+\tfrac12t-\tfrac18t^2+\cdots

with

t=h/4

:

\sqrt{4+h}=2\sqrt{1+\tfrac h4}=2\Big(1+\tfrac12\cdot\tfrac h4-\tfrac18\cdot\tfrac{h^2}{16}+\cdots\Big)=2+\tfrac h4-\tfrac{h^2}{64}+\cdots

Both polynomials fall out at once.

(a) $P_1(x)=2+\tfrac14(x-4)$ , $P_2(x)=2+\tfrac14(x-4)-\tfrac1{64}(x-4)^2$ . (b) $P_1(4.2)=2.05$ , $P_2(4.2)=2.049375$ , true $2.0493902$ ; errors $6.1\times10^{-4}$ and $1.5\times10^{-5}$ . (c) $f''<0$ : the tangent sits above a curve that bends down; the quadratic term restores the bend.

Remember

$f''(a)$ is not the coefficient — $f''(a)/2$ is. Writing $-\tfrac1{32}(x-4)^2$ instead of $-\tfrac1{64}(x-4)^2$ is the classic lost mark here. And always substitute the increment $x-a$ , not $x$ : $P_1(4.2)$ uses $0.2$ , not $4.2$ .

Problem 3medium

Let $f(x,y)=e^{x}\sin y$ . (a) Compute $\nabla f$ and the Hessian $H$ at $(0,0)$ . (b) Write the second-order Taylor polynomial $Q(x,y)$ of $f$ about $(0,0)$ . (c) Use $Q$ to approximate $f(0.1,0.2)$ , compare with the exact value, and say where the error comes from.

What this tests. The two-variable quadratic clone — five partial derivatives, the $\tfrac12$ in front and the $2$ on the cross term — and reading the Hessian as a shape. Plan. List all five partials before evaluating any; substitute the centre; assemble $Q$ in the form value + gradient term + half the quadratic form; then evaluate and compare, and name the order of the first neglected term.

Show the full solution

(a) Step 1 — the five partials. The exponential survives every

x

-derivative; the sine cycles under

y

-derivatives:

\begin{aligned} f_x&=e^{x}\sin y, & f_y&=e^{x}\cos y,\\ f_{xx}&=e^{x}\sin y, & f_{xy}&=e^{x}\cos y, & f_{yy}&=-e^{x}\sin y. \end{aligned}

At

(0,0)

,

e^{0}=1,\ \sin0=0,\ \cos0=1

, so

f=0,\ f_x=0,\ f_y=1,\ f_{xx}=0,\ f_{xy}=1,\ f_{yy}=0

:

\nabla f(0,0)=\begin{bmatrix}0\\1\end{bmatrix},\qquad H(0,0)=\begin{bmatrix}0&1\\1&0\end{bmatrix}.

(b) Step 2 — assemble the clone. With centre

(0,0)

the increments are

h=x,\ k=y

:

\begin{aligned} Q(x,y)&=f+\big(f_x\,x+f_y\,y\big)+\tfrac12\big(f_{xx}x^2+2f_{xy}xy+f_{yy}y^2\big)\\ &=0+(0\cdot x+1\cdot y)+\tfrac12\,(0+2\cdot1\cdot xy+0)\\ &=y+xy. \end{aligned}

(c) Step 3 — the numbers.

Q(0.1,0.2)=0.2+(0.1)(0.2)=0.22

. Exactly,

f(0.1,0.2)=e^{0.1}\sin0.2=1.105171\times0.198669=0.2195636,

so

Q

overshoots by

4.4\times10^{-4}

.

Step 4 — where the error comes from. Multiply the one-variable series and keep track of total degree:

\begin{aligned} e^{x}\sin y&=\Big(1+x+\tfrac{x^2}{2}+\tfrac{x^3}{6}+\cdots\Big)\Big(y-\tfrac{y^3}{6}+\cdots\Big)\\ &=y+xy+\underbrace{\tfrac12x^2y-\tfrac16y^3}_{\text{degree }3}+\cdots \end{aligned}

This is the product-of-series shortcut:

Q=y+xy

appears with no Hessian at all. The first neglected terms are cubic, and at

(0.1,0.2)

they give

0.001-0.001333=-3.3\times10^{-4}

; the quartic terms

\tfrac16x^3y-\tfrac16xy^3

add

-1.0\times10^{-4}

, and together they account for almost the whole gap of

-4.4\times10^{-4}

. Doubling the step to

(0.2,0.4)

gives

f=0.475637

against

Q=0.48

, a gap of

-4.4\times10^{-3}

— ten times larger for a step twice as long, exactly what a cubic-led error (

2^3=8

, plus the quartic share) predicts.

Step 5 — read the Hessian as a shape.

H=\begin{bmatrix}0&1\\1&0\end{bmatrix}

has trace

0

and determinant

-1

, so its eigenvalues solve

\lambda^2-1=0

:

\lambda=\pm1

. The pure curvatures

f_{xx},f_{yy}

vanish and all the bending lives in the cross term — the quadratic part

xy

is a saddle, rising along

y=x

and falling along

y=-x

. (The origin is not a critical point, since

f_y=1

, so this is a statement about the clone's shape, not a classification of an extremum.)

(a) $\nabla f(0,0)=(0,1)^{\top}$ , $H(0,0)=\begin{bmatrix}0&1\\1&0\end{bmatrix}$ . (b) $Q(x,y)=y+xy$ . (c) $Q(0.1,0.2)=0.22$ against $f=0.2195636$ ; error $4.4\times10^{-4}$ , third order, mostly $\tfrac12x^2y-\tfrac16y^3$ .

Remember

The quadratic term is $\tfrac12\big(f_{xx}h^2+2f_{xy}hk+f_{yy}k^2\big)$ — a $\tfrac12$ outside and a $2$ on the cross term. Dropping either is where this problem sheds marks. When both functions have known series, multiply them and truncate by total degree: it is faster and it also shows you the size of the next term.

Problem 4mediumadded — covers the remainder bound

Bound the lie. (a) The small-angle rule $\cos x\approx1-\tfrac{x^2}{2}$ is used at $x=0.5$ . Using the remainder formula, bound the error before computing anything; then compute the actual error and compare. (b) How many terms of the MacLaurin series of $e^{x}$ are needed at $x=1$ to guarantee an error below $10^{-6}$ ? Confirm by summing.

What this tests. Using $R_n=f^{(n)}(c)(x-a)^n/n!$ as a guarantee: choosing $n$ , bounding $|f^{(n)}(c)|$ over the whole interval, and reading "how many terms" off a factorial. Plan. First decide which $n$ the remainder formula should carry (count a zero coefficient as a kept term); replace the unknown $c$ by the worst case on the interval; only then compute the true value and check that it sits inside the bound.

Show the full solution

(a) Step 1 — pick $n$ honestly. The MacLaurin coefficients of

\cos

are

1,\ 0,\ -\tfrac12,\ 0,\ \tfrac1{24},\dots

because

f'''(0)=\sin0=0

. So

1-\tfrac{x^2}{2}

is not only

P_2

but also

P_3

: the degree-3 term is present and equals zero. The remainder formula therefore applies with

n=4

:

R_4(x)=\frac{f^{(4)}(c)}{4!}\,x^4=\frac{\cos c}{24}\,x^4,\qquad c\ \text{between}\ 0\ \text{and}\ x.

Step 2 — bound the unknown $c$ away. Whatever

c

is,

|\cos c|\le1

, so

|R_4(0.5)|\le\frac{0.5^4}{24}=\frac{0.0625}{24}=0.0026042.

That is a promise made before any cosine was evaluated: the rule cannot lie by more than

0.0026

.

Step 3 — the actual error.

\cos0.5=0.8775826

and

1-\tfrac{0.25}{2}=0.875

, so the error is

0.0025826

— inside the bound, and almost equal to it (ratio

0.99

). The bound is nearly tight because

\cos c\approx1

for every

c

in

(0,0.5)

: the next term of the series,

x^4/24

, is essentially the error. Had we lazily used

n=3

, the bound

|\sin c|\,0.5^3/6\le0.0208

would still be true, but eight times looser.

(b) Step 4 — write the bound. Keeping the terms

k=0,\dots,n-1

of

e^{x}=\sum x^k/k!

leaves the remainder

R_n=e^{c}x^n/n!

with

c\in(0,1)

. At

x=1

,

e^{c}\le e<3

, so

|R_n|\le\frac{e}{n!}<\frac{3}{n!}.

Step 5 — find the first $n$ that works.

\frac{3}{9!}=\frac{3}{362\,880}=8.27\times10^{-6}\ (\text{too big}),\qquad \frac{3}{10!}=8.27\times10^{-7}<10^{-6}.

So

n=10

: keep the terms

k=0,1,\dots,9

— ten terms, up to

x^9/9!

.

Step 6 — confirm by summing.

\sum_{k=0}^{9}\frac1{k!}=2.7182815\quad\text{against}\quad e=2.7182818,

an error of

3.0\times10^{-7}

, below

10^{-6}

as promised. And nine terms genuinely fail:

\sum_{k\le8}1/k!=2.7182788

, error

3.1\times10^{-6}

. The bound was honest, not lazy.

(a) $n=4$ : $|R_4|\le0.5^4/24=0.0026042$ ; actual error $\cos0.5-0.875=0.0025826$ , inside the bound. (b) $3/n!<10^{-6}$ first at $n=10$ : ten terms ( $k=0,\dots,9$ ); the sum is $2.7182815$ , error $3.0\times10^{-7}$ .

Remember

In the remainder formula $n$ is one more than the highest degree you kept — and a zero coefficient still counts as kept, which is why $1-x^2/2$ earns the $x^4$ bound. Two habits lose marks: bounding $f^{(n)}(c)$ by its value at $a$ instead of its maximum over the interval, and confusing "number of terms" with "degree".

Problem 5easyadded — covers the secret c

Find the secret $c$ . Take $f(x)=x^3$ on $[0,2]$ , anchored at $a=0$ . Taylor's theorem promises a $c\in(0,2)$ with $f(2)=P_{n-1}(2)+\dfrac{f^{(n)}(c)}{n!}\,2^n$ . (a) For $n=1$ (the mean value theorem), find $c$ . (b) For $n=2$ , find $c$ . (c) For $n=3$ , what happens?

What this tests. That the $c$ in the remainder is a definite number fixed by $f$ , $a$ , $x$ and $n$ — not a free parameter — and that it moves when $n$ moves. Plan. Write the theorem with the given $n$ , fill in every known value ( $f$ and its derivatives at the anchor, the power of $x-a$ ), and solve the one equation left for $c$ ; check it lands strictly inside the interval.

Show the full solution

(a) Step 1 — $n=1$ is the mean value theorem.

f(2)-f(0)=f'(c)(2-0)

with

f'(x)=3x^2

:

8-0=3c^2\cdot2=6c^2\ \Rightarrow\ c^2=\tfrac43\ \Rightarrow\ c=\tfrac{2}{\sqrt3}=1.1547.

The negative root

-1.1547

lies outside

[0,2]

; the positive one is inside, as promised. Check: the chord has slope

8/2=4

, and

f'(1.1547)=3\cdot\tfrac43=4

. ✓

(b) Step 2 — $n=2$ . Now

P_1(x)=f(0)+f'(0)\,x=0+0\cdot x=0

: at the anchor the cubic is flat, so its tangent line is the

x

-axis and the whole of

f(2)

is remainder. With

f''(x)=6x

:

8=\frac{f''(c)}{2!}\,2^2=\frac{6c}{2}\cdot4=12c\ \Rightarrow\ c=\tfrac23.

Again

c\in(0,2)

. The curvature that explains the climb to height

8

is the curvature measured at

x=\tfrac23

.

(c) Step 3 — $n=3$ .

f'''(x)\equiv6

, so

R_3(x)=\frac{f'''(c)}{3!}\,x^3=\frac66\,x^3=x^3=f(x)

for every

c

. The remainder is the entire function: a cubic is its own degree-3 Taylor polynomial, so

P_3=f

,

R_4\equiv0

and

f^{(4)}\equiv0

. The theorem is true for all

c

at once and stops telling you anything.

Step 4 — $c$ depends on $n$ and on $x$ . Redo (a) and (b) with a general endpoint

x

instead of

2

:

x^3=3c^2x

gives

c=x/\sqrt3

, and

x^3=3c\,x^2

gives

c=x/3

. At

x=2

these are

1.1547

and

0.6667

; at

x=1

they would be

0.5774

and

0.3333

. The secret point slides with the endpoint and jumps when the order changes.

(a) $c=2/\sqrt3=1.1547$ . (b) $c=2/3$ . (c) $f'''\equiv6$ makes $R_3=x^3=f(x)$ : every $c$ works, because a cubic equals its own $P_3$ . In general $c=x/\sqrt3$ for $n=1$ and $c=x/3$ for $n=2$ .

Remember

The theorem promises that some $c$ strictly between $a$ and $x$ makes the formula exact; it does not tell you which one, and it changes with $n$ and $x$ . Marks go when a student sets $c=a$ or $c=x$ "for simplicity", or forgets the $(x-a)^n$ factor next to $f^{(n)}(c)/n!$ .

Problem 6medium

Let $f(x,y)=x^3-3xy+3y^2$ . (a) Find all critical points. (b) Form the Hessian and classify each point with the second-derivative test. (c) State the local-minimum value, and confirm the classification by the eigenvalue route.

What this tests. The full routine — solve $\nabla f=0$ , build $H$ , read $D=f_{xx}f_{yy}-f_{xy}^2$ and then $f_{xx}$ — at two points where the Hessian is not constant. Plan. Use the simpler gradient equation to eliminate a variable; keep $H$ symbolic and only then substitute each point; write $D$ 's sign before $f_{xx}$ 's; finish with trace and determinant for the eigenvalues as an independent check.

Show the full solution

(a) Step 1 — solve $\nabla f=\mathbf 0$ .

\nabla f=\big(3x^2-3y,\ -3x+6y\big)

. The second component gives

x=2y

; substitute into the first:

3(2y)^2-3y=12y^2-3y=3y(4y-1)=0\ \Rightarrow\ y=0\ \text{or}\ y=\tfrac14.

Then

x=2y

gives the critical points

(0,0)

and

\big(\tfrac12,\tfrac14\big)

.

(b) Step 2 — the Hessian, kept symbolic.

f_{xx}=6x,\ f_{xy}=-3,\ f_{yy}=6

:

H=\begin{bmatrix}6x&-3\\-3&6\end{bmatrix},\qquad D=f_{xx}f_{yy}-f_{xy}^2=36x-9.

Step 3 — the verdicts. At

(0,0)

:

D=-9<0

⇒ saddle (no need to look at

f_{xx}

). At

\big(\tfrac12,\tfrac14\big)

:

D=36\cdot\tfrac12-9=9>0

and

f_{xx}=3>0

⇒ local minimum.

(c) Step 4 — the value.

f\big(\tfrac12,\tfrac14\big)=\tfrac18-3\cdot\tfrac12\cdot\tfrac14+3\cdot\tfrac1{16}=\tfrac{2}{16}-\tfrac{6}{16}+\tfrac{3}{16}=-\tfrac1{16}.

Step 5 — the eigenvalue route. At the minimum

H=\begin{bmatrix}3&-3\\-3&6\end{bmatrix}

has trace

9

and determinant

18-9=9

, so its eigenvalues solve

\lambda^2-9\lambda+9=0

:

\lambda=\frac{9\pm\sqrt{81-36}}{2}=\frac{9\pm\sqrt{45}}{2}=\frac{9\pm6.708}{2}=7.854,\ 1.146,

both positive ⇒ positive-definite ⇒ a bowl. At the origin

H=\begin{bmatrix}0&-3\\-3&6\end{bmatrix}

has determinant

-9

, so

\lambda=3\pm\sqrt{18}=7.243,\ -1.243

: opposite signs ⇒ indefinite ⇒ saddle. For a symmetric

2\times2

matrix

\det H=\lambda_1\lambda_2

, which is exactly why

D<0

forces a saddle.

(a) $(0,0)$ and $\big(\tfrac12,\tfrac14\big)$ . (b) $(0,0)$ : $D=-9$ , saddle; $\big(\tfrac12,\tfrac14\big)$ : $D=9,\ f_{xx}=3$ , local minimum. (c) $f_{\min}=-\tfrac1{16}$ ; eigenvalues there $(9\pm\sqrt{45})/2\approx7.854,\ 1.146$ .

Remember

Ask $D$ 's sign first, then $f_{xx}$ 's — a saddle needs no second question. The mark most often lost here: evaluating $H$ once and reusing it, when $f_{xx}=6x$ changes from point to point. And $f_{xx}=0$ at the origin does not make it "neither"; $D=-9$ already says saddle.

Problem 7medium

A model is trained by minimising the quadratic loss $f(x,y)=2x^2+2xy+3y^2-4x-2y$ . (a) Find the unique critical point. (b) Show in two ways — Sylvester's criterion and the eigenvalues — that the Hessian is positive-definite, and conclude the point is the global minimum. (c) State the minimum value.

What this tests. That "positive-definite Hessian" can be certified by leading minors or by eigenvalue signs, and that for a quadratic a constant positive-definite Hessian makes the local minimum global. Plan. Solve the linear gradient system; write the constant $H$ ; check $\Delta_1=f_{xx}$ and $\Delta_2=\det H$ ; get the eigenvalues from trace and determinant; evaluate $f$ at the point, and if time allows complete the square to prove the minimum is global.

Show the full solution

(a) Step 1 — the critical point.

\nabla f=(4x+2y-4,\ 2x+6y-2)=\mathbf 0

, i.e.

2x+y=2

and

x+3y=1

. From the first,

y=2-2x

; substitute:

x+3(2-2x)=1\ \Rightarrow\ -5x=-5\ \Rightarrow\ x=1,\ y=0.

The critical point is

(1,0)

.

(b) Step 2 — Sylvester. Because

f

is quadratic the Hessian is the same everywhere:

H=\begin{bmatrix}4&2\\2&6\end{bmatrix},\qquad \Delta_1=4>0,\quad \Delta_2=\det H=24-4=20>0.

Both leading principal minors are positive ⇒

H

is positive-definite.

Step 3 — eigenvalues. Trace

10

, determinant

20

:

\lambda=\frac{10\pm\sqrt{100-80}}{2}=5\pm\sqrt5=7.236,\ 2.764,

both positive ⇒ positive-definite again. Since

H\succ0

at every point,

f

is strictly convex: it has exactly one stationary point and that point is the global minimum.

(c) Step 4 — the value.

f(1,0)=2-4=-2

.

Step 5 — completing the square proves "global" by hand. Shift to the critical point with

x=1+u,\ y=v

:

\begin{aligned} f&=2(1+u)^2+2(1+u)v+3v^2-4(1+u)-2v\\ &=-2+\big(2u^2+2uv+3v^2\big)\\ &=-2+2\big(u+\tfrac v2\big)^2+\tfrac52v^2\ \ \ge\ -2, \end{aligned}

with equality only at

u=v=0

. One manoeuvre certifies definiteness and pins the value.

Step 6 — how elongated is the bowl? The condition number

\lambda_{\max}/\lambda_{\min}=7.236/2.764=2.618

(exactly

(3+\sqrt5)/2

). The contours are ellipses whose axes are in the ratio

\sqrt{2.618}\approx1.62

: steepest along the

\lambda\approx7.24

eigenvector, gentlest along the

\lambda\approx2.76

one. The larger this ratio, the more gradient descent zig-zags before it settles.

(a) $(1,0)$ . (b) $H=\begin{bmatrix}4&2\\2&6\end{bmatrix}$ : minors $4>0,\ 20>0$ ; eigenvalues $5\pm\sqrt5\approx7.236,\ 2.764$ ; $H\succ0$ everywhere ⇒ global minimum. (c) $f_{\min}=-2$ ; condition number $\approx2.618$ .

Remember

Sylvester uses the leading minors: the top-left $1\times1$ entry and then the full determinant. Checking $f_{yy}>0$ instead of $f_{xx}>0$ is not the criterion (it happens to agree here). For a quadratic, $H$ is constant, so positive-definite once means convex everywhere — and "local" upgrades to "global" for free.

Problem 8medium

Consider $g(x,y)=x^4+y^4$ and $h(x,y)=x^4-y^4$ . (a) Show that for both, the origin is a critical point and the Hessian there is the zero matrix, so the second-derivative test is inconclusive. (b) Determine the true nature of the origin for each by a direct argument. (c) In one line, say why the Hessian could not tell them apart.

What this tests. What $D=0$ actually means — "no verdict", not "flat" — and how to classify a point when the quadratic clone is silent. Plan. Compute gradient and Hessian at the origin for both; when the test is silent, argue directly from the sign of the function (a global inequality, or two directions with opposite signs); the ray slice $(t\cos\theta,t\sin\theta)$ is the systematic version of that.

Show the full solution

(a) Step 1 — gradients and Hessians.

\begin{aligned} \nabla g&=(4x^3,\ 4y^3), & H_g&=\begin{bmatrix}12x^2&0\\0&12y^2\end{bmatrix},\\ \nabla h&=(4x^3,\ -4y^3), & H_h&=\begin{bmatrix}12x^2&0\\0&-12y^2\end{bmatrix}. \end{aligned}

At the origin both gradients vanish (critical point) and both Hessians are the zero matrix, so

D=\det H=0

for each: the test says nothing.

(b) Step 2 — $g$ by a global inequality.

g(x,y)=x^4+y^4\ge0

for every

(x,y)

, with equality only at the origin ⇒ a strict local (indeed global) minimum.

Step 3 — $h$ by two directions. Along the

x

-axis,

h(t,0)=t^4>0

; along the

y

-axis,

h(0,t)=-t^4<0

. A point that is a minimum in one direction and a maximum in another is a saddle.

Step 4 — the ray slice does both at once. Put

(x,y)=t(\cos\theta,\sin\theta)

:

\begin{aligned} g&=t^4\big(\cos^4\theta+\sin^4\theta\big)=t^4\big(1-\tfrac12\sin^2 2\theta\big)\ \ge\ \tfrac12t^4>0,\\ h&=t^4\big(\cos^4\theta-\sin^4\theta\big)=t^4\cos2\theta, \end{aligned}

using

\cos^4+\sin^4=(\cos^2+\sin^2)^2-2\sin^2\cos^2

and

\cos^4-\sin^4=(\cos^2-\sin^2)(\cos^2+\sin^2)

. Every ray of

g

climbs; the rays of

h

climb for

|\theta|<45^\circ

and fall for

45^\circ<\theta<135^\circ

. A sign that changes with direction is the signature of a saddle.

(c) Step 5 — why the Hessian was blind. The Hessian records second-order behaviour only; here the first non-zero behaviour is fourth order, and the quadratic clone of both functions is the same flat plane

z=0

.

(a) Both have $\nabla=\mathbf 0$ and $H=0$ at the origin, so $D=0$ . (b) $g\ge0$ ⇒ minimum; $h=t^4$ along $x$ and $-t^4$ along $y$ ⇒ saddle (ray slice: $g=t^4(1-\tfrac12\sin^22\theta)$ , $h=t^4\cos2\theta$ ). (c) The difference is quartic; a second-order test cannot see it.

Remember

$D=0$ is a refusal to judge, not a verdict of "flat" or "neither". Writing "inconclusive" and stopping loses the marks: the question asks for the true nature, which needs a direct argument — an inequality, two opposite directions, or a ray slice.

Problem 9easy

Find the local extreme values of $f(x,y)=xy-x^2-y^2-2x-2y+4$ .

What this tests. The three-step routine on a function whose Hessian is constant: critical point, $D$ , verdict, value. Plan. Solve the two linear gradient equations by substitution; write the three second partials (they are numbers here); compute $D$ and then look at $f_{xx}$ ; substitute back to get the value, and cross-check with the eigenvalues.

Show the full solution

Step 1 — critical points.

f_x=y-2x-2=0

and

f_y=x-2y-2=0

. From the first,

y=2x+2

; into the second:

\begin{aligned} x-2(2x+2)-2=0\ &\Rightarrow\ -3x-6=0\ \Rightarrow\ x=-2,\\ y&=2(-2)+2=-2. \end{aligned}

The only critical point is

(-2,-2)

.

Step 2 — second partials and $D$ .

f_{xx}=-2,\ f_{yy}=-2,\ f_{xy}=1

, constants:

D=f_{xx}f_{yy}-f_{xy}^2=(-2)(-2)-1^2=4-1=3>0.

Step 3 — verdict and value.

D>0

and

f_{xx}=-2<0

⇒ local maximum. Its value:

f(-2,-2)=4-4-4+4+4+4=8.

Step 4 — cross-check by eigenvalues.

H=\begin{bmatrix}-2&1\\1&-2\end{bmatrix}

has trace

-4

and determinant

3

, so

\lambda^2+4\lambda+3=0

gives

\lambda=-1,\ -3

: both negative ⇒ negative-definite ⇒ a dome. Because

H

is constant,

f

is concave everywhere and

8

is in fact the global maximum.

One critical point $(-2,-2)$ ; $D=3>0$ , $f_{xx}=-2<0$ : a local (indeed global) maximum with $f(-2,-2)=8$ .

Remember

A maximum needs both $D>0$ and $f_{xx}<0$ ; a negative $f_{xx}$ with $D<0$ is a saddle, not a maximum. The marks that go missing here are almost always a sign slip in the two-line linear system — substitute your point back into both gradient equations before moving on.

Problem 10medium

Find the local extreme values of $f(x,y)=x^3+y^3-3xy+1$ .

What this tests. A nonlinear gradient system with a real-root argument, and a Hessian that must be re-evaluated at each point. Plan. Turn each gradient equation into a substitution, reduce to one polynomial equation, keep only the real roots; evaluate $f_{xx},f_{yy},f_{xy}$ separately at each point; give both the verdict and the value, and say whether the minimum is global.

Show the full solution

Step 1 — critical points.

f_x=3x^2-3y=0\ \Rightarrow\ y=x^2,\qquad f_y=3y^2-3x=0\ \Rightarrow\ x=y^2.

Substituting

y=x^2

into the second:

x=x^4

, so

x(x^3-1)=0

. The real roots are

x=0

and

x=1

(the other two roots of

x^3=1

are complex). Hence the critical points are

(0,0)

and

(1,1)

.

Step 2 — the Hessian entries.

f_{xx}=6x,\ f_{yy}=6y,\ f_{xy}=-3

— the diagonal changes from point to point, the off-diagonal does not.

Step 3 — at $(0,0)$ .

f_{xx}=0,\ f_{yy}=0,\ f_{xy}=-3

:

D=0\cdot0-(-3)^2=-9<0\ \Rightarrow\ \text{saddle},\qquad f(0,0)=1.

Step 4 — at $(1,1)$ .

f_{xx}=6,\ f_{yy}=6,\ f_{xy}=-3

:

\begin{aligned} D&=6\cdot6-(-3)^2=36-9=27>0,\quad f_{xx}=6>0\\ &\Rightarrow\ \text{local minimum},\qquad f(1,1)=1+1-3+1=0. \end{aligned}

Step 5 — eigenvalue check, and local versus global. At

(1,1)

,

H=\begin{bmatrix}6&-3\\-3&6\end{bmatrix}

has eigenvalues

6\pm3=9,\ 3

(both positive: bowl); at

(0,0)

,

H=\begin{bmatrix}0&-3\\-3&0\end{bmatrix}

has eigenvalues

\pm3

(saddle). The minimum value

0

is local only:

f(-2,0)=-8+1=-7

, and a cubic runs to

-\infty

— the test describes a neighbourhood, nothing more.

$(0,0)$ : $D=-9$ , saddle, $f=1$ . $(1,1)$ : $D=27,\ f_{xx}=6$ , local minimum, $f=0$ (not global: $f(-2,0)=-7$ ).

Remember

$f_{xx}=f_{yy}=0$ at the origin does not make $D=0$ : the cross term $f_{xy}=-3$ gives $D=-9$ and a clear saddle. And for cubics, local is not global — say so when you report a minimum.

Problem 11hard

Find the absolute maximum and minimum of $f(x,y)=x^2+y^2-x-y$ on the region $D=\{(x,y):x^2\le y\le1\}$ .

What this tests. Absolute extrema on a closed, bounded region: interior critical points and every piece of the boundary and the corners, then a comparison. Plan. Draw the region first; collect the interior critical points that lie inside it; reduce each boundary curve to a one-variable function and find its critical points; add the corners; make one list of values and pick the largest and smallest — the second-derivative test is not needed for any of this.

Show the full solution

Step 1 — draw the region in words.

D

is the set of points above the parabola

y=x^2

and below the line

y=1

: a parabolic bowl with a flat lid, meeting at the corners

(-1,1)

and

(1,1)

. It is closed and bounded, so an absolute maximum and minimum exist, and each lives either at an interior critical point or somewhere on the boundary. The second-derivative test can only classify flat points in the open interior; on the boundary the gradient need not vanish at all, so the boundary must be searched separately.

Step 2 — interior critical points.

f_x=2x-1=0

and

f_y=2y-1=0

give

\big(\tfrac12,\tfrac12\big)

. Is it inside

D

?

\big(\tfrac12\big)^2=\tfrac14\le\tfrac12\le1

, yes. Its value:

f\big(\tfrac12,\tfrac12\big)=\tfrac14+\tfrac14-\tfrac12-\tfrac12=-\tfrac12.

(Here

H=2I

, a bowl, so this is a local minimum — but that says nothing about the boundary.)

Step 3 — the parabola arc $y=x^2$ , $-1\le x\le1$ . Substitute:

\begin{aligned} g(x)&=f(x,x^2)=x^2+x^4-x-x^2=x^4-x,\\ g'(x)&=4x^3-1=0\ \Rightarrow\ x=\big(\tfrac14\big)^{1/3}=0.62996. \end{aligned}

Then

y=x^2=0.39685

, and since

4x^3=1

we have

x^4=x/4

, so

g=\tfrac x4-x=-\tfrac34x=-0.47247

. The arc's endpoints are the corners:

f(-1,1)=1+1+1-1=2,\qquad f(1,1)=1+1-1-1=0.

Step 4 — the lid $y=1$ , $-1\le x\le1$ .

h(x)=f(x,1)=x^2+1-x-1=x^2-x

, with

h'(x)=2x-1=0

at

x=\tfrac12

:

f\big(\tfrac12,1\big)=\tfrac14+1-\tfrac12-1=-\tfrac14.

Its endpoints are the same two corners, already counted.

Step 5 — compare. The candidate values are

\{-\tfrac12,\ -0.47247,\ 2,\ 0,\ -\tfrac14\}

. The absolute maximum is

2

at the corner

(-1,1)

; the absolute minimum is

-\tfrac12

at the interior point

\big(\tfrac12,\tfrac12\big)

.

Step 6 — a geometric cross-check. Completing squares,

f=\big(x-\tfrac12\big)^2+\big(y-\tfrac12\big)^2-\tfrac12

:

f

is the squared distance from

\big(\tfrac12,\tfrac12\big)

, minus

\tfrac12

. The minimum is at that centre, which lies in

D

; the maximum is at the point of

D

farthest from it — the corner

(-1,1)

, at squared distance

1.5^2+0.5^2=2.5

, giving

2.5-0.5=2

. ✓ The arc point

(0.63,0.40)

is merely the foot of the perpendicular from the centre to the parabola — the closest point of the arc, never a contender for either extreme. Note

\nabla f(-1,1)=(-3,1)\neq\mathbf 0

: the maximum sits where the gradient does not vanish, which is exactly why the boundary must be checked by hand.

Interior: $f(\tfrac12,\tfrac12)=-\tfrac12$ . Parabola: $f(0.62996,0.39685)=-0.47247$ , corners $f(-1,1)=2$ , $f(1,1)=0$ . Lid: $f(\tfrac12,1)=-\tfrac14$ . Absolute maximum $2$ at $(-1,1)$ ; absolute minimum $-\tfrac12$ at $(\tfrac12,\tfrac12)$ .

Remember

On a closed, bounded region keep three lists — interior critical points, each boundary piece reduced to one variable, and the corners — and only then compare. The marks vanish when the corners are skipped (that is where the maximum lives here) or when the arc's $-0.47$ is reported as the minimum without checking it against $-\tfrac12$ .

Problem 12hardadded — covers the judge's identity

The judge's identity, proved and used. Let $Q(h,k)=h^2f_{xx}+2hkf_{xy}+k^2f_{yy}$ with the partials taken at a critical point, and $D=f_{xx}f_{yy}-f_{xy}^2$ . (a) Prove that $f_{xx}\,Q=(hf_{xx}+kf_{xy})^2+D\,k^2$ by expanding the right-hand side. (b) Deduce the three verdicts: $D>0,\ f_{xx}>0$ ⇒ $Q>0$ for all $(h,k)\neq\mathbf 0$ ; $D>0,\ f_{xx}<0$ ⇒ $Q<0$ ; $D<0$ ⇒ $Q$ takes both signs. (c) For a symmetric $H=\begin{bmatrix}a&b\\b&c\end{bmatrix}$ , show $\lambda_1\lambda_2=ac-b^2=D$ and $\lambda_1+\lambda_2=a+c$ ; hence $D<0$ exactly when the eigenvalues have opposite signs, and $D>0$ with $a>0$ forces $c>0$ and both eigenvalues positive. (d) Apply (c) to $H=\begin{bmatrix}3&-3\\-3&6\end{bmatrix}$ .

What this tests. The one identity behind the whole second-derivative test, and why the discriminant $D$ and the eigenvalue signs are the same criterion in two languages. Plan. Expand, cancel, factor; for each verdict start from the sign of the right-hand side and divide by $f_{xx}$ with the correct flip; for the saddle, exhibit two explicit directions; for (c) write the characteristic polynomial and read off sum and product of roots; finish with the numbers.

Show the full solution

(a) Step 1 — expand the right-hand side.

\begin{aligned} (hf_{xx}+kf_{xy})^2&=h^2f_{xx}^2+2hk\,f_{xx}f_{xy}+k^2f_{xy}^2,\\ D\,k^2&=f_{xx}f_{yy}\,k^2-f_{xy}^2\,k^2. \end{aligned}

Adding, the two

k^2f_{xy}^2

terms cancel and what is left has a common factor

f_{xx}

:

h^2f_{xx}^2+2hk\,f_{xx}f_{xy}+k^2f_{xx}f_{yy}=f_{xx}\big(h^2f_{xx}+2hkf_{xy}+k^2f_{yy}\big)=f_{xx}\,Q.\ \ \blacksquare

The identity is nothing more than completing the square in

h

, with everything multiplied by

f_{xx}

so that no fractions appear.

(b) Step 2 — verdict 1: $D>0$ and $f_{xx}>0$ . The right-hand side is a square plus

D\,k^2

, so it is

\ge0

. It can be

0

only if

k=0

and

hf_{xx}+kf_{xy}=0

, i.e.

hf_{xx}=0

, i.e.

h=0

. So for

(h,k)\neq\mathbf 0

we have

f_{xx}Q>0

, and dividing by

f_{xx}>0

gives

Q>0

: uphill in every direction — a bowl, a local minimum.

Step 3 — verdict 2: $D>0$ and $f_{xx}<0$ . The same right-hand side is still strictly positive for

(h,k)\neq\mathbf 0

— its sign never depended on the sign of

f_{xx}

. Dividing by

f_{xx}<0

flips the inequality:

Q<0

in every direction — a dome, a local maximum. (Note that

D>0

forces

f_{xx}\neq0

, since

f_{xx}f_{yy}>f_{xy}^2\ge0

, so the division is always legal in verdicts 1 and 2.)

Step 4 — verdict 3: $D<0$ . Test two explicit directions. Along

(h,k)=(1,0)

:

Q(1,0)=f_{xx}.

Along

(h,k)=(-f_{xy},\,f_{xx})

:

\begin{aligned} Q(-f_{xy},f_{xx})&=f_{xy}^2f_{xx}-2f_{xy}f_{xx}f_{xy}+f_{xx}^2f_{yy}\\ &=f_{xx}\big(f_{xx}f_{yy}-f_{xy}^2\big)=f_{xx}\,D, \end{aligned}

so

f_{xx}\cdot Q(-f_{xy},f_{xx})=f_{xx}^2D<0

whenever

f_{xx}\neq0

: the two values

Q(1,0)=f_{xx}

and

Q(-f_{xy},f_{xx})=f_{xx}D

have opposite signs — up one way, down another, a saddle. If

f_{xx}=0

, then

D=-f_{xy}^2<0

means

f_{xy}\neq0

, and along

(h,k)=(1,t)

:

Q(1,t)=2t\,f_{xy}+t^2f_{yy}=t\,(2f_{xy}+t\,f_{yy}),

which for small

t

has the sign of

t\,f_{xy}

— opposite signs at

t=\pm\varepsilon

. (When

f_{yy}\neq0

one can instead swap the roles of

x

and

y

and run the identity with

f_{yy}

.) Either way

Q

changes sign: saddle.

(c) Step 5 — eigenvalues of a symmetric $2\times2$ .

\det(H-\lambda I)=(a-\lambda)(c-\lambda)-b^2=\lambda^2-(a+c)\lambda+(ac-b^2).

For a monic quadratic with roots

\lambda_1,\lambda_2

, the sum of the roots is the negated middle coefficient and the product is the constant term:

\lambda_1+\lambda_2=a+c=\operatorname{tr}H,\qquad \lambda_1\lambda_2=ac-b^2=\det H=D.

(The roots are real because the discriminant

(a+c)^2-4(ac-b^2)=(a-c)^2+4b^2\ge0

.) Hence

D<0\iff\lambda_1\lambda_2<0\iff

the eigenvalues have opposite signs. And if

D>0

with

a>0

:

ac>b^2\ge0

with

a>0

forces

c>0

, so the sum

a+c>0

and the product

D>0

— two real numbers with positive sum and positive product are both positive. (By the same token

a<0

forces

c<0

and both eigenvalues negative.) This is why "

D

and

f_{xx}

" and "eigenvalue signs" are one criterion in two languages.

(d) Step 6 — the numbers.

H=\begin{bmatrix}3&-3\\-3&6\end{bmatrix}

: trace

9

, determinant

18-9=9=D

. So

\lambda^2-9\lambda+9=0

:

\lambda=\frac{9\pm\sqrt{81-36}}{2}=\frac{9\pm\sqrt{45}}{2}=7.854,\ 1.146.

Check: sum

9

, product

9

. ✓ Both positive ⇒ positive-definite ⇒ a bowl — this is the Hessian of Problem 6 at its minimum

\big(\tfrac12,\tfrac14\big)

. Sylvester says the same in one line (

3>0,\ 9>0

), and the identity of part (a) shows it explicitly:

3Q=(3h-3k)^2+9k^2>0

for

(h,k)\neq\mathbf 0

.

(a) Expanding gives $f_{xx}(h^2f_{xx}+2hkf_{xy}+k^2f_{yy})=f_{xx}Q$ . (b) $D>0$ : the right side is positive for $(h,k)\neq\mathbf 0$ , so $Q$ has the sign of $f_{xx}$ ; $D<0$ : $Q(1,0)=f_{xx}$ and $Q(-f_{xy},f_{xx})=f_{xx}D$ have opposite signs (or $Q(1,\pm\varepsilon)$ when $f_{xx}=0$ ). (c) $\lambda_1\lambda_2=D$ , $\lambda_1+\lambda_2=a+c$ . (d) $\lambda=(9\pm\sqrt{45})/2\approx7.854,\ 1.146$ , both positive: bowl.

Remember

The whole test is one line of completed square: $f_{xx}Q=(hf_{xx}+kf_{xy})^2+Dk^2$ . Read it as "a square plus $D$ times a square" and the three verdicts follow, and $D=\det H=\lambda_1\lambda_2$ is why the discriminant's sign is the eigenvalue story in disguise. Marks are lost by forgetting the $f_{xx}=0$ case in the saddle argument, or by claiming $D>0$ alone gives a minimum.

Next up

Unit 9 · Gradient Descent →

You now know what a landscape looks like near a flat point and how to tell a bowl from a saddle. Unit 9 starts walking: the descent rule proved, the step-size speed limit the Hessian's eigenvalues set and the zig-zag it causes, exact line search, learning-rate schedules, gradient checking — and stochastic gradient descent, why a random spoonful of the data points the right way.

← All units