$Derivatives and the Chain Rule

Rate of change, partial derivatives, and the composition rule that makes backpropagation possible.

Aug 3, 2026

A derivative answers: if I nudge this input slightly, how much does the output move, and in which direction? It is the rate of change of one quantity with respect to another — the slope of the curve at a point.

f(x) = x^2        f'(x) = 2x
                  at x = 3, nudging x up by 0.001 raises f by about 0.006

Positive slope means increasing the input increases the output; negative means the reverse; zero means you are at a flat spot.

Partial derivatives and the gradient

When a function has many inputs, the partial derivative with respect to one of them is the same question asked while holding all the others fixed. Collect the partial derivatives for every input into a list and you have the gradient — a vector pointing in the direction of steepest increase.

Gradient descent steps the opposite way: to reduce a loss, move each parameter a little in the direction that reduces it. With learning rate $\eta >0$, subtract $\eta \nabla \mathcal{L}$, the scaled gradient of the loss with respect to $\theta$, from the current parameters.

The chain rule

If a value passes through one function and then another, the derivatives multiply:

y = g(x)  and  z = f(y)      then      dz/dx = dz/dy * dy/dx

That is the rule for one path through the computation. A loss computed at the end of a hundred stacked operations can still have a derivative with respect to a parameter at the beginning: multiply local derivatives along each path, then add the contributions when paths meet.

Backpropagation is the bookkeeping that does this efficiently: sweep backwards once from the loss, and each operation converts the gradient of its output into the gradient of its input and of its own parameters. Every parameter gets a gradient in a backward sweep. For common dense transformers, planning estimates often put that sweep at roughly twice the cost of the forward pass, instead of requiring one forward pass per parameter. The exact ratio depends on the operations and implementation.

One practical consequence

Where the forward pass copies or shares a value, the backward pass sums the gradients from the copies, because each use contributed separately to the loss. Missing one of those contributions is a common hand-written gradient bug.