$Nonlinearity

Why a stack of matrix multiplications needs a nonlinear operation between the layers, or the depth buys nothing.

Aug 8, 2026

Multiplying by one matrix and then another is the same as multiplying by a single matrix:

(x @ A) @ B  ==  x @ (A @ B)

$A@B$ can be computed once, in advance. So a stack of linear layers collapses to one matrix multiplication. Biases make the layers affine rather than linear, but a stack of affine layers still collapses to one affine operation. An intermediate bottleneck can restrict the resulting matrix’s rank; it does not add expressive power. Depth only adds new functions when something nonlinear happens between the layers.

That is the entire job of an activation function: an elementwise operation, applied to each number independently, whose graph is not a straight line. It lets a deep network represent functions that a single matrix cannot.

The common ones

ReLU(x)  = max(0, x)              a hinge at zero; cheap, and a historical default
GELU(x)  = x * Φ(x)               smooth; Φ is the standard normal CDF
SiLU(x)  = x * sigmoid(x)         also called swish; smooth and allows negative outputs

SiLU is the one that appears inside SwiGLU. Written out, x / (1 + exp(-x)).

Gating

A gated layer goes further. Instead of applying a fixed shape to every input, it computes two projections and lets one scale the other elementwise:

out = (SiLU(x @ W_gate) * (x @ W_up)) @ W_down

The difference is where the shape comes from. A plain activation applies the same curve to everything; a gate computes its multiplier from the input, so the layer can suppress its own activations conditionally. SwiGLU is this construction with SiLU as the activation.

The nonlinearity is still essential. Gating refines it rather than replacing it.