$Mean and Variance

Average and spread — and why controlling spread is most of what odd-looking constants in neural networks are for

Aug 3, 2026

The mean is the average: add the numbers, divide by how many there are.

The variance used here is the population average of the squared distances from the mean, dividing by the number of values. It measures how spread out the numbers are. Squaring keeps negatives from cancelling positives, but it also changes the units. Its square root, the standard deviation, is in the same units as the numbers themselves. Calling it a “typical distance from the mean” is useful intuition, not an exact guarantee about where any fixed fraction of values lies.

values      2, 4, 4, 4, 5, 5, 7, 9
mean        5
variance    4
std dev     2        most values sit within about 2 of the mean

A related quantity appears in normalization layers: the root mean square is the square root of the average of the squared values, without subtracting the mean first. The exact relationship is $\mathrm{RMS}(x{)}^2=\mathrm{Var}(x)+\mathrm{mean}(x{)}^2$. For numbers centred at zero, RMS equals the standard deviation; near zero, the two are close.

Why it keeps appearing

Two facts drive most of the design decisions:

• Adding $n$ independent, zero-mean numbers of variance 1 gives a result of variance $n$, so its standard deviation grows as $\sqrt{n}$. Under those assumptions, a dot product over 128 dimensions has standard deviation about 11 rather than 1.
• Several training-sensitive operations work best when their inputs stay in a controlled range. Softmax can saturate and produce very small gradients; scale drift can make deep stacks harder to optimize.

Put together, variance control explains many odd-looking constants in neural networks. Dividing attention scores by $\sqrt{d_k}$, choosing initialization scales from layer dimensions, scaling residual projections in some architectures, and using normalization layers all aim to put numbers on a scale the next operation handles well. The exact rule depends on the architecture.