$Logarithms

The inverse of exponentiation: the power a base must be raised to in order to produce a number.

Aug 2, 2026

A logarithm answers the question “what power?”. ${\log }_{b}x$ is the number $y$ such that $b^y=x$. For real-valued logarithms, $x>0$, $b>0$, and $b\ne 1$.

log2(8)    = 3      because 2^3 = 8
log2(1024) = 10     because 2^10 = 1024
log10(1000) = 3     because 10^3 = 1000

Three properties do most of the work:

• $\log (ab)=\log a+\log b$ — multiplication becomes addition, which is why logs turn compounding growth into straight lines.
• $\log (x^n)=n\log x$ — exponents come down to the front.
• ${\log }_{b}x$ grows very slowly. Doubling $x$ adds exactly 1 to ${\log }_{2}x$, no matter how large $x$ already is.

Why base 2 means “bits”

${\log }_{2}x$ is the information needed to identify one item out of $x$ equally likely ones, measured in bits. Sixteen possibilities take exactly 4 bits. A fixed-length binary label for one of a thousand possibilities needs $\lceil {\log }_{2}1000\rceil =10$ bits; the information content is about 9.97 bits.

Turned around, an event of probability $p$ carries $-{\log }_{2}p$ bits of information: a certainty carries 0, a fair coin outcome carries 1, and a one-in-a-thousand event carries about 10. Individual prefix codes use whole bits, but entropy coding can approach fractional average lengths over many events. An ideal entropy coder driven by a model scoring 1 bit per byte would use about one-eighth of the raw byte payload, before overhead.

Bits and nats

The same quantity measured in a different base gets a different name. Base 2 gives bits; the natural logarithm, base $e$, gives nats.

1 nat = 1 / ln(2) bits ≈ 1.443 bits
1 bit = ln(2) nats     ≈ 0.693 nats

Nothing conceptual changes — a nat is a bit measured with a different ruler. Machine-learning code often reports nats because common log functions use the natural logarithm unless another base is requested. If a reported loss does not name its base, check the formula or implementation: 2 nats and 2 bits are not the same claim.

A millinat is a thousandth of a nat, a convenient unit for small differences in loss.

Loss in bits, perplexity as $2^{\mathcal{L}}$, and the log-log plots used to fit Power Laws.