$Probability Distribution

An assignment of non-negative numbers summing to one over a set of outcomes, and the sampling it supports.

Aug 3, 2026

A discrete probability distribution assigns a number to each possible outcome such that every number is at least zero and all of them sum to exactly 1.

{ " mat": 0.31, " floor": 0.09, " table": 0.04, ... }   sums to 1.0

For a finite discrete set, that constraint is the whole definition. It makes the numbers comparable across outcomes. Given the text so far, a language model produces one such distribution for the next token.

Sampling

To sample from a distribution is to pick one outcome with probability equal to its number. Do it many times and the outcome frequencies approach the distribution. Sampling repeatedly from a model’s next-token distribution, feeding each choice back in, is how text is generated.

Picking the single highest-probability outcome instead is greedy decoding. It is deterministic for fixed logits and tie-breaking, and it can become repetitive because it never explores a lower-probability alternative.

Softmax

Neural networks emit unconstrained real numbers (logits), which are not a distribution: they can be negative and do not sum to anything in particular. Softmax converts them into one by exponentiating each and dividing by the total. Here $\exp (z)$ means $e^z$, where $e\approx 2.718$.

softmax(z)_i = exp(z_i) / sum_j exp(z_j)

Exponentiating makes everything positive; dividing by the sum makes it total 1. Larger logits get exponentially more mass, so softmax is sensitive to the gaps between logits rather than their absolute size. For a positive temperature $T$, softmax is applied to $z/T$: $T>1$ flattens the distribution, while $0<T<1$ sharpens it. Greedy decoding is the limiting case as $T$ approaches zero, not softmax evaluated at $T=0$.

The output of every language model, the attention weights inside one, the routing weights that choose experts, and the $-{\log }_{2}p$ scoring in Logarithms.