$Vectors and Matrix Multiplication

Dot products, matrix shapes, and the one rule that governs whether two arrays can multiply.

Aug 3, 2026

A vector is an ordered list of numbers. A matrix is a rectangular grid of them, described by its shape: an $m\times n$ matrix has $m$ rows and $n$ columns.

The dot product

The dot product of two vectors of the same length multiplies them element-by-element and sums the results:

[1, 2, 3] · [4, 5, 6] = 1*4 + 2*5 + 3*6 = 32

One number out. For unit-length vectors, it is positive when they point in a similar direction, zero when they are perpendicular, and negative when they oppose. For unnormalized vectors, magnitude also affects the result. Attention learns query and key vectors so this score can help decide what information to read.

Matrix multiplication

Multiplying an $m\times n$ matrix by an $n\times p$ matrix produces an $m\times p$ matrix, in which entry $(i,j)$ is the dot product of row $i$ of the first with column $j$ of the second.

(m × n) @ (n × p)  ->  (m × p)
       ^^^^^^^ these must match

The inner dimensions must agree and they vanish from the result. This rule catches many shape errors and lets you check the dimensions claimed by an architecture diagram.

In code this is usually written A @ B. It is not commutative: A @ B and B @ A are different operations and often have incompatible shapes.

The transpose of a matrix swaps its rows and columns. If $K$ has shape $T\times d$, then $K^T$ has shape $d\times T$. This is why the attention score product $Q{K}^T$ has shape $T\times T$: it contains one score for each pair of token positions.

Why it dominates the cost

A single $m\times n$ by $n\times p$ multiply performs $m\times n\times p$ multiply-adds. Under the common convention that counts a multiplication and an addition separately, that is about $2mnp$ FLOPs. Matrix multiplication accounts for most arithmetic in common dense transformer layers, which is why model cost is often quoted in FLOPs and accelerators are designed around this operation.

An embedding lookup is equivalent to selecting a row rather than performing a dense multiplication. After that lookup, the query/key/value projections, attention scores, feed-forward block, and final vocabulary projection are all dominated by matrix multiplications, with normalization and other elementwise operations between them.