$The Log is the Architecture

Recently, as I looked more closely at different systems, I kept running into the same structure. The systems did different jobs. Some moved events between services. Some recovered after crashes. Some kept several machines in agreement. But beneath the specialised terminology, one shape kept returning: a log.

The older and more general computer-science idea: an ordered sequence of records that grows by appending new records to the end.

Jay Kreps made this idea famous to a generation of distributed-systems engineers in his 2013 essay The Log, describing the log as a unifying abstraction that appears in databases, replication systems, data pipelines, and other infrastructure. The more systems I study, the more useful that lens becomes.

The interesting thing about a log is not only that it stores records. It gives those records a chosen order.

That sounds almost too simple to be important. But ordering gives us something subtle. Knowing what happened and in what order, we can often reconstruct the system’s current state.

The word chosen matters. A log does not discover a universal ordering of reality. A system defines an ordering scope and a rule for deciding which record occupies each position. In a single local log that may be straightforward. In a partitioned or replicated system, deciding what is ordered with what — and who is allowed to decide — becomes an architectural choice.

Start with the smallest possible log

Imagine a notebook that nobody can edit. You can write only the next line.

0   UserRegistered   user=alice
1   EmailChanged     user=alice  email=alice@example.com
2   PlanUpgraded     user=alice  plan=pro
3   OrderCreated     order=100   user=alice
4   PaymentReceived  order=100   amount=79

Three properties matter.

1. Records have an order

Record 3 happened after record 2. When two operations depend on each other, the sequence shows which came first.

2. We append instead of rewriting history

When Alice changes her email address, we do not need to edit an old line. We write a new fact at the end. The history stays intact.

3. A position in the log is meaningful

If a reader says “I have processed everything through record 2,” that position tells us exactly where it is in the history. Kafka calls this an offset. Other systems use different terms, but the idea is older and broader than Kafka.

The first hard question: what exactly is ordered?

It is easy to say “the log is ordered” and accidentally imply more than the system guarantees.

A single sequence gives a total order over the records in that sequence. But large systems often have many sequences. Kafka, for example, guarantees order within a topic-partition; it does not impose a single global order across every partition. A database WAL has an ordering relevant to recovery. A Raft group agrees on one command sequence for one replicated state machine. Those are different scopes.

This distinction matters because ordering is not free. Broader ordering usually requires more coordination, and coordination constrains throughput, availability, or both. Partitioning improves parallelism precisely by allowing independent histories to advance without forcing every write through one global sequencer.

So when someone says a system “preserves order,” I want to ask four more questions:

Ordered within what boundary?
Who chooses the next position?
When does that position become authoritative?
Can a failure cause an uncommitted suffix to disappear or change?

Wall-clock timestamps do not answer these questions. Two machines can disagree about time; messages can be delayed or reordered in transit. The useful order is normally the one the system itself establishes and can defend under its failure model.

History and state are different things

This distinction is the first big payoff.

A log tells us what happened. A table, object, cache entry, or materialised view usually tells us what is true now.

The “current order” above is a derived view. If the history is complete and the transition rules are deterministic, reprocessing the ordered records can rebuild the same state.

This does not mean every application should be event-sourced. It means history and state are different representations of information, and a log provides a disciplined way to move from one to the other.

initial state + replayable ordered changes + stable interpretation = current state

The extra terms are important. Replay is a contract, not magic. A historical record must still mean something when it is read months later. Code changes, schema changes and deleted reference data can all make yesterday’s record impossible to interpret using today’s assumptions.

A replayable system therefore has to decide what remains stable:

Are event schemas versioned?
Are old transition rules preserved or migrated?
Does replay depend on external services or mutable reference data?
Are side effects separated from state reconstruction?
Where do snapshots establish a new replay starting point?

That relationship is one reason logs appear often in recovery and replication. If a machine can recover an earlier state and replay missing changes in order, it can catch up without reconstructing every piece of state from scratch. Snapshots reduce how much history must be replayed, but they introduce another invariant: the snapshot and the log position it represents must agree.

A log lets readers move at their own pace

Now add another simple idea. The writer and the reader do not need to advance together.

One process can append new records while others remember their own positions. A fast reader can stay near the tail. A slow reader can lag behind. A new reader can start at the beginning. A debugging tool can move backwards and replay old records.

A retained log separates “the data exists” from “this particular reader has processed it.” This is a very different mental model from a transient message that disappears when delivered.

This property matters because systems rarely run at the same speed. Search indexing, billing, analytics, and backups can have different latency requirements without forcing the producer to coordinate synchronously with all of them. That changes the character of a messaging system. If old records remain available, the stream is not just a delivery mechanism; it is a history that can be revisited.

But a reader position introduces its own correctness boundary. “I fetched record 8273,” “I processed record 8273,” and “the effects of record 8273 are durable” are distinct statements.

Imagine a consumer that charges a customer and then records its progress:

read event
-> call payment provider
-> persist offset

If it crashes after the payment succeeds but before the offset is persisted, the event may be delivered again. If it persists the offset first and then crashes before charging, the event may be skipped. The log did its job. The ambiguity lies at the boundary between consumption and the external side effect.

This is why robust consumers need some combination of idempotency, transactional coupling, deduplication, fencing, or application-specific reconciliation. “Exactly once” is not a property you get merely because records have offsets. You must define exactly once with respect to which state transition, then make the commit boundary include everything that definition requires.

Why does “append-only” help so much?

Changing the end of a sequence is mechanically and logically simpler than coordinating arbitrary in-place mutations.

That simplicity appears at several layers. Storage engines can batch appended records into larger writes and segment files. Recovery has a natural replay order. Replicas can compare positions and request a suffix. Checkpoints can name a prefix already known to be safe. Consensus protocols can agree on an ordered command stream rather than synchronising an opaque heap of mutable state.

The performance story is more nuanced than “sequential writes are fast.” Modern systems involve page caches, filesystems, SSD controllers, replication, and flush policies. The deeper advantage is that append-oriented designs create predictable work that can be batched, checksummed, replicated, and recovered incrementally.

Real writes pass through caches, filesystems, flash-translation layers, and device firmware. The useful point is that append-oriented workloads are easier to batch and lay out predictably than arbitrary small rewrites. It is not that the application controls the exact physical order of flash cells.

None of these benefits is automatic. A badly designed log can still be slow or incorrect. But the abstraction aligns well with properties systems exploit: batching, sequential processing, checksums, checkpoints, replay, segment rotation, and prefix comparison.

If two deterministic machines begin from the same state and apply the same commands in the same order, they should end in the same state.

That simple observation is one of the foundations behind replicated state machines. The Raft paper describes consensus as maintaining a replicated log of commands; each server executes the same ordered commands so its state machine produces the same state and outputs.

Durable, replicated and committed are not synonyms

Once a log crosses machine boundaries, another distinction becomes critical: where is the acknowledgement boundary?

A record can exist in memory, in an operating-system cache, on one machine’s durable storage, on several replicas, or in a position that a consensus protocol considers committed. Those states provide different guarantees.

accepted     -> one process has received it
written      -> bytes reached some local write path
durable      -> survives the failures covered by the local durability contract
replicated   -> copies exist in additional failure domains
committed    -> the protocol says this position is authoritative
applied      -> a state machine or consumer has incorporated it

The exact definitions vary by system, but collapsing them into the word “stored” hides the most important failure semantics.

Suppose a leader acknowledges a write before any follower has it. Latency is low, but a leader failure may lose an acknowledged suffix. Waiting for more replicas shrinks the failure window but adds network coordination to the acknowledgement path. Waiting for a quorum under consensus gives a stronger notion of commitment but only while enough members can communicate to make progress.

That is a recurring architecture trade-off:

The point at which you acknowledge a write is the point at which you choose what failures the caller is allowed to observe.

The same shape, doing different jobs

Once the abstraction is familiar, it appears everywhere. The implementations are not interchangeable, and the word log can refer to very different record formats and guarantees. What repeats is the basic shape: ordered records appended over time, then consumed or replayed for some purpose.

Kafka: the log as shared infrastructure

Kafka organises records into partitioned logs. Producers append records; consumers track positions and can replay retained history. Ordering is scoped to a partition, which is also Kafka’s unit of replication. That boundary is fundamental. Adding partitions increases parallelism, but there is no single total order across all partitions. Kafka adds replication, batching, retention, compaction, and consumer-group coordination around that core abstraction.

8271  OrderCreated #100
8272  PaymentReceived #100
8273  OrderCreated #101

WAL: the log as a recovery path

PostgreSQL describes write-ahead logging around one central rule: changes to data files are written only after WAL records describing those changes have been flushed according to the database’s durability rules. This lets the database acknowledge a transaction without first forcing every modified data page to its final location. After a crash, recovery can redo changes in WAL that were not yet reflected in the data files.

Notice the job of this log: WAL is primarily a recovery and replication mechanism for database state. Its records are not automatically a stable domain-event API for arbitrary application consumers. Sharing the word log does not mean sharing the same abstraction boundary.

LSN 91A0  update page 42
LSN 91D8  insert tuple
LSN 9220  commit tx 781

Raft: the log as agreed history

Raft manages a replicated log. A leader appends commands and works to reproduce that ordering on followers. Entries may exist on a leader before they are committed. Commitment is a protocol-level property, not merely “the leader wrote it.” Once entries are committed, state machines apply committed commands in log order.

That distinction matters during leader changes. An uncommitted suffix cannot be treated as authoritative merely because some server once stored it.

term 8  set x=4
term 8  set y=9
term 9  delete z

Journaling: the log as a crash-safety tool

Linux’s ext4 documentation describes a journal that protects filesystem consistency across crashes. Journal transactions have commit records and can be replayed after an interrupted update. By default, ext4’s data=ordered mode journals filesystem metadata rather than treating all file contents as one durable application event stream. Again, the common shape is a log. The semantics are specific to the subsystem.

descriptor block
metadata blocks
commit record

These systems are solving different problems. Kafka is not “a database WAL with an API.” Raft is not “Kafka for consensus.” An ext4 journal is not an event-streaming platform. Treating them as equivalent would erase the interesting parts.

But learning the shared abstraction means you no longer see each system as an entirely new creature. You can ask a familiar set of questions:

What gets appended: commands, physical changes, logical events, or something else?
What is the scope of ordering: global, shard, partition, key, transaction?
Who chooses the next position, and what happens when that writer fails?
When is an entry durable, replicated, committed and applied?
Can an acknowledged suffix ever be lost or rewritten?
How do readers track position, and what does advancing that position promise?
Can old entries be replayed under today's schemas and code?
When can history be truncated, compacted or replaced by a snapshot?
What state is derived from the log, and where do external side effects occur?
What happens when a consumer is slower than the producer for hours or days?

Those questions are useful far beyond any single technology.

The design lives in the trade-offs around the log

Once the abstraction is understood, the interesting work moves outward. A design review is rarely about whether a log exists. The interesting work is choosing the boundaries and failure semantics around it.

A few trade-offs recur:

ordering scope        <-> parallelism and throughput
acknowledgement point <-> latency and tolerated data loss
retention             <-> replay window, storage cost and compliance
compaction            <-> smaller history and historical fidelity
partitioning          <-> scale and cross-partition invariants
consumer independence <-> lag, backpressure and operational recovery
replayability         <-> schema/versioning discipline
side-effect coupling  <-> duplicate handling and coordination cost

None of these has a universally correct setting. The right answer depends on the invariant the system is protecting.

For a payment ledger, losing an acknowledged record may be unacceptable, and duplicates need explicit reconciliation. For analytics, a few seconds of lag may be harmless, while replayability matters enormously. For a cache invalidation stream, old history may have little value once newer state supersedes it. For consensus metadata, the entire point is that all replicas agree on one authoritative sequence before applying commands.

The abstraction is shared. The engineering judgement is in deciding which guarantees are worth paying for.

Lag turns retention into a capacity equation

Independent consumers sound simple until one of them falls behind. Then retention, throughput and recovery time become coupled.

A consumer that normally keeps up at 50 MB/s but can only process 50 MB/s while the producer is still writing 50 MB/s has no catch-up capacity. After an outage, its lag will remain roughly constant. To recover, sustained consumer throughput must exceed sustained ingress.

catch-up rate = consumer throughput - producer throughput
required replay window > outage duration + catch-up time

That turns “keep seven days of data” from a storage preference into a recovery claim. Seven days is enough only if the worst expected outage plus the time required to catch up fits inside that window. If not, the consumer can fall off the retained history and require a different recovery path.

The same reasoning affects capacity planning. Retention cost grows with ingest rate, retained duration and replication; compaction and compression change the shape but do not remove the need for headroom. A production design therefore needs to know not only steady-state throughput, but also the rate at which it can recover from being unhealthy.

That is why useful operational signals are usually about distance and headroom: consumer lag, oldest retained position, replication lag, disk utilisation, append/commit latency, replay throughput and the estimated time to catch up.

A log is powerful because it is incomplete

The simplicity is also the limitation.

If you ask a log, “What is Alice’s current email address?”, the naive answer may require replaying each relevant change. If you ask, “Give me every order over £100 from last month,” a raw log is not the best access path. If history grows forever, storage becomes a problem. If two writers must establish one order, the system needs some sequencing rule. If consumers lag beyond the retention window, replay may no longer be possible. If schemas evolve carelessly, old records can become unreadable. If a record contains sensitive data, “append-only forever” can conflict with retention and deletion requirements.

Operationally, logs also accumulate failure modes that the clean abstraction does not show: hot partitions, stuck consumers, poison records, replay storms, disk pressure, replication lag and snapshots that take too long to restore. These are not arguments against logs. They remind us that an elegant primitive still needs capacity planning, observability, and recovery procedures.

This is why real systems layer other structures around logs:

log + index        -> efficient lookup
log + snapshot     -> faster recovery
log + compaction   -> bounded history / latest values
log + replication  -> fault tolerance
log + consensus    -> agreed ordering
log + consumers    -> derived views and side effects
log + schemas      -> replayable interpretation
log + observability -> lag, throughput and recovery signals

The log is not the whole system.

A useful review technique: name the invariant first

When evaluating a log-backed design, I find it useful to begin with the invariant rather than the technology.

“Every order transition for one order must be observed in order” leads to a different design from “all orders in the company need one global order.”

“An acknowledged payment instruction must survive a single-node failure” leads to a different acknowledgement rule from “telemetry may lose the final few seconds during a regional failure.”

“A new projection must be rebuildable from six months of history” creates a retention and schema-compatibility requirement. “Only the latest value per key matters” may make compaction more attractive.

The sequence I want in a design discussion is therefore:

State the invariant.
Name the failure model.
Define the ordering scope.
Define the acknowledgement / commit point.
Define the replay and retention contract.
Decide how consumers make side effects safe.
Then choose the implementation.

Starting with “we should use Kafka” or “we should event-source this” reverses that reasoning. Products are implementations. The invariant is the reason the architecture exists.

Why does this idea keep returning?

Because a mutable world is difficult to reason about directly.

At any instant, a large system contains thousands or millions of pieces of current state. Some are cached. Some are replicated. Some are temporarily inconsistent. Some are in flight. Looking only at current state often hides how it got there.

A log gives us another axis: time.

Instead of only asking “What is the value?”, we can ask “What sequence of changes produced this value?” Instead of copying an opaque blob of state, one replica can copy a history prefix and catch up. Instead of forcing every downstream system to execute synchronously, it can consume the changes it cares about at its own pace.

That does not make distributed systems simple. But it gives us a simple object to reason around.

A useful mental shift. When you encounter a new system, do not ask only where it stores its state. Ask whether it maintains an ordered history of how that state changes — then ask who defines that order, when the history becomes authoritative, how long it remains replayable, and what happens at the boundary between replay and external side effects. Those answers usually reveal the system’s real durability, replication and recovery model.

References and further reading

1. Jay Kreps, “The Log: What every software engineer should know about real-time data’s unifying abstraction” (16 December 2013). The original LinkedIn Engineering essay introduces the log as an append-only, totally-ordered sequence of records ordered by time and follows the abstraction through databases, distributed systems, and data integration. LinkedIn Engineering.
2. Apache Kafka design documentation. Kafka’s design describes partition-scoped ordering, consumer offsets and rewinding, batching, replication, retention and compaction. Apache Kafka Design.
3. PostgreSQL documentation, Write-Ahead Logging. The WAL introduction describes logging changes before data-file updates and using log replay for recovery. PostgreSQL WAL introduction.
4. Diego Ongaro and John Ousterhout, “In Search of an Understandable Consensus Algorithm” (Raft). The paper describes consensus as management of a replicated log whose commands are applied in order by replicated state machines, so that each server executes the same commands in the same order to produce the same state and outputs. Raft paper.
5. Linux Kernel documentation, ext4 Journal (jbd2). The documentation explains journal transactions, commit records, replay after crashes and the differences between ext4 journaling modes. ext4 journal documentation.