Module 13 / Replication

Module 13 · lesson 2 of 9 · Read 9 min

Leader and followers

One writer, many readers, and how changes flow.

What you'll learn

  1. The basic shape
  2. What goes in the log
  3. Order matters
  4. Adding a new follower
  5. When a follower falls behind or restarts
  6. When the leader dies
  7. Why it's the default

Most databases you'll use replicate the same way: one copy takes the writes, and every other copy follows it. PostgreSQL, MySQL, SQL Server, MongoDB, Redis, and each partition of a Kafka topic all work like this. Module 5 used it for Tiffin's read replicas without looking inside.

This lesson looks inside. What exactly does the leader send? How does a brand new copy catch up with a database that's 2 TB and changing every millisecond? And what does a follower do after it's been offline for an hour?

The basic shape

One replica is the leader (also called primary or master). Every write goes to it. The others are followers (replicas, secondaries, standbys). They never accept writes from clients. They only apply changes the leader sends them.

%% caption: Writes go only to the leader. The leader streams its changes to every follower, which applies them in the same order.
flowchart TD
  OS["Order service"] -- "writes" --> L[("Leader<br/>applies the write, records it in its log")]:::hot
  L -. "changes, in order" .-> FA[("Follower A")]
  L -. "changes, in order" .-> FB[("Follower B")]
  MS["Menu service"] -- "reads" --> FA
Writes go only to the leader. The leader streams its changes to every follower, which applies them in the same order.

For each write, the leader:

  1. Applies it to its own data.
  2. Records the change in its replication log, in order.
  3. Sends that log to every follower, as a continuous stream.

Each follower reads the stream and applies each change in exactly the order the leader recorded it. Reads can go to the leader or to any follower.

In leader-follower replication, one replica accepts all writes and sends an ordered log of changes to the others, which apply the changes in the same order.

The big advantage is that there's one place where writes happen, so there's one order of events. Two customers can't make conflicting changes on two different copies, because there is only one copy that takes changes. Most of the pain in multi-leader and leaderless replication, later in this module, comes from giving that up.